llms.txt: What the File Does and What It Cannot Do

llms.txt is a proposed Markdown index, normally published at a site's /llms.txt path. Its intended purpose is to give language models and agents a concise description of the site plus annotated links to useful resources. Jeremy Howard published the proposal in September 2024; it is not an IETF or W3C web standard, and the format does not require any provider to fetch or use it. It also does not grant access, restrict access or replace a crawler policy.

TL;DR

• Three files, three different roles. robots.txt communicates crawl rules to cooperating crawlers, sitemap.xml supports URL discovery, and llms.txt proposes an annotated orientation layer. None is authentication or a guarantee that a page will be used.
• The format has exactly one mandatory element — an H1 with the site or project name. The blockquote summary, the prose sections and the H2 link lists are all optional structure.
• Google Search does not use it. Google's guidance on generative AI features says you do not need to create "new machine readable files, AI text files, markup, or Markdown" to appear in Google Search, that "Google Search itself doesn't use them", and that publishing them "will neither harm nor help your site's visibility or rankings in Google Search".
• Publishing support is not the same as consumer adoption. Documentation platforms and developer sites generate or publish the file, but that does not prove that major AI search products use third-party llms.txt files for retrieval or ranking.
• The proposal also asks for clean Markdown copies of pages at the same URL with .md appended, and index.html.md where the URL has no filename.
• llms-full.txt is an ecosystem convention, not part of the proposal. It is the whole corpus in one file rather than an index of it.
• It cannot rescue a site a crawler cannot read. A pointer file does not fix pages that render client-side only, return errors under bot traffic, or contain no self-contained answer.
• It can be inexpensive when generated from public source data. It can also expose stale, gated or unreleased information if the generation rules are careless. Maintenance and information governance matter more than the file's syntax.
• It is not a citation guarantee. Nothing in the format promises retrieval, and no provider promises to read it.

What the file actually looks like

The proposal at llmstxt.org defines a deliberately small shape. Only the H1 is mandatory. Everything after it is optional, and the order is fixed so that a parser can stop reading early and still have something coherent:

1. An H1 with the name of the site, project or company.
2. A blockquote with a short summary containing the key information needed to understand the rest.
3. Zero or more Markdown sections of free prose with more detail.
4. Zero or more H2 sections containing file lists. The proposal's own words: "Each 'file list' is a markdown list, containing a required markdown hyperlink [name](url), then optionally a : and notes about the file."
5. Optionally, a section literally named ## Optional, whose links carry secondary information that "can often be skipped" when a shorter context is needed.

The Optional heading has a defined meaning inside the proposal: a consumer may omit those URLs when it needs a shorter context. It is a curation instruction for a parser that chooses to follow the proposal, not a ranking signal for search engines.

Here is a complete, valid file for a fictional direct-to-consumer hardware brand:

# Northsound Audio

> Northsound Audio designs and sells studio monitors and audio interfaces direct
> to musicians in the UK and EU. Founded in Glasgow, ships from one warehouse,
> supports hardware for the full warranty term with in-house engineers.

Prices on product pages include VAT and are updated daily. Stock status is live.
Support is answered by the engineering team, not an outsourced desk.

## Products
- [Studio monitors](https://example.com/monitors.md): the full monitor range with cabinet sizes, frequency response and room-size guidance
- [Audio interfaces](https://example.com/interfaces.md): channel counts, connectivity and latency figures for each model
- [Warranty and returns](https://example.com/warranty.md): warranty term, return window, and who pays return shipping

## Company
- [About](https://example.com/about.md): founding date, team, manufacturing partners, registered entity
- [Contact](https://example.com/contact.md): support hours, phone, email, registered address

## Optional
- [Engineering notes](https://example.com/blog/index.html.md): technical articles on monitor placement and acoustic treatment

The blockquote gives a consumer an early summary, while the notes after each link explain why a resource is useful. Those annotations are what distinguish a curated llms.txt from a sitemap copied into Markdown. Keep both factual: a consumer may quote, paraphrase, ignore or truncate any part of the file.

Three files, three jobs

Most of the confusion around llms.txt disappears once the three root-level files are put side by side. They answer different questions and have different authority.

robots.txt is the only one of the three that expresses crawler-specific rules, but it is still a protocol for cooperating crawlers rather than a security boundary. Which AI agents publish tokens, which say they honour the file and how to verify or block requests is covered in AI crawlers: GPTBot, ClaudeBot, PerplexityBot and what to allow.

Is llms.txt a replacement for robots.txt AI rules?

No. There is no field in llms.txt that grants or refuses crawling, training or inference use. Prose such as “do not train on this content” is not a directive defined by the proposal. Express preferences through the controls each provider documents, obtain legal advice where rights or licences matter, and use authentication or network enforcement when content must not be accessible. llms.txt is orientation; robots.txt is a crawler policy, not access control.

Glossary

• llms.txt — a Markdown file at a domain's root that names the site, summarises it, and lists its most important pages with notes.
• llms-full.txt — a companion file convention that concatenates the full text of a site's key pages into one document rather than linking to them; it is not part of the llms.txt proposal.
• Markdown mirror — a clean .md copy of a page served at the same URL with .md appended, so a consumer gets prose without navigation or scripts.
• Retrieval — the step where a system fetches live pages or an index at question time, as opposed to relying on what a model absorbed during training.
• Grounding — attaching retrieved source text to a generated answer so the answer can cite it.
• .well-known/ — a reserved URI path prefix used for registered site-wide metadata. Some platforms additionally place llms.txt there, but that location is a platform convention rather than the proposal's primary path.
• Curation note — the text after the colon in a link list, describing what the linked page answers. The part that gives the file value over a sitemap.

Who actually reads llms.txt

This is where honesty matters more than enthusiasm, because the answer is fewer things than the marketing around the file suggests, and the gap is widest exactly where buyers assume it is narrowest: search.

Google is explicit. Its guidance on optimising for generative AI features in Search states that you do not need to create "new machine readable files, AI text files, markup, or Markdown" to appear in Google Search, that "Google Search itself doesn't use them", and that creating them "will neither harm nor help your site's visibility or rankings in Google Search". That is not ambiguity to be interpreted around. If the business case for llms.txt is Google AI Overviews or AI Mode, there is no business case.

The other major providers are quieter, which is a different problem. Their public crawler and bot documentation describes user agents, robots.txt handling and IP ranges — it does not describe reading a third-party llms.txt. Silence is not a denial, but it is also not support, and you cannot build a deliverable on it.

The clearest adoption is on the publishing side of documentation ecosystems. Mintlify automatically generates /llms.txt, /llms-full.txt and Markdown page variants for hosted documentation. OpenAI's developer documentation also points automated readers to a documentation index in llms.txt. These examples show that the format can be useful when a user or agent deliberately asks for technical documentation. They do not establish automatic discovery of arbitrary marketing sites, and they are not evidence of a ranking benefit.

The most testable use case is a retrieval system you control. A support assistant, internal search or sales-enablement tool can be programmed to read the file, and its answers can be evaluated against a defined test set. In that case llms.txt is an information architecture choice, not an SEO assumption.

What belongs in the file, and what does not

The test for every line is whether a consumer that read only this file would come away with a true and useful picture of the business. That rules out both extremes: marketing copy with no facts in it, and a raw dump of every URL.

For consumers that load the file into model context, length creates token and latency costs. Keep the summary concise, annotate links rather than dumping page text, and use ## Optional for genuinely secondary material. A consumer may process the file differently or ignore it altogether, so do not rely on front-loading as a visibility mechanism.

Why llms.txt will not fix a site models cannot read

Publishing llms.txt does not repair the resources it links to. If the intended consumer cannot access a page, cannot extract its main content or finds conflicting facts, a better index does not resolve the underlying issue.

An index is a set of pointers. Test those pointers using the actual consumer or a representative retrieval method rather than assuming that every assistant fetches pages the same way:

• Content dependent on client-side rendering. Some consumers render JavaScript and others perform a simpler fetch. Put critical information in robust rendered HTML and test what the intended consumer receives; do not infer failure from page source alone.
• Bot rules at the edge. A WAF challenge or an aggressive bot-management setting returns 403 to an unfamiliar client regardless of what robots.txt or llms.txt say. The file is a request to read pages that are refusing to be read.
• Slow or unstable responses. Different clients have different timeouts, but repeated errors and timeouts undermine any retrieval path. Monitor HTTP status, response reliability and server capacity separately from browser Core Web Vitals.
• Facts available only through inaccessible media. Images need meaningful text alternatives where appropriate. PDFs can be readable, but they still need discoverable links, accessible text, correct status codes and current versions. Keep decision-critical terms available in an accessible page when practical.
• Thin or ambiguous content. A fetched page still needs to answer the user's question with current facts, context and evidence. Google explicitly says it does not require “chunking” pages into tiny blocks, so structure content for people rather than an imagined llms.txt ranking rule.

The sensible sequence is to make the canonical pages accurate, accessible and testable first, then publish an index when a defined consumer or low-cost experiment justifies it.

How we approach this at Space Ads

Space Ads publishes llms.txt on its own site and treats it as a maintained public summary, not as a ranking lever. For client work, the decision should start with a consumer and maintenance owner: a documentation agent, an internal retrieval system or a deliberately limited experiment. If neither exists, crawlability, source accuracy and people-first content have higher priority. The broader work is described on our AI SEO service page.

Our public file extends beyond the proposal's minimal link-list shape. It contains company facts, registration details, markets, capabilities and an article index. This makes maintenance and disclosure review important: every statement must already be appropriate for public distribution, and dynamic lists should come from the same content source as the site. Generated and editorial sections can coexist, but both need an owner and validation.

Our robots.txt also carries a non-standard hint line pointing at it:

Sitemap: https://www.spaceads.agency/sitemap.xml

# LLM-readable site summary (proposed standard, see https://llmstxt.org)
LLM-content: https://www.spaceads.agency/llms.txt

LLM-content is not defined by RFC 9309 or by the llms.txt proposal, and no provider documentation reviewed for this article assigns it a function. It should therefore be read only as local documentation, not as access control, discovery support or a standard to copy. A conventional comment in robots.txt or project documentation can record the same engineering intent without resembling a recognised directive.

In a sensible delivery order, the file follows the facts and pages it describes. First reconcile the organisation, offer and policies on canonical pages; then decide whether an annotated machine-readable index serves a defined use case. The implementation effort depends on the CMS, number of resources, generation path, review process and risk of exposing information that should remain gated.

A seven-step way to publish it and keep it true

1. Name the consumer before writing the file. Internal retrieval tool, coding agents reading your docs, or "unknown, cheap to try" — all acceptable. "It was on a checklist" is not, because nobody will maintain it.
2. Generate repeatable sections from a source of truth. Company facts should agree with the source used by visible pages and structured data; article lists can come from the same content index that produces the sitemap. Keep editorial text under review rather than pretending all facts can be generated safely.
3. Front-load the facts. Blockquote and first section carry what must survive truncation: what the business is, where it operates, what it sells, how to contact it.
4. Write notes, not link text. Each entry gets a clause saying what question that page answers. If a note could be swapped between two links without anyone noticing, it is not doing anything.
5. Reconcile with access and canonical decisions. Do not accidentally expose or promote gated, staging, noindex, non-canonical or obsolete resources. A deliberate internal use case may justify a different list, but it should not live publicly by accident.
6. Watch access logs for /llms.txt. A verified request proves only that the endpoint was requested from an address you could attribute; it does not prove the file affected retrieval, an answer or a citation. Analytics may omit automated traffic, so use server or edge logs.
7. Review it by risk and event. Recheck after a migration, rebrand, policy or pricing change, and set a cadence based on how often the underlying content changes. Revisit the provider case only when documentation supplies new evidence.

Common mistakes and what to do instead

FAQ

What is llms.txt?

llms.txt is a plain Markdown file published at the root of a domain — example.com/llms.txt — that names a site, summarises it in a blockquote, and lists its most important pages with a short note on each. It was proposed by Jeremy Howard in September 2024 at llmstxt.org and remains a proposal rather than a ratified web standard. Its purpose is orientation for language models and agents, not access control.

Does Google use llms.txt?

No. Google's guidance on generative AI features in Search states that you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, that Google Search itself does not use them, and that creating them will neither harm nor help a site's visibility or rankings in Google Search. Publishing the file for another consumer is fine; expecting it to affect Google Search is not.

Is llms.txt the same as robots.txt?

No. robots.txt is a formalised protocol (RFC 9309) for stating which paths a given user agent may fetch, and it expresses rules. llms.txt is a Markdown summary and index describing what a site contains, and it expresses nothing that a consumer must obey. They sit at the same root path and answer different questions: permission versus orientation.

Do AI crawlers follow llms.txt instead of robots.txt?

No. AI crawler behaviour is documented against robots.txt and user-agent tokens, not against llms.txt. There is no field in the llms.txt format that grants or withholds access, so it cannot substitute for robots.txt AI rules. Access decisions belong in robots directives, and enforcement belongs in the layer that can refuse a request.

Where should the llms.txt file be placed?

The proposal's primary location is the site's root path: https://example.com/llms.txt, with optional files in subpaths for individual projects or sections. Serve the Markdown as readable text with a correct successful response. Some platforms additionally publish /.well-known/llms.txt, but that is not the primary location defined by the proposal.

What is llms-full.txt?

llms-full.txt is a widely used convention that concatenates the full text of a site's key pages into a single file, rather than linking to them as llms.txt does. It is not part of the original proposal. It is common on documentation sites, where platforms generate it automatically, and it is heavy: the whole corpus in one file has to be regenerated whenever the content changes.

Will llms.txt get my brand cited in ChatGPT or Perplexity?

No provider guarantees that outcome. A consumer must first choose to read the file, follow a link and use the resulting page; source selection then depends on product logic the proposal does not define. llms.txt may orient a consumer that deliberately supports it, but it is not a citation mechanism. Focus first on accurate, accessible pages that satisfy the user's question.

Should llms.txt list every page on the site?

No. Completeness is the sitemap's job. llms.txt is a curated shortlist — the pages a person would need to understand what the business is and to make a decision. A file that lists everything carries no priority information and costs the reader more context to process.

Do I need Markdown versions of my pages?

Only if you can maintain them. The proposal suggests serving clean Markdown copies at the same URL with .md appended, using index.html.md where the URL has no filename, so a consumer gets prose without navigation or scripts. It is genuinely useful for documentation. For a marketing site it doubles the surface you have to keep in sync, and a stale Markdown mirror contradicting the live page is worse than not having one.

Can llms.txt stop AI training on my content?

No. Nothing in the format expresses a use restriction that a parser is obliged to honour. Use the controls each provider publishes for training, search or snippets, understanding that robots.txt communicates preferences to cooperating crawlers rather than enforcing confidentiality. Use authentication or network controls for private content and obtain legal advice for licensing or rights questions. A prohibition written in llms.txt records intent but has no defined effect.

Sources and further reading

• llmstxt.org — The /llms.txt file proposal
• Google Search Central — Optimizing for generative AI features in Google Search
• Google Search Central — AI features and your website
• IETF — RFC 9309: Robots Exclusion Protocol
• sitemaps.org — Sitemaps XML protocol
• Mintlify — llms.txt documentation
• OpenAI Developers — Documentation index guidance and llms.txt
• Perplexity — Perplexity crawlers

Statements about provider behaviour above reflect vendor and Google documentation as of July 2026. Providers change agent names, documentation and support without notice, so re-check the source pages before changing a live configuration.

In short

• llms.txt is a Markdown index and summary at a domain's root. It describes a site; it does not control access to it.
• The only mandatory element is an H1 with the site name. The value lives in the blockquote summary and the note after each link.
• Google states that Search does not use the file and that publishing it neither helps nor harms rankings. Build the case on another consumer or not at all.
• Clear adoption today is on the publishing side of documentation and developer tooling. That does not establish automatic use of third-party marketing-site files by major AI search products.
• The file cannot compensate for linked pages the intended consumer cannot access, render reliably or interpret accurately.
• Generate repeatable sections from a source of truth, keep editorial facts reviewed, reconcile public links with access and canonical decisions, and treat log requests as evidence of a fetch — not of use or citation.

Continue reading

• AI Crawlers: GPTBot, ClaudeBot, PerplexityBot and What to Allow
• AI SEO: How to Optimize for Search Engines, AI Overviews and LLMs
• LLM SEO: Make Your Brand Easier for AI Tools to Cite
• Core Web Vitals: How to Fix LCP, INP and CLS
• What Is an SEO Audit and How to Do It Properly
• AI SEO · Marketing audit · Contact Space Ads