llms.txt Explained: The robots.txt of AI — Who Actually Reads It and Should Your Site Have One?

If you've spent any time in SEO circles this year, you've seen the pitch: add a single text file to your website and AI assistants like ChatGPT, Claude and Perplexity will finally understand your content. The file is called llms.txt, it's been described as "robots.txt for AI", and thousands of sites have already published one.

Here's the uncomfortable question almost nobody selling llms.txt services will answer: who actually reads it?

We've dug through the spec, the crawler behaviour and the public statements from the companies that matter. This is the honest version — what llms.txt is, what it genuinely does today, what it doesn't do, and a clear decision framework for whether your site should have one. (Spoiler: ours does — you can see it at tempestdigital.com.au/llms.txt — but probably not for the reason you'd expect.)

What is llms.txt?

llms.txt is a proposed web standard: a Markdown file placed at the root of your website (yoursite.com/llms.txt) that gives large language models a curated, machine-friendly index of your most important content.

It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI (and the researcher behind fast.ai). The full specification lives at llmstxt.org, and the format is deliberately simple:

# Your Business Name

> One-paragraph summary of what your site is and who it serves.

## Key Pages

- [Services](https://yoursite.com/services): What you offer and for whom
- [Pricing](https://yoursite.com/pricing): Current plans and inclusions

## Resources

- [Guides](https://yoursite.com/guides): In-depth how-to content

That's it — an H1 with your name, a blockquote summary, then H2 sections containing annotated link lists. There's also a heavier sibling, llms-full.txt, which inlines the complete text of your key pages into one giant file rather than linking out to them.

The problem it's trying to solve

The reasoning behind the standard is sound. When an AI system reads your website, it faces three genuine problems:

1. HTML is noisy. Your actual content is buried in navigation, scripts, cookie banners and layout markup. An AI model has to burn context window space wading through it.
2. Context windows are finite. Even models with large context limits can't ingest a whole site. They need to know which pages matter.
3. JavaScript is a wall. Most AI crawlers don't execute JavaScript at all. If your content only exists after client-side rendering, many AI systems simply never see it. (This one is worth fixing regardless of llms.txt — it's a core part of technical SEO in 2026.)

llms.txt proposes to fix all three with one clean, curated Markdown file: no noise, no JavaScript, and an explicit editorial signal about what matters.

llms.txt vs robots.txt: the comparison is backwards

The "robots.txt for AI" framing is catchy but misleading, and understanding why tells you a lot about what llms.txt actually is.

robots.txt is access control. It tells crawlers where they may not go, and crawlers from major companies actually honour it. It's a directive with real enforcement behind decades of convention.

llms.txt is an invitation. It tells AI systems where you'd like them to look. Nothing obliges any AI company to fetch it, parse it, or weight it. It restricts nothing and guarantees nothing.

If you want to control which AI crawlers can access your site — allowing or blocking GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends — that still happens in your actual robots.txt. llms.txt plays a completely different game.

Who actually reads llms.txt?

This is the part that matters, so let's be precise about the two sides of the ledger.

Who publishes it

Adoption on the publishing side is real and substantial. Anthropic (the maker of Claude) publishes one for its documentation. Developer-tool companies like Cloudflare and Zapier maintain them. Documentation platforms such as Mintlify generate them automatically, which put llms.txt on thousands of software documentation sites overnight. Community directories track hundreds more.

Notice the pattern, though: it's overwhelmingly developer documentation. That's not an accident — docs are exactly the content that AI coding assistants get pointed at deliberately.

Who consumes it

Here's the honest state of play at the time of writing:

• No major AI provider has publicly confirmed that its crawlers use llms.txt to decide what to retrieve, cite or train on. Not OpenAI, not Anthropic, not Perplexity, not Google.
• Google has been openly dismissive. Google's John Mueller has compared llms.txt to the old keywords meta tag — a label sites give themselves that search systems have little reason to trust. Google's AI features (AI Overviews, AI Mode) are built on Google's ordinary index, not on llms.txt files.
• Server logs back this up. Site owners who check their access logs consistently report that the big AI crawlers request their normal pages far more than their llms.txt — many never request it at all.
• The genuine consumers are on-demand tools. AI coding assistants (Cursor and similar), agentic browsers and custom AI workflows do fetch llms.txt — when a user points them at a site or the tool is configured to look for it. This is why docs sites lead adoption: a developer who tells their AI assistant "use the Stripe docs" benefits directly from Stripe having a clean llms.txt.

So the honest answer to "who reads it?" is: almost no one automatically, some tools on request, and possibly everyone later. It's a standard with real publishing momentum waiting for consumption to catch up — and standards have flipped from ignored to essential before. Schema.org markup spent years as a nice-to-have before it started powering rich results.

Should your site have one?

Our decision framework, by site type:

Documentation or SaaS product sites: yes, clearly. Your users are already pointing AI assistants at your docs. An llms.txt (and ideally llms-full.txt) makes those interactions measurably better today. This is the one category with a real, current payoff.

Service businesses and content sites: yes, as cheap insurance. It takes under an hour, carries zero risk, and positions you for the scenario where a major assistant starts consuming it. Just be clear-eyed: you're buying a lottery ticket with a good expected value because the ticket is nearly free, not a ranking boost.

"Set and forget" sites: only if you'll maintain it. An llms.txt that links to deleted pages or describes services you no longer offer is worse than none — the one thing worse than being invisible to AI is being confidently misrepresented by it.

Two things llms.txt will not do, no matter what a vendor tells you:

• It will not improve your Google rankings. Google doesn't use it.
• It will not, by itself, get your brand cited in ChatGPT or Perplexity answers. Citation happens because your content is present, crawlable and authoritative in the indexes those systems actually retrieve from.

How to write a good one

If you're going to do it, do it properly:

1. Follow the spec. H1 site name, blockquote summary, H2 sections with [link](url): description lists. Plain Markdown, UTF-8, at your domain root.
2. Curate ruthlessly. Ten well-described core pages beat a hundred-line site dump. The file is an editorial statement about what matters.
3. Write descriptions for machines that will be read to humans. Each link's annotation should say what a visitor gets from that page — an AI assistant may relay it verbatim.
4. Keep your robots.txt decisions separate. llms.txt is not where you allow or block crawlers.
5. Review it quarterly. Stale llms.txt files misinform the exact systems you published them to help.

You can see how we've structured ours at /llms.txt — a business summary, our service pages with plain-language descriptions, and our key resources.

What actually gets you cited by AI in 2026

Since llms.txt alone won't do it, here's what does move the needle for AI visibility — the things we check when we audit a site:

• Server-rendered, clean HTML. AI crawlers don't run JavaScript. If your content isn't in the initial HTML response, you don't exist to them.
• Presence in the underlying indexes. ChatGPT's browsing leans on Bing; Perplexity and Google's AI features have their own crawls. Being properly indexed — everywhere, not just Google — is the entry ticket.
• Structured data. Schema markup remains the most reliable machine-readable layer that AI systems demonstrably consume.
• Entity consistency. Your business name, location, services and claims should match across your site, your Google Business Profile, directories and social profiles — LLMs cross-reference sources and hedge when they disagree.
• Content that answers questions directly. AI answers are assembled from passages, not pages. Clear headings, direct answers and genuine expertise get quoted; marketing fluff gets skipped.

If you want to know where your site stands right now, our free AI Search Score tool checks your site against these factors in about a minute. And if the results sting, that's what our AI SEO service is for.

The bottom line

llms.txt is a sensible, well-designed standard that the AI industry hasn't committed to yet. Publish one — it costs almost nothing and the option value is real. Just don't let anyone convince you it's a strategy. The sites getting cited by AI assistants today earned it the old-fashioned way: clean technical foundations, real authority, and content worth quoting.