llms.txt explained: should your website have one?

If you've seen a website with a file at /llms.txt and wondered what it's for, the short answer is: it's a curated summary of the site, written for AI systems instead of people. It's the same idea as a table of contents, formatted so a language model can read it quickly instead of crawling your whole site to figure out what matters.

It's a genuinely useful idea. It's also not an official standard, not required by any search engine or AI company, and not something you should expect to transform your traffic overnight. Here's what it actually is, how it differs from files you already know, and whether it's worth the twenty minutes it takes to add one.

What llms.txt actually is

llms.txt is a plain markdown file placed at the root of a domain, at yoursite.com/llms.txt. The convention was proposed in late 2024 by Jeremy Howard (co-founder of Answer.AI) as a lightweight way to help large language models understand a site's structure without having to parse full HTML pages, navigation menus, cookie banners, and JavaScript-rendered content.

A typical llms.txt file has a short H1 with the site or company name, a one-line summary of what it does, and then a series of markdown links grouped under headings, pointing to the pages that matter most: docs, pricing, key guides, an about page. Some sites also publish an llms-full.txt, which is the same idea but with the full text of key pages inlined rather than just linked, so a model can ingest the content in one request instead of following links.

How it differs from robots.txt and sitemap.xml

It's easy to lump llms.txt in with the other root-level files, but the three do different jobs and none of them replace each other.

robots.txt is a permissions file — it tells automated crawlers, including AI crawlers like GPTBot or PerplexityBot, which paths they're allowed to fetch. It says nothing about what your site is or which pages matter most. sitemap.xml is exhaustive by design; it's meant to list every URL so an indexer doesn't miss anything, and it's XML, not something a person or a language model reads for context. llms.txt is the opposite of exhaustive. It's a hand-picked shortlist, written in plain language, meant to say "if you're only going to look at a few pages on this site, look at these."

Does it actually change what gets cited?

This is the honest, slightly unsatisfying part. As of now, none of the major AI companies — OpenAI, Anthropic, Google, Perplexity — has publicly confirmed that their models or crawlers specifically fetch and prioritize llms.txt files when deciding what to cite or summarize. Some developer tools and documentation platforms have adopted it enthusiastically because it's genuinely useful for feeding docs into coding assistants and custom AI workflows that are explicitly built to look for it. Whether general-purpose consumer AI systems are actively using it as a ranking or citation signal today is unconfirmed and likely varies by system and is likely to keep changing.

What is true, and doesn't require any company's confirmation, is that a well-written llms.txt forces you to articulate what your site is and what its most important pages are, in plain language, in one place. That's a useful exercise regardless of whether any crawler reads the file, because the same clarity tends to show up in your actual page copy too.

Should your site have one?

For most small business sites, yes, but treat it as a low-cost, low-certainty addition rather than a priority project. It won't hurt — there's no known downside to publishing one, since AI crawlers that don't recognize it will simply ignore it like any other unrecognized file. It's genuinely cheap to create: a single markdown file, no build process, no ongoing maintenance beyond updating it when your key pages change.

Where it's more clearly worth the effort: documentation-heavy sites, SaaS products with an API or developer audience, and any business whose customers might reasonably ask an AI assistant to research it before making a decision. Where it matters less: a single-location local business with five static pages, where a good sitemap and clear on-page content already cover the same ground.

1. 1
   Write one summary paragraph

   What your business does, in plain language, no marketing fluff — the same sentence you'd use to explain the business to a stranger.
2. 2
   Pick 5-15 key pages

   Your most useful docs, guides, pricing, and about page — the pages you'd want someone to read first.
3. 3
   Format as markdown

   H1 title, one-line summary, then H2-grouped link lists pointing to each key page with a short description.
4. 4
   Publish at /llms.txt

   Root of your domain, plain text file, no authentication required — same visibility as robots.txt.
5. 5
   Revisit when your site changes

   Update it when you add major new pages or restructure navigation — it goes stale exactly like a sitemap does.

• ✓llms.txt is a curated, human-readable summary file for AI systems, not an official web standard.
• ✓It differs from robots.txt (permissions) and sitemap.xml (exhaustive URL list) by being a short, hand-picked shortlist.
• ✓No major AI company has confirmed it directly affects citation or ranking today.
• ✓It costs almost nothing to add and forces useful clarity about what your site actually offers.
• ✓Prioritize it more if you run a documentation-heavy or developer-facing site; it matters less for a simple local business site.