llms.txt turns up on every AI-visibility checklist. It is a reasonable idea, it costs an hour, and it is nowhere near as important as the checklists imply. Here is the honest version.

What is llms.txt?

llms.txt is a proposed file at the root of a website — yourdomain.com/llms.txt — written in markdown, that tells an AI assistant in one page what the site is and which of its URLs are worth reading. It was proposed as a convention rather than a standard, and it works like a curated index: a short description of the organisation followed by grouped links to the pages that best answer questions about it.

The reasoning behind it is sound. An assistant answering a question about your company has a limited budget of pages it can fetch and read. Left to itself it may spend that budget on your careers page and a five-year-old press release. A short, honest index that says "here is what we do, and here are the six pages that actually explain it" is a genuinely useful thing to hand it.

Do I need an llms.txt file?

Not urgently. No search engine or AI assistant is known to require one, and adoption remains thin, so a site without llms.txt is not faulty and will not be penalised. It is cheap insurance rather than a fix: worth publishing if you have an hour, but it will not rescue a site that AI crawlers are blocked from or whose content only appears after JavaScript runs.

We take that position seriously enough to build it into the product. TrustCtrl checks for llms.txt on every crawl, and a missing file is reported as a plain fact, never as a problem to fix. Reporting an absent llms.txt as a fault would be inventing urgency that the evidence does not support, and a scanner that cries wolf about conventions nobody has adopted teaches you to ignore it.

The order of operations matters more than the file. If the AI crawlers cannot fetch your pages, llms.txt changes nothing — see are you blocking AI crawlers. If your content only exists after JavaScript runs, llms.txt changes nothing either. Fix those first; publish this after.

What should an llms.txt file contain?

Keep it to a single readable page: a heading with your organisation name, one or two sentences describing what you do, then grouped markdown links to the pages that genuinely answer questions — documentation, pricing, guides, product pages. Each link should carry a short note saying what the page covers. It is an index for a reader in a hurry, not a sitemap, so a file that has grown past a few thousand words has stopped doing its job.

The failure mode is predictable: somebody automates it, and llms.txt becomes a dump of every URL on the site. At that point it is a worse sitemap.xml with none of the tooling, and the one thing it was supposed to provide — curation — is gone. TrustCtrl flags a file that has grown past roughly a hundred kilobytes for exactly this reason.

  • Write it for a reader, not a parser. The consumer is a language model. Plain descriptive sentences beat keyword lists.
  • Link to pages that answer questions. Pricing, documentation, guides, product detail. Not the careers page, not the press archive.
  • Say what each link covers. A bare URL forces the assistant to fetch it to find out; a one-line note may save the fetch entirely.
  • Keep it current. A file pointing at three URLs that now 404 is worse than no file, because it burns the budget you were trying to direct.

Is llms.txt the same as robots.txt?

No, and they are not alternatives. Robots.txt controls access — which crawlers may fetch which paths — and it is honoured by the major operators. llms.txt offers guidance about content and grants no permissions at all. Publishing llms.txt does nothing for a site whose robots.txt blocks the AI crawlers in the first place.

It is worth being blunt about this, because the two files sit next to each other at the site root and get discussed in the same breath. One is a lock. The other is a leaflet. Publishing the leaflet while the lock is closed accomplishes nothing.

The useful part: what the check finds when the file is missing

Here is the part that earns the request, and it has little to do with AI. When TrustCtrl asks for /llms.txt on a site that does not have one, the interesting answer is not "absent" — it is how the server says absent.

A correctly configured server answers a request for a file it does not have with 404. Search engines treat a 404 as final: the URL is dropped from the index and stops being re-crawled. Some servers instead answer 500, because a framework's catch-all route maps "no such resource" onto a server error. Search engines treat a 500 as temporary: they keep the URL, retry it, and come back. Every mistyped link and every long-dead URL keeps consuming crawl budget indefinitely, and keeps sitting in the index.

We did not learn this from a specification. We found it on our own site, where exactly that catch-all was turning missing files into server errors. Asking for a file we knew was absent turned out to be a clean way to test how the server handles absence in general — which is why the check earns its place on a crawl that already fetches robots.txt and sitemap.xml.

There is a third answer worth knowing about: 200 with an HTML page. That is a soft 404 — the server responding to any unknown path with its ordinary page. TrustCtrl treats an HTML body as absent rather than as a published file, because otherwise every site with a catch-all would be reported as having an llms.txt it never wrote.

Where it fits

Publish llms.txt if you have the hour. Do not expect it to move anything on its own, and do not let it distract from the three things that actually decide whether an assistant can cite you: crawler access, content that exists in the HTML, and answers written so they can be lifted in one piece. That is the subject of how to get cited by AI search.

TrustCtrl checks all of it on the same visit — llms.txt, robots.txt, AI-crawler access, whether pages render without JavaScript, plus titles and descriptions, speed, accessibility and security settings. Every finding comes with the evidence behind it and a fix you can hand to whoever maintains the site.