llms.txt: What the File Can Do, What It Cannot, and Whether It Is Worth It

Where the proposal comes from

The idea comes from Jeremy Howard and was published on 3 September 2024 on llmstxt.org; the page was last changed on 10 August 2026 (accessed 30 October 2026). It describes itself as "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website". So it is a proposal, not an adopted standard. According to the page, the specification is "open for community input".

The basic idea, according to llmstxt.org: language models have a limited context window, and a normal website consists of much more than the actual content. A short Markdown file with the most important pages and a summary is meant to help a model or agent find the right thing quickly when it needs information about your website.

How the file is structured

The file sits at /llms.txt in the root directory and is written in Markdown. According to the specification, it contains the following, in this order:

In addition, llmstxt.org suggests providing a clean Markdown version of important pages at the same address with .md appended, so page.html.md or page.md. A file called llms-full.txt, which you often see, is not part of the specification on llmstxt.org.

How it differs from robots.txt and sitemap.xml

The three files sit side by side in the root directory, but they have different jobs. llmstxt.org itself writes: "robots.txt and llms.txt have different purposes".

Two consequences follow. An llms.txt blocks nothing. If you do not want your text to go into the training of AI models, you control that in robots.txt; how to do it is explained in AI crawlers in robots.txt. And an llms.txt does not replace a sitemap. llmstxt.org sees it as an addition: the sitemap lists all pages, the llms.txt a selection.

What is proven and what is not

Google: ignored by search

Google is the only major provider with an explicit statement. The guide to optimising for generative AI features says in its "Mythbusting" section (Google Search Central, page dated 10 July 2026, accessed 30 October 2026):

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

It goes on to say that it is perfectly fine to maintain llms.txt for other services. "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google added this clarification on 15 June 2026 (Google Search Central: Latest documentation updates, accessed 30 October 2026). The page on AI features also says that AI Overviews and AI Mode do not require any new machine-readable files (Google Search Central: AI features and your website, accessed 30 October 2026).

OpenAI, Anthropic, Perplexity: no statement

The crawler documentation of OpenAI, Anthropic and Perplexity says nothing about whether their search or user crawlers read an llms.txt or take it into account in answers (checked on 30 October 2026). We have not found any other official statement from these providers either that ChatGPT, Claude or Perplexity evaluate the llms.txt of other websites. That does not mean it never happens. It means nobody has committed to it.

What the providers do: they publish llms.txt files for their own developer documentation. On 30 October 2026, developers.openai.com/llms.txt, platform.claude.com/llms.txt and ai.google.dev/gemini-api/docs/llms.txt were reachable. This shows what the providers use the format for themselves, but it is no evidence that their search systems read other websites' files.

Measurements: few fetches

The largest analysis so far comes from Ahrefs (Ahrefs: llms.txt study, Louise Linehan, 15 June 2026, accessed 30 October 2026). It was based on 137,210 domains with data from Ahrefs Web Analytics for May 2026:

• Around 28% of these domains had an llms.txt. Ahrefs itself calls this an upper bound, because its own customers are more technically minded than the rest of the web.
• 97% of these files were not fetched a single time in May.
• Of the measured fetches, 233 came from bots that answer user questions (for example OAI-SearchBot or PerplexityBot). Ahrefs puts this at 1.1%, but in the text relates the figure to AI bot fetches and in the table to all fetches. Training crawlers accounted for 1,179 fetches, about five times as many.
• A fetch is no evidence that the file was actually used. Ahrefs points this out explicitly.

Ahrefs also quotes Google's John Mueller, who described llms.txt as a "temporary crutch, perhaps to save some tokens" for AI coding tools, not as a topic for search. We have only checked this quote via Ahrefs, not in the original.

In summary

When it still makes sense

The file takes little effort, and according to Google it does no harm. Whether it is worth it for you depends on who is meant to read it:

• Developer documentation, APIs, software. This is where the benefit is most plausible. If you work with a coding assistant, you can give it a documentation address; an llms.txt with links to Markdown versions saves the tool from working its way through the whole website. This is exactly what OpenAI, Anthropic and Google use the format for themselves.
• Websites with many subpages and a clear structure. A curated list can help when an agent is looking specifically for prices, terms or contacts. This benefit is not proven.
• As a fact sheet. Writing the file forces you to pin down what the business is called, what it offers and which pages matter. That is a useful exercise even without AI.

It does not make sense as a substitute for the essentials: pages whose content is in the source code without JavaScript, a robots.txt that does not lock out search crawlers, and an up-to-date sitemap.

Example: creating an llms.txt

An example for a joinery business. The first line is the required heading with a hash sign and a space in front, here # Example Joinery. Then comes:

> Example Joinery in Freiburg designs and builds fitted furniture and made-to-measure kitchens for private customers in the Freiburg area. On-site consultation, manufacturing in our own workshop.

Workshop open Monday to Friday, 8 am to 5 pm. Please send enquiries via the contact form.

## Services

- [Fitted wardrobes](https://www.example.com/fitted-wardrobes): made to measure for sloping ceilings and alcoves
- [Kitchens](https://www.example.com/kitchens): design, manufacturing and installation

## Company

- [About us](https://www.example.com/about-us): team, workshop, history
- [Contact](https://www.example.com/contact): address, phone, enquiry form

## Optional

- [References](https://www.example.com/references): photos of completed projects

How to go about it:

1. Make a selection. Take the 5 to 20 pages someone needs to understand what you offer. Not the whole sitemap.
2. Write a summary. Two or three sentences in the blockquote: who, what, where, for whom. Only details that also appear on the website in the same form.
3. Links with notes. A short description after the colon for each link. Only public, permanent addresses that are not blocked in your robots.txt.
4. Upload it. The file goes into the root directory as llms.txt, reachable at https://your-domain.com/llms.txt, in UTF-8. The AI providers serve their own files as text/plain or text/markdown (checked on 30 October 2026).
5. Check it. Open the file in the browser, check for status code 200 and readable text, and click every link once.

Keeping it up to date

An outdated llms.txt sends tools to deleted pages or gives old details. Three rules help:

• Tie it to the navigation. When the menu, services or contact details change, the llms.txt belongs on the same list as the sitemap and the legal notice.
• Generate it rather than type it, if possible. llmstxt.org names documentation platforms such as Mintlify that support the format. With your own CMS, the file can be built on publishing from the same data as the sitemap.
• Check the links regularly. A link checker that also works through /llms.txt will find dead entries.

Where Helferlain stands

In the AI visibility area, Helferlain's SEO audit checks whether your website has an llms.txt, together with the access of 13 AI and search crawlers according to robots.txt and the question of whether content only appears via JavaScript. As an export, alongside the report as JSON, there is also a draft llms.txt that you can review, adapt and upload yourself. More on the SEO audit page.