llms.txt: How to Implement It and What (Not) to Expect

TL;DR: llms.txt is a community proposal (Jeremy Howard, September 2024) for a Markdown file that maps your site’s most important content for AI systems. Implementing it takes under an hour. But according to Ahrefs’ server-log study, 97% of published llms.txt files received zero requests in May 2026, and no major LLM provider — OpenAI, Anthropic, or Google — officially supports the file. Treat it as a cheap bet, not a lever.

What is llms.txt?

llms.txt is a proposed convention — not an official standard — for a Markdown file placed at the root of your website that gives AI systems a curated map of your most important content. It was proposed by Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, and published on 3 September 2024 at llmstxt.org (2024). No standards body — not the IETF, not the W3C — has adopted it, and you will sometimes see it misspelled as “llm txt” in searches; the correct filename is llms.txt, plural, in the site root.

The idea addresses a real constraint: LLM context windows are limited, and full HTML pages carry navigation, ads, and scripts that waste tokens. A single Markdown file saying “here is what this site is about, and here are the pages that matter” would, in theory, let a model get to the substance faster. Whether anything actually reads that file is a separate question — and the honest answer, covered below, is: almost nothing does.

It helps to place llms.txt next to the two files it gets confused with:

• robots.txt controls crawler access — which bots may fetch which URLs. OpenAI and Google both document robots.txt as the control mechanism for their AI crawlers.
• sitemap.xml lists all indexable URLs for search engines, with no editorial judgment.
• llms.txt proposes a curated, annotated reading list for AI systems — a claim about what matters, with no enforcement and, so far, no confirmed consumers.

What does an llms.txt file look like?

An llms.txt file is plain Markdown with one required element: an H1 heading with the name of the project or site — the llms.txt specification (2024) states this is “the only required section”. After the H1, the spec allows a blockquote with a short summary, optional free-form paragraphs, and zero or more H2 sections containing link lists in the format [name](url): optional notes.

Here is a minimal, spec-compliant example:

# Example Docs

> Example Docs is the documentation hub for the Example API,
> covering authentication, endpoints, and rate limits.

Details that don't fit the summary can go here as plain paragraphs.

## Guides

- [Quickstart](https://example.com/quickstart.md): install, authenticate, first request
- [Authentication](https://example.com/auth.md): API keys and OAuth flows

## Optional

- [Changelog](https://example.com/changelog.md): full version history

Two details of the spec matter more than they look. First, the H2 section named Optional has special meaning: per the spec (2024), URLs listed there can be skipped when a system needs a shorter context — so put your must-read pages in other sections and the nice-to-haves under Optional. Second, the proposal (2024) also suggests serving clean Markdown versions of individual pages at the same URL with .md appended, and the ecosystem — notably the documentation host Mintlify (2024) — added a variant called llms-full.txt, which compiles all of a site’s documentation into a single Markdown file.

How do you implement llms.txt step by step?

Implementation is a five-step job that most site owners can finish in under an hour — the file is static text, requires no plugins or build tools, and carries little risk to your existing setup.

1. Pick 10–30 pages that define your site. Choose evergreen, high-value pages: pillar guides, documentation, product and methodology pages. Skip tag archives, pagination, and thin posts — this is editorial curation, not a sitemap dump.
2. Write the file in Markdown. Start with an H1 naming the site, add a short blockquote summary, then group links into H2 sections (e.g. “Guides”, “Services”). Use the [name](url): note format; notes should describe what a page answers, not marketing copy.
3. Add an Optional section. Move secondary material — changelogs, about pages, archives — under an H2 named “Optional” so context-constrained systems know what to drop first.
4. Upload it to the site root. The file must resolve at https://yourdomain.com/llms.txt with HTTP 200 and plain text. On WordPress, upload via SFTP or a file-manager plugin; on static hosts, drop it into the public directory.
5. Verify and maintain. Fetch the URL in an incognito window, confirm raw Markdown (not a styled 404), and add the file to your release checklist so links don’t rot.

That is the entire implementation. Now for the part most guides skip: what the file actually does.

Who publishes llms.txt — and why the adoption numbers mislead

Publisher-side adoption is genuinely high, but it says nothing about whether the file is used. An Ahrefs study (2026) of 137,210 domains with verified traffic found that 28% — roughly 38,000 sites — published a valid llms.txt (HTTP 200 with real Markdown, not a soft 404) as of May 2026.

A large share of that adoption happened without anyone deciding anything. On 20 November 2024, Mintlify (2024) enabled automatic generation and hosting of /llms.txt and /llms-full.txt for every documentation site on its platform — instantly adding llms.txt support to every docs site hosted on its platform, including Anthropic, Windsurf, and Bolt.new. When you see “even Anthropic uses llms.txt” cited as proof of relevance, this is what it usually refers to.

And there is a crucial asymmetry hidden in that example. Anthropic does publish an llms.txt for its own developer documentation at platform.claude.com/docs/llms.txt (accessed 2026) — but as Ahrefs (2026) notes, Anthropic nowhere states that its crawlers read anyone else’s llms.txt files. Publishing your own file is not the same as consuming the standard.

Does anything actually read llms.txt files?

Almost nothing does — that is the central finding of the only large log-based study on the subject. The Ahrefs llms.txt study (2026), based on Ahrefs Web Analytics data, found that 97% of published llms.txt files received zero requests in May 2026. The files sit on servers, and neither humans nor bots come for them.

The remaining 3% look no better under a microscope. Among llms.txt files that were requested at all, 96% of requests came from bots, only 19.5% of those bot requests came from named AI tools, and AI retrieval bots — the category that matters for showing up in AI answers — accounted for just 1.1% of all requests (Ahrefs, 2026). The study’s authors put it bluntly: “If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration.”

The provider side confirms the logs. Per Ahrefs (2026), “no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google.” OpenAI’s official crawler documentation (2025) describes access control for GPTBot, OAI-SearchBot, and ChatGPT-User exclusively through robots.txt and published IP ranges — llms.txt is not mentioned once.

Google has been unusually direct. In April 2025, Google’s John Mueller compared llms.txt to the keywords meta tag — a self-declared claim about your own site — and noted that “none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it)” (Search Engine Journal, 2025). In June 2026 he went further, calling the file “purely speculative for now” and pointing out it “has existed for years, yet none of the AI systems use it”, while flagging WebMCP as the more promising direction (Search Engine Journal, 2026).

llms.txt is a cheap bet, not a lever: the file costs an hour and carries little risk, but as of mid-2026 there is no log evidence and no provider commitment suggesting it moves AI visibility at all. Implement it if you like — just don’t put it in the results column.

What controls AI systems today, if llms.txt doesn’t?

The mechanisms that demonstrably work are robots.txt for access control and ordinary crawlable content for visibility — not any AI-specific text file. Google’s official documentation on AI features and your website (2025) states it outright: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”

In other words: the boring infrastructure carries the weight. Clean crawlable HTML, correct robots.txt directives, and content structured for extraction come first — we walk through that sequence in our guide on how to prepare a website for generative engines, and the strategic background is in our generative engine optimization (GEO) guide. GEO — creating and optimizing content so that it gets cited in AI-generated answers — depends on what the file points to, not on the file.

Should you implement llms.txt in 2026?

Yes, if it costs you an hour and zero expectations; no, if you want something that moves results. Ryan Law, who wrote Ahrefs’ analysis, answers the “is it needed?” question with “In my opinion, no, not yet” — while noting the file is just “a Markdown file, hosted on your own website” with “little risk” (Ahrefs, 2026). That is our position too: a cheap bet, not a lever. If the standard ever gains a real consumer, you are ready early; if it never does, you lost an hour.

What llms.txt cannot do is substitute for the work that actually correlates with AI visibility: citable content, source-anchored facts, and presence in the places generative engines already pull from. If the shift from rankings to AI answers is new territory for you, start with what changes between GEO and classic SEO. And if you want to know which levers are worth your hours before you spend them — an llms.txt file will not tell you, but a measurement will: our AI visibility audit shows where your brand already appears in generative engine answers and what is actually holding it back.

FAQ

Does llms.txt improve visibility in ChatGPT, Perplexity, or Google AI Overviews?

No — there is no evidence it does. Per the Ahrefs study (2026), 97% of llms.txt files got zero requests in May 2026, and AI retrieval bots accounted for just 1.1% of all llms.txt requests. Google’s documentation adds that no AI text files are needed to appear in its AI features (Google, 2025).

Is llms.txt the same as robots.txt?

No. robots.txt controls crawler access, and major providers document it as their control mechanism — OpenAI documents GPTBot and OAI-SearchBot control via robots.txt (OpenAI, 2025), and Google-Extended is a robots.txt token (Google, accessed 2026). llms.txt merely suggests reading priorities, and no major provider has committed to reading it.

What is llms-full.txt?

llms-full.txt is an ecosystem extension, not part of the original proposal’s required format: one Markdown file containing a site’s entire documentation, so an AI system can ingest everything in a single fetch. Documentation host Mintlify popularized it by auto-generating both /llms.txt and /llms-full.txt for all hosted docs from November 2024 (Mintlify, 2024).