‹ all guides

Sources checked 2026-11-03

llms.txt explained

An llms.txt file offers a curated map of a site for clients that choose to read it. Learn its structure, discovery links and limits.

llms.txt is a plain text file that tells AI agents and language models what a site contains and where the important content lives. It sits at /llms.txt at the root of a site, or at any path inside it. A file at such a path covers the pages under that path, and the most specific file applies. It works like a guide written for machines. A human reads the rendered page, an agent reads llms.txt and follows the links it lists.

File structure

The file opens with the site name as an H1, then a short summary as a blockquote, then the key pages and resources as markdown links grouped under H2 headings. Only the H1 is required. Everything after it, the summary, the headings, and the grouped links, is optional but recommended. The format does expect each item in a file list to be a markdown hyperlink, and a minimal file with just the name is still a valid one. Some sites also publish llms-full.txt, a single file that bundles the full text of the site so an agent can read everything in one request.

The llms.txt of turva.dev departs from the file-list format on purpose. It lists its agent endpoints as plain addresses, for two reasons. First, two of them answer a plain GET with something other than 200 by design, the MCP endpoint with 405 and the x402 endpoint with 402. Second, the whole group keeps one plain format, with no mix of linked and unlinked lines within it. Its pricing, business details, engagement model and contact sections also carry plain values, so those lines were never links to begin with.

The proposal reached v2 in August 2026 and the file format did not change. What changed is how an agent finds the machine-readable forms. A page now names them with two standard link relations, given either as HTML link elements or as an HTTP Link header. The relation rel="alternate" type="text/markdown" is for the markdown version of the page, and rel="describedby" is for the llms.txt that covers it.

v2 also accepts both address forms for a markdown version, page.html.md and page.md. It drops the context expansion tooling that v1 described, so the Optional section is a convention for secondary links and carries no mechanical meaning any more.

How clients find it

A client finds llms.txt at the root of the site, /llms.txt, or through the rel="describedby" link a page names. It finds the markdown version of a page through rel="alternate" or by asking for text/markdown.

The llms.txt proposal recommends the markdown address and the two link relations and does not require that negotiation. HTTP lets a server that has no markdown version disregard the Accept header and send HTML. A client that asks for text/markdown therefore reads the Content-Type of the answer and falls back to the declared markdown address.

For a client that does fetch it, the file saves cost and adds clarity. A normal HTML page wraps the content in navigation markup and scripts that an agent has to read past, and that spends tokens and invites mistakes. An llms.txt file, paired with markdown content negotiation, lets an agent fetch a clean text version and skip the noise.

On turva.dev the markdown version of a page costs a fraction of the HTML. That lowers the chance that a long page runs past a reading tool's content limit and arrives cut short. turva.dev publishes llms.txt and llms-full.txt, serves markdown on request, and publishes the markdown version of every page at its own .md address with both v2 link relations. Whether a client fetches any of it depends on the client.

What it does not replace

llms.txt is not a ranking trick and it does not replace a sitemap or robots.txt. A sitemap lists the URLs a site wants crawlers to find. Where the same content sits at more than one address, it lists the canonical one the site prefers, and not necessarily every URL that exists. robots.txt sets crawl rules. llms.txt is a curated, human-written map of what matters, aimed at models. The three work together. Google states that Search, including its generative features, ignores the file, so publish it for the clients that fetch it rather than as a route into Google or into search rankings.

Does llms.txt really work?

It works as a map for a client that fetches it, but in one large log study 97% of files received no request in a month. It has not been shown to earn AI citations, so publish it for coding agents rather than as a route to visibility.

Ahrefs measured requests across 137 210 domains in May 2026. About 38 000 of them had a valid file, and 97% of those files received no request that month. GPTBot and Claude Code were among the most frequent AI requesters in that data. Ahrefs notes that a request shows a file was fetched and does not show that anything acted on it. SE Ranking modeled nearly 300 000 domains and found no link between having the file and how often a domain is cited. Both are vendor studies of their own samples, and neither measures every client.

The strongest case in this evidence is documentation read by coding agents. As of November 2026, Google's Gemini API docs publish an llms.txt index and recommend a project rule that tells coding agents to consult its docs MCP server or that index, so an agent without the server can start from the index. A site whose readers work through coding agents has a stronger reason to publish the file than a site that hopes to be cited.

Validate your file

A validator reads the file the way a client would. It checks that the H1 is present, that each link is a well-formed absolute address and that the grouping under each heading is well formed. The free validator on this site does not fetch the pages the links point to, so request each one once before publishing to confirm that it still answers.

Check any site's llms.txt structure with the free validator, which reads the file and the home page without signing up.

Frequently asked

What is llms.txt?

llms.txt is a plain text file that tells AI agents and language models what a site contains and where the important content lives. It sits at the root of a site or at any path inside it, where it covers the pages under that path. It does not replace robots.txt or a sitemap, and both keep their own jobs.

Does llms.txt help with search ranking?

No. It is not a ranking trick. Google states that the file neither harms nor helps visibility or rankings in Search, because Search ignores it. What llms.txt gives is a curated map of the content to those clients that do fetch it.

Does llms.txt really work?

It works as a map for a client that fetches it, but in one large log study 97% of files received no request in a month. It has not been shown to earn AI citations, so publish it for coding agents rather than as a route to visibility.

What does an llms.txt file contain?

The site name and a short summary, then the key pages and resources as markdown links, often grouped under headings. Some sites also publish llms-full.txt, which bundles the full text so an agent can read everything in one request.

What changed in v2 of llms.txt?

The file format did not change. v2 added two standard link relations so an agent finds a page's markdown version and its llms.txt without guessing. It also accepted page.md alongside page.html.md as the address of a markdown version and defined what an llms.txt in a subpath covers. It dropped the context expansion tooling along with the mechanical meaning of the Optional section.

Sources