‹ all guides

Sitemaps, robots.txt and AI crawler access

Sources checked 2026-09-15

Sitemaps describe published URLs, while robots.txt communicates crawler rules. Neither file guarantees retrieval or replaces access control.

robots.txt and the sitemap are the oldest machine-readable files on the web, and they still shape whether an agent is welcomed in and what it can find. A well-behaved agent reads robots.txt to learn the rules and the sitemap to learn the map before it reads any page. Not every client does either, so these files set the terms for the agents that follow them rather than for all traffic.

On this page

Crawl preferences

robots.txt sets crawl rules, and it can name AI crawlers explicitly, so a site states whether it welcomes GPTBot and similar clients rather than leaving them to guess. A Content-Signal directive can go further and declare how content may be used, separating ordinary search from AI input and training, which states a granular preference instead of an all-or-nothing block. GPTBot and OAI-SearchBot serve different purposes for the same provider. OpenAI uses OAI-SearchBot for search and GPTBot for training, so blocking one does not block the other, and a page can still be named in an assistant's answer through a source other than the blocked crawler.

URL discovery

The sitemap answers the other question, which is what exists. A complete sitemap lists every canonical URL, so an agent can find the real pages without inferring them from navigation. A last-modified date is optional in the sitemaps protocol and still worth publishing, because it tells a returning client what changed. The sitemap is a hint to the client rather than a guarantee that anything gets fetched. A page that is not in it is still a page an agent may never reach.

Content Signals

A Content-Signal directive is a stated preference and not an enforcement mechanism. The Content Signals documentation says plainly that some automated systems may ignore it. Publishing the directive tells a compliant client what is welcome. It does not by itself stop a client that chooses not to read it or not to honour it.

Checks to run

Getting these files wrong is quietly expensive. A robots.txt that blocks an AI crawler by accident stops that crawler from fetching the pages and can keep the content out of what it feeds, though it does not by itself remove the site from an assistant's answers, since an answer can still name a page through another source. A stale sitemap hides new pages. The files are small and the fix is fast, which is why they are the first thing a readiness review checks. In a scan of 567 company sites finished in September 2026, robots.txt and the sitemap were the two most frequent first-fix subjects among the 74 sites that read Level 0, 45 and 38 of 68 notes, and 29 named both, usually a robots.txt the CMS shipped by default and a sitemap that was missing or never announced in it.

turva.dev declares AI bot rules and Content Signals in robots.txt and keeps a complete sitemap.

Frequently asked

How do robots.txt and the sitemap affect AI agents?

A well-behaved agent reads robots.txt to learn the rules and the sitemap to learn the map before it reads any page, though not every client does either. robots.txt can name AI crawlers explicitly, and the sitemap lists every canonical URL so an agent finds the real pages without inferring them from navigation.

What is a Content-Signal directive in robots.txt?

A Content-Signal directive declares how content may be used, separating ordinary search from AI input and training. It states a granular preference instead of an all-or-nothing block, and its own documentation says some automated systems may ignore it.

Can robots.txt name AI crawlers specifically?

Yes. robots.txt can name AI crawlers explicitly rather than treating every client the same, and a Content-Signal directive separates ordinary search from AI input and training. Both express a preference that a client can ignore.

Sources