Response headers for AI clients

Response headers describe formats, discovery links and request policy. Check both what a header declares and whether the server behaves accordingly.

Response headers are the metadata a server sends with every page, and the right ones let an AI client work without parsing the full HTML. They are the cheapest place to make a site more legible to automated clients, because a client reads them before it reads the body. The reason headers matter is order. A client fetches the response, reads the status and headers first, and decides what to do next from them. If the headers already say where the structured data is and what formats are available, the client can skip the expensive step of parsing a page built for human display.

Content type

Content-Language and a clean content type tell the client what it is reading. The content type names the media type, so the client knows how to parse the body. Content-Language names the language of the intended audience, which RFC 9110 keeps separate from every language the text may contain, so it is a hint rather than a guarantee. Neither header identifies which version of a page arrived. An ETag or a Last-Modified date supports comparison and revalidation, and a strong validator can confirm exact representation identity; a weak validator, marked with a W/ prefix, only supports semantic equivalence and does not by itself guarantee the client has the byte-for-byte same representation.

Link and Vary

A Link header can point a client straight at a site's machine-readable resources, such as an API catalog or a markdown version of the page, so the client finds them without crawling. A Vary header that includes Accept tells caches that the site can return different formats for the same URL, so a cached markdown response does not reach a client that asked for HTML, or the other way round. The server can still negotiate without it. Where a shared cache stores the response, a missing Vary: Accept can let that cache serve the wrong format to the next client that asks, so check the cache settings and Cache-Control alongside it. A Cache-Control immutable directive set on the wrong response can also stop a client from seeing an update.

Rate limits

RateLimit-Policy states the quota a server enforces, and RateLimit adds the remaining allowance per client where the server tracks one, so a well-behaved client can throttle itself instead of guessing. Sending a RateLimit-Policy header does not by itself prove the server enforces the stated quota. Checking enforcement means sending requests past the stated limit and confirming the server responds accordingly, not reading the header alone. This site sends the policy header on every response its Worker renders and applies it with Cloudflare's rate-limiting binding; a static asset served directly, such as an image, carries no RateLimit-Policy header. The binding keeps an approximate count per client IP in each Cloudflare location, so a burst can pass more requests than the policy states before the first 429. It answers only whether a request may pass and never how much of the quota remains, so the site sends no RateLimit header.

Verification

As of 30 October 2026 the IETF draft is at revision 11, dated 23 May 2026. Datatracker lists it as an active Internet-Draft that expires on 24 November 2026, and it has not become a standard. Checking a site's headers means requesting a page and reading the response headers directly, then confirming the behaviour they describe, such as a different response for a different Accept value or a request that is actually throttled once a stated limit is passed. The fix for a missing or wrong header is usually small and lives at the edge, which on turva.dev is a Cloudflare Worker that sets these headers on every response. For an audit of a site's response headers and the behaviour behind them, contact info@turva.dev.

Frequently asked

Which response headers help AI agents?

A Link header points an agent at machine-readable resources such as an API catalog or a markdown version of the page. A Vary header that includes Accept keeps a cache from serving the wrong format when the same URL answers in both markdown and HTML. A RateLimit-Policy header, and a RateLimit header where the server tracks a per-client allowance, let a well-behaved agent throttle itself, and Content-Language with a clean content type tells the client the intended audience language and the media type it received.

Why do response headers matter to agents?

An agent reads the status and headers before the body and decides what to do from them. If the headers already say where the structured data is and what formats are available, the agent can skip parsing a page built for human display.

Which header keeps a cache from mixing up markdown and HTML?

A Vary header that includes Accept. It tells a cache that the same URL can return more than one representation of the page, so the cache does not hand a markdown response to a client that asked for HTML, or the other way round.