What an agent pays to read your site
2026-06-26
This article examines the publisher's influence on the content a text-based client receives, using a dated HTML-versus-Markdown comparison.
When an AI agent visits your site to check a price or finish a task for someone, it pays to read the page. That cost is counted in tokens, and a normal HTML page is expensive. Navigation, styling, scripts and structured data all arrive whether the agent needs them or not. The agent either spends its budget getting past that markup or runs out of room and reads only part of the page. Both outcomes are yours to deal with: a page an agent can read in full is more likely to give it your facts right, though this post measures page size rather than whether an agent completed a task or read a fact correctly.
Your surface sets the cost
Most advice about agent token cost is aimed at the people building agents. Cache the prompt, route easy work to a cheaper model. That is real, but it misses the half of the bill that the publisher controls. If your page is heavy and agents read the same markup browsers get, every agent that reads it pays for that weight, on every visit. You cannot tune someone else's model, but you can decide how much your own content costs to read.
The same page, served as clean text
The mechanism is content negotiation. The site keeps serving its normal HTML to browsers, and when an agent asks for the markdown form of a page it gets the same content with the markup stripped out. Nothing is hidden and nothing is duplicated. One URL answers in the format the client asked for.
On turva.dev the homepage as markdown costs roughly a third of the HTML, a couple of thousand tokens against several thousand. An llms.txt sits alongside it as a map of the whole site, so an agent can read the structure in one request instead of crawling it page by page.
What it buys you
A cheaper page is easier to read in full. When the content fits comfortably inside the agent's budget, the agent can read all of it instead of stopping halfway, so it is more likely to quote your real price and your real terms rather than a guess. For anything that ends in a transaction, that can be the difference between a completed action and a wrong one. This post measures page size, not whether an agent completed a task.
It also widens who can reach you. The assistants that answer questions and cite sources may read more of a page that costs less to process, so your pages are more likely to be used in full and represented accurately. The work an agent does against your site gets cheaper for whoever runs it, which makes you the easier site to integrate with when an agent is choosing where to act.
The benefit is measurable. An independent scanner checks for markdown content negotiation and for an llms.txt, and the result shows up as a higher score in the categories that name it. You do not take the improvement on faith. You read the number before the change and after it.
A small change that lasts
None of this is a rebuild. It is a small piece of code at the edge that picks the response format from the request header, and it keeps working as the site grows. The Worker that does it on turva.dev is public, so you can read exactly what it does before deciding whether it belongs on your own site.
For an audit of how cheaply agents can read your site, contact info@turva.dev.
Frequently asked
Why does a normal HTML page cost an agent more to read?
Navigation, styling, scripts and structured data all arrive whether the agent needs them or not. The agent either spends its budget getting past that markup or runs out of room and reads only part of the page.
How much cheaper is a markdown version of a page?
On this site the homepage as markdown costs roughly a third of the HTML, a couple of thousand tokens against several thousand. An llms.txt sits alongside it, so an agent can read the structure of the site in one request instead of crawling it page by page.
Does serving markdown mean duplicating the site?
No. It is content negotiation. The site keeps serving HTML to browsers, and when an agent asks for the markdown form of a page it gets the same content with the markup stripped out. One URL answers in the format the client asked for.
Corrected 2026-09-27. A paragraph said a cheaper page is a more reliable one. This post measures page size and not task success, so it now says a cheaper page is easier to read in full. No number changed.
Corrected 2026-09-28. The opening sentence said a cheaper page decides whether an agent gets your facts right. It now says a page an agent can read in full is more likely to, and names that this post measures page size rather than task success.
Corrected 2026-09-28. A sentence said assistants read better from clean text and are therefore more likely to use a page in full and represent it accurately. It now ties that to the lower cost of reading the page rather than claiming reading quality itself improves.