‹ all posts

The checks that pass for the wrong reason

2026-07-29

The Model Context Protocol cut revision 2026-07-28 on 28 July. I read the released tree that evening, the way I read any spec I am about to depend on, and the finding was not in the prose. It was in the step that produced the release.

Thirteen links that resolved

Cutting a revision copies the living draft tree into a dated one, and every reference inside the copy has to be dated with it. That is the whole promise a dated snapshot makes. Reading the 2026-07-28 tree a year from now should give what it gives today.

The promote step rewrote one link pattern when it copied the spec tree, and that part worked. Two other patterns were not covered. Three links still pointed at the draft schema, including the one whose own next line calls it the source of truth for all protocol messages and structures. Nine pointed at the draft docs tier, eight of those at the security best practices page. The changelog's compare link was the thirteenth.

Not one of them was broken. Every one resolved, just to the living tier rather than the frozen one, which is presumably why a preview check did not catch them. The reader who follows a link labelled 2026-07-28 in a year gets whatever the draft has become by then, and nothing anywhere reports a problem.

I filed it with the line numbers and the two greps I had run to find them. It was closed the same day. The fixed release carries thirteen corrected links, and the promote step now runs all three rewrites over all three copied trees with the changelog compare pinned to the new tag. The second half is the half that matters. A file fix is good for one revision and a workflow fix is good for every revision after it.

The same shape, one day later, in my own gate

The revision is a large breaking one. Sessions are gone, the initialize handshake is gone, the protocol is stateless, and there is a new server discovery request that servers must implement. Both dependencies my own MCP server needed went stable within two days of the cut, so I moved it over.

My site has a verification script that proves the signed MCP server card describes the server that is actually running. It exists because a card once declared capabilities the server did not implement, for weeks, invisibly. The way it proved parity was to run the initialize handshake against the live server and compare what came back.

The new revision has no initialize. The library keeps a compatibility lane that still answers it.

So the gate would have kept passing. It would have reported the same clean count it always reports, and it would have been measuring the old lane while the new one went untested. Not a failure, not a warning, just a number that no longer meant what it said. I rewrote the parity block around the new discovery request before the migration shipped rather than after, and added two checks the old transport had no equivalent for.

What the two have in common

Both are the same defect wearing different clothes. In the release it was links that resolve, so nothing is broken. In my gate it was a check that passes, so nothing is red. In both cases the text and the tooling were fine in isolation and the thing between them had quietly stopped being true.

The practical version, for anyone maintaining a checker: a gate that only fails on breakage cannot tell you that it has started measuring the wrong thing. When the thing under test changes shape, read what your own check actually speaks to. Mine was one line of code and a protocol version, and I would not have found it by running it.

The lesson I took from reading three specs this way is narrower than I expected. On a mature spec, the prose has been over hundreds of eyes and the review process works. The publishing machinery around it has been over far fewer. That is where a fresh reader still has something to contribute.

Related