My agent's notes were already a graph

2026-11-06

My Claude Code sessions keep their working memory in plain files: a decision log, a trap log, a measurement log, a dated changelog and a memory folder with one fact per file. On 5 November I built a reference graph and a full-text search over those files. The next day I tuned it three times, twice against fixed query sets and once against real searches from earlier sessions, and ran two experiments with agents. I hoped it would become the way my sessions find things. It stays as a tool, and it replaced nothing. The measurements show that the agents' answers did not change, and my reading is that the documentation was already doing most of that work.

Where did the idea come from?

I saw an example where a language model reads a long PDF and builds a knowledge graph of the people, companies, places and dates in it. I did not test that model, so this post says nothing about how well it works. My problem is a different one. My sessions ask which decision covers a rule and whether that decision still holds. Those links are already written into the files as decision numbers, dated log tags, links between memory facts, file paths and skill names.

So the graph reads those references and uses no model. On 5 November it counted 810 decisions, 1 329 trap entries, 746 measurement entries, 1 918 changelog entries and 312 memory files, with about 29 200 references between them. Each one was written by a session or by me for the next reader. The search uses SQLite's full-text index with trigrams and folded diacritics, because older entries are written without the Finnish letters ä and ö and newer ones with them, and a Finnish compound word is found only as a substring. The index is never saved. A topic search builds it in memory on every run, because several sessions write to the same files at once and the folder is under constant sync, and a lookup by reference does not build it at all.

What did the first measurement promise?

On a test set I derived from the rules, a lookup by reference found the right location 30 times out of 30 and the right status 16 times out of 16. Its output was smaller than an unbounded grep 28 times out of 30.

Then I measured real searches from 124 earlier sessions, 20 lookups by reference and 20 searches by topic. A lookup was smaller than what the session had actually done only 9 times out of 20, because my sessions already narrow their own searches. After I shortened the default output, the lookup was smaller 15 times out of 20. That cost something: the target's own location appeared in the output 5 times out of 19 instead of 8. Two of those 19 cases later turned out to be picking errors in my measurement.

The derived test promised more than the real searches gave. That was the first sign of what the rest of the work showed.

What did tuning change?

Tuning 1 used 1 275 queries fixed before any change, of which 1 007 could be scored, with about 30 percent held out. A candidate stayed only if it did not lower how often the answer appeared in the first five results, and it either raised the first-place hits, raised the mean reciprocal rank or cut the output by at least 5 percent at the same hit level. Bug fixes were kept apart from that rule. Six candidates failed it. On the 300 held-out queries the right answer came first 263 times instead of 195, and appeared in the first five 275 times instead of 200. The main reason was one form. My rules cite a log entry as the file name followed by a date tag, and in the tuning part of the set the lookup resolved that form 0 times out of 147 before tuning and 147 times out of 147 after it.

Search by topic stayed weak. Tuning 2 wrote a new set of 1 099 queries for 240 targets. A subagent read each target and wrote a scenario, and a different one wrote the queries from the scenario alone, without seeing the target or the graph. On the 362 held-out queries, the answer appeared in the first five 195 times instead of 36. By class the gap is plain:

A word search rarely matches an English question to Finnish text. When an agent first translated its own English query into Finnish, it closed 94 percent of the question gap and 69 percent of the keyword gap. Those two figures come from the part the tuning used, so they are optimistic. The search took about 1,8 seconds before and after.

Tuning 3 started from what real sessions do after a search. I read the searches in 663 main session and 2 034 subagent transcripts from 7 October on, with a rule fixed in advance for which read follows which search. Sessions often needed the place where a thing is used and not its definition, so the lookup now also prints up to three such places. The output overlapped the lines the session went on to read in 46 of 98 tuning searches instead of 42, and in 5 of 15 held-out searches instead of 3. Fifteen searches show only a direction. The median output grew by 33 percent on the tuning searches and 74 percent on the held-out ones. Of the 18 tuning searches that ended in a memory fact, 15 still missed. In 7 of those the fact names a date as plain text and not as a reference, which is a job for grep.

Did the agents do better with it?

No. Not in a way I could measure.

The first experiment ran 8 tasks under three conditions: the brief named the graph, did not mention it, or forbade it. Two repeats made 48 Sonnet runs. Two graders scored the answers from 0 to 2 with the tool name masked, though some answers still hinted at the condition. Averaged, the scores were 1,97, 2,00 and 2,00, and every condition found the answers. The condition that did not mention the graph made 15 graph calls across its 16 runs anyway, because my instruction file names the graph, so only the forbidden condition was a clean comparison. Runs that named the graph were the slowest, with a median of 27,6 seconds against 25,6 for the forbidden condition. Both repeats ran as parallel batches, so machine load is in those times. In the runs that named the graph, 8 of the 25 graph calls ended in a usage error or an empty result, and tuning 2 fixed both causes.

The second experiment used the graph as it stood after tuning 2, 12 tasks meant to separate right answers from wrong ones, two conditions and three repeats, 72 runs in all. Both conditions averaged 1,93, and 3 of 36 runs in each scored below 2. I had fixed in advance that the tasks separate the conditions only if at least 6 runs in one of them scored below 2, so they did not. Correctness sat near the ceiling, and a small difference would not have shown. With the graph the summed duration was 11,1 percent lower, the summed tokens 16,0 percent lower and the tool calls 20,7 percent lower. None of those differences held up task by task. The p values were 0,77 for duration and 0,39 for tokens, and 0,065 for tool calls. One more figure did separate: the context of the last API call was smaller with the graph in 10 of 12 tasks, with a p value of 0,039, but that is the size of the final context and not what a run consumed. I measured the durations while another experiment ran on the same machine. None of the 49 graph calls ended in an error or an empty result. Tuning 3 came after both experiments, so no agent experiment covers it.

The searches found the right entry far more often, and in both experiments the answers stayed the same.

Why did it change so little?

I did not measure how sessions follow references, so this is my reading. Many rules in my instruction file name the decision or the log entry they come from, and the instruction file names the home of each kind of record. A session that greps a cited tag lands on the entry, and a session that knows where decisions live opens that file. The graph does that in one call and adds the status and who cites the entry. The answers came out the same either way.

A graph read from references finds only the links someone wrote down. Where nobody wrote them, a model that extracts entities is the obvious thing to try. I have tested neither that nor a messier knowledge base.

What did measuring it teach?

The measurement was wrong first, three times. The baseline of tuning 1 had four errors in my own scoring. One looked for targets in headings only, so 41 of the 42 references it called missing were entries that have only an index line. In the first agent experiment, the token figure turned out to be the context of the last API call and not what the run consumed. The real searches of tuning 3 had two errors in how I picked them, and two cases in the first measurement of real searches were exactly those errors. The scoring errors surfaced only when I opened the misses and read the tool's actual output.

Load distorts timing. A subagent measured a search at 2 553 milliseconds while other runs shared the machine. Run one after another, the same versions took about 1,8 seconds.

A held-out set gets used up. The first one found a real gap, and once I fixed it, those queries no longer tested anything independently. Tuning 2 wrote a new held-out set before it started.

What stays?

The graph stays as a read-only tool next to grep. My instruction file points sessions to it for a lookup by reference: where an entry lives, its status and who cites it, in one call. The lookup takes about 0,37 seconds and a topic search about 1,8. Grep and my other search scripts keep their jobs.

The most lasting part may be a gate built on the same parser. It checks that every decision number, dated log tag, memory link and tools path in my instruction file and my procedure files resolves to a real entry. Its first run checked 1 469 references and found two that pointed nowhere, both of them examples in the text. After I fixed those, it checked 1 467 and found none.

Related

• What agent memory in local files gets me
• The checks that pass for the wrong reason
• TLA+ on my own tools, in numbers