diff --git a/site-astro/src/content/blog/an-index-cannot-answer-twice.md b/site-astro/src/content/blog/an-index-cannot-answer-twice.md index 20eae30..9e174a1 100644 --- a/site-astro/src/content/blog/an-index-cannot-answer-twice.md +++ b/site-astro/src/content/blog/an-index-cannot-answer-twice.md @@ -23,7 +23,7 @@ Asking "what calls this function" without a graph means a text search, then read The same applies to reverse imports, to inheritance chains, and to the general class of question where the answer is a relationship rather than a location. Text search is bad at those. A graph is good at them. Fine. -## The thing it does not change +## You cannot traverse a graph until you know where to enter it What a graph makes cheaper is the individual hop. What it does not touch is how many hops you need. @@ -33,7 +33,7 @@ You cannot traverse a graph until you know where to enter it. Finding the entry When I traced what these tools do at that first step, the answer was blunter than I expected. The entry point is found by text search: a grep, essentially, against the repository, with the results then sorted using the graph. In the common case, the graph isn't being traversed at all. It's used only to rank the output of a text search. -## Ranking by popularity has a specific failure +## The ranking never sees your question That ranking step is worth looking at closely, because the way it usually works has a consequence people do not expect. @@ -59,7 +59,11 @@ The edge count was several times larger than the one I maintain. Once builtins a If you are evaluating one of these, the useful question isn't how many edges there are. It's what fraction of call sites (the individual places in the code where a call happens) resolved at all, and of those, how many had exactly one candidate. Those two numbers are the honest ones and they are rarely published. -## The part that actually bothers me +The same applies to the three-dimensional rendering, since it comes up. It is for you, not for the agent. The agent receives text; it never sees the picture. A beautiful graph view is a fine thing to build and I understand why it demos well, but it is evidence about a presentation layer, not about whether an agent finished a task in fewer steps. + +The metric that would settle these arguments is unglamorous: for a fixed set of tasks, how many actions did the agent take before it had the right answer. Steps, not tokens, because tokens follow from steps. I have not seen that number published by anyone, including by me, and I am wary of any comparison that leads with something else. + +## The two hours that nothing recorded Everything so far is a quality argument, and quality arguments can be answered with better engineering. Someone can improve the resolver, add the query to the ranking, expose confidence. The next objection cannot be fixed that way, because it is about what kind of object a graph is. @@ -130,18 +134,14 @@ That is the claim in one line. A graph makes each hop cheaper. It does not reduc
A graph makes each hop cheaper. It does not reduce how many hops you need, and because everything in it was already in the source, it cannot hold a conclusion, so the second occurrence of a question costs what the first one did.
-## A note about the visualisation - -Since it comes up: the three-dimensional rendering of your codebase is for you, not for the agent. The agent receives text. It never sees the picture. A beautiful graph view is a fine thing to build and I understand why it demos well, but it is evidence about the tool's presentation layer and not about whether an agent finished a task in fewer steps. - -The metric that would settle these arguments is unglamorous: for a fixed set of tasks, how many actions did the agent take before it had the right answer. Steps, not tokens, because tokens follow from steps. I have not seen that number published by anyone, including by me, and I am wary of any comparison that leads with something else. +## Why the missing record has to be authored -## What has to be different +If the missing thing is a conclusion, two properties follow, and neither is negotiable. -If the missing thing is a conclusion, then the record has to be written by whoever reached it, and it has to be a separate object from the derived index. +It has to be written by whoever reached it. A conclusion is not recoverable from the source afterwards: that is exactly what made it a conclusion rather than a lookup. The only party holding the invariant, the hypothesis that was checked and cleared, and the reason two files are coupled is the one that just spent the two hours. -That is the split coldstart is built around. There is a static index, and it is deliberately ordinary: it ranks files against the words in your question using paths, symbol names, exports, and references, and it can show you the shape of a file and who uses it. It is the cheap-hop layer and I make no larger claim for it. +And it has to be a separate object from the derived index, because it fails in the opposite direction. A derived index cannot go stale; regenerate it and it is correct by construction. An authored record can, and does, the moment someone edits the file it was about. Keeping both in one structure means either regenerating the notes away or serving them with the index's confidence, and both are worse than keeping them apart. -Separately there is a notebook. After a real task, the agent writes down what it worked out, anchored to the files it actually used. That record is not derivable from the source, which is the whole point, and it is also the reason it can go stale in a way the index cannot. So each note carries the state of the files it was based on. If those files changed, the note is not served as truth. It is served as a claim that needs re-checking, which is roughly what a colleague saying "this was true last month" gives you. +That is the split coldstart is built around, and it is worth stating briefly because the argument does not depend on it. There is a static index, deliberately ordinary: it ranks files against the words in your question using paths, symbol names, exports, and references, and it can show you the shape of a file and who uses it. It is the cheap-hop layer and I make no larger claim for it. Separately there is a notebook. After a real task, the agent writes down what it worked out, anchored to the files it actually used, and each note carries the state of those files at the time it was written. If they have changed since, the note is not served as truth. It is served as a claim that needs re-checking, which is roughly what a colleague saying "this was true last month" gives you. -Fewer notes that are true beat a large derived structure that cannot hold a conclusion. Both layers are useful. They are answering different questions, and the mistake worth avoiding is expecting the first one to do the second one's job. +Both layers are useful, and the mistake worth avoiding is expecting the first to do the second's job. The general form, whatever you end up building: anything derived from your source can only tell you what is in the source. It cannot tell you what was already checked and ruled out, and on the second occurrence of a hard question, that is most of what you wanted to know. diff --git a/site-astro/src/content/blog/codebase-memory-is-not-agent-memory.md b/site-astro/src/content/blog/codebase-memory-is-not-agent-memory.md index ed66cb9..869a08f 100644 --- a/site-astro/src/content/blog/codebase-memory-is-not-agent-memory.md +++ b/site-astro/src/content/blog/codebase-memory-is-not-agent-memory.md @@ -6,14 +6,28 @@ keywords: "codebase memory, agent memory, persistent memory for coding agents, A kicker: "Memory" ogDescription: "Remembering a person and remembering a place are different problems. Codebase memory needs its own shape, not a copy of conversational memory." publishDate: 2026-08-08 -readingTime: "6 min" +readingTime: "7 min" tags: ["memory", "notebook", "codebase-memory"] next: "why-coldstart-makes-zero-llm-calls" --- Most things called "memory" for an AI agent are about a person. The agent talked to you last week, and it should remember your name, your preferences, the project you mentioned, the decision you already made so it doesn't ask again. That is a real problem and a reasonable one to solve with a store of facts about a user, retrieved by similarity when a new conversation starts. -I build a different kind of memory, and for a while I described it the same way, because the word was already sitting there. It took getting the design wrong once to notice it is not the same problem at all. +I build a different kind of memory, and for a while I described it the same way, because the word was already sitting there. Borrowing the word turned out to mean borrowing an assumption with it, and I did not notice until the assumption was already shipped. + +## The retrieval keys that never expired + +The mistake was in the retrieval keys. + +Every note in coldstart's notebook carries a handful of short alias strings: the words a later search should be able to find it by. I let them accumulate. Each time an agent updated a note it contributed whatever vocabulary it had been using, and nothing was ever removed. + +That is the right behaviour for memory about a person. You told me in March that you prefer typed configs. That is still true in August unless you say otherwise, so a store of facts about you should union and hold, and forgetting would be the bug. + +A note about a file is not like that, and the reason is specific. Much of the vocabulary an agent brings to a file arrives from an incident. Something was broken, the symptoms had names, and those names are what ended up written down. Once the bug is fixed, the symptom stops being a description of the file. It is a description of a week. + +So the aliases silted up. Going back through my own notebook I found notes still keyed on phrases like "capture never fires" and "stop hook fires rarely": accurate accounts of a regression that had been fixed a while earlier, still attached to a note whose actual summary had moved on, still pulling that note to the surface whenever those words came up. Nothing was corrupt. Every one of those strings had been true when it was written. They had simply outlived the thing they described, and nothing in the design gave them any way to expire, because I had built them on the assumption that a fact once stated holds until it is contradicted. + +The fix was to split the field in two. Names for stable things about the file accumulate, as they should. Words tied to whatever the note currently claims get replaced wholesale by the next write that changes the note's substance, so silence means "not carried forward" rather than "keep the old ones." That is a small change. The reason it was needed is the entire subject of this post: I had answered a codebase-memory question with a conversational-memory answer, and the two problems disagree about whether facts expire. ## What a codebase already remembers @@ -70,12 +84,14 @@ That's the part conversational memory tooling doesn't have to solve and codebase
A preference has no ground truth to check against, so it just holds until you say otherwise. A note about code sits next to the file it describes, and that's what lets the system catch the note going stale instead of quietly acting on it, not extra ceremony.
-## Where the notes actually come from +## Who is qualified to write one + +One more difference falls out of the same root. For a person, the agent is the only witness: it heard what you said, and that is the end of it. For code the ground truth is on disk and anyone can re-read it, which means the note worth having is precisely the one that *isn't* readable off the file, and the only party holding that is the agent that just spent the turns working it out. That argument has its own post, [made at length](/blog/notes-should-be-written-by-whoever-read-the-code/). The short form: a codebase note is a byproduct of the task, not a later extraction pass over a log. -The other difference is who's qualified to write the note. For a person, the agent is the only witness. For code, the agent that just spent real turns tracing a call path, reading three files to find where a value actually gets set, running the failing case down to the line, is also the only witness that has the full context right now. A note written after the fact, by summarizing a transcript, is a paraphrase of a paraphrase, not the same thing at all. +## Remembering a person, remembering a place -So the natural place to write a codebase note is the moment the work happens, by the agent doing the work, not as a separate memory-extraction pass over a log later. The note is a byproduct of the task, not a second task. +I stopped calling this "agent memory" because the name imports the wrong answers to questions it does not look like it is answering. -## Naming the category +A memory about a person is a record of what was said, and saying it is what made it true. There is nothing else to check it against, so it holds until it is restated. A note about code is a record of what was *found*, and the thing that made it true is still sitting on disk, changing, without telling you. That single difference is what forces the address, the freshness stamp, the expiry, and the split between a name and a symptom. None of it is extra ceremony. It is what remembering a place costs when the place can move. -I stopped calling this "agent memory" because it invites the wrong comparison. This is memory for a codebase specifically, not a bigger or smarter version of remembering a user's preferences: address it to the code, expire it against the code, and let the agent that did the reading be the one who writes it down. Once you name it as its own thing, the design questions get a lot easier to answer, because you stop importing answers from a problem that isn't the one in front of you. +Every design question I had got easier the moment I stopped answering it from the other problem. diff --git a/site-astro/src/content/blog/from-four-tools-to-two.md b/site-astro/src/content/blog/from-four-tools-to-two.md index 364e52e..2f7ab18 100644 --- a/site-astro/src/content/blog/from-four-tools-to-two.md +++ b/site-astro/src/content/blog/from-four-tools-to-two.md @@ -17,7 +17,9 @@ It started with four. There was an overview operation for locating files, a stru Both traversal operations were deleted. The path there is more instructive than the destination. -## Deletion one: the two tools that were really missing fields +Four removals follow. They are grouped by what each one turned out to be rather than numbered, because the count was never the point: each was a different wrong assumption I had to be shown, and only one of them was about having too many tools. + +## The two tools that were really missing fields The important thing about removing the traversal tools is what did not get removed. The graph stayed. The import edges, the call edges, the reverse lookups, all of it survived and is still there. @@ -95,7 +97,7 @@ Which leaves two real options. Default it on and accept the cost, or move it to The general version: if a capability requires the agent to opt in, and the agent only knows to opt in after seeing the result, it will not get used. Make it the default or fold it into something else. A third option of explaining it more clearly does not exist. -## Deletion three: the labels +## The labels that were guesses in the costume of facts The first version tried to tell the agent what each file *was*. @@ -109,7 +111,7 @@ The deeper problem is a division of labour. The agent reading my output is a lan So the rule I ended up with: return evidence, never classification. Paths, symbol names, exports, references, line numbers, the matched lines themselves. Things I can point at in a file. No role labels, no capability tags, and no generated descriptions of what a file is for. Let the model do the interpreting, since it is better at it than my heuristics and it is going to redo the work anyway. -## Deletion four: the duplicate +## One operation, two names A smaller one, included because the lesson is not the obvious one. @@ -123,9 +125,9 @@ So the tools were renamed to match the shell verbs exactly, the duplicate verb w The cost of an extra name isn't the code: it's that everything you write about your tool now has to be conditional. -## Deletion five: the server that served +## The daemon that served nothing -The third removal is the one I would not have predicted. +The last removal is the one I would not have predicted. The architecture had a background process, a daemon, that held the index in memory and answered queries over a local connection. This is the obvious shape. The index is expensive to build, so build it once, keep it warm, and have the command line be a thin client that asks the running process. There was a bridge layer and an HTTP daemon (a background server, listening for requests the way a website's server does) to do exactly this. diff --git a/site-astro/src/content/blog/how-your-own-benchmark-lies-to-you-first.md b/site-astro/src/content/blog/how-your-own-benchmark-lies-to-you-first.md index bd9ed9f..1631c8e 100644 --- a/site-astro/src/content/blog/how-your-own-benchmark-lies-to-you-first.md +++ b/site-astro/src/content/blog/how-your-own-benchmark-lies-to-you-first.md @@ -11,33 +11,17 @@ tags: ["benchmark", "methodology", "cost"] next: "what-a-graph-cannot-see" --- -[Why most token savings tools lie](/blog/why-most-token-savings-tools-lie/) is about a number someone else is showing you, and what to check before believing it. This one is earlier in the process: how the number gets built at all, and the specific ways a benchmark run in good faith still comes out wrong. All four of these are mistakes I made on coldstart's own eval, caught before the number shipped, not mistakes I'm describing from watching someone else make them. +A change I was testing came back with 2,136 resolved reference edges against the baseline's 2,131. Five edges missing, same repository, from a change that had no business touching resolution at all. That is exactly the shape of a regression worth an afternoon. -## Don't let the query know the answer +Before spending the afternoon I ran the unchanged baseline against itself a second time. It came back at 2,135. -A recall number (of the files a correct answer actually needs, how many the tool surfaced) is only honest if the query that produced it could plausibly have come from an agent that hadn't already seen the answer. That sounds obvious and it's easy to violate by accident. Early on, I hand-wrote a test query against a component I'd already found on disk, using a phrase pulled straight from its filename. The tool ranked it first. Of course it did. The query wasn't testing whether the tool could find that file. It was testing whether the tool could match a string against itself, because I'd already done the finding and encoded the result into the question. - -Writing more careful queries doesn't fix this. A query written by someone who already knows the target file can't be trusted as evidence, no matter how careful they are: the leak is in the fact that the answer was already visible when the question was written, not in how the query is worded. The only queries that count are the ones a real agent actually asked while working a task blind, mined from the run's own transcript, not authored afterward to look plausible. If you don't have real transcripts yet, you don't have a recall number yet either. - -## One arm, one variable - -A comparison is only informative if exactly one thing differs between the arms being compared (an "arm" here just means one side of the comparison, like a run with coldstart on versus a run with coldstart off). This one breaks less obviously: I ran a no-tool baseline where the agent, left without navigation help, delegated part of the task to a sub-agent. That's a reasonable thing for an agent to do on its own, and it silently corrupted the comparison two different ways. - -First, sub-agent delegation is itself a way of managing context, arguably the main alternative to a navigation tool. A baseline that uses it quietly stops being a "no tool" comparison: delegating is a different tool, so the cost difference between arms stopped being attributable to the thing I was actually trying to test. Second, and worse: a sub-agent's reads and greps happen in a separate context that doesn't appear in the main transcript. If you're building your ground-truth file list (the pre-decided list of files a correct answer has to include) from what the transcript shows was read, and part of the real exploration happened somewhere the transcript can't see, your ground truth is quietly missing files, and every recall number computed against it is wrong in a way that doesn't announce itself. - -The fix is a flat rule, not a judgment call: forbid sub-agent delegation in both arms, verify from the transcript that neither one used it, and if delegation itself is worth measuring, make it a third arm with its own baseline, never a variable that leaks into an arm meant to isolate something else. - -## Lock the environment, or re-run both arms together +The regression was sitting inside the range the baseline produces on its own, from nothing but being run again. [Why most token savings tools lie](/blog/why-most-token-savings-tools-lie/) is about a number someone else is showing you and what to check before believing it. This one is earlier in the process: how a number goes wrong while you are still building it, in ways that are invisible from inside the run. All four of the guardrails below exist because I broke them on coldstart's own eval and caught it before the number shipped. The one above is simply the cheapest to demonstrate, because the wrong conclusion was sitting there in two integers. -The agent's working environment is part of what you're measuring, whether you intend it to be or not. Rules files, IDE and CLI versions, the model checkpoint behind the API: all of it shapes how an agent behaves, and none of it is the variable a token-savings benchmark is trying to isolate. Running the baseline arm one week and the tool-enabled arm the next, with the rules file having changed in between, means the gap between arms is now partly a gap between environments, and there's no way to retroactively separate the two once the earlier state is gone. - -The fix costs nothing and is easy to skip anyway because it feels like overhead: pin the rules file for the duration of a benchmark cycle, record the tool and model versions alongside the results, and when you can't be certain the environment held steady, re-run both arms back to back rather than trusting a comparison across drift. Keeping your actual day-to-day environment active during the run is fine, arguably better than a sterile one. The requirement is only that both arms sit inside the same environment, not a clean one. +## Why the baseline moves when nothing changed -## Find the noise floor before you trust a delta +That first one isn't a discipline problem. It's a fact about the system that's easy not to know until it costs you a wrong conclusion. -This one is different: not a discipline problem, but a fact about the system that's easy to not know until it costs you a wrong conclusion. coldstart's own indexer parses files in parallel batches, and completion order feeds into how its resolver breaks ties. Run the exact same code, on the exact same repository, twice in a row, and the count of resolved reference edges (the links coldstart's index draws between files, "this function calls that one") can come out different both times, not because anything changed, but because parallel completion order isn't guaranteed to replay identically. This is what people mean by a noise floor: how far a measurement moves on its own, from nothing but re-running it, with no real change behind it. - -I hit this directly: a change under test showed 2,136 resolved edges against a baseline's 2,131 on the same repository, a five-edge gap that looked exactly like a regression worth chasing. Before chasing it, I ran the unchanged baseline against itself a second time. It came back at 2,135. The "regression" was sitting inside the range the baseline produces on its own, from nothing but rerunning it. +coldstart's own indexer parses files in parallel batches, and completion order feeds into how its resolver breaks ties. Run the exact same code, on the exact same repository, twice in a row, and the count of resolved reference edges (the links coldstart's index draws between files, "this function calls that one") can come out different both times, not because anything changed, but because parallel completion order isn't guaranteed to replay identically. This is what people mean by a noise floor: how far a measurement moves on its own, from nothing but re-running it.
@@ -74,10 +58,36 @@ I hit this directly: a change under test showed 2,136 resolved edges against a b
The baseline moved four edges just from being run twice, nothing else changed. The branch's five-edge gap from the first baseline run was inside that same band: a real difference would have had to clear it, not sit one edge past where the baseline's own noise already reaches.
-Six of the seven repositories in that sweep came back bit-identical between runs. Only the one that looked like a regression turned out to be the one whose count isn't guaranteed to come out the same twice, which is exactly backwards from what it looked like at first glance. Without the second baseline run, that five-edge gap gets chased as a bug or merged past as a false confirmation, and either way the conclusion is wrong for a reason that has nothing to do with the code. The general rule: before attributing any gap to the change under test, run the baseline against itself and see how far it moves on nothing. A gap smaller than that movement isn't evidence yet. +Six of the seven repositories in that sweep came back bit-identical between runs. Only the one that looked like a regression turned out to be the one whose count isn't guaranteed to come out the same twice, which is exactly backwards from what it looked like at first glance. Without the second baseline run, that five-edge gap gets chased as a bug or merged past as a false confirmation, and either way the conclusion is wrong for a reason that has nothing to do with the code. + +The general rule: before attributing any gap to the change under test, run the baseline against itself and see how far it moves on nothing. A gap smaller than that movement isn't evidence yet. -## The checklist +## The query I wrote from the answer -Four checks, applied before a number is trusted enough to write into a sentence with a percentage in it. Was every query pulled from a real run where the answer wasn't visible yet, not authored afterward by someone who already knew it? Do both arms differ in exactly one thing, with sub-agent delegation either banned in both or measured as its own arm? Did the environment hold steady across both arms, or get re-run together when it might not have? And has the baseline been run against itself at least once, so a gap has an actual noise floor to clear before it counts as real? +A recall number (of the files a correct answer actually needs, how many the tool surfaced) is only honest if the query that produced it could plausibly have come from an agent that hadn't already seen the answer. That sounds obvious and it's easy to violate by accident. Early on, I hand-wrote a test query against a component I'd already found on disk, using a phrase pulled straight from its filename. The tool ranked it first. Of course it did. The query wasn't testing whether the tool could find that file. It was testing whether the tool could match a string against itself, because I'd already done the finding and encoded the result into the question. + +Writing more careful queries doesn't fix this. A query written by someone who already knows the target file can't be trusted as evidence, no matter how careful they are: the leak is in the fact that the answer was already visible when the question was written, not in how the query is worded. The only queries that count are the ones a real agent actually asked while working a task blind, mined from the run's own transcript, not authored afterward to look plausible. If you don't have real transcripts yet, you don't have a recall number yet either. + +## The baseline that quietly used a different tool + +A comparison is only informative if exactly one thing differs between the arms being compared (an "arm" here just means one side of the comparison, like a run with coldstart on versus a run with coldstart off). This one breaks less obviously: I ran a no-tool baseline where the agent, left without navigation help, delegated part of the task to a sub-agent. That's a reasonable thing for an agent to do on its own, and it silently corrupted the comparison two different ways. + +First, sub-agent delegation is itself a way of managing context, arguably the main alternative to a navigation tool. A baseline that uses it quietly stops being a "no tool" comparison: delegating is a different tool, so the cost difference between arms stopped being attributable to the thing I was actually trying to test. Second, and worse: a sub-agent's reads and greps happen in a separate context that doesn't appear in the main transcript. If you're building your ground-truth file list (the pre-decided list of files a correct answer has to include) from what the transcript shows was read, and part of the real exploration happened somewhere the transcript can't see, your ground truth is quietly missing files, and every recall number computed against it is wrong in a way that doesn't announce itself. + +The fix is a flat rule, not a judgment call: forbid sub-agent delegation in both arms, verify from the transcript that neither one used it, and if delegation itself is worth measuring, make it a third arm with its own baseline, never a variable that leaks into an arm meant to isolate something else. + +## The environment is an arm too + +The agent's working environment is part of what you're measuring, whether you intend it to be or not. Rules files, IDE and CLI versions, the model checkpoint behind the API: all of it shapes how an agent behaves, and none of it is the variable a token-savings benchmark is trying to isolate. Running the baseline arm one week and the tool-enabled arm the next, with the rules file having changed in between, means the gap between arms is now partly a gap between environments, and there's no way to retroactively separate the two once the earlier state is gone. + +The fix costs nothing and is easy to skip anyway because it feels like overhead: pin the rules file for the duration of a benchmark cycle, record the tool and model versions alongside the results, and when you can't be certain the environment held steady, re-run both arms back to back rather than trusting a comparison across drift. Keeping your actual day-to-day environment active during the run is fine, arguably better than a sterile one. The requirement is only that both arms sit inside the same environment, not a clean one. + +## What was left standing + +None of these were rules I had in advance. Each one is the residue of a run that produced a confident number I then had to throw away, which is why they read as questions rather than principles. + +Four checks, then, applied before a number is trusted enough to write into a sentence with a percentage in it. Was every query pulled from a real run where the answer wasn't visible yet, not authored afterward by someone who already knew it? Do both arms differ in exactly one thing, with sub-agent delegation either banned in both or measured as its own arm? Did the environment hold steady across both arms, or get re-run together when it might not have? And has the baseline been run against itself at least once, so a gap has an actual noise floor to clear before it counts as real? Arches's 64% and JMRI's 31%, the two numbers in the post before this one, are the numbers that were left standing after all four. Not because the methodology is exotic (none of these checks require anything more than re-running something you'd otherwise only run once), but because skipping any one of them produces a number that looks exactly as confident as the honest one, right up until someone tries to reproduce it. + +What changed for me is smaller than a methodology. I no longer trust a number because I remember being careful when I ran it. I trust it because I can say which of these four it survived, and when I can't remember, it goes back through them. diff --git a/site-astro/src/content/blog/notes-should-be-written-by-whoever-read-the-code.md b/site-astro/src/content/blog/notes-should-be-written-by-whoever-read-the-code.md index b902200..64ad5ae 100644 --- a/site-astro/src/content/blog/notes-should-be-written-by-whoever-read-the-code.md +++ b/site-astro/src/content/blog/notes-should-be-written-by-whoever-read-the-code.md @@ -7,14 +7,12 @@ ogDescription: "A transcript is not the same thing as context. The agent that di lead: "I maintain a tool that keeps notes about a codebase. An agent does a task, and at the end it writes down what it worked out. There is an easier way to build this. I chose not to take it." publishDate: 2026-07-25 readingTime: "6 min" -wordCount: 1086 +wordCount: 1128 tags: ["notebook", "agent-memory", "capture"] next: "codebase-memory-is-not-agent-memory" --- -I maintain a tool that keeps notes about a codebase. An agent does a task, and at the end it writes down what it worked out. There is an easier way to build this. I chose not to take it. - -The easier design is a background job. Coding agents already leave transcripts behind, so a tool can wake up later, read the finished sessions, decide whether anything durable happened, and write notes from the log. +The easier design I turned down is a background job. Coding agents already leave transcripts behind, so a tool can wake up later, read the finished sessions, decide whether anything durable happened, and write notes from the log. That design has obvious advantages. It never interrupts you. It can backfill old sessions. You can improve the summarizer later and run it again over the whole archive. I understand the appeal. I still think it produces worse notes, for a reason that has nothing to do with how clever the summarizer is. @@ -82,7 +80,7 @@ In practice, long sessions get compacted. The tool notices the conversation is r That means transcript-based capture loses the most detail on the sessions that probably mattered most: the long ones, where the hard problem was solved. -## What I do instead +## Write the note while the files are still open The capture runs at the end of a turn, inside the session that just happened. @@ -92,7 +90,7 @@ Then it hands that list back to the agent that just did the work and asks it to The rule I settled on is short: if writing the note needs a read you have not already done, do not write the note. An agent that has to go look something up in order to write a note isn't remembering, it's guessing, and a guess written into a file that another agent will trust later is worse than no file at all. -## Freshness is what makes the note usable +## A note has to know when the ground moved Because the note is written by the agent that read the files, the tool knows which files it came from. It records a content hash for each one, a fingerprint of the file's exact contents at that moment, so any later change to the file produces a different fingerprint. @@ -110,8 +108,8 @@ It also does not know anything on day one. There is no bulk import of your git h I took those costs because fewer notes that are true seem more valuable than a large pile of notes reconstructed from partial context. -## How coldstart fits into this +## A transcript is not the context that produced it -coldstart's notebook follows this design. Notes are written by agents after real work, stored in the repo, anchored to concrete files and symbols, and surfaced later only as reference data. If the evidence changed, the note says so. +coldstart's notebook is one implementation of the rule and not a complicated one. Notes are written by agents at the end of real work, stored in the repo, anchored to the files and symbols they actually came from, stamped against the contents of those files, and surfaced later as reference data rather than instructions. If the evidence changed, the note says so. -The point isn't to build a perfect memory: it's to stop pretending that a transcript is the same thing as the working context that produced it. +The rule is the portable part and it needs none of that machinery. Whoever read the code writes the note, while they are still holding it, or nobody does. A transcript is a record that the reading happened. It is not the reading, and the gap between those two things is most of what you would actually have wanted written down. diff --git a/site-astro/src/content/blog/the-tool-the-agent-doesnt-call.md b/site-astro/src/content/blog/the-tool-the-agent-doesnt-call.md index e993d57..cfef541 100644 --- a/site-astro/src/content/blog/the-tool-the-agent-doesnt-call.md +++ b/site-astro/src/content/blog/the-tool-the-agent-doesnt-call.md @@ -17,6 +17,26 @@ On a large Java repository, I checked how many of the agent's file reads had bee That's not a defect report about the model, it's a fact about how tool use actually happens, and it applies to whatever you are building too. +The first thing I did with that number was look for a bug in my own plumbing, on the theory that an agent cannot bypass an intervention that is genuinely in its path. I found one. It is worth describing before anything else, because it is the failure nearest to hand and the cheapest to rule out, and because ruling it out is what forced the rest of this. + +## The hook that fired and did nothing + +There is a specific engineering error here that I want to describe properly, because I made it and then found the same shape in another project while reading its source, which suggests it is a trap rather than an oversight. + +Most agent harnesses (the program actually running the agent loop, like Claude Code or Cursor) let you register a hook that fires before a tool runs. The obvious use is a gentle intervention: when the agent is about to run a search, notice, and point it at the better path first. + +So you register the hook on the search tools. The dedicated grep tool, the file glob tool. Reasonable. That is where searching happens. + +Then the agent runs a shell command that happens to contain `grep -r`. + +Your hook is registered on the search tools. A shell call is not a search tool. Nothing fires. The agent searches the entire repository, pays for it, and your intervention was never in the path. The other project had the same structure, gated on exactly the tools an agent uses when it is being formal, and blind to the shell it uses when it is being quick. + +Mine was slightly worse, and I only found it by reading my own code with this question in mind. The pattern that decides which calls to intercept did include the shell. The handler that runs afterwards only recognised one specific command inside it. So the hook fired on every shell call and then declined to act on almost all of them. It looked wired up. It was matching a broader surface than it could handle, which is the kind of bug that survives review because the tests pass and the logs look busy. + +The general lesson is worth stating plainly. Instrument the surface the agent actually uses, not the one your tool taxonomy says it should. A general-purpose shell defeats every category-based gate, because anything can happen inside it. + +That bug was real and worth fixing. It was not the explanation. With the intervention properly in the path, the agent still reached for the search it already trusted, which meant the assumption underneath had to be the thing that was wrong: that availability, documentation and an instruction add up to use. + ## Three things that look like adoption and are not The mental model I had was a chain. Make the tool available, describe it clearly, and instruct the agent to use it. Each link seemed necessary and the set seemed sufficient. None of the three does what it appears to. @@ -37,22 +57,6 @@ A custom tool asks for something extra. It asks the agent to trust a ranking it The bar people assume is "is my tool better than a text search." The real bar is "is my tool better by enough, at the exact moment of choosing, to overcome a habit that is already working." Those are different bars, and only the second one predicts behaviour. -## The mistake I shipped, and so did someone else - -There is a specific engineering error here that I want to describe properly, because I made it and then found the same shape in another project while reading its source, which suggests it is a trap rather than an oversight. - -Most agent harnesses (the program actually running the agent loop, like Claude Code or Cursor) let you register a hook that fires before a tool runs. The obvious use is a gentle intervention: when the agent is about to run a search, notice, and point it at the better path first. - -So you register the hook on the search tools. The dedicated grep tool, the file glob tool. Reasonable. That is where searching happens. - -Then the agent runs a shell command that happens to contain `grep -r`. - -Your hook is registered on the search tools. A shell call is not a search tool. Nothing fires. The agent searches the entire repository, pays for it, and your intervention was never in the path. The other project had the same structure, gated on exactly the tools an agent uses when it is being formal, and blind to the shell it uses when it is being quick. - -Mine was slightly worse, and I only found it by reading my own code with this question in mind. The pattern that decides which calls to intercept did include the shell. The handler that runs afterwards only recognised one specific command inside it. So the hook fired on every shell call and then declined to act on almost all of them. It looked wired up. It was matching a broader surface than it could handle, which is the kind of bug that survives review because the tests pass and the logs look busy. - -The general lesson is worth stating plainly. Instrument the surface the agent actually uses, not the one your tool taxonomy says it should. A general-purpose shell defeats every category-based gate, because anything can happen inside it. - ## Injection is not adoption either The next thing you try is stronger. Do not wait to be called. Inject the relevant context into the conversation when the user submits their prompt, so it arrives before the agent decides anything. @@ -75,10 +79,12 @@ The third is to say when the answer is nothing. A tool that returns an empty res What I stopped doing is trying to steer. Instructions telling the agent when not to read a file did nothing measurable in my tests. The interventions that worked were all about the quality of what gets surfaced, never about the discipline of the agent receiving it. -## How this shaped coldstart +## The rule I carried into coldstart The design follows from the failure rather than from an ideal. The commands are shell commands first, because the shell is where the agent already is. Results are ranked with the matched lines shown inline, so the common case is answered without opening anything. An empty result is phrased as a finding about the repository. And notes written after previous work are surfaced automatically at the start of a turn, because the note that has to be requested is the note that never gets read. +The rule underneath all of it is one sentence: you cannot instruct your way into being chosen, so the only lever left is making the tool cheaper to choose than the habit it is competing with, at the exact moment of the choice. Everything on the list above is an application of that. Everything I tried that failed was an attempt to argue with the agent instead of out-competing the alternative it already had. + I would not claim this is solved. My own measurement says the agent still bypasses the tool often, and I would rather publish that than a story where instructions worked. If you are building something in this space, measure the bypass rate before you measure anything else. It is the number that tells you whether you have a tool or a feature nobody reaches for. diff --git a/site-astro/src/content/blog/what-a-graph-cannot-see.md b/site-astro/src/content/blog/what-a-graph-cannot-see.md index df8215d..fafcb33 100644 --- a/site-astro/src/content/blog/what-a-graph-cannot-see.md +++ b/site-astro/src/content/blog/what-a-graph-cannot-see.md @@ -67,4 +67,12 @@ A smarter graph wouldn't fix this. Some couplings are structurally invisible to What ended the pattern was writing the incident down the moment it was understood, as a note attached to the files involved, saying plainly that these three scripts share a convention and any change to the marker shape has to be checked against all three. The next time someone, human or agent, opens one of those files, that note is sitting right there, and the fix starts from the actual failure history instead of from three files that each look complete on their own. -None of this argues against building a graph. coldstart's own `gs` relies on one, and it's exactly the right tool for the relationships it can see: real imports, real calls, real reference edges. The argument is narrower: a graph has an edge it fundamentally cannot draw, and no amount of extra thoroughness fixes that. What fixes it is remembering, in a form attached to the code, the things that broke because the graph didn't know to warn you. +## The missing edge was a conclusion + +None of this argues against building a graph. coldstart's own `gs` relies on one, and it's exactly the right tool for the relationships it can see: real imports, real calls, real reference edges. The argument is narrower than "graphs are insufficient." It's that the edge missing here was never a parsing failure at all. + +Nobody could have read that convention off the three files, because at the time it was written it wasn't in them. It came into existence the way most real couplings do: someone changed one script, something broke in another, and the relationship between them was established by that incident rather than by any statement in the code. The edge is the output of an investigation. Regenerating a graph over the same source, however carefully, cannot produce it, because the source is not where it lives. + +Which is what the notebook is for here, and the claim is small. It doesn't discover couplings; it stores the ones that someone already paid to discover, attached to the files they concern, so the next change to that marker shape starts from the failure history rather than from three files that each read clean on their own. + +The rule I'd take out of this, whatever you're building: static structure will show you the relationships the code states. It will never show you the ones the code only assumes. Those are found by breaking something, and they survive only if whoever broke it wrote them down. diff --git a/site-astro/src/content/blog/where-the-tokens-go.md b/site-astro/src/content/blog/where-the-tokens-go.md index fd26f4e..e299744 100644 --- a/site-astro/src/content/blog/where-the-tokens-go.md +++ b/site-astro/src/content/blog/where-the-tokens-go.md @@ -6,16 +6,26 @@ keywords: "agent token cost, Claude Code token usage, prompt caching, cache_read kicker: "Cost" ogDescription: "Most of what a session costs is not the answer. It is the same context, re-sent on every turn." publishDate: 2026-07-25 -readingTime: "7 min" +readingTime: "8 min" tags: ["cost", "context-windows", "prompt-caching"] next: "an-index-cannot-answer-twice" --- I spent a while building tooling for coding agents while assuming the thing I should optimise was output size. Return fewer lines. Trim the file listing. Compress the search result. It seemed obvious. The tool prints text into the conversation, the conversation costs money, so smaller output costs less. -Then I actually decomposed a session, and the picture was not the one I had in my head. +Then I took one real session, pulled its usage records off disk, and added them up turn by turn. The picture was not the one I had in my head. -## What a turn actually bills for +## What one session actually billed for + +The session was fifteen turns of ordinary work in a real repository. Decomposed, it looked like this. + +Around 90% of the total was cache reads: conversation that had already been sent, being sent again. The fixed base alone, meaning the harness system prompt plus the schema for every tool the agent could call, was roughly 25,000 tokens, resident and unchanged on every one of those fifteen turns. Re-reading that single block came to about half the entire session's bill on its own, and it never contained a word about the task. + +The output of my own tool, the part I had spent weeks trimming, was around 2%. + +I had been optimising the 2%. The arithmetic that explains why is not complicated, and it is worth doing slowly, because it also names the one term that responds to design. + +## Why the same tokens get billed fifteen times A chat with a model is stateless underneath, meaning the model itself has no memory between calls; it doesn't remember the previous turn on its own. Every time the agent takes an action, the entire conversation so far is sent again: the system prompt, the tool definitions, every file that was read, every command that was run, every result that came back. @@ -49,7 +59,7 @@ The practical consequence is that the resident term is mostly not yours to shrin ## Output is a rounding error -Here is the part that killed my original assumption. +That 2% is not a quirk of the session I happened to pick. What the model writes is a small fraction of what a session bills. Not a modest fraction. Small enough that halving it changes almost nothing. The reason is structural: output is generated once, then it becomes part of the conversation and is re-billed as cached context on every subsequent turn, at a much lower rate. The single largest share of a long session is re-reading context that was already established. @@ -128,15 +138,17 @@ A search that returns eight paths, says which ones define the thing you asked ab Same logic in the other direction. Ten small precise calls are worse than two calls that each carry more. Batch what you can. Answer the follow-up before it is asked. -## Measuring your own +## Decomposing your own transcript -None of this needs to be taken on faith. If you use a harness (the program running the agent loop, like Claude Code or Cursor) that writes session transcripts to disk, the numbers are already there. +None of this needs to be taken on faith, and you should not take mine. If you use a harness (the program running the agent loop, like Claude Code or Cursor) that writes session transcripts to disk, the numbers are already there. -Each assistant message carries a usage record with separate counts for fresh input, cache reads, cache writes, and output. The total billed for that turn is the sum of all four. Two things to be careful about. Dedupe by message id first, because a streamed message can appear more than once and double counting will flatter or wreck your result. And group by turn, so you can watch the resident context climb rather than seeing one aggregate. +Each assistant message carries a usage record with separate counts for fresh input, cache reads, cache writes, and output. The total billed for that turn is the sum of all four, and the common mistake is to read only the first one, which makes a session look almost free. Dedupe by message id before you add anything up, because a streamed message can appear in the log more than once and double counting will either flatter your result or wreck it. Then group by turn rather than aggregating the whole file, so you can watch the resident context climb instead of seeing one number at the end. -Then plot the per-turn total across the session. You are looking for two things: how fast the line rises, which tells you what is accumulating, and how many turns there are, which is the thing you can act on. Compare the same task done two ways and count turns, not tokens. Turns are the honest metric because tokens follow from them. +Plot the per-turn total across the session and you are looking for two things. How steeply the line rises tells you what is accumulating: a step change usually means a large file was read and is now being paid for on every remaining turn. How many turns there are tells you the thing you can act on. -I would suggest doing this on your own sessions rather than trusting anyone's published figures, mine included. The shape holds across harnesses. The exact proportions depend on your system prompt, your instruction files, how many tools you have connected, and how large the files in your repository are. A codebase with long files behaves differently from one with short ones. +Then compare the same task done two ways and count turns, not tokens. Turns are the honest metric because tokens follow from them. + +The exact proportions will not be mine. They depend on your system prompt, your instruction files, how many tools you have connected, and how large the files in your repository are: a codebase with long files behaves differently from one with short ones. The shape has held across every harness I have looked at. The specific 90% has not, and I would not expect it to. ## What I changed @@ -146,4 +158,11 @@ In practice that meant a few things. Ranked results instead of a list, so the ag That last one is why coldstart is two commands rather than the larger set I started with. `find` locates the files for a concept and ranks them by evidence. `gs` takes one file and returns its shape along with who uses it. There is no third operation, and cutting the others was not a simplification for its own sake. Each one was rent. -The general form of the lesson is short. Work out the cost model of the thing you are building for before you optimise anything, because the obvious target and the real one are often not the same, and in this case they are not even close. + +## What the decomposition proves, and what it doesn't + +It is one session, on one repository, through one harness. It does not establish that 90% is a constant, and I have not run enough sessions to claim a distribution. Anyone quoting my percentage back at me, including me, is over-reading it. + +What it does establish is an ordering, and the ordering is what the design decisions actually rest on. Re-sent context was larger than everything else combined. Fixed overhead that had nothing to do with the task was larger than the task. Tool output was small enough that halving it could not have moved the total. You do not need a second session to see that trimming output was aimed at the wrong term; you need one session, decomposed honestly. + +Which is the general form of it. Work out the cost model of the thing you are building for before you optimise anything, because the obvious target and the real one are often not the same, and in this case they were not even close. diff --git a/site-astro/src/content/blog/why-coldstart-makes-zero-llm-calls.md b/site-astro/src/content/blog/why-coldstart-makes-zero-llm-calls.md index 3e39b50..d68f39c 100644 --- a/site-astro/src/content/blog/why-coldstart-makes-zero-llm-calls.md +++ b/site-astro/src/content/blog/why-coldstart-makes-zero-llm-calls.md @@ -11,11 +11,19 @@ tags: ["architecture", "embeddings", "search"] next: "why-most-token-savings-tools-lie" --- -coldstart is built from two parts. One is a notebook: an agent writes down what it worked out about your code after a real task, so the next session doesn't re-discover the same thing from scratch. Those notes are AI-written, full stop. The other part is a navigation layer (`find` and `gs`) that decides which files are relevant to a question and maps how they connect. This post is about that second part, and the claim is narrower than the title alone tells you: the navigation layer never calls out to an AI model of its own to decide what's relevant, even though the notebook sitting right next to it is nothing but AI-written notes. +Here is the kind of thing a coding agent actually asks while it is working. Where is the function named `resolveImports`. Who calls this symbol. Which file defines the class this stack trace is pointing at. Which of these three similarly named files is the one the config actually loads. -Some code-search tools build that navigation layer by having a model "understand" your concept instead of matching your words. Ask for "auth" and get back a file called `session_token_validator.py`, even though the word auth never appears in it. That's a real capability, coldstart doesn't have it, and I could have built it in. I chose not to, and it's worth being honest about why. +Every one of those is answered by something the code already declares about itself: a filename, a path segment, an exported name, an import edge. Not one of them requires anything to work out what the code *means*. I bring this up first because it is the observation the rest of the post rests on, and because it is easy to design past. -## What that trick actually costs +coldstart is built from two parts. One is a notebook: an agent writes down what it worked out about your code after a real task, so the next session doesn't re-discover the same thing from scratch. Those notes are AI-written, full stop. The other is a navigation layer (`find` and `gs`) that decides which files are relevant to a question and maps how they connect. This post is about that second part, and the claim is narrower than the title alone tells you: the navigation layer never calls out to an AI model of its own to decide what's relevant, even though the notebook sitting right next to it is nothing but AI-written notes. + +## Why the semantic version looks like the right answer + +Some code-search tools build that navigation layer by having a model "understand" your concept instead of matching your words. Ask for "auth" and get back a file called `session_token_validator.py`, even though the word auth never appears in it. + +That is a real capability, coldstart doesn't have it, and when it lands it is plainly the better answer: the query you were able to think of finds the file you were not able to name. I could have built it in. I wanted it to be the answer for a while. It's worth being honest about why I stopped reaching for it. + +## What the trick costs after the demo The way that "auth" match works: a small AI model reads each file and converts it into an *embedding*, a long list of numbers meant to capture what the file is about. Your query gets turned into a list of numbers the same way, and the tool returns whichever files' numbers land closest to yours. That whole system, the model doing the converting plus the database holding all those number-lists, is usually called a vector index or vector store. @@ -62,15 +70,17 @@ The stability problem is quieter but matters more for a tool an agent calls doze
Declared identity answers the same query with the same file at the same rank, every time. Embedding-based matching has no such guarantee: adding unrelated files anywhere in the repo can push yesterday's rank-one result to rank four, with nothing about the query having changed.
-## What coldstart uses instead +## What it uses instead Files already declare their own identity: a filename, the segments of its path, the names it exports. Most of the time, the thing you're looking for is named close to what you'd call it, because someone wrote that name for exactly the reason you're now searching for it: so the next person reading the codebase could find it. coldstart ranks files by how many of your query terms they actually cover, using that declared identity plus a real repo-wide text search (backed by [ripgrep](https://github.com/BurntSushi/ripgrep), a fast plain-text search tool, where it's available; plain `git grep` where it isn't). +## Where coldstart is the worse tool + This is a worse tool than an embedding-based search for a genuinely fuzzy conceptual query, one where nothing in your vocabulary overlaps with anything in the file. I don't think that's a gap worth pretending away. If you want that kind of retrieval, point an embedding-based tool at the same repo. coldstart isn't trying to be the same thing done differently, it's trying to be exact where exactness is available, and honest about the rest. ([More on where that line actually falls.](/vs/vector-rag/)) -## Why the tradeoff wins for the common case +## Why the exact half is the common half -The thing I noticed watching real coding sessions is that most navigation questions during an actual task aren't conceptual. They're literal: where's the function named `resolveImports`, who calls this symbol, which file defines the class this error is coming from. The code already answers these definitively, in its names and its structure. Paying for an AI model call, plus the staleness and drift that come with it, to answer a question the filenames already settle is a bad trade, made worse by the fact that it happens on almost every call, not occasionally. +The questions at the top of this post are not a flattering sample. They are what watching real sessions turns up: most navigation during an actual task isn't conceptual at all, it's literal, and the code already answers the literal kind definitively, in its names and its structure. Paying for an AI model call, plus the staleness and drift that come with it, to answer a question the filenames already settle is a bad trade, made worse by the fact that it happens on almost every call, not occasionally. There's a second reason that matters less philosophically and more practically: coldstart runs a background process that keeps its index current as you edit, patching just the changed files within a few seconds of a save. That only works cheaply because there's no AI model to call and no number-list to recompute. A patch is a few milliseconds of parsing per changed file. Re-running a model over every file that frequently, for every keystroke-adjacent save across a session, isn't something you'd want to pay for even if you could. diff --git a/site-astro/src/content/blog/why-most-token-savings-tools-lie.md b/site-astro/src/content/blog/why-most-token-savings-tools-lie.md index 89171ff..3880a91 100644 --- a/site-astro/src/content/blog/why-most-token-savings-tools-lie.md +++ b/site-astro/src/content/blog/why-most-token-savings-tools-lie.md @@ -11,47 +11,29 @@ tags: ["benchmark", "cost", "methodology"] next: "how-your-own-benchmark-lies-to-you-first" --- -There's a temptation with any benchmark: run it once, get a number that looks good, put it on the site. I did that once with coldstart's own numbers, caught it before it shipped, and the fix taught me more about what I was actually measuring than the number itself did. +The JMRI sweep came back at 39% fewer tokens. That is a strong number, it was measured on a real repository with a real agent, and I nearly put it on the site. -## The shortcut that breaks +What stopped me was wall-clock time. The baseline runs in that sweep were taking noticeably longer than the coldstart runs, by more than the token counts alone explained. The machine running them was also doing other work at the time, and the baseline's heavier reliance on repeated full-file reads and directory walks made it more sensitive to that contention than coldstart's smaller, targeted calls were. Some part of that 39% wasn't the tool. It was a machine under uneven load. -Most token-savings numbers you'll see for agent tooling are measuring the wrong thing, and the reason is mechanical, not a matter of anyone rounding generously. I covered the mechanism in [Where the tokens go in an agent session](/blog/where-the-tokens-go/): on a real trace I decomposed turn by turn, about 90% of total tokens were cache reads, the accumulated conversation getting re-billed on every single turn. Coldstart's own output, the part a tool can shrink directly, was roughly 2% of the bill. - -That ratio is why compressing an individual call doesn't tell you what you think it tells you. If a tool takes a grep result, a directory listing, or a batch of retrieved context and returns a smaller version of it, that's a real, measurable reduction of one artifact. It says nothing about the session, because it doesn't touch turn count, and turn count is what the other 98% is riding on. Worse, it can go the other way: give an agent cheaper access to more context and it often just asks for more of it, or issues more calls to compensate for a thinner answer, and the total goes up even though the one number that got measured, the size of a single response, went down. - -So the only claim worth calling a savings number is total tokens (or dollars) across an entire session, one full task done with the tool against the same task done without it, counted from the session's own usage records afterward. Anything narrower, a single call's output, a compressed context batch, a shorter prompt, is a real measurement of something, just not of that. +I reran it in isolation and got 31%. Everything else here is downstream of those eight points, so the measurement comes before anything I draw from it. -## When smaller costs more +## What the two arms actually were -There's a version of this that doesn't just measure the wrong thing, it can point in the wrong direction entirely, and the mechanism is worth understanding because it isn't obvious from outside. - -Prompt caching is what makes a long agent session affordable at all, and every provider that offers it works the same way underneath: the cache is keyed to an exact prefix, the literal, unchanged beginning of the conversation. Send that same prefix again on the next turn and you pay a fraction of the price for it. Change one byte anywhere before the cache boundary, summarize an earlier exchange, drop a tool call judged no longer relevant, reorder something to save space, and the match breaks from that point forward. Every token after the edit gets billed fresh on the very next turn, at full price, regardless of how much smaller the edited version is. - -That means a tool can genuinely shrink what's sitting in the context window and still raise the bill for that session, because it converted tokens that were about to be cheap cache reads back into expensive cache writes. A dashboard reading "40% smaller context" isn't lying about the context. It's just not the same claim as "40% cheaper session," and on the same transcript, the two numbers can point in opposite directions. - -The ratio this rides on is large enough to make the bet a bad one by default. On a real session I decomposed for [the tokens post](/blog/where-the-tokens-go/), the fixed base alone, system prompt plus tool schemas (the descriptions of every tool the agent can call, sent on every turn whether it uses one or not), sat around 25,000 tokens, resident and unchanged on every turn. Across 15 turns, re-reading that one unchanged block was roughly half the entire session's bill, and every one of those re-reads was a cheap cache hit, precisely because nothing before that point in the conversation ever moved. Touch anything earlier in the transcript to save space and you're betting against that ratio, not for it: you're risking the cheap half of the bill to shrink a number that was never the expensive part. - -So the question worth asking of anything that compresses or rewrites context already in the conversation, as opposed to a single new call's output, is whether its savings number survives being measured after the edit, from the resulting usage record, on the same real task, rather than compared against the size of the thing it rewrote. If the number only holds at the instant the edit is applied, it isn't a session cost number yet. It's a description of the edit. - -## What I actually measured - -That's the standard I held coldstart's own numbers to: turns and total tokens across a whole session, coldstart wired in against a no-tool baseline doing the same task with plain file reads, greps, and directory listings. Call each of those two setups an arm, the standard term for one side of a comparison like this. Each arm is a real agent session against a real repository, tokens read off that session's own usage records, not estimated and not summed from a printed total. Both arms ran the same fixed list of queries, once each, no retries. +The standard, stated before the numbers: turns and total tokens across a whole session, coldstart wired in against a no-tool baseline doing the same task with plain file reads, greps, and directory listings. Call each of those two setups an arm, the standard term for one side of a comparison like this. Each arm is a real agent session against a real repository, tokens read off that session's own usage records, not estimated and not summed from a printed total. Both arms ran the same fixed list of queries, once each, no retries. I ran this on two real open-source applications rather than a synthetic benchmark repo, because a repo built to be benchmarked tends to have suspiciously clean naming and structure that a real codebase doesn't. Arches is a Python and Django application, 27 queries. JMRI is a Java application, considerably more verbose per file and with a heavier build surface, 25 queries. Both query lists are navigation questions representative of what an agent actually asks while working a real task in that codebase: where's the thing that handles this, who calls this function, what does this error trace back to. Recall, meaning how many of the files a correct answer actually needed were the ones the agent found, was scored against a fixed, pre-written list decided before either arm ran, not reconstructed afterward from whichever arm did better. -## The number that was too good +## The number after the machine was quiet The Arches sweep, 27 queries, came out to a 64% reduction in tokens against the no-tool baseline, with recall two points better on top of that, because the baseline sometimes ran out of turns before it found the right file at all and gave a worse answer, not just a slower one. -The JMRI sweep, 25 queries, first came out at 39%. That's a strong number and I almost left it there. What made me distrust it was noticing the baseline runs in that sweep were taking noticeably longer in wall-clock time than the coldstart runs, more than the token counts alone explained. The machine running the sweep was also doing other work at the time, and the baseline's heavier reliance on repeated full-file reads and directory walks made it more sensitive to that contention than coldstart's smaller, targeted calls were. The savings number wasn't purely measuring the tool. It was partly measuring a machine under uneven load. - -I reran the JMRI sweep in isolation, no other processes competing for CPU. The real number is 31%, recall at parity rather than better. Fifteen points lower than the first run. +The JMRI sweep, 25 queries, is the one from the top of this post. Rerun in isolation with no other processes competing for CPU, it comes out at 31%, with recall at parity rather than better. Eight points lower than the run I almost published, and a weaker claim on recall as well.
The number that was too good, and the one that replaced it -Three bars. Arches at sixty-four percent fewer tokens. JMRI's first run at thirty-nine percent, flagged as measured on a machine under contention. JMRI's isolated rerun at thirty-one percent, fifteen points lower and the number that actually gets published. +Three bars. Arches at sixty-four percent fewer tokens. JMRI's first run at thirty-nine percent, flagged as measured on a machine under contention. JMRI's isolated rerun at thirty-one percent, eight points lower and the number that actually gets published. @@ -72,34 +54,54 @@ I reran the JMRI sweep in isolation, no other processes competing for CPU. The r 64% 39%, provisional 31% -reran in isolation, 15 points lower +reran in isolation, 8 points lower Arches: 27 queries JMRI: first run machine under contention JMRI: isolated rerun
-
The first JMRI number looked real and wasn't quite: the baseline was running on a machine under contention, inflating the gap. Isolating the rerun brought it down fifteen points, to the number that actually gets published.
+
The first JMRI number looked real and wasn't quite: the baseline was running on a machine under contention, inflating the gap. Isolating the rerun brought it down eight points, to the number that actually gets published.
## Why the correction is the point It would have been easy to keep the first number. Nobody auditing a percentage on a website reruns your benchmark. But if you can't explain how you got a number, it doesn't count as a number, only as a claim, and the difference matters most exactly when nobody's checking. The methodology has to be something a skeptical reader could redo: real repository, fixed query list, controlled machine, counted from the actual transcript's usage records rather than eyeballed from a printed summary, same as the process in the tokens post. -The gap between the two real numbers is informative in its own right. Arches at 64% and JMRI at 31% are not the same win, and averaging them into one blended figure would have erased the reason they differ. Java's verbosity and build tooling change what a single turn costs in that codebase, which changes how much a tool that reduces turns can save. A single number implies the saving is a property of the tool. Two numbers, from two different codebases, show it's a property of the tool interacting with the codebase, which is the truer and less flattering thing to say. - Both numbers also stayed inside the standard from the top of this post: whole session, both arms, same task, counted from the transcript. A tool that only ever shows you a percentage without saying what it was measured against, whole session or single call, is hiding the one fact that would let you check it. -## What this number doesn't cover +## What these numbers don't cover Both sweeps measure navigation: an agent asking where something lives, who calls it, what it depends on. Coldstart also keeps a notebook, and after a real task it writes down what it worked out, which costs its own tokens on top of whatever the navigation calls cost. That write is not in either number above, because a pure navigation query doesn't trigger it, and I haven't yet run a sweep built to isolate what a full task, edits and note-writing included, costs with coldstart against the same task without it. I'd rather say that plainly than fold in a number from a different kind of run just to make this one sound more complete. When that sweep exists, it gets its own post, not a quiet edit to this one. Same caveat on repeats: each arm here ran once per query, not several times averaged. Token counts on a coding agent vary run to run even with nothing else changed, so a single run tells you what happened, not the full spread of what could happen. Wider than that, I haven't measured. -## The test that actually catches it +## The shortcut most published numbers take + +Most token-savings numbers you'll see for agent tooling are measuring the wrong thing, and the reason is mechanical, not a matter of anyone rounding generously. I covered the mechanism in [Where the tokens go in an agent session](/blog/where-the-tokens-go/): on a real trace I decomposed turn by turn, about 90% of total tokens were cache reads, the accumulated conversation getting re-billed on every single turn. Coldstart's own output, the part a tool can shrink directly, was roughly 2% of the bill. + +That ratio is why compressing an individual call doesn't tell you what you think it tells you. If a tool takes a grep result, a directory listing, or a batch of retrieved context and returns a smaller version of it, that's a real, measurable reduction of one artifact. It says nothing about the session, because it doesn't touch turn count, and turn count is what the other 98% is riding on. Worse, it can go the other way: give an agent cheaper access to more context and it often just asks for more of it, or issues more calls to compensate for a thinner answer, and the total goes up even though the one number that got measured, the size of a single response, went down. + +So the only claim worth calling a savings number is total tokens (or dollars) across an entire session, one full task done with the tool against the same task done without it, counted from the session's own usage records afterward. Anything narrower, a single call's output, a compressed context batch, a shorter prompt, is a real measurement of something, just not of that. + +## When smaller costs more + +There's a version of this that doesn't just measure the wrong thing, it can point in the wrong direction entirely, and the mechanism is worth understanding because it isn't obvious from outside. + +Prompt caching is what makes a long agent session affordable at all, and every provider that offers it works the same way underneath: the cache is keyed to an exact prefix, the literal, unchanged beginning of the conversation. Send that same prefix again on the next turn and you pay a fraction of the price for it. Change one byte anywhere before the cache boundary, summarize an earlier exchange, drop a tool call judged no longer relevant, reorder something to save space, and the match breaks from that point forward. Every token after the edit gets billed fresh on the very next turn, at full price, regardless of how much smaller the edited version is. + +That means a tool can genuinely shrink what's sitting in the context window and still raise the bill for that session, because it converted tokens that were about to be cheap cache reads back into expensive cache writes. A dashboard reading "40% smaller context" isn't lying about the context. It's just not the same claim as "40% cheaper session," and on the same transcript, the two numbers can point in opposite directions. + +The ratio this rides on makes the bet a bad one by default. On the session I decomposed in [the tokens post](/blog/where-the-tokens-go/), the fixed base alone, system prompt plus tool schemas, came to roughly half the entire session's bill, and every one of those re-reads was a cheap cache hit precisely because nothing before that point in the conversation ever moved. Touch anything earlier in the transcript to save space and you're betting against that ratio, not for it: you're risking the cheap half of the bill to shrink a number that was never the expensive part. + +So the question worth asking of anything that compresses or rewrites context already in the conversation, as opposed to a single new call's output, is whether its savings number survives being measured after the edit, from the resulting usage record, on the same real task, rather than compared against the size of the thing it rewrote. If the number only holds at the instant the edit is applied, it isn't a session cost number yet. It's a description of the edit. + +## Two questions to ask of any savings number None of this needs a benchmark to check. Take any published savings number and ask two questions. First, is it measured across the whole session, from the final usage record, or against the size of one call, one context batch, or one edit? Second, if it involves rewriting or summarizing context that was already in the conversation, does the number still hold after accounting for the cache writes that rewrite forces on every token that comes after it? A number that survives both is measuring a session. A number that only survives the first is measuring a call. A number that fails the second was measuring the wrong direction the whole time. -## The two numbers, stated plainly +## What a savings number is actually a property of + +Arches at 64% and JMRI at 31% came out of the same method, on the same day, from the same tool. Averaging them would have produced one tidier figure and destroyed the only interesting thing about them, which is that they disagree. Java's verbosity and build tooling change what a single turn costs in that codebase, which changes how much a tool that reduces turns can save there. -Arches: 27-query sweep, 64% fewer tokens than the no-tool baseline, recall two points better. JMRI: 25-query sweep, 31% fewer tokens, recall at parity. Both are navigation-only, one run per arm, measured the same way, on real applications, against a baseline doing the same task without coldstart. If you run this on your own codebase and get a different number, that's expected, not a contradiction, because the number was never a property of the tool alone. +So the honest form of a savings number is not a percentage. It's a percentage with a codebase attached, and a task attached, and a note about how many times it was run. A saving is a property of a tool interacting with a particular repository on a particular kind of work, never a property of the tool by itself. If you run this on your own codebase and get something different, that isn't a contradiction. It's the number behaving the way the quantity actually behaves.