Repository navigation
Streamed usage: the AI ledger reads a call's whole input (#30 item 8) - #72
Merged
Merged
Conversation
…answer (#30 item 8) The gateway asks for stream + stream_options.include_usage and folds the event stream back into the ChatResponse the agents, the recorder and replay already read: text and refusal text joined, tool-call arguments joined by index, finish/stop reasons and the last usage kept. A gateway that ignores stream and answers one JSON body is read as before. An error event, a stream cut before it finished or a non-JSON chunk is a 502; a stall is a 504. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude streams no argument piece for a call that takes none, so the fold gave
"" and 9router refused the next call that sent it back (400 tool_use.input:
Input should be an object). A whole answer carries "{}"; the fold now does too.
Found by the first live recorded turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…em 8) One recorded app-helpdesk turn on Claude Opus 5.5 through 9router: 107,305 input tokens (179 uncached, 103,158 cache read, 3,968 cache write) and 1,779 output, about $0.08 warm and $0.34 cold at list prices. The streamed prompt total is the whole input plus 9router's 2,000 per call; the cache split comes from 9router's own log because its OpenAI-shape usage drops it. The env example says calls are streamed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… item 8) Streamed usage moves sys_ai_runs.prompt_tokens from the uncached remainder to the whole input, about 17x on a builder turn (measured 115,084 tokens a turn, was ~8,000). The 200,000 default then admitted 2 builder turns a tenant a day instead of 26. 3,000,000 admits 27, pinned by a test against the measured turn. The budget is documented in .env.example and passed through docker compose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… an index-less tool call opens on a new id [DONE] alone no longer finishes an answer (partial tool arguments were accepted), and a stream that closes after the finish with neither its usage nor [DONE] is a 502 instead of an answer that escapes the budget. A proxy that streams parallel tool calls without an index gets one call per id instead of one merged call. .env.example says what happens with a gateway that rejects stream_options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps at the logger #70 gave toolExecutor a *zap.Logger and AccessProbes two probes (PacksOf, AggregateScalar) on a base without #68's read-only pin, so main fails the pin since both landed. Both probes are SELECTs and are named as reads; the logger is a listed sink the walk does not enter, and the catch test proves a zapcore.Core reached another way is still reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #30 (item 8). Items 4 and 5 are on main (#70, #69); item 8 stays open for the cache split, which the dev gateway cannot deliver (below).
What changed
internal/ai/gateway.go:Gateway.Completeasks forstream: true+stream_options.include_usageand folds the event stream back into the oneChatResponsethe agents, the recorder and replay already read (foldStream): text and refusal text joined, tool-call arguments joined by index (id/type/name taken as sent,"{}"when no argument piece came), last finish reason, choice and top-levelstop_reason, last usage. A gateway that ignoresstreamand answers one JSON body is read as before. A proxy that streams parallel tool calls without anindexgets one call per id. An answer is finished by a finish reason (or a top-levelstop_reason) and closed by[DONE]or its usage; an error event, a stream that stops before the finish ([DONE]alone included), a stream that closes after the finish with neither usage nor[DONE], a non-JSON chunk or a cut connection isErrGateway(502), key scrubbed; a stall isErrTimeout(504).stream: false, nostream_options). Both demo cassettes replay unchanged (unit test + both demo loops).internal/config/ai_config.go). Streamed,prompt_tokensis the whole input, about 17× the uncached remainder on a builder turn (115,084 tokens a turn, was ~7,958). The old default of 200,000 would have let a tenant run 2 builder turns a day instead of 26. The default is now 3,000,000, which allows 27.TestBudgetGuard_DefaultBudgetAdmitsADayOfBuilderTurnspins that number, using the realLoadAIConfigdefault and the measured turn.budget.go's overshoot note now counts input too (about 1M worst case against 3M).AI_DAILY_TOKEN_BUDGETis documented in.env.examplewith the per-turn figure and passed throughdocker-compose.yml(empty means the default). ⌘K's per-user share, 50%, scales with it.evals/builder-bench/report.md: a new section, "What one app really costs (measured 2026-10-27)"..env.examplesays calls are streamed, and that a gateway which rejectsstream_optionsanswers 400 on every call (502AI_GATEWAY_ERROR, quoted in the server log), so put LiteLLM in front of it.TestToolExecutor_ReachesReadOnlyDependenciesOnly, which fails on maine9e17a8). A pack table delete stays a card the builder confirms: honest card, honest reply, bench expectation (#30 item 4) #70 added a logger and two probes on a base that did not yet have MCP follow-ups: propose_field scope, /mcp brake and thread caps, read-only pin, dev /mcp, live agent thread, class-aware note (#22) #68's pin. The probesPacksOfandAggregateScalar(both SELECT) are now named as reads.*zap.Loggeris a listed sink the walk does not enter. The catch test proves that azapcore.Corereached any other way is still reported. Origin/main is merged in.Prove which: 9router (0.5.99) streamed vs not
Probes through
http://127.0.0.1:20128/v1, modelcc/claude-opus-5-5. "9router log" is its ownusageHistoryrow for the same call.prompt_tokensprompt_tokensis the whole input + 2,000. Not streamed, it is the uncached remainder + 2,000, which is the old floor.app/.next-cli-build/server/chunks/:5330.js(module 73483):gadds2e3toinput_tokens,prompt_tokensandtotal_tokens.hfilters usage per client format. The OpenAI list isprompt_tokens, completion_tokens, total_tokens, cached_tokens, reasoning_tokens, prompt_tokens_details, completion_tokens_details, estimated.8895.js, not streamed:O.usage = WL(O9(O.usage)), after a Claude→OpenAI conversion that keeps onlyinput_tokens/output_tokens.8895.js, streamed, on the finish chunk:a.usage = WL(O9(C.usage), o).C.usageis the translator's state (8499.js, module 29675,message_start/message_delta):prompt_tokens = input + cache_read + cache_creation, with the split undercache_read_input_tokens/cache_creation_input_tokens, which the filter drops./v1/messages(Anthropic shape) passes the usage through unpadded, with the cache fields (one probe). It would fill the columns, but it is a second wire protocol and was not built here.One real builder turn, recorded (
AI_GATEWAY_MODE=record, demo stack,app-helpdeskprompt)The turn ended
outcome okwith the draft kindsentity, app, access, in 3 model calls. Thesys_ai_runsrow:prompt_tokenslist_packs(no arguments) streamed no argument piece, so the fold gave"". 9router refused the next call with400 tool_use.input: Input should be an object. A test now pins"{}".Tests (red first, then each fix broken once)
gateway_stream_test.go:Refused()shape from Declined turns stay declined: outcome per message, every refusal shape, a clean next turn (#30 item 5 leftovers) #69, plus a tool call, a token-id stop_reason andlength;ai_refusal_test.go: the API-level refusal runs streamed too, through the real gateway, both agents and the handlers.include_usage;"{}";stop_reasondropped;Verifier round 1
.env.example, compose passthroughTestBudgetGuard_DefaultBudgetAdmitsADayOfBuilderTurns: redadmits 2 builder turns of 115084 tokens a day; want 27indexmerged into oneFoldsTheStream/parallel tool calls streamed without an index…[DONE]with no finish accepted partial arguments[DONE]closes the stream but does not finish the answerStreamFailures/[DONE] before the answer finished[DONE]or a usage must come (9router's usage rides on the finish chunk; a gateway that ignoresinclude_usagestill sends[DONE])StreamFailures/the stream closes after the finish, before the usage and [DONE]stream_options.env.example, no code fallback (no such gateway seen)TestToolExecutor_ReachesReadOnlyDependenciesOnlyred on the merge, green now; catch test extendedEach fix was broken once on purpose, and its test turned red each time: default back to 200000;
if !seenwithout the id check; the usage/[DONE] check disabled;!doneput back in the finish check; the sink check disabled;PacksOf|AggregateScalarremoved.Gates (verbatim)
Round 0, before the merge:
demo loop passed: 18 steps, on bothcassettes/demo-loop.jsonlanddemo-loop-en.jsonl. For this head, CI ondfca320ran the cassettes through both demo-loop jobs, andTestReplay_TheNonStreamingCassettesStillAnswerreplays every call. All seven checks passed:Not done
sys_ai_runs.cache_read_tokens/cache_write_tokensare still NULL through 9router. Its OpenAI-shape filter drops the split on both paths. A gateway that forwards it fills the columns: OpenAIprompt_tokens_details, and LiteLLM's Anthropic fields are parsed since feat(ai): record prompt-cache tokens when the gateway reports them (partial step for #30 item 8) #44. Ways forward:prompt_tokens_details;/v1/messageswire, which is a second protocol and a config knob;prompt_tokens. The engine cannot tell 9router from another gateway. It overcounts by 6% on a builder call, which errs on the safe side for the daily budget, and the 3M default's 27 turns already include it. The report subtracts it.make test-integrationand the web gates. CI'sgo integrationandwebjobs ran both on this head and passed.check-orphan-endpointsandacceptance-acldo not apply (no route or entity change).🤖 Generated with Claude Code