Skip to content

Streamed usage: the AI ledger reads a call's whole input (#30 item 8) - #72

Merged
doxuta merged 8 commits into
mainfrom
slice/streaming-usage
Oct 27, 2026
Merged

doxuta merged 8 commits into
mainfrom
slice/streaming-usage

Conversation

@doxuta

@doxuta doxuta commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Refs #30 (item 8). Items 4 and 5 are on main (#70, #69); item 8 stays open for the cache split, which the dev gateway cannot deliver (below).

What changed

  • internal/ai/gateway.go: Gateway.Complete asks for stream: true + stream_options.include_usage and folds the event stream back into the one ChatResponse the agents, the recorder and replay already read (foldStream): text and refusal text joined, tool-call arguments joined by index (id/type/name taken as sent, "{}" when no argument piece came), last finish reason, choice and top-level stop_reason, last usage. A gateway that ignores stream and answers one JSON body is read as before. A proxy that streams parallel tool calls without an index gets one call per id. An answer is finished by a finish reason (or a top-level stop_reason) and closed by [DONE] or its usage; an error event, a stream that stops before the finish ([DONE] alone included), a stream that closes after the finish with neither usage nor [DONE], a non-JSON chunk or a cut connection is ErrGateway (502), key scrubbed; a stall is ErrTimeout (504).
  • Cassettes are unchanged in format: the recorder writes the folded answer, and the request as the agent built it (stream: false, no stream_options). Both demo cassettes replay unchanged (unit test + both demo loops).
  • The daily token budget is re-based to the new unit (internal/config/ai_config.go). Streamed, prompt_tokens is the whole input, about 17× the uncached remainder on a builder turn (115,084 tokens a turn, was ~7,958). The old default of 200,000 would have let a tenant run 2 builder turns a day instead of 26. The default is now 3,000,000, which allows 27. TestBudgetGuard_DefaultBudgetAdmitsADayOfBuilderTurns pins that number, using the real LoadAIConfig default and the measured turn. budget.go's overshoot note now counts input too (about 1M worst case against 3M). AI_DAILY_TOKEN_BUDGET is documented in .env.example with the per-turn figure and passed through docker-compose.yml (empty means the default). ⌘K's per-user share, 50%, scales with it.
  • evals/builder-bench/report.md: a new section, "What one app really costs (measured 2026-10-27)". .env.example says calls are streamed, and that a gateway which rejects stream_options answers 400 on every call (502 AI_GATEWAY_ERROR, quoted in the server log), so put LiteLLM in front of it.
  • Main's red pin (TestToolExecutor_ReachesReadOnlyDependenciesOnly, which fails on main e9e17a8). A pack table delete stays a card the builder confirms: honest card, honest reply, bench expectation (#30 item 4) #70 added a logger and two probes on a base that did not yet have MCP follow-ups: propose_field scope, /mcp brake and thread caps, read-only pin, dev /mcp, live agent thread, class-aware note (#22) #68's pin. The probes PacksOf and AggregateScalar (both SELECT) are now named as reads. *zap.Logger is a listed sink the walk does not enter. The catch test proves that a zapcore.Core reached any other way is still reported. Origin/main is merged in.

Prove which: 9router (0.5.99) streamed vs not

Probes through http://127.0.0.1:20128/v1, model cc/claude-opus-5-5. "9router log" is its own usageHistory row for the same call.

request path reported prompt_tokens 9router log: whole input / read / write
one-line prompt not streamed 2,042 42 / 0 / 0
one-line prompt streamed 2,042 42 / 0 / 0
44 KB system prompt, call A streamed 18,836 16,836 / 0 / 16,820
same, call B streamed 18,836 16,836 / 16,820 / 0
  • The +2,000 pad is on both paths. Streamed, prompt_tokens is the whole input + 2,000. Not streamed, it is the uncached remainder + 2,000, which is the old floor.
  • The cache split never reaches the client. No cache field appears in either answer.
  • Source, app/.next-cli-build/server/chunks/:
    • 5330.js (module 73483): g adds 2e3 to input_tokens, prompt_tokens and total_tokens. h filters usage per client format. The OpenAI list is prompt_tokens, completion_tokens, total_tokens, cached_tokens, reasoning_tokens, prompt_tokens_details, completion_tokens_details, estimated.
    • 8895.js, not streamed: O.usage = WL(O9(O.usage)), after a Claude→OpenAI conversion that keeps only input_tokens/output_tokens.
    • 8895.js, streamed, on the finish chunk: a.usage = WL(O9(C.usage), o). C.usage is the translator's state (8499.js, module 29675, message_start/message_delta): prompt_tokens = input + cache_read + cache_creation, with the split under cache_read_input_tokens/cache_creation_input_tokens, which the filter drops.
  • /v1/messages (Anthropic shape) passes the usage through unpadded, with the cache fields (one probe). It would fill the columns, but it is a second wire protocol and was not built here.

One real builder turn, recorded (AI_GATEWAY_MODE=record, demo stack, app-helpdesk prompt)

The turn ended outcome ok with the draft kinds entity, app, access, in 3 model calls. The sys_ai_runs row:

outcome  prompt_tokens  output_tokens  cache_read_tokens  cache_write_tokens  tool_calls  latency_ms  model
ok       113305         1779           NULL               NULL                5           19339       cc/claude-opus-5-5
call streamed prompt_tokens whole input (9router log) uncached cache read cache write output
1 35,531 33,531 175 33,356 0 285
2 38,448 36,448 2 33,356 3,090 772
3 39,326 37,326 2 36,446 878 722
turn 113,305 107,305 179 103,158 3,968 1,779
  • 113,305 − 3 × 2,000 = 107,305, the whole input to the token. The same calls not streamed would have recorded 6,179.
  • Cost at Claude Opus 5.5 list prices: about $0.08 per app with a warm cache (writes at the 5-minute rate, which 9router uses for the conversation tail; $0.089 at the 1-hour rate). About $0.34 cold: the 33,356-token persona and tool prefix is written at the 1-hour rate.
  • The live run found a bug that is fixed here (fca5d0d). list_packs (no arguments) streamed no argument piece, so the fold gave "". 9router refused the next call with 400 tool_use.input: Input should be an object. A test now pins "{}".
  • The cassette (3 lines, about 300 KB of persona and schemas) is not committed.

Tests (red first, then each fix broken once)

  • New tests in gateway_stream_test.go:
    • the request asks for a stream with usage;
    • the fold: text, two tool calls in pieces, a call without arguments, 9router's verbatim stream with no [DONE], comments and event lines, CRLF, cache usage;
    • a streamed answer equals the whole-body answer for every Refused() shape from Declined turns stay declined: outcome per message, every refusal shape, a clean next turn (#30 item 5 leftovers) #69, plus a tool call, a token-id stop_reason and length;
    • failures: an error event, a stream cut before the finish, a bad chunk, a cut connection, a mid-stream stall;
    • a streamed call recorded, then replayed;
    • both demo cassettes replay every call unchanged.
  • ai_refusal_test.go: the API-level refusal runs streamed too, through the real gateway, both agents and the handlers.
  • Mutations, each turning red:
    • no include_usage;
    • arguments overwritten instead of joined;
    • no "{}";
    • refusal text dropped;
    • choice stop_reason dropped;
    • an unfinished stream accepted;
    • an error event ignored;
    • usage dropped;
    • no JSON fallback;
    • finish reason dropped (the handler's streamed subtests).

Verifier round 1

finding fix test (red first)
MEDIUM: the 200k default allowed 2 builder turns a day default 3,000,000 (27 turns), .env.example, compose passthrough TestBudgetGuard_DefaultBudgetAdmitsADayOfBuilderTurns: red admits 2 builder turns of 115084 tokens a day; want 27
LOW: parallel tool calls without index merged into one a new id opens a new call FoldsTheStream/parallel tool calls streamed without an index…
LOW: [DONE] with no finish accepted partial arguments [DONE] closes the stream but does not finish the answer StreamFailures/[DONE] before the answer finished
LOW: a finish, then a clean close before the usage, accepted with zero usage after the finish, [DONE] or a usage must come (9router's usage rides on the finish chunk; a gateway that ignores include_usage still sends [DONE]) StreamFailures/the stream closes after the finish, before the usage and [DONE]
LOW: a gateway that rejects stream_options documented in .env.example, no code fallback (no such gateway seen) —
LOW: CI go job red on main's pin pin names the two probes, logger is a sink TestToolExecutor_ReachesReadOnlyDependenciesOnly red on the merge, green now; catch test extended
LOW: cache columns NULL through 9router unchanged, still Refs #30 (below) —

Each fix was broken once on purpose, and its test turned red each time: default back to 200000; if !seen without the id check; the usage/[DONE] check disabled; !done put back in the finish check; the sink check disabled; PacksOf|AggregateScalar removed.

Gates (verbatim)

gofmt -l . -> [] exit=0
go vet ./... exit=0
go test ./... exit=0 ok=42 fail=0
go build ./cmd/server/ exit=0
go test -race ./internal/ai/ ./internal/config/ ./internal/handlers/: ok ok ok
run-integration: PASS=760 SKIP=0 go-test-exit=0   (-timeout 50m -run 'Test' ./internal/ai/ ./cmd/aieval/)
run-integration: PASS=88 SKIP=0 go-test-exit=0    (-timeout 50m -run 'Test(AI|MCP|Chat|Assistant)' ./internal/handlers/)

Round 0, before the merge: demo loop passed: 18 steps, on both cassettes/demo-loop.jsonl and demo-loop-en.jsonl. For this head, CI on dfca320 ran the cassettes through both demo-loop jobs, and TestReplay_TheNonStreamingCassettesStillAnswer replays every call. All seven checks passed:

careful guardrail (hook test suite) | success
go (vet · gofmt · test -race · build) | success
go integration (MySQL 8 · migrate up · -race) | success
web (build · tsc · vitest · eslint) | success
one-command demo (docker compose up · seed · sign in as the rep) | success
demo loop (compose · AI replayed from the cassette · builder and rep in Chrome) | success
demo loop in English (compose · AI replayed from the English cassette · builder and rep in Chrome) | success

Not done

  • sys_ai_runs.cache_read_tokens / cache_write_tokens are still NULL through 9router. Its OpenAI-shape filter drops the split on both paths. A gateway that forwards it fills the columns: OpenAI prompt_tokens_details, and LiteLLM's Anthropic fields are parsed since feat(ai): record prompt-cache tokens when the gateway reports them (partial step for #30 item 8) #44. Ways forward:
    • patch 9router to keep prompt_tokens_details;
    • add an Anthropic /v1/messages wire, which is a second protocol and a config knob;
    • accept the gap.
  • The 2,000 pad stays in prompt_tokens. The engine cannot tell 9router from another gateway. It overcounts by 6% on a builder call, which errs on the safe side for the daily budget, and the 3M default's 27 turns already include it. The report subtracts it.
  • The bench was not re-run. Its tables are still floors; the new section is one measured turn.
  • The budget default comes from one measured builder turn. A Discovery turn (up to 12 steps) costs more, roughly 0.5M worst case, so about 6 of those fit in a day, about as many as the 200k default allowed before the stream.
  • Not run locally: verifier round 2, the full make test-integration and the web gates. CI's go integration and web jobs ran both on this head and passed. check-orphan-endpoints and acceptance-acl do not apply (no route or entity change).

🤖 Generated with Claude Code

doxuta and others added 8 commits October 27, 2026 14:39
…answer (#30 item 8)

The gateway asks for stream + stream_options.include_usage and folds the
event stream back into the ChatResponse the agents, the recorder and replay
already read: text and refusal text joined, tool-call arguments joined by
index, finish/stop reasons and the last usage kept. A gateway that ignores
stream and answers one JSON body is read as before. An error event, a stream
cut before it finished or a non-JSON chunk is a 502; a stall is a 504.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude streams no argument piece for a call that takes none, so the fold gave
"" and 9router refused the next call that sent it back (400 tool_use.input:
Input should be an object). A whole answer carries "{}"; the fold now does too.
Found by the first live recorded turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…em 8)

One recorded app-helpdesk turn on Claude Opus 5.5 through 9router: 107,305
input tokens (179 uncached, 103,158 cache read, 3,968 cache write) and 1,779
output, about $0.08 warm and $0.34 cold at list prices. The streamed prompt
total is the whole input plus 9router's 2,000 per call; the cache split comes
from 9router's own log because its OpenAI-shape usage drops it. The env
example says calls are streamed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… item 8)

Streamed usage moves sys_ai_runs.prompt_tokens from the uncached remainder to the
whole input, about 17x on a builder turn (measured 115,084 tokens a turn, was ~8,000).
The 200,000 default then admitted 2 builder turns a tenant a day instead of 26.
3,000,000 admits 27, pinned by a test against the measured turn. The budget is
documented in .env.example and passed through docker compose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… an index-less tool call opens on a new id

[DONE] alone no longer finishes an answer (partial tool arguments were accepted), and a
stream that closes after the finish with neither its usage nor [DONE] is a 502 instead of
an answer that escapes the budget. A proxy that streams parallel tool calls without an
index gets one call per id instead of one merged call. .env.example says what happens
with a gateway that rejects stream_options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps at the logger

#70 gave toolExecutor a *zap.Logger and AccessProbes two probes (PacksOf, AggregateScalar)
on a base without #68's read-only pin, so main fails the pin since both landed. Both probes
are SELECTs and are named as reads; the logger is a listed sink the walk does not enter,
and the catch test proves a zapcore.Core reached another way is still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@doxuta
doxuta merged commit b7e6a36 into main Oct 27, 2026
7 checks passed
@doxuta
doxuta deleted the slice/streaming-usage branch October 27, 2026 08:47
@doxuta doxuta mentioned this pull request Oct 27, 2026
8 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant