Skip to content
doxutaPublic

About

⚡ Fail-closed spending circuit breaker for autonomous AI agents — single Go binary, zero deps, meters token spend live from the SSE stream and cuts the connection when the budget is gone.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

caudao ⚡

A fail-closed spending circuit breaker for autonomous AI agents. Single Go binary, zero dependencies. Point ANTHROPIC_BASE_URL at it and an unattended overnight agent physically cannot burn $300.

Named after the household circuit breaker: when the load gets dangerous, it doesn't warn. It cuts.

$ caudao demo
⚡ caudao demo — daily budget: $0.05, mock model at $500/MTok output
→ an 'agent' starts an expensive streaming request through the breaker...

  streaming...   5 deltas, spent $0.0185
  streaming...  10 deltas, spent $0.0335
  streaming...  15 deltas, spent $0.0485

💥 {"type":"error","error":{"type":"caudao_budget_exhausted","message":"caudao: daily total budget exhausted ($0.0515 of $0.05) — circuit opened mid-stream"}}

⏱  stream cut after 15 deltas in 431ms — final spend $0.0515 (cap $0.05)
→ follow-up request while the breaker is open:
  HTTP 402 {"error":{"message":"caudao: daily total budget exhausted ($0.0515 of $0.05) — request refused","type":"caudao_budget_exhausted"},"type":"error"}

That demo needs no API key — a mock upstream ships in the binary.

Why

Agent frameworks retry. Loops run overnight. A prompt bug can turn one task into ten thousand Opus calls. Client-side budget flags only work if every client honors them; caudao sits under all of them, at the network layer, and enforces the budget on the live token stream:

  • Metered mid-stream — input and cache tokens come from message_start, plus whatever message_delta adds: a reply that runs a server tool (web search, web fetch) reports more input at the end, and each web search adds its fee ($10 per 1,000, server_tool_use.web_search_requests). Output tokens are reported only by message_delta, which a real reply sends once, after its last byte (Proof), so caudao estimates them from the bytes of each content delta as it flows and trips on that. Reported usage replaces the estimate when it arrives; a reply cut before then keeps it.
  • Fail-closed — a model with no configured price is refused; a $0 price is refused at startup; a POST to an endpoint caudao doesn't meter is refused; and a reply caudao cannot read (compressed, or not JSON on a metered endpoint) is answered with 502 rather than forwarded unmetered — and charged a pessimistic estimate anyway, because upstream already billed those tokens and a free refusal would let a retrying agent loop forever with the ledger at zero. Upstream is asked for Accept-Encoding: identity so the meter can always see the stream.
  • Durable ledger — daily spend survives restarts (atomic JSON writes); a tripped breaker stays open until local midnight, and the day only ever rolls FORWARD, so a backwards clock step cannot refill the budget.
  • Reserved, not raced — each in-flight request holds a small reservation against its budget, so concurrent streams cannot all pass the same check and spend the same headroom (before this, 24 concurrent streams overshot a $0.10 cap by 48x).
  • Invisible otherwise — bytes pass through unmodified (no response buffering: one line on the common path, one event while reassembling a multi-line data: field, ~115 MB/s metering throughput); auth headers are never read, stored, or logged.

Use

go install github.com/doxuta/caudao/cmd/caudao@latest
caudao init            # writes caudao.json — set YOUR budgets and prices
caudao serve           # :8484

export ANTHROPIC_BASE_URL=http://localhost:8484
# run your agent as usual; check spend anytime:
caudao status          # {"day":"2026-08-26","total_usd":3.41,...}

Config is deliberately boring JSON — daily total cap, per-model caps, and a price table you own (prices change; keeping them current is your job, being suspicious of unknown models is caudao's):

{
  "daily_total_usd": 25.0,
  "daily_per_model_usd": { "claude-opus": 15.0 },
  "prices": {
    "claude-opus":  { "input_per_mtok": 15.0, "output_per_mtok": 75.0 },
    "claude-opus-5-5": { "input_per_mtok": 4.0, "output_per_mtok": 20.0 },
    "claude-sonnet": { "input_per_mtok": 3.0, "output_per_mtok": 15.0 }
  }
}

A broad prefix prices every model it matches, so keep it at the highest of their rates: claude-opus covers Claude Opus 4.1 at $15 / $75, still served on some cloud platforms. Give the models you run their own narrower entry (Claude Opus 5.5 at $4 / $20, pricing page fetched 2026-11-18). caudao init writes that shape for Opus 5.5, Sonnet 5 and 5.5, and Haiku 5.5.

daily_per_model_usd keys are model-name prefixes, and each prefix is one shared ceiling: with "claude-opus": 15.0, claude-opus-4-1 and claude-opus-4-5 together may spend $15, not $15 each. The longest matching prefix owns a model, so a narrower entry such as "claude-opus-5": 5.0 is its own pool. Models matching no prefix are held only by daily_total_usd.

Cache tokens are billed too (1.25× input for 5-minute cache writes, 2× for 1-hour writes, 0.1× for reads; cache_write_mult, cache_write_1h_mult and cache_read_mult override them per model), and so is each web search ($10 per 1,000, web_search_per_1k per model).

A model priced by prompt length gets a second tier. Claude Haiku 5.5 bills every token of a request whose prompt (input + cache writes + cache reads) is over 100,000 tokens at 5×:

"claude-haiku-5-5": {
  "input_per_mtok": 0.10, "output_per_mtok": 0.50,
  "long_prompt_over_tokens": 100000,
  "long_prompt": { "input_per_mtok": 0.50, "output_per_mtok": 2.50 }
}

The tier is chosen per request, on everything reported so far: a reply whose server tool pushes the prompt over the line is re-priced in full. A long_prompt cheaper than its base, or without a threshold, is refused at startup.

Fast mode has its own price. A request sent with "speed": "fast" (Claude Code's /fast) is charged at the entry's fast rates, with the same cache multipliers and search fee unless the tier sets its own:

"claude-opus-5-5": {
  "input_per_mtok": 4, "output_per_mtok": 20,
  "fast": { "input_per_mtok": 8, "output_per_mtok": 40 }
}

A fast request for a model whose entry has no fast is refused with caudao_unpriced_speed, before it reaches the API: what caudao cannot price, it does not forward. A fast cheaper than its base, or with a tier of its own, is refused at startup.

US-only inference is priced on top. A request sent with "inference_geo": "us" is charged 1.1x on every token category (input, output, cache writes and reads), on the long-prompt tier and on fast mode alike; the web search fee is not a token price and does not change. "global", or no field, is the standard price. Any other geo is refused with caudao_unpriced_geo before it reaches the API.

Proof

Measured 2026-11-05 on an M3 Pro, with claude-haiku-5-5 through Claude Code 2.1.312.

Real streams report output once, at the end. In all 4 replies captured (--include-partial-messages; 153, 240, 172 and 289 content deltas) there was exactly one message_delta, and it was the second-to-last event. That is what these replies did, not a promise about the API. Until 2026-11-05 caudao charged output only from that event, and its own mock sent one after every delta, so every test passed while a real reply could not be cut.

The cut, on a test server with that event order. Replies of 400 deltas, each 64 bytes, 8,000 tokens reported at the end, $4.00 at the test price, against a $0.80 cap:

streams cut mid-reply deltas reaching the client ledger after the stream
before the fix 0 of 50 400 of 400 $4.00, 5x the cap
after the fix 1,000 of 1,000 at most 49 of 400 at most $0.8015

From the moment upstream wrote the delta that crossed the cap:

over 1,000 streams p50 p99 max
client has the breaker event 268 µs 521 µs 2.2 ms
upstream sees the hang-up 1.26 ms 1.35 ms 2.7 ms

This is loopback and the proxy's own cost, nothing more. The test server only checks for a hang-up between deltas, which are 1 ms apart. Run it: CAUDAO_PROOF_N=1000 go test -run TestCutProof -v -count=1 .

One real reply through a real proxy. A 500-word essay with a $0.004 cap (output priced at a placeholder $5/MTok, input near zero). The client got 136 deltas and 1,593 bytes of text and no message_delta; the same prompt uncapped ran to 3,157 bytes. The ledger held 801 output tokens, 798 of them the estimate.

Server tools grow the input after message_start. Captured 2026-11-11, Claude Code 2.1.312's WebSearch call on claude-haiku-5-5: message_start said 2,914 input tokens and the closing message_delta 14,949, its usage being cumulative. Until v0.3.1 caudao charged the 2,914 and missed 12,035, 80% of that reply's input. The three plain replies in the same session repeated their input and cache counts unchanged in message_delta, so v0.3.1 charges them nothing extra. The regression test replays those numbers in that event order.

A web search costs more than the tokens around it. The same stream 3 reported web_search_requests: 1. At the published Haiku 5.5 rates ($0.10 / $0.50 per MTok, pricing) its 14,949 input and 725 output tokens cost $0.0019; the search costs $0.01 on top, 84% of that reply's bill. Until v0.3.2 caudao charged only the tokens. Now each search is charged by how much the cumulative count grew, from the stream or from a non-streaming body, at web_search_per_1k per price entry (default $10). TestWebSearchFeeIsCharged covers 1 and 3 searches, a count repeated from message_start, a free web fetch and a non-streaming body.

One rate per model under-charged long Haiku prompts 5×. The pricing page lists Claude Haiku 5.5 twice: $0.10 / $0.50 per MTok for prompts up to 100,000 tokens, $0.50 / $2.50 over it, cache reads included in the count. Until v0.3.3 a prefix had one rate. With Haiku 5.5's published low rate in the table, a 120,000-token prompt (110,000 of it cache reads) with 1,000 output tokens was charged $0.0026 instead of $0.0130, and the breaker opened five times too late. The old example config priced every claude-haiku at 0.8 / 4, above both tiers, so a config left at that example over-charged instead. TestLongPromptTierIsCharged covers that case, exactly 100,000 (not over, still the low rate), a server tool that pushes the prompt from 90,000 to 130,000 inside one reply, and a non-streaming body. No live call: the numbers are the published rates.

Fast mode was charged at the standard rate. Claude Opus 5.5 costs $4 / $20 per MTok, and $8 / $40 with "speed": "fast" (pricing page, fetched 2026-11-15; Fable 5.2 doubles the same way). Until v0.3.4 caudao never read speed. With Opus 5.5's published rate in the table, a fast reply of 3,000 input, 20,000 cache-read and 800 output tokens was charged $0.0320 instead of $0.0640: half its bill. The old example config priced every claude-opus at 15 / 75, above every published fast rate, so a config left at that example over-charged instead. TestFastModeIsCharged covers the streaming and non-streaming settle, the estimate held while a reply streams, the estimate charged when a stream ends without usage, the unmeterable-reply estimate and the in-flight reservation; each held half the right amount on v0.3.3. That Claude Code sends the field was checked on 2026-11-15 with a local server that records only the request body and answers 400 (so no API call, $0): Claude Code 2.1.312 with --settings '{"fastMode":true}' sent "speed": "fast" at the top level of all 6 requests to /v1/messages, and none without it. Through caudao serve with no fast price, v0.3.3 forwarded all 6; v0.3.4 refused the first with caudao_unpriced_speed and forwarded none. The fast price itself is the published rate, not a measured bill.

US-only inference was charged at the global rate. On Claude 4.6 and later, "inference_geo": "us" costs 1.1x on every token category, Haiku 5.5's long tier included, and stacks on fast mode (pricing page and data-residency page, fetched 2026-11-17). Until v0.3.5 caudao never read the field, so every US-only request was charged 10% short and the breaker opened that much late. The same 3,000 input, 20,000 cache-read and 800 output tokens at Opus 5.5's rate: $0.0320 instead of $0.0352; with fast mode too, $0.0640 instead of $0.0704. TestUSInferenceIsCharged checks hand-computed dollar values for the streaming and non-streaming settle, cache writes, the long-prompt tier, fast mode, the mid-stream and end-of-stream estimates, the unmeterable-reply estimate and the in-flight reservation (each short by 10% on v0.3.4), that the search fee stays $10 per 1,000, and that an unknown geo is refused before upstream. The multiplier is the published one, not a measured bill.

1-hour cache writes were charged at the 5-minute rate. A cache write with a 1-hour TTL costs 2x the input price, a 5-minute one 1.25x (prompt caching page, fetched 2026-11-18); the reply counts the 1-hour part in usage.cache_creation.ephemeral_1h_input_tokens, inside cache_creation_input_tokens. Until v0.3.6 caudao read only the total, so every 1-hour write was charged 37.5% short. 10,000 1-hour writes at Opus 5.5's rate: $0.050 instead of $0.080; a streamed reply of 3,000 input, those writes and 800 output tokens: $0.078 instead of $0.108. Claude Code can ask for the 1-hour cache (ENABLE_PROMPT_CACHING_1H since 2.1.108, promptCacheTtl since 2.1.243), and its own CHANGELOG for 2.1.286 fixes the same miss in the Claude apps gateway's spend meter. The 1-hour writes are charged as an extra 0.75x on top of the total, so the bill is the same whichever stream event carries the split. TestOneHourCacheWritesAreCharged checks hand-computed dollar values for a non-streaming reply, mixed 5-minute and 1-hour writes, the split in message_start and only in message_delta, fast mode, US-only inference, the long-prompt tier and the cache_write_1h_mult override on both tiers; each of the first eight was short on v0.3.5. No live call: the multiplier is the published one, and where the split sits in a real stream was not captured.

At the cap the agent stops, and is told why. Until v0.3.0 a budget refusal was a 429. The breaker stays open until midnight, so every retry was wasted. Measured 2026-11-06, Claude Code 2.1.312 and anthropic 2.6.0 for Python, against a local server that always answers with the same refusal:

refusal Claude Code requests what Claude Code prints Python SDK requests
429 11, the last at 183 s Server is temporarily limiting requests (not your usage limit), then caudao's text 3
429 with x-should-retry: false 1 the same wrong line not run
402, what caudao sends now 1 402 caudao: daily total budget exhausted, exit 1 1

Through a real caudao serve on the real API, one capped reply: cut, exit 1, 7 seconds from start to finish. The run before the fix (2026-11-05) spent 137 seconds in 429 retries after its cut. The TypeScript and Go SDKs have the same retry rule (408, 409, 429, every 5xx) in their source; only Python was run.

What still fails.

  • Hidden thinking is invisible until the end. thinking_delta carried 0 bytes in all 4 replies while 197 to 3,076 thinking tokens were billed, 16% to 74% of each reply's output. Those tokens are charged at the final message_delta, so a long thinking pass is not cut; the visible text after it is.
  • The estimate is deliberately high. At 2 bytes a token it never under-counted visible text here (measured 3.19 and 3.16 English, 2.31 Vietnamese, 2.28 Korean), but it over-counts English about 1.6x, so a reply near the cap can be cut early.
  • A cut reply is charged an estimate. Upstream never reports it, and may bill for tokens generated before it noticed the hang-up, which is not measured.
  • A harness with its own retry loop may still loop. Only Claude Code and the Python SDK were measured. Anything else that retries every failure keeps hitting a 402; each one is answered locally and spends nothing. Match on the error type caudao_budget_exhausted, not on the status.
  • Server-tool input arrives at the end, so it cannot cut that reply. It is charged then, and the next request is the one refused.
  • The long-prompt tier after a server tool is a reading, not a measurement. The pricing page counts "all of its input tokens"; caudao applies that to the cumulative input a server tool reports, the higher of the two readings. A request whose prompt crosses 100,000 only that way was not observed live.
  • Fast mode is read from the request, not from usage.speed. Claude Opus 4.6 can accept "speed": "fast" and run at standard speed; caudao then charges the fast rate, too much, on the safe side. Microsoft Foundry's fast deployments use their own model IDs ending in -fast and send no speed: give them their own prefix entry at the fast rate.
  • A workspace that defaults to US-only inference is not seen. caudao reads inference_geo from the request. A workspace whose default_inference_geo is "us" bills a request without the field at 1.1x, and caudao charges it at the global rate; so does a Microsoft Foundry deployment of the US Data Zone Standard type, which sends no inference_geo. Raise those prices by 10% in the table (a request that also says "us" is then charged 1.21x, on the safe side).
  • A reply without the cache_creation split is charged at the 5-minute rate. If you only ever write the 1-hour cache and your replies leave the split out, set cache_write_mult to 2.
  • The search fee also lands at the end. Like the extra input, it trips the next request, not the reply that ran the search.
  • Code execution is not metered. It is billed by container hours (1,550 free a month, then $0.05 an hour) and is free alongside web_search_20260209 or web_fetch_20260209; the usage object reports a request count, not hours.
  • The cap in the message is rounded to cents, so a $0.004 test cap reads $0.00.
  • A 502 is still retried, on purpose. A reply caudao cannot read is charged and answered 502, so a retrying client walks the ledger to the cap and then gets the 402.
  • One live cut is one live cut. The numbers above are the test server's.

What it is not

Not an auth gateway, not a router, not multi-provider, not a proxy for your secrets — one job, done fail-closed. It meters POST /v1/messages (streaming and non-streaming), passes GETs and count_tokens through, and refuses everything else.

Provenance

caudao is the honest, minimal public distillation of the budget layer I run in production on my own self-hosted LLM gateway that routes all my coding agents' traffic (per-model cost caps, deploy locks, allowlists). That system is private; this is the part of it everyone running agents actually needs, rebuilt clean with tests: SSE metering verified byte-for-byte, breaker behavior tested mid-stream, ledger race-clean under -race.

Tiếng Việt

Cầu dao chống cháy ví cho AI agent: proxy Go một binary, đo token ngay giữa SSE stream và ngắt kết nối đúng lúc ngân sách ngày cạn — agent chạy qua đêm không thể đốt quá số tiền bạn cho phép. Fail-closed: model không có giá thì từ chối, endpoint không đo được thì chặn. caudao demo xem cầu dao sập trong 60 giây, không cần API key.

MIT © Xuan Tai Doan — built with an AI coding agent under human review; see DESIGN.md.

About

⚡ Fail-closed spending circuit breaker for autonomous AI agents — single Go binary, zero deps, meters token spend live from the SSE stream and cuts the connection when the budget is gone.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages