Skip to content
Merged
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ services:
AI_GATEWAY_BASE_URL: ${AI_GATEWAY_BASE_URL:-}
AI_GATEWAY_API_KEY: ${AI_GATEWAY_API_KEY:-}
AI_MODEL: ${AI_MODEL:-claude-opus-5-5}
# Tokens per tenant per UTC day; empty = the default 3,000,000, ~27 builder turns (nexus-platform/.env.example).
AI_DAILY_TOKEN_BUDGET: ${AI_DAILY_TOKEN_BUDGET:-}
# live | record | replay (nexus-platform/.env.example); a cassette path only with record or replay.
AI_GATEWAY_MODE: ${AI_GATEWAY_MODE:-live}
AI_GATEWAY_CASSETTE: ${AI_GATEWAY_CASSETTE:-}
Expand Down
38 changes: 35 additions & 3 deletions evals/builder-bench/report.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,17 +91,49 @@ turns, including a turn the panel sent on its own.
- **The prompt side is a floor.** A model call reported 2,046 prompt tokens on average over the cheap model's 213
calls, and 2,049 over Opus's 65, whatever its size. A one-line test request through the same gateway reported 2,042.
A builder call sends about 70 KB (the persona, 26 tool schemas, the workspace's tables). This gateway reports an
uncached slice of the input; the cached bulk is not in the count.
uncached slice of the input; the cached bulk is not in the count. The engine now streams its calls and reads the
whole input (#30 item 8); one app measured that way is below.
- **The output side is exact:** 2,666 output tokens per successful app on the cheap model, 2,039 on Opus.
- **"KB sent" is the input actually carried:** 14,835 KB for the cheap model's 60 attempts, 4,574 KB for Opus's 20.
- **No dollar figure.** List prices exist, but the input count they would multiply is not known here (cache reads and
writes are not reported), so a dollar amount would be invented.
- **No dollar figure for these runs.** List prices exist, but the input count they would multiply is not known here
(cache reads and writes are not reported), so a dollar amount would be invented. One app priced from a measured
run is below.

**Totals.** The 80 attempts recorded 734,940 tokens; stage 1 alone used 93,221, far under the 1.5M that gates stage 2.
Before the runs, one probe of the cheap model's id through the gateway used 2,046. One attempt, aborted by a driver bug
(fixed in `1853ae6`) and discarded, used 5,626 measured tokens; a third call was cut mid-flight and its usage is
unknown. All in, about 742,600 tokens, under the 5M cap.

## What one app really costs (measured 2026-10-27)

One builder turn on the `app-helpdesk` prompt, after the engine started streaming its model calls with
`stream_options.include_usage` (#30 item 8). Claude Opus 5.5 (`cc/claude-opus-5-5`) through the same local gateway
(9router 0.5.99), on the one-command demo stack in `AI_GATEWAY_MODE=record`, sent as the builder through
`POST /api/v1/admin/ai/builder/chat`. The turn ended `ok` with a draft of every expected kind (entity, app, access) in
three model calls; it was not applied or published, so it is one app's turn, not a bench attempt.

| call | prompt tokens reported | whole input | uncached | cache read | cache write | output |
|---|---|---|---|---|---|---|
| 1 (list_roles, list_packs) | 35,531 | 33,531 | 175 | 33,356 | 0 | 285 |
| 2 (propose_entity, propose_app, propose_access) | 38,448 | 36,448 | 2 | 33,356 | 3,090 | 772 |
| 3 (the reply) | 39,326 | 37,326 | 2 | 36,446 | 878 | 722 |
| **the turn** | **113,305** (`sys_ai_runs`) | **107,305** | **179** | **103,158** | **3,968** | **1,779** |

- **What the gateway reports, streamed:** the whole input plus 2,000 per call. 9router adds 2,000 to `prompt_tokens`
on both paths and keeps only `prompt_tokens`, `completion_tokens` and `total_tokens` in its OpenAI-shape usage, so
the cache split never reaches the engine and `sys_ai_runs.cache_read_tokens` / `cache_write_tokens` stay NULL
through it. The split above is 9router's own usage log for the same three calls. A gateway that forwards the
split (OpenAI's `prompt_tokens_details`, LiteLLM's Anthropic fields) fills those columns; the engine reads both.
- **Not streamed** (every attempt in this report), the same three calls would have read 6,179 prompt tokens: the
uncached part plus 2,000 each. The whole input is 17 times that.
- **In dollars,** at Claude Opus 5.5's list prices ($4 / MTok input, $20 output, $0.20 cache read, $5 cache write for
5 minutes, $8 for 1 hour): 179 uncached $0.0007, 103,158 read $0.0206, 3,968 written $0.0198, 1,779 output
$0.0356, so **about $0.08 for the app** while the cache is warm. 9router marks the conversation tail with the
5-minute cache, and these writes are the tail; at the 1-hour rate they would make it $0.089.
- **Cold,** the first turn after the cache expires writes the persona and the tool schemas (33,356 tokens, marked
with the 1-hour cache) instead of reading them: $0.26 more, **about $0.34 for the app**. The turn before this one
wrote exactly that.

## The run

- **Product under test:** origin/main `d300dfc` (#21), this branch's base. The bench's commits change no server or web
Expand Down
8 changes: 8 additions & 0 deletions nexus-platform/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,20 @@ REDIS_URL=redis://127.0.0.1:6379
# To turn it ON, set BOTH of these — a gateway must actually be reachable:
#AI_ENABLED=true
# Any OpenAI-compatible /v1/chat/completions gateway works (LiteLLM, vLLM, Ollama, a hosted endpoint).
# Calls are streamed with stream_options.include_usage, so the ledger gets the whole input; a gateway
# that ignores stream and answers one JSON body works too. One that rejects stream_options answers 400
# and every call fails (502 AI_GATEWAY_ERROR, the server log quotes the gateway): put LiteLLM in front.
#AI_GATEWAY_BASE_URL=http://127.0.0.1:4000/v1
# Put the real key in .env only (never commit it). The model id must be one the gateway lists (GET /v1/models):
#AI_GATEWAY_API_KEY=
#AI_MODEL=claude-opus-5-5
# Tool-loop depth per turn (Discovery mode needs ~14; default 12):
#AI_MAX_STEPS=14
# Tokens per tenant per UTC day, every AI call (prompt + output); past it a turn answers 429
# AI_BUDGET_EXCEEDED until midnight UTC. A call counts its whole input, cached or not: one builder turn
# that makes an app is ~115,000 tokens, so the default 3,000,000 is ~27 a day. 0 turns the cap off.
# One user's ⌘K assistant gets at most ASSISTANT_DAILY_TOKEN_SHARE_PCT (default 50) percent of it.
#AI_DAILY_TOKEN_BUDGET=3000000
# Probe: GET /api/v1/ai/health → {"data":{"enabled":true|false}} — the FE reads this
# to show an honest on/off state BEFORE the user types.
# Record/replay, for demos, CI and evals without a live model: live (default) | record | replay.
Expand Down
9 changes: 5 additions & 4 deletions nexus-platform/internal/ai/budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,11 @@
// Accepted overshoot (spec §12): the check runs PRE-turn against a ledger that only reflects
// COMPLETED turns, so every turn admitted while others are still in flight can independently pass
// the same budget snapshot before any of them commits usage back. Up to MaxConcurrent turns can be
// in that window at once, each spending up to MaxSteps × MaxTokens — so the day total may overshoot
// the configured budget by at most MaxConcurrent × MaxSteps × MaxTokens (default
// 2 × 12 × 4096 ≈ 98k, against a 200k default budget — MaxSteps was raised 6→12 for Discovery
// M10, so one worst-case turn is ~49k). That is bounded and not worth a mid-turn abort.
// in that window at once, each spending up to MaxSteps calls of whole input plus MaxTokens output — so
// the day total may overshoot the configured budget by MaxConcurrent such turns. A builder call's input
// is ~35-40k tokens (evals/builder-bench/report.md), so one worst-case 12-step turn (MaxSteps was raised
// 6→12 for Discovery M10) is ~0.5M and the overshoot ~1M, against a 3M default budget. That is bounded
// and not worth a mid-turn abort.
package ai

import (
Expand Down
40 changes: 40 additions & 0 deletions nexus-platform/internal/ai/budget_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,46 @@ func TestBudgetGuard_DailyBudget(t *testing.T) {
}
}

// measuredBuilderTurn is the sys_ai_runs row of one real builder turn that made an app, its calls streamed
// through the dev gateway (evals/builder-bench/report.md, "What one app really costs"): 113,305 prompt tokens
// (the whole input of three calls, cache reads and writes included) + 1,779 output.
const measuredBuilderTurn = 113305 + 1779

// Issue #30 item 8 moved prompt_tokens from the uncached remainder to the whole input, about 17 times more on a
// builder turn, so the default daily budget is sized in that unit. It admits the working day it admitted before
// the stream: 26 turns at the ~8,000 tokens a turn recorded then under 200,000; 27 at the measured 115,084 now.
// A third turn a day (what 200,000 admits now) would 429 AI_BUDGET_EXCEEDED a live demo until UTC midnight.
func TestBudgetGuard_DefaultBudgetAdmitsADayOfBuilderTurns(t *testing.T) {
for _, k := range []string{"AI_DAILY_TOKEN_BUDGET", "AI_GATEWAY_MODE", "AI_GATEWAY_CASSETTE"} {
t.Setenv(k, "")
}
loaded, err := config.LoadAIConfig()
if err != nil {
t.Fatalf("LoadAIConfig: %v", err)
}
cfg := guardCfg() // the rate and concurrency gates out of the way: the budget alone decides
cfg.DailyTokenBudget = loaded.DailyTokenBudget
ledger := &fakeTokenLedger{}
g := newBudgetGuard(cfg, ledger, nil, time.Now)

turns := 0
for ; turns < 1000; turns++ {
release, err := g.Acquire(context.Background(), "tenant-a")
if errors.Is(err, ErrBudgetExceeded) {
break
}
if err != nil {
t.Fatalf("turn %d: Acquire error = %v", turns+1, err)
}
release()
ledger.total += measuredBuilderTurn
}
if turns != 27 {
t.Fatalf("the default budget %d admits %d builder turns of %d tokens a day; want 27",
cfg.DailyTokenBudget, turns, measuredBuilderTurn)
}
}

// TestBudgetGuard_BudgetZero_CheckDisabled: DailyTokenBudget=0 is the explicit opt-out — the
// ledger must never be consulted.
func TestBudgetGuard_BudgetZero_CheckDisabled(t *testing.T) {
Expand Down
Loading
Loading