Skip to content

Engine gaps found by builder-bench #30

Closed

Description

@doxuta

builder-bench (#28, report) read every failure from its transcript. The engine gaps it found, most important first:

Activity

  1. doxuta commented on Oct 21, 2026

    @doxuta
    OwnerAuthor

    Item 8 — partial step merged in #44: the gateway now parses cache reads/writes (OpenAI and Anthropic shapes, each input token once), stores them in sys_ai_runs (migration 000099) and builder-bench prints them ("—" when not reported). Still a floor through today's dev gateway: 9router drops the cache fields on the non-streaming path and pads prompt_tokens by +2000. Remaining for this item: request streaming usage (stream_options.include_usage) and record one real run showing non-NULL cache columns.

  2. doxuta commented on Oct 21, 2026

    @doxuta
    OwnerAuthor

    Item 5 — #48 ends a declined turn with outcome "refused" and shows one line in the Studio panel and ⌘K. Left for this item (verifier LOWs): a reopened session still shows the refused turn as an empty assistant message (sys_ai_messages has no per-message outcome; max_steps has the same gap); Refused() does not map finish_reason content_filter, OpenAI message.refusal, or a top-level stop_reason; buildHistory resends the refused prompt as two consecutive user messages on the next turn; no test pins the too-broad hint for max_steps.

  3. doxuta commented on Oct 27, 2026

    @doxuta
    OwnerAuthor

    All eight items are on main. Item 4: #70. Item 5: #48 and #69. Item 7: #47. Item 8: #44 and #72. Usage is now streamed, so prompt_tokens records the whole input. One measured Opus 5.5 builder turn used 107,305 input tokens where the non-streamed path recorded 6,179, about $0.077 per app with a warm cache and $0.34 cold (evals/builder-bench/report.md). Known limit, accepted: 9router, the dev gateway, drops the cache read/write split on both paths, so those two columns stay NULL behind it. They fill in with gateways that send the split, in either the OpenAI or the Anthropic shape. 9router also pads prompt_tokens by 2,000 per call; the report subtracts it, and the engine counts it, which keeps the daily budget on the safe side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions