Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
builder-bench (#28, report) read every failure from its transcript. The engine gaps it found, most important first:
- (fix(ai): propose_layout sees the same draft's table and pending index (#30 items 1–2) #39)
propose_layoutrefuses a table proposed in the same turn (ENTITY_NOT_FOUND … Apply entity đó trước).propose_relationshipandpropose_accessaccept such a table; the layout tool does not. Because of this, no model can passapp-recruiting("a table, its entry screen, an app"). The same refusal likely explainsgolden.json'swhole_app_all_kindsbaseline note. - (fix(ai): propose_layout sees the same draft's table and pending index (#30 items 1–2) #39) The layout validator ignores an index proposed in the same turn. Asked for a list sorted by
name, the model proposed the index (edit_field indexed: true) and retried. The validator read the live schema and refused again (LAYOUT_INVALID), and the index change applied at once, leaving nothing to publish. - (fix(ai): propose_field refuses a duplicate, reserved names in the tool contract, canonical table lookups (#30 items 3, 6) #40)
propose_fieldaccepts the same field twice in one turn.propose_entityrefuses a duplicate. In one attempt the second card failed at apply ("Trường đã tồn tại"), costing 11 model calls and 28k tokens against about 10k for a passing run. - (A pack table delete stays a card the builder confirms: honest card, honest reply, bench expectation (#30 item 4) #70) Decide the policy for "delete this pack table and all its data now, don't ask again". Both models drafted a
delete_entitycard (soft delete; the builder must retype the name) instead of refusing. Either refuse at propose time for pack tables, or keep the draft and change the bench's expectation; write the decision down. - (fix(ai): show "the model declined" instead of an empty answer (#30 item 5) #48; leftovers in comments) An API-level refusal (
finish_reason: "refusal", no text) shows the builder an empty panel. Show a short "the model declined this request" line instead. - (fix(ai): propose_field refuses a duplicate, reserved names in the tool contract, canonical table lookups (#30 items 3, 6) #40) Reserved field names are not in the tool description. Six attempts named a person field
owner, gotFIELD_NAME_RESERVEDand retried, costing one model call each. List the reserved names inpropose_field's description. - (fix(seed): the demo's sales rep owns a book of CRM sample records (#30 item 7) #47) The seed shows the rep empty CRM tables. The rep's
view: ownergrants cover 2 of the 10 CRM tables with data, because the installer owns the seeded rows.acceptance-acl.pyfails 7 CRM tables on a fresh stack before any AI change. Seed rows the rep can see, or grant the rep's roleview: allon the read-mostly tables. - (feat(ai): record prompt-cache tokens when the gateway reports them (partial step for #30 item 8) #44, Streamed usage: the AI ledger reads a call's whole input (#30 item 8) #72) The gateway's token counts omit cached input. A ~70 KB call reports about 2,046 prompt tokens. Record cache reads and writes in
sys_ai_runs, so the cost per app is real and not a floor.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
fix(ai): look a table up by one canonical name in the registry and th…
fix(ai): propose_layout sees the same draft's table and pending index (…
fix(ai): propose_field refuses a duplicate, reserved names in the too…
fix(ai): a malformed usage field never fails the model's answer
fix(aieval): builder-bench prints "—" for a cache nobody reported
feat(ai): record prompt-cache tokens when the gateway reports them (p…
doxuta commentedon Oct 21, 2026
Item 8 — partial step merged in #44: the gateway now parses cache reads/writes (OpenAI and Anthropic shapes, each input token once), stores them in sys_ai_runs (migration 000099) and builder-bench prints them ("—" when not reported). Still a floor through today's dev gateway: 9router drops the cache fields on the non-streaming path and pads prompt_tokens by +2000. Remaining for this item: request streaming usage (stream_options.include_usage) and record one real run showing non-NULL cache columns.
doxuta commentedon Oct 21, 2026
Item 5 — #48 ends a declined turn with outcome "refused" and shows one line in the Studio panel and ⌘K. Left for this item (verifier LOWs): a reopened session still shows the refused turn as an empty assistant message (sys_ai_messages has no per-message outcome; max_steps has the same gap); Refused() does not map finish_reason content_filter, OpenAI message.refusal, or a top-level stop_reason; buildHistory resends the refused prompt as two consecutive user messages on the next turn; no test pins the too-broad hint for max_steps.
fix(seed): the demo's sales rep owns a book of CRM sample records (#30 …
fix(ai): show "the model declined" instead of an empty answer (#30 it…
Declined turns stay declined: outcome per message, every refusal shap…
A pack table delete stays a card the builder confirms: honest card, h…
fix(ai): size the default daily token budget in whole-input tokens (#30…
Streamed usage: the AI ledger reads a call's whole input (#30 item 8) (…
doxuta commentedon Oct 27, 2026
All eight items are on main. Item 4: #70. Item 5: #48 and #69. Item 7: #47. Item 8: #44 and #72. Usage is now streamed, so prompt_tokens records the whole input. One measured Opus 5.5 builder turn used 107,305 input tokens where the non-streamed path recorded 6,179, about $0.077 per app with a warm cache and $0.34 cold (evals/builder-bench/report.md). Known limit, accepted: 9router, the dev gateway, drops the cache read/write split on both paths, so those two columns stay NULL behind it. They fill in with gateways that send the split, in either the OpenAI or the Anthropic shape. 9router also pads prompt_tokens by 2,000 per call; the report subtracts it, and the engine counts it, which keeps the daily budget on the safe side.