Skip to content

fix(redesign): honor title-only requests behind any determiner#565

Merged
HomenShum merged 1 commit into
mainfrom
fix/response-shape-determiner
Jul 17, 2026
Merged

fix(redesign): honor title-only requests behind any determiner#565
HomenShum merged 1 commit into
mainfrom
fix/response-shape-determiner

Conversation

@HomenShum

Copy link
Copy Markdown
Owner

What

detectRequestedResponseShape matched return only the title but not return only its title. The determiner group listed articles (a/the) and no possessives, so the possessive phrasing fell through to the five-section memo and silently overrode an explicit user output constraint on a paid run.

The determiner set now lives in one TITLE_DETERMINER constant shared by every title pattern — it was duplicated across two regexes, which is how they could disagree in the first place.

Why this is the only P1 left from the 2026-07-16 audit

The audit listed five P1s. Re-verifying each against main rather than trusting the doc:

Audit P1 Verdict
Mobile wide-mode collapses chat to 70px RESOLVED#550 wrote the mobile rule at specificity (0,3,0) and lost a tie-break; #561 prepended .rd-shell → (0,4,0). Cause was misattributed: the rule was in agent-workspace.css, not primitives.css, and no "persisted wide-mode flag" exists (nb-wide is prototype-only) — the collapse was unconditional at ≤760px.
Double-click submit cancels the new run RESOLVED#550 added CANCEL_ARM_DELAY_MS + cancelArmed. The audit captured a ~66-minute window between #548 and #550.
Exact output constraints lose to memo template HALF LIVE → fixed here. exactly two bullets already worked; return only its title did not.
Preflight withholds provider/model RESOLVED#550 deleted Provider names appear in the trace, not here. and now renders Paid · {provider} · {model} pre-submit.
Evidence lineage too coarse for "best source" RESOLVED for the audited casesanitizeUnsupportedSuperlatives already rewrites the superlative and appends a source-needed limitation. One residual remains, see below.

The audit was taken against production hours before #550 landed, so it reads as stale against main.

Why the suite stayed green while production failed

responseShape.test.ts:29 asserted "Return only the title."the one phrasing that works. The test encoded the bug's blind spot. The new tests therefore use the audit's verbatim production prompts, plus negative cases ("explain why its title is misleading", "job titles") to pin the false-positive risk the wider pattern introduces.

Both test files are added to the CI runtime-smoke allowlist, which is an explicit file list, not a full-suite run. Without that, this guard would never execute in CI — repeating the same blind spot one level up.

Also included

  • DEFAULT_TIERS pinned to the runtime's modelForTier with a parity test. They were hand-maintained mirrors with no test binding them; drift would make the paid preflight advertise a model the runtime never runs (and never prices).
  • UniversalComposer header comment corrected — it still claimed "no provider names in UI / provider names appear only in the trace", the opposite of the contract shipped in fix(redesign): harden live agent chat contracts #550, so a future reader could "restore" it and re-open that finding.

Verification

  • npx tsc --noEmit --pretty false → 0 errors
  • npx tsc -p convex --noEmit --pretty false → 0 errors
  • npm run build → clean
  • Targeted → 16 passed
  • Guard proven: reverting only the detector fix makes the audit's verbatim prompt fail with expected { kind: 'memo' } to deeply equal { kind: 'title_only' } (2 failed / 11 passed); restored → 16 passed
  • Redesign sweep → 141 passed; 3 failures in ScratchnodeEventsSurface.test.tsx confirmed pre-existing on clean main with all edits stashed, and unrelated

No visual proof: no rendered surface changed — the response body is runtime-produced and the composer edit is a comment.

Known gaps, not fixed here

  1. Evidence gate is run-level, not claim-level. sanitizeUnsupportedSuperlatives fires only when zero supported URLs exist run-wide. A mixed run (one valid URL + five cached labels) still lets "strongest supported claim with its best source" through unsanitized. Fixing it is a product-design question, not a bug fix.
  2. ScratchnodeEventsSurface.test.tsx is red on main (ImportRecapButton:448 reads .product of undefined) and CI structurally cannot see it — it is not in the runtime-smoke allowlist.
  3. No 390px computed-geometry regression test. The audit's release order asked for one; it never landed. The existing AgentWorkspaceResponsiveCss.guard.test.ts is a readFileSync + toContain string match and cannot catch a new higher-specificity competitor.
  4. Cancel arming is a 400ms timer, not a spatially distinct control. Windows' double-click interval is user-configurable to ~900ms, so a slow/assistive double-click can still arm.
  5. detectRequestedResponseShape covers only title_only and bullets. "one sentence", "a table", "JSON", "under 50 words" still fall through to the memo — a feature gap, not this bug.

🤖 Generated with Claude Code

detectRequestedResponseShape matched "return only the title" but not "return
only its title": the determiner group listed articles and no possessives, so
the possessive phrasing fell through to the five-section memo and silently
overrode an explicit user output constraint on a paid run.

The determiner set now lives in one TITLE_DETERMINER constant shared by every
title pattern. It was previously duplicated across two regexes, which is how
the two could disagree in the first place.

Found by re-verifying the 2026-07-16 production audit against main instead of
trusting it. Four of its five P1s were already fixed by #550/#561/#564 within
hours of the run; only this one survived. The suite stayed green because every
title case it asserted used the one phrasing that worked, so the new tests use
the audit's verbatim production prompts and cover the negative cases the wider
pattern could now over-match.

Both test files are added to the CI runtime-smoke allowlist, which is an
explicit file list rather than a full-suite run. Without that, the regression
guard would never execute in CI and would repeat the same blind spot one level
up.

Also pins DEFAULT_TIERS to the runtime's modelForTier with a parity test. The
two were hand-maintained mirrors with no test binding them; drift would make
the paid preflight advertise a model the runtime never runs. The composer
header comment still claimed "provider names appear only in the trace", the
opposite of the disclosure contract shipped in #550.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 17, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nodebench-ai Ready Ready Preview, Comment Jul 17, 2026 10:35am

Request Review

@github-actions

Copy link
Copy Markdown

✅ Dogfood Visual QA Gate: PASSED

Check Status
Screenshots 7 captured (pass)
Walkthrough 9 chapters (pass)
Key Frames 9 extracted (pass)
Scribe Steps 8 how-to steps (pass)
Build success
Artifacts

Download the dogfood-evidence-100553a artifact from the Actions tab for full screenshots, frames, and walkthrough video.


Generated by Dogfood QA Gate

@HomenShum
HomenShum merged commit b2d12ae into main Jul 17, 2026
17 checks passed
@HomenShum
HomenShum deleted the fix/response-shape-determiner branch July 17, 2026 10:39
@github-actions

Copy link
Copy Markdown

Demo: walkthrough of the surfaces this PR changed is available as a workflow artifact (pr-demo-565) at https://github.com/HomenShum/NodeBenchAI/actions/runs/29574213219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants