fix(redesign): honor title-only requests behind any determiner#565
Merged
Conversation
detectRequestedResponseShape matched "return only the title" but not "return only its title": the determiner group listed articles and no possessives, so the possessive phrasing fell through to the five-section memo and silently overrode an explicit user output constraint on a paid run. The determiner set now lives in one TITLE_DETERMINER constant shared by every title pattern. It was previously duplicated across two regexes, which is how the two could disagree in the first place. Found by re-verifying the 2026-07-16 production audit against main instead of trusting it. Four of its five P1s were already fixed by #550/#561/#564 within hours of the run; only this one survived. The suite stayed green because every title case it asserted used the one phrasing that worked, so the new tests use the audit's verbatim production prompts and cover the negative cases the wider pattern could now over-match. Both test files are added to the CI runtime-smoke allowlist, which is an explicit file list rather than a full-suite run. Without that, the regression guard would never execute in CI and would repeat the same blind spot one level up. Also pins DEFAULT_TIERS to the runtime's modelForTier with a parity test. The two were hand-maintained mirrors with no test binding them; drift would make the paid preflight advertise a model the runtime never runs. The composer header comment still claimed "provider names appear only in the trace", the opposite of the disclosure contract shipped in #550. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
HomenShum
enabled auto-merge (squash)
July 17, 2026 10:34
✅ Dogfood Visual QA Gate: PASSED
ArtifactsDownload the Generated by Dogfood QA Gate |
|
Demo: walkthrough of the surfaces this PR changed is available as a workflow artifact ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
detectRequestedResponseShapematchedreturn only the titlebut notreturn only its title. The determiner group listed articles (a/the) and no possessives, so the possessive phrasing fell through to the five-section memo and silently overrode an explicit user output constraint on a paid run.The determiner set now lives in one
TITLE_DETERMINERconstant shared by every title pattern — it was duplicated across two regexes, which is how they could disagree in the first place.Why this is the only P1 left from the 2026-07-16 audit
The audit listed five P1s. Re-verifying each against
mainrather than trusting the doc:.rd-shell→ (0,4,0). Cause was misattributed: the rule was inagent-workspace.css, notprimitives.css, and no "persisted wide-mode flag" exists (nb-wideis prototype-only) — the collapse was unconditional at ≤760px.CANCEL_ARM_DELAY_MS+cancelArmed. The audit captured a ~66-minute window between #548 and #550.exactly two bulletsalready worked;return only its titledid not.Provider names appear in the trace, not here.and now rendersPaid · {provider} · {model}pre-submit.sanitizeUnsupportedSuperlativesalready rewrites the superlative and appends a source-needed limitation. One residual remains, see below.The audit was taken against production hours before #550 landed, so it reads as stale against
main.Why the suite stayed green while production failed
responseShape.test.ts:29asserted"Return only the title."— the one phrasing that works. The test encoded the bug's blind spot. The new tests therefore use the audit's verbatim production prompts, plus negative cases ("explain why its title is misleading","job titles") to pin the false-positive risk the wider pattern introduces.Both test files are added to the CI runtime-smoke allowlist, which is an explicit file list, not a full-suite run. Without that, this guard would never execute in CI — repeating the same blind spot one level up.
Also included
DEFAULT_TIERSpinned to the runtime'smodelForTierwith a parity test. They were hand-maintained mirrors with no test binding them; drift would make the paid preflight advertise a model the runtime never runs (and never prices).UniversalComposerheader comment corrected — it still claimed "no provider names in UI / provider names appear only in the trace", the opposite of the contract shipped in fix(redesign): harden live agent chat contracts #550, so a future reader could "restore" it and re-open that finding.Verification
npx tsc --noEmit --pretty false→ 0 errorsnpx tsc -p convex --noEmit --pretty false→ 0 errorsnpm run build→ cleanexpected { kind: 'memo' } to deeply equal { kind: 'title_only' }(2 failed / 11 passed); restored → 16 passedScratchnodeEventsSurface.test.tsxconfirmed pre-existing on clean main with all edits stashed, and unrelatedNo visual proof: no rendered surface changed — the response body is runtime-produced and the composer edit is a comment.
Known gaps, not fixed here
sanitizeUnsupportedSuperlativesfires only when zero supported URLs exist run-wide. A mixed run (one valid URL + five cached labels) still lets "strongest supported claim with its best source" through unsanitized. Fixing it is a product-design question, not a bug fix.ScratchnodeEventsSurface.test.tsxis red on main (ImportRecapButton:448reads.productof undefined) and CI structurally cannot see it — it is not in the runtime-smoke allowlist.AgentWorkspaceResponsiveCss.guard.test.tsis areadFileSync+toContainstring match and cannot catch a new higher-specificity competitor.detectRequestedResponseShapecovers onlytitle_onlyandbullets."one sentence","a table","JSON","under 50 words"still fall through to the memo — a feature gap, not this bug.🤖 Generated with Claude Code