fix(redesign): ground best-source superlatives on their own citation#571
Merged
Conversation
applyDeterministicResponsePolicy only rewrote unsupported "best/strongest source|claim" superlatives when the run had zero URL-backed evidence rows. In a mixed run - one real URL plus several cached section labels like Setup or Rising Action - the superlative passed through, so an answer could claim "strongest supported claim with its best source" while the rendered best source was a label with no URL. On a paid runtime path that is false confidence, not honest degradation. The gate is now sentence-level: a superlative stands only if its own sentence carries a [N] citation resolving to a URL-backed evidence row; otherwise that sentence is rewritten and the source-needed limitation is appended. A superlative citing a cached label, or citing nothing, is rewritten even when a URL-backed row exists elsewhere in the run. The best/strongest pattern now lives in one shared constant used by both the sanitizer and the gate so they cannot drift. Closes the residual left open by the 2026-07-16 audit follow-up (#565 fixed four of five P1s; this is the fifth). Closes #566 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
HomenShum
enabled auto-merge (squash)
July 17, 2026 19:09
|
Demo: walkthrough of the surfaces this PR changed is available as a workflow artifact ( |
HomenShum
added a commit
that referenced
this pull request
Jul 17, 2026
* fix(scratchnode): repair red handoff tests and close CI allowlist gap Three ScratchnodeEventsSurface tests failed on clean main with a TypeError because the api mock predated ImportRecapButton's domains.product.scratchnodeImport query, and CI never saw the red because the runtime-smoke job runs an explicit allowlist that omitted the file. The api mock is now a Proxy that degrades unknown function references to unresolved queries (the surface's honest gates render nothing) instead of crashing at property access, the convex/react mock gains the useMutation the component already imports, two scenarios cover the import affordance's published/unresolved gates, and the file joins the runtime-smoke allowlist so future drift turns CI red. Closes #567 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(composer): make run cancellation spatially stable, not timer-only The submit->Stop same-slot swap meant a double-click slower than the 400ms arming window (Windows accessibility settings allow ~900ms) could still cancel the run it just created. Stop and submit are now both always rendered: Stop holds a reserved slot (visibility:hidden while idle) and submit stays at identical coordinates while streaming, merely disabled — so the second click of a double-click lands on an inert control at every interval. The arming delay remains as defense in depth. Closes #568 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(chat): honor sentence, paragraph, and word-limit response shapes detectRequestedResponseShape recognized only title-only and bullets, so "in one sentence", "a single paragraph", and "under 50 words" silently rendered the five-section memo despite the user's explicit constraint. Three new shapes are detected, instructed (via an exhaustive responseShapeSystemInstructions switch replacing the ternary chain), and deterministically enforced in applyDeterministicResponsePolicy: first sentence extraction, prose collapse, and sentence-accumulating word budgets. Honesty survives compaction — unsupported runs carry a "Source needed" prefix or inline limitation instead of busting the user's limit. JSON/table shapes stay model-side deliberately: the memo render path is markdown prose, and a deterministic policy cannot reconstruct valid structured output from parsed memo fields. Closes #569 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): pin 390px chat geometry by computed grid, seal audit lane The audit's mobile collapse (grid 70px 320px, chat clipped to a 70px strip) was invisible to both the CSS-source string guard and the document-overflow boolean. one-flow-regression (Tier B, per-PR) now asserts the computed grid-template-columns of .rd-shell__main resolves to one track and main#main-content stays >300px at 390x844. Reversion- proved by reinjecting a higher-specificity two-column rule: computed geometry returns exactly the audited 70px/320px shape, which the new assertions reject. Also seals the audit-residual work in the redesign-chat changelog lane and backfills the #565/#571 entries' pending merge markers. Closes #570 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: hshum <hshum@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
applyDeterministicResponsePolicy(convex/domains/redesign/chatRuns.ts) sanitized "best/strongest source|claim" superlatives only when the run had zero URL-backed evidence rows. In a mixed run — one real URL plus several cached section labels (Setup,Rising Action) — the superlative passed through, so an answer could assert "strongest supported claim with its best source" while the rendered best source was a label with no URL. On a paid runtime path that is false confidence, not honest degradation.Closes #566. This is the fifth and last P1 from the 2026-07-16 production audit; #565 fixed the other four (or verified them already fixed by #550/#561/#564).
The fix — sentence-level citation gate
A superlative now stands only if its own sentence carries a
[N]marker resolving to a URL-backed evidence row. Otherwise that sentence is rewritten to "source or claim requiring verification" and the source-needed limitation is appended.[1](the URL-backed row) → kept.[2](a cached label) → rewritten, even though[1]exists elsewhere in the run.Mechanics:
evidenceparam to carryidx(production rows already have it — set at thegeminiEvidence/fallbackEvidence/memoryEvidenceconstruction; only the local param type omitted it).idxis optional so existing callers compile.supportedIdx= theidxset of rows that are URL-backed and not blocked/unsupported/fetch_blocked — computed from the samesupportedRowsfilter that already drivessupportedUrls(no duplicated predicate).UNSUPPORTED_SUPERLATIVE_SOURCEconstant shared by the sanitizer (replace) and the gate (test) so they can't drift — the same class of bug as the determiner fix in fix(redesign): honor title-only requests behind any determiner #565.Verification
npx tsc -p convex --noEmit --pretty false→ 0 errorsnpx vitest run convex/domains/redesign/chatRuns.responseShape.test.ts→ 16 passed (3 new)expected 'acme is the strongest supported claim…' not to contain 'strongest supported claim'(2 failed / 14 passed); restored → 16 passedNo visual proof: the answer body is runtime-produced; no rendered surface markup changed.
Deliberately not in scope
Full claim→source binding at claim granularity (the
claim.sourceRefIdsmodel inshared/productAnswerControl.ts) is a larger design change touching both renderers and is not attempted here. This PR closes the specific audited honesty gap; issue #566 can track the broader model if desired.🤖 Generated with Claude Code