Repository navigation
Evidence gaps and run polling - #17
Merged
Merged
Conversation
…rsioned eval corpora Lands the in-flight change set for scoped report rechecks, report-bound evidence snapshots, unified fetch/execution boundaries and versioned dataset evaluation, together with the fixes from reviewing it. Review fixes in this commit: - Evidence goals annotate, they no longer adjudicate. The price/route/time/ quantity/scope detectors are bounded keyword probes over a few domains, so a pattern miss said nothing reliable about the claim, yet it was rewriting verdict/confidence/truth_probability/correction to `insufficient`. Gaps now only record `evidence_gaps` plus a note and drive the follow-up retrieval round; the rule and LLM judges keep adjudication. This removes a systematic false-`insufficient` risk on any claim whose evidence is phrased outside the patterns, and it stops pattern growth from quietly moving replay scores. ADR 0008 and docs/evidence-goals.md now describe the shipped behaviour. - Replay thresholds match measured scores. The hard gate asked for 0.75 against an actual 62.5% (identical at HEAD, so the threshold was simply set above reality) and would have failed CI. Every gate is now a ratchet pinned just below its 2026-09-30 score, documented inline, and tight enough that losing a single case fails the job. The covid19-health gate is annotated as a regression detector only: it sits far below that corpus's 87.5% majority-class baseline and must not be read as accuracy. - Run status polling no longer takes a write lock. `get`/`event_page` read without `BEGIN IMMEDIATE` and sweep expired leases at most once per interval, instead of running a grouped TTL query and lease UPDATE on every poll while the worker appended events to the same SQLite file. `/analyze` also backs its poll off from 50ms to 500ms rather than asking 20x a second for minutes. - Supplemental evidence is cached in a request scope, not stashed on the request model via `object.__setattr__`. Three layers call the merge, so a cache that failed to stick meant silently re-fetching up to five external pages. - `reliable_get` names its retry ceiling and warns when clamping, instead of silently ignoring a larger `URL_FETCH_MAX_RETRIES`. - `_error_response` is public `error_response` rather than a private symbol imported across modules. Verified: check_contracts, ruff, 1561 backend tests, frontend typecheck/182 tests/build, dataset audit, agent replay, and all five replay gates. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.