Add the policy layer: evidence floors per path, gate-blocking misses - #18
Merged
Conversation
correctful.json at the repo root declares rules — paths plus a floor (min_tier, optionally a required mechanism and scope). Evaluation is per changed file, and the tie between a file and its evidence is STRUCTURAL, never assumed: a verified claim speaks for a file only when sourced from it or when a reference site in shipped code names it. A matched file nothing ties evidence to misses with exactly that reason; misses block the gate like refutations. Test files are exempt and the exemption is counted. The policy digest (SHA-256 over the exact bytes) is the second chain field, rendered beside the change. A malformed policy fails loudly before any probe runs — a broken floor must never fail open. Evidence.CountsFor moves the LLM edge gate into the schema so weighing and policy evaluation share one definition and can never disagree. Measured live on a real annotated repo's branch: a T1 floor matched 26 files, exempted 8 test files, and correctly blocked on all 18 evaluated files — the branch's verified claims are test-name-sourced with no reconciled code annotations, so no evidence structurally ties to the changed code. Schema 0.0.11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp
correctful receipt23 claims — ✅ 23 verified · ❌ 0 refuted · Change:
|
| Tier | Claim | Statement |
|---|---|---|
| T1-assertion | TestLoadValidatesAndDigests |
Load Validates And Digests |
| T1-assertion | TestPatternMatching |
Pattern Matching |
| T1-assertion | TestEvaluateFloors |
Evaluate Floors |
| T1-assertion | TestEvaluateScopeFloorAndLLMGate |
Evaluate Scope Floor And LLMGate |
| T1-assertion | INV-001 |
Remainder Contains Every Unverified Claim |
| T1-assertion | INV-002 |
Refuted Claim Never Enters Remainder |
| T1-assertion | TestVerifiedClaimCarriesProbeTier |
Verified Claim Carries Probe Tier |
| T1-assertion | INV-007 |
Refutation Dominates Verification |
| T1-assertion | TestT0PassConfersNoVerification |
T0 Pass Confers No Verification |
| T1-assertion | TestScopeExclusionsAndInputDigestAreDisclosed |
Scope Exclusions And Input Digest Are Disclosed |
| T1-assertion | TestSuppressedMentionsAreDisclosed |
Suppressed Mentions Are Disclosed |
| T1-assertion | TestRemainderSectionAlwaysRenders |
Remainder Section Always Renders |
| T1-assertion | TestCoverageDisclosesUnreadFiles |
Coverage Discloses Unread Files |
| T1-assertion | TestUnreadCausesRenderSeparately |
Unread Causes Render Separately |
| T1-assertion | TestAnchoringSummaryAndMarkers |
Anchoring Summary And Markers |
| T1-assertion | TestLLMProposedMarker |
LLMProposed Marker |
| T1-assertion | TestRefutedDetailComesFromFailingProbe |
Refuted Detail Comes From Failing Probe |
| T1-assertion | TestReceiptSanitizesPathsAndPinsSHAs |
Receipt Sanitizes Paths And Pins SHAs |
| T1-assertion | TestDetailSanitizationIsCategorical |
Detail Sanitization Is Categorical |
| T1-assertion | TestLLMPassCountsOnlyWithConfirmedEdge |
LLMPass Counts Only With Confirmed Edge |
| T1-assertion | TestLLMVerifiedRowKeepsProvenance |
LLMVerified Row Keeps Provenance |
| T1-assertion | TestReceiptCarriesToolVersion |
Receipt Carries Tool Version |
| T1-assertion | TestPolicySectionRenders |
Policy Section Renders |
Harvest coverage: 9 files — 3 claimed · 6 scanned · 0 unread
2 spec-id mention(s) not minted as claims — the repo defines no spec-id corpus, so a reference has no possible referent
schema 0.0.11 · correctful v0.0.0-20260817220144-3a9319fac090 3a9319f · exit gate: refuted claims block; the remainder informs, never fails
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The end state's second commitment delivered:
correctful.jsondeclares per-path evidence floors (min tier + required mechanism + required scope). Per-file evaluation with structural ties only (claim source file or reference site); misses block the gate; test files exempt and counted; policy digest is the second chain field; malformed policy fails loudly.Evidence.CountsFormoves the LLM edge gate to the schema — one definition for weighing and policy. Live-measured on a real annotated repo (correctly blocked 18/18 untied files). Schema 0.0.11.🤖 Generated with Claude Code
https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp