Skip to content

Bind LLM-proposed claims to changed tests behind two mechanical gates - #13

Merged
joshft merged 1 commit into
masterfrom
llm-probe-binding
Aug 17, 2026
Merged

Bind LLM-proposed claims to changed tests behind two mechanical gates#13
joshft merged 1 commit into
masterfrom
llm-probe-binding

Conversation

@joshft

@joshft joshft commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Closes the extract→bind→verify loop for the LLM path without giving the model certification power. The model proposes an EDGE (an optional test field naming a changed Go test); two mechanical gates decide whether it counts:

  1. Existence (mint time) — the test's definition must appear in the diff the model was shown, on an added or context line of a changed _test.go. Deleted definitions do not count. An ambiguous name (defined in two changed test files) fails closed. The claim's file must be shipped Go code.
  2. Execution (run time) — the probe run is instrumented; the pass counts only when the coverage profile shows execution reaching the claim's file (binding: file-covered). file-not-reached refutes the edge; a missing profile raises nothing. Both discards are disclosed on the remainder row.

Refutation keeps unconditional dominance. Verified rows keep the [llm-proposed] marker in both renderers. Schema 0.0.8.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp

The model may now name, in an optional "test" field, a changed Go test
that checks its proposed claim — but the edge only counts after two
machine checks, because a wrong claim→test association plus a passing
test would mint exactly the false verification this tool exists to
prevent:

1. Existence, at mint time: the test's definition must appear in the
   diff the model was shown (added or context line of a changed
   _test.go; deleted definitions don't count), exactly one changed test
   file may define it, and the claim's file must be shipped Go code.
   A failed edge mints the claim probe-less, as before.
2. Execution, at run time: the probe run is instrumented, and the pass
   counts only when the coverage profile shows execution reaching the
   claim's file (binding "file-covered"). A profile that never touched
   the file refutes the edge ("file-not-reached"); no profile at all
   raises nothing. Both discards are disclosed on the remainder row.

Refutation keeps unconditional dominance — a failing test blocks the
gate no matter whose edge bound it; a mis-attributed refutation is
fail-safe because the failure is real. Verified rows now carry the
[llm-proposed] marker in both renderers, so verification never launders
the provenance away. Schema 0.0.8; binding vocabulary moved to named
schema constants.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp
@github-actions

Copy link
Copy Markdown

correctful receipt

33 claims — ✅ 33 verified · ❌ 0 refuted · ⚠️ 0 unverified

Change: origin/master...HEAD (124324c..08ade70 · input:269689fbd15f) — 11 files

⚠️ Unverified remainder (0) — what nothing checked

Empty — every harvested claim reached a probe.

✅ Verified (33)
Tier Claim Statement
T1-assertion TestExtractMintsValidatedProposals Extract Mints Validated Proposals
T1-assertion TestFenceWrappedOutputTolerated Fence Wrapped Output Tolerated
T1-assertion TestProseWrappedOutputFailsLoud Prose Wrapped Output Fails Loud
T1-assertion TestAPIErrorSurfaces APIError Surfaces
T1-assertion TestByteCapIsHardAndDisclosed Byte Cap Is Hard And Disclosed
T1-assertion TestContextCancellationPropagates Context Cancellation Propagates
T1-assertion TestSectionFileParsing Section File Parsing
T1-assertion TestDotDirSectionsNeverReachTheModel Dot Dir Sections Never Reach The Model
T1-assertion TestGithubSectionsIncludedAfterCode Github Sections Included After Code
T1-assertion TestGithubSectionCannotCrowdOutCode Github Section Cannot Crowd Out Code
T1-assertion TestLLMEdgeExistenceGate LLMEdge Existence Gate
T1-assertion TestLLMEdgeFlowsThroughHarvest LLMEdge Flows Through Harvest
T1-assertion TestParseCoverProfileRealShape Parse Cover Profile Real Shape
T1-assertion TestCoverageProvenBinding Coverage Proven Binding
T1-assertion TestFileBindingFor File Binding For
T1-assertion TestLLMEdgeFileBinding LLMEdge File Binding
T1-assertion INV-001 Remainder Contains Every Unverified Claim
T1-assertion INV-002 Refuted Claim Never Enters Remainder
T1-assertion TestVerifiedClaimCarriesProbeTier Verified Claim Carries Probe Tier
T1-assertion INV-007 Refutation Dominates Verification
T1-assertion TestT0PassConfersNoVerification T0 Pass Confers No Verification
T1-assertion TestScopeExclusionsAndInputDigestAreDisclosed Scope Exclusions And Input Digest Are Disclosed
T1-assertion TestSuppressedMentionsAreDisclosed Suppressed Mentions Are Disclosed
T1-assertion TestRemainderSectionAlwaysRenders Remainder Section Always Renders
T1-assertion TestCoverageDisclosesUnreadFiles Coverage Discloses Unread Files
T1-assertion TestUnreadCausesRenderSeparately Unread Causes Render Separately
T1-assertion TestAnchoringSummaryAndMarkers Anchoring Summary And Markers
T1-assertion TestLLMProposedMarker LLMProposed Marker
T1-assertion TestRefutedDetailComesFromFailingProbe Refuted Detail Comes From Failing Probe
T1-assertion TestReceiptSanitizesPathsAndPinsSHAs Receipt Sanitizes Paths And Pins SHAs
T1-assertion TestDetailSanitizationIsCategorical Detail Sanitization Is Categorical
T1-assertion TestLLMPassCountsOnlyWithConfirmedEdge LLMPass Counts Only With Confirmed Edge
T1-assertion TestLLMVerifiedRowKeepsProvenance LLMVerified Row Keeps Provenance

Harvest coverage: 11 files — 5 claimed · 6 scanned · 0 unread
2 spec-id mention(s) not minted as claims — the repo defines no spec-id corpus, so a reference has no possible referent

schema 0.0.8 · exit gate: refuted claims block; the remainder informs, never fails

@joshft
joshft merged commit 28db0ad into master Aug 17, 2026
1 check passed
@joshft
joshft deleted the llm-probe-binding branch August 18, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant