Measure LLM round 2: recall, repeatability, and .github visibility - #8
Merged
Conversation
Round 1 measured only precision. Round 2 adds the other two axes, against blinded claim inventories written before any run, on two real merged changes: - Precision held: 115/115 proposals accurate across six runs, zero hallucinated files or mechanisms. - Recall: 11/12 and 9/11. Every core behavioral claim surfaced; the misses are exactly the hard shapes — a subtle negative (a path deliberately not instrumented), a deliberate design omission, a documentation-meta claim. - Repeatability: exact-text stability across four identical runs is ZERO (the model rewords everything, so content-hashed ids never collide) while semantic stability is effectively complete. The README now states the consequence: LLM remainder rows are stable in meaning, not in id or wording. The one code change is the measured scope fix: .github diff sections now reach the model — it is project-owned behavior (this repository's merge gate lives there), not installed tooling — ordered AFTER every non-hidden section so they can never crowd shipped code out of the byte cap, the exact failure the dot-dir rule was measured against. Live A/B: pre-fix the workflow claim was structurally unreachable; post-fix the model minted it from the workflow section, exact to the step name and setting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp
correctful receipt10 claims — ✅ 10 verified · ❌ 0 refuted · Change:
|
| Tier | Claim | Statement |
|---|---|---|
| T1-assertion | TestExtractMintsValidatedProposals |
Extract Mints Validated Proposals |
| T1-assertion | TestFenceWrappedOutputTolerated |
Fence Wrapped Output Tolerated |
| T1-assertion | TestProseWrappedOutputFailsLoud |
Prose Wrapped Output Fails Loud |
| T1-assertion | TestAPIErrorSurfaces |
APIError Surfaces |
| T1-assertion | TestByteCapIsHardAndDisclosed |
Byte Cap Is Hard And Disclosed |
| T1-assertion | TestContextCancellationPropagates |
Context Cancellation Propagates |
| T1-assertion | TestSectionFileParsing |
Section File Parsing |
| T1-assertion | TestDotDirSectionsNeverReachTheModel |
Dot Dir Sections Never Reach The Model |
| T1-assertion | TestGithubSectionsIncludedAfterCode |
Github Sections Included After Code |
| T1-assertion | TestGithubSectionCannotCrowdOutCode |
Github Section Cannot Crowd Out Code |
Harvest coverage: 3 files — 1 claimed · 2 scanned · 0 unread
schema 0.0.6 · exit gate: refuted claims block; the remainder informs, never fails
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last open item from the second external review. Protocol: blinded claim inventories written before any extractor run, two real merged changes as cases, six live runs total.
.githubdiff sections now reach the model, ordered after all non-hidden sections so they can never crowd shipped code out of the byte cap. Unit tests pin the ordering and the crowd-out case; live A/B on a real workflow-touching change shows the workflow claim going from structurally unreachable to minted exact from the workflow section.What the receipt cannot check here: the live measurements ran against past merged diffs in detached worktrees; the receipt covers the extractor change and its tests.
🤖 Generated with Claude Code
https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp