Skip to content

Measure LLM round 2: recall, repeatability, and .github visibility - #8

Merged
joshft merged 1 commit into
masterfrom
llm-round-2
Aug 17, 2026
Merged

Measure LLM round 2: recall, repeatability, and .github visibility#8
joshft merged 1 commit into
masterfrom
llm-round-2

Conversation

@joshft

@joshft joshft commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Closes the last open item from the second external review. Protocol: blinded claim inventories written before any extractor run, two real merged changes as cases, six live runs total.

  • Precision held: 115/115 proposals accurate, zero hallucinated files or mechanisms.
  • Recall (new axis): 11/12 and 9/11 — every core behavioral claim surfaced; misses concentrate in subtle negatives, deliberate design omissions, and documentation-meta claims.
  • Repeatability (new axis): exact-text stability across four identical runs is zero — the model rewords every proposal, so content-hashed ids never collide — while semantic stability is effectively complete (all runs cover the same claim set). The README states the consequence honestly: LLM remainder rows are stable in meaning, not in id or wording.
  • .github visibility fix (the one code change, measured): .github diff sections now reach the model, ordered after all non-hidden sections so they can never crowd shipped code out of the byte cap. Unit tests pin the ordering and the crowd-out case; live A/B on a real workflow-touching change shows the workflow claim going from structurally unreachable to minted exact from the workflow section.

What the receipt cannot check here: the live measurements ran against past merged diffs in detached worktrees; the receipt covers the extractor change and its tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp

Round 1 measured only precision. Round 2 adds the other two axes,
against blinded claim inventories written before any run, on two real
merged changes:

- Precision held: 115/115 proposals accurate across six runs, zero
  hallucinated files or mechanisms.
- Recall: 11/12 and 9/11. Every core behavioral claim surfaced; the
  misses are exactly the hard shapes — a subtle negative (a path
  deliberately not instrumented), a deliberate design omission, a
  documentation-meta claim.
- Repeatability: exact-text stability across four identical runs is
  ZERO (the model rewords everything, so content-hashed ids never
  collide) while semantic stability is effectively complete. The README
  now states the consequence: LLM remainder rows are stable in meaning,
  not in id or wording.

The one code change is the measured scope fix: .github diff sections
now reach the model — it is project-owned behavior (this repository's
merge gate lives there), not installed tooling — ordered AFTER every
non-hidden section so they can never crowd shipped code out of the
byte cap, the exact failure the dot-dir rule was measured against.
Live A/B: pre-fix the workflow claim was structurally unreachable;
post-fix the model minted it from the workflow section, exact to the
step name and setting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015GeUG1gboWiZSnFyzQghyp
@github-actions

Copy link
Copy Markdown

correctful receipt

10 claims — ✅ 10 verified · ❌ 0 refuted · ⚠️ 0 unverified

Change: origin/master...HEAD (ea2a528..8fa8d18 · input:2b956763aa06) — 3 files

⚠️ Unverified remainder (0) — what nothing checked

Empty — every harvested claim reached a probe.

✅ Verified (10)
Tier Claim Statement
T1-assertion TestExtractMintsValidatedProposals Extract Mints Validated Proposals
T1-assertion TestFenceWrappedOutputTolerated Fence Wrapped Output Tolerated
T1-assertion TestProseWrappedOutputFailsLoud Prose Wrapped Output Fails Loud
T1-assertion TestAPIErrorSurfaces APIError Surfaces
T1-assertion TestByteCapIsHardAndDisclosed Byte Cap Is Hard And Disclosed
T1-assertion TestContextCancellationPropagates Context Cancellation Propagates
T1-assertion TestSectionFileParsing Section File Parsing
T1-assertion TestDotDirSectionsNeverReachTheModel Dot Dir Sections Never Reach The Model
T1-assertion TestGithubSectionsIncludedAfterCode Github Sections Included After Code
T1-assertion TestGithubSectionCannotCrowdOutCode Github Section Cannot Crowd Out Code

Harvest coverage: 3 files — 1 claimed · 2 scanned · 0 unread

schema 0.0.6 · exit gate: refuted claims block; the remainder informs, never fails

@joshft
joshft merged commit 094f434 into master Aug 17, 2026
1 check passed
@joshft
joshft deleted the llm-round-2 branch August 17, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant