Repository navigation
feat(bench): fable-bench v0 — behavioral regression harness - #8
Merged
Merged
Conversation
…nstruction layer Automates the CONTRIBUTING.md RED/GREEN loop: per scenario x condition (baseline | treatment) x k, provisions an isolated git workspace from a fixture, installs the skills + activation block for treatment, drives headless Claude Code, and grades two tiers (deterministic checks + judge with substring-verified verbatim rationalization capture). Seven seed scenarios ported from the manual record; s5 runs against a real fixture repo. Mock mode replays recorded transcripts so the 90-test suite and CI spend zero API money. Live-verified: s5 acceptance on Opus 4.8 (k=3) reproduces the scope-discipline flip (baseline 2/3 with a captured bundling rationalization, treatment 3/3).
…te tiers, fixture recorder - inline condition (D21): skill body prepended to the prompt, nothing installed — reproduces the manual GREEN protocol and isolates skill-text effect from install-shape effect (v0 flip happened with zero Skill calls). - pack flags (D22): --skills-dir / --claude-md / --scenarios-dir; installer copies any child dir containing a SKILL.md. Defaults preserve v0 behavior. - shell-arg quoting (D24): LiveExecutor quotes args so allowed-tool patterns with spaces/parens survive the Windows shell:true spawn. - quote verification (D23): third formatting-insensitive tier — a verbatim quote that dropped markdown backticks/emphasis still verifies. - record-fixtures.mjs (D25): converts a real run dir into mock-fixture layout, replacing hand-authored reconstructions with real captures. - report gains an inline column and 3-part pass@k/pass^k cells when inline data is present; 2-part table unchanged otherwise. 110 tests green; mock e2e, validate, and report re-render all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ships the eval-harness roadmap item: automates the CONTRIBUTING.md RED/GREEN loop.
What
A TypeScript CLI (
bench/, Node 20+, one runtime dep) that runs pressure scenarios against headless Claude Code in baseline vs treatment (and nowinline) conditions, k times each, and grades two tiers: deterministic checks (diff size, file contents, reply regexes) and a judge model whose FAIL verdicts quote the rationalization verbatim, substring-verified against the transcript.Verified
cd bench && npm test→ 110/110 (mock mode, zero API spend).validateandreportre-render pass.Seed suite
All seven pressure scenarios ported; s5 runs against a real fixture repo and is graded on its actual diff. Mock transcripts for the harness's own tests are labeled reconstructions;
scripts/record-fixtures.mjsreplaces them with real captures.CI
bench-ci.yml— mock suite on push, zero API money.bench.yml— live,workflow_dispatchonly, model/scenario/k inputs, report uploaded as artifact.Notes for review
--setting-sources project), weaker than the manual Opus+superpowers baseline — stated in report.md and SPEC D5.inlinecondition (D21) exists to separate skill-text effect from install-shape effect before publishing treatment-effect claims.bench/SPEC.md; authoring guide indocs/bench-authoring.md.Do not squash-merge blindly — the two commits separate v0 from the upgrade wave.