Skip to content

feat(bench): fable-bench v0 — behavioral regression harness - #8

Merged
DizzyMii merged 2 commits into
mainfrom
feat/fable-bench-v0
Jul 13, 2026
Merged

DizzyMii merged 2 commits into
mainfrom
feat/fable-bench-v0

Conversation

@DizzyMii

Copy link
Copy Markdown
Owner

Ships the eval-harness roadmap item: automates the CONTRIBUTING.md RED/GREEN loop.

What

A TypeScript CLI (bench/, Node 20+, one runtime dep) that runs pressure scenarios against headless Claude Code in baseline vs treatment (and now inline) conditions, k times each, and grades two tiers: deterministic checks (diff size, file contents, reply regexes) and a judge model whose FAIL verdicts quote the rationalization verbatim, substring-verified against the transcript.

Verified

  • cd bench && npm test → 110/110 (mock mode, zero API spend).
  • Full mock e2e (7 scenarios x 2 conditions x k=3) green; validate and report re-render pass.
  • Live: s5 scope-discipline acceptance on Opus 4.8, k=3 → baseline 2/3 (bundling failure with the model's own justification captured), treatment 3/3, FLIP. Live smoke on Haiku reproduces the plumbing.

Seed suite

All seven pressure scenarios ported; s5 runs against a real fixture repo and is graded on its actual diff. Mock transcripts for the harness's own tests are labeled reconstructions; scripts/record-fixtures.mjs replaces them with real captures.

CI

  • bench-ci.yml — mock suite on push, zero API money.
  • bench.yml — live, workflow_dispatch only, model/scenario/k inputs, report uploaded as artifact.

Notes for review

  • Baseline = stock model with no user config (--setting-sources project), weaker than the manual Opus+superpowers baseline — stated in report.md and SPEC D5.
  • The v0 flip happened with zero explicit Skill-tool calls; the inline condition (D21) exists to separate skill-text effect from install-shape effect before publishing treatment-effect claims.
  • Design + 26 logged decisions in bench/SPEC.md; authoring guide in docs/bench-authoring.md.

Do not squash-merge blindly — the two commits separate v0 from the upgrade wave.

DizzyMii added 2 commits July 13, 2026 11:08
…nstruction layer

Automates the CONTRIBUTING.md RED/GREEN loop: per scenario x condition
(baseline | treatment) x k, provisions an isolated git workspace from a
fixture, installs the skills + activation block for treatment, drives
headless Claude Code, and grades two tiers (deterministic checks + judge
with substring-verified verbatim rationalization capture).

Seven seed scenarios ported from the manual record; s5 runs against a
real fixture repo. Mock mode replays recorded transcripts so the 90-test
suite and CI spend zero API money. Live-verified: s5 acceptance on Opus
4.8 (k=3) reproduces the scope-discipline flip (baseline 2/3 with a
captured bundling rationalization, treatment 3/3).
…te tiers, fixture recorder

- inline condition (D21): skill body prepended to the prompt, nothing
  installed — reproduces the manual GREEN protocol and isolates skill-text
  effect from install-shape effect (v0 flip happened with zero Skill calls).
- pack flags (D22): --skills-dir / --claude-md / --scenarios-dir; installer
  copies any child dir containing a SKILL.md. Defaults preserve v0 behavior.
- shell-arg quoting (D24): LiveExecutor quotes args so allowed-tool patterns
  with spaces/parens survive the Windows shell:true spawn.
- quote verification (D23): third formatting-insensitive tier — a verbatim
  quote that dropped markdown backticks/emphasis still verifies.
- record-fixtures.mjs (D25): converts a real run dir into mock-fixture layout,
  replacing hand-authored reconstructions with real captures.
- report gains an inline column and 3-part pass@k/pass^k cells when inline
  data is present; 2-part table unchanged otherwise.

110 tests green; mock e2e, validate, and report re-render all pass.
@DizzyMii
DizzyMii merged commit 6e99cd0 into main Jul 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant