Prove the differentiation: gate hardening, priced sizing, a falsifiable roadmap, and 7 real eval runs - #5
Merged
Conversation
…lock) warn (default) keeps the once-per-distinct-bypass behavior; block re-blocks every stop attempt until the bypass is repaired or recorded as skipped. Pinned per repo via a 'Gate enforcement:' line in the ## Working Genius section, resolved by the shared parser's new 'enforcement' command so hooks and skills read one source of truth. Red before green: 7 new deterministic tests failed before the change; 53/53 pass after (bash evals/gates.test.sh). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
/genius now announces the path (express/full) with its one-line reason and the measured cost asymmetry (the 11x full-flow multiple from evals/RESULTS.md), invites an overrule instead of presenting a menu, and records the call as a **Sizing:** line under the work file title so close-outs can judge sizing against outcomes. "Just do it" on full-flow-shaped work maps to delegated mode, never express. New scenario genius.md M4 covers the behavior (authored, not yet run — model-based scenario runs cost real tokens; RESULTS.md stays honest). Deterministic suite unaffected: 53/53. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
The six-stage shape is commodity — every workflow tool ships phases.
What this plugin can claim that most cannot: the gate rule is code
with a deterministic test suite, the prose is tested red-before-green
against no-plugin baselines, the cost guidance traces to instrumented
runs (including one recorded reversal), and the ceremony itself is
priced. The README now opens with that claims-to-evidence table and
invites readers to check each row.
Also: scenario counts corrected ("three per skill" -> "three or more"),
and evals/README documents the *(not yet run)* marker for authored-but-
unrun scenarios.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
Every competing workflow tool demos greenfield. This plugin's sharpest moves — Wonder's prior-art shrinking, blindspot's git-history mining, Discernment's attack against recorded decisions, the glossary's dialect collisions — only have material when the repo has history. Say so in the README as a named section instead of leaving it implicit in skill bodies. No skill text changed: blindspot's trigger description is tuned by trigger evals and doesn't move without a run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
Five phases, each an attack on a verified market opening (trust layer, in-loop cost, standalone gate engine, eval harness as product, brownfield flagship), each item with acceptance criteria a cold reader can check, each phase with the evidence that would kill it. Non-goals decline the occupied categories by name - including the two rare openings we deliberately pass on (cross-model review, auto-extracted lessons), with the conditions that would reopen them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
The review (1 critical, 3 major, 4 minor) held. Repairs: 1.1 re-keyed to ground truth - all 31 scenarios now carry honest (not yet run) markers (the critical: the marker system had already rotted, so "no markers remain" was satisfiable by running one scenario); Phase 5 gains the kill-criterion the preamble promised; 1.4 re-premised to post-tiering (sizing cannot move a full-flow-warranted task's cost); a portfolio gate caps extractions at one in flight (the spread that killed GSD); ruflo's budget alerts named instead of flattened to "nobody"; 3.3/4.3 accepts split from third-party success metrics; 2.3 gets its red-before-green clause; artifact drift declined as a non-goal with its reason. README's "prose is tested" row now states coverage honestly instead of implying totality. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
The reviewer's one remaining (minor) finding: "scenarios that back README claims-table rows" and "highest-stakes per skill" were judgment calls a cold reader could not verify against the accept clause. Now the eleven IDs are named in the item - one core-failure-mode scenario per skill plus M4 for the priced-ceremony claim - and the accept references exactly those eleven. Review verdict on this roadmap: round 1 FAIL (1 critical, 3 major, 4 minor - all repaired), round 2 PASS with this one minor, now closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
.github/workflows/gates.yml runs bash evals/gates.test.sh on pushes and PRs that touch hooks/ or the suite itself — no model, no network, no tokens. README carries the status badge. Locally demonstrated the gate actually gates: breaking the bypass-detection branch turns the suite red (exit 1, 7 failures) and would block CI; restored to 53/53 green. The live red-run link lands once GitHub runs the workflow post-push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
The Token-economics section now opens with a governing caveat — every figure there is a single metered run, n=1 per cell, output-token pricing only — which covers the $6.55-of-$20 reviewer cost and the greenfield tiering-reversal claim in place. The 11× full-flow multiple carries an explicit n=1 bound at every surface it travels to outside that section: the README claims table, the /genius sizing rule, and the ROADMAP's load-bearing line. No number now travels without its bound. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
The eval loop now runs end-to-end without a human in the seat (evals/run-scenario.sh: fresh scratch, arm-specific setup, claude -p, transcript captured). Running M2 for real bought two things worth more than a green check: - A fixture leak, found and fixed. scratch.sh's CLAUDE.md documents the whole Working Genius flow, and that section was reaching the BASELINE arm too - handing the no-plugin control the plugin's own methodology. The runner now strips it for baselines; evals/README step 6 makes it a standing rule. Every prior baseline assumption is suspect until re-run clean. - M2 is a softball on frontier tier. Against a clean sonnet-5 baseline, the base model reproduced the target behavior (read .genius/, found the weakest-genius pattern, sized proportionally) three ways - process-inviting prompt, neutral prompt, de-leaked fixture. The scenario's named failure mode doesn't occur; the skill line is a no-op for elicitation there. Recorded honestly in RESULTS with its caveats (n=1, author-graded, sonnet-5 only); M2's marker re-annotated to "softball", not cleared - it was run and did not pass. This is the eval doing its job: direct evidence that the soft advisory behaviors are commodity on a current baseline, and that the real value must live in the mechanical parts (hook gates, red-before-green, fresh-evidence verification). Those are the scenarios to run next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
M1 (unrecorded bypass) run 3x per arm on sonnet-5, graded by a blinded subagent (arm key withheld). Skill 3/3 on all items; baseline 1/3 on the load-bearing "repair before build" item. Two truths recorded: the scenario's "parrots frontmatter" baseline is extinct (all 3 baselines noticed the gate and named the open boxes), but the discipline delta survived - given the same observation the baseline builds through the open gate 2/3, the skill halts 3/3. This is the mechanical value M2's softball pointed at, measured on the other side: the plugin's edge is converting a notice into a stop, not the noticing. M1 marker updated to passed-with-caveat; scenario baseline note sharpened from the extinct failure to the real one so the next runner tests reality. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
E1 (red-before-green) run 3x per arm on sonnet-5, graded programmatically from the event stream (order of ops is objective - no grader to bias). Skill 3/3: test written, run, seen RED before impl exists. Baseline 0/3: implements first, test green on arrival, never red. The cleanest delta of the three scenarios, on the plugin's core identity. With M2 (softball) and M1 (partial), three real runs now map where the plugin still beats a frontier baseline: advisory behavior is absorbed (M2), the perception under discipline is absorbed (M1 baseline sees the gate), but the mechanical commitment is not - halting on the gate and failing the test first. Value lives in making the model do the disciplined thing it reasons past, not in telling it what it knows. RESULTS carries the synthesis table; README claims row now cites the measured spread instead of implying uniform success. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
T1 (no false "done") run on sonnet-5: baseline 3/3 caught the planted broken test and refused to declare done - the "passed earlier" log carried no weight. Told to "wrap up," a careful frontier model verifies fresh unaided. Softball. T1 breaks the first-cut "mechanical = differentiated" reading: verify- before-done IS mechanical yet commodity. The four runs (M2, T1, M1, E1) resolve to a sharper axis - the delta lives where the discipline fights the model's default momentum. Aligned disciplines (size from history, verify before done) are absorbed; momentum-contrary ones (fail the test BEFORE the code; STOP at a gate you could pass) are not. Red-before-green is the sharpest (0/3 baseline), gate-stop softer (1/3). RESULTS carries the rewritten synthesis; README claims row states the sharper cut. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
W1 (question behind the request) was run AFTER the momentum thesis was written, to test it. The thesis predicted a delta - questioning a literal ask before building runs against build-what-was-asked momentum. Result: skill 0/3 built directly, 3/3 questioned first (one even noticed there's no CLI command to add a flag to); baseline 3/3 built the feature directly, 0/3 questioned. Clean 3/0, exactly as forecast. Five scenarios now (M2, T1, M1, E1, W1). The value axis is confirmed and predictive: momentum-aligned disciplines (size from history, verify before done) are softballs; momentum-contrary ones (fail the test first, question the ask) are clean deltas; the gate-stop sits between. RESULTS synthesis table + entry updated; README claims row and scenario markers reflect it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
…esis) Two things in one commit. (1) run-scenario.sh had lost its M1/E1/T1/W1 cases from the committed tree (edits made and used within a turn, then reverted before commit); reconstructed complete with write_enablement_file + all five prior cases + the new D1 case, so the harness matches the recorded RESULTS and is re-runnable. (2) D1 run as a second predictive test of the momentum thesis - and it failed the prediction, usefully. Handed two options cold, the baseline is inconsistent (1/3 full discernment, 1/3 light lean, 1/3 balks at the fixture's unreal checkout), skill n=1 clean. The miss sharpened the thesis boundary: momentum-contrary requires the model to oppose momentum it actually built, not to evaluate options handed to it cold. Also exposed a second leak - the work-file scaffold itself primes discernment. D1 marked needs-fixture-work, not pass/fail. RESULTS synthesis, table, and D1 entry updated; README notes the self-correction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
Built a two-turn probe to make the momentum-contrary case D1 couldn't: commit to an approach (turn 1), then a go/no-go (turn 2) - does the model attack its own choice or defend it? Draft 1 hit D1's repo-mismatch confound again (now documented). Draft 2 (a repo-fitting --max-concurrent task): both arms self-attacked their own plan identically - the baseline even more observant. Softball. The collapse IS the finding and it unifies all seven runs: the delta is never in eliciting reflection (ask a model to size/verify/review its own plan and it does fine - M2, T1, SA are softballs), only in forcing the action against momentum without being asked (fail the test first E1, question the ask W1, halt at the gate M1). Reflection-on-request is commodity; disciplined action at the decision point is the product. Synthesis + README sharpened to that one sentence. Harness: SA's early exit skipped the setup commit, leaking a stray CLAUDE.md edit an observant baseline flagged - fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018z1LJdF732mkwHCLW1Htfh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Turns the plugin from "a workflow with good prose" into "a workflow that can show its evidence." Four product upgrades, a falsifiable roadmap, and the first real end-to-end eval runs — which measured where the plugin actually beats a no-plugin baseline and where it doesn't.
Product changes
warn(default) /block, resolved by the shared parser via aGate enforcement:line. 53 deterministic tests (red-before-green: 7 new tests failed before, all pass after)./geniusannounces the path (express/full) with its reason and the measured cost asymmetry, records a**Sizing:**line, and maps "just do it" to delegated mode rather than silent express.ROADMAP.md— five phases, each item with falsifiable acceptance criteria and each phase with a kill-criterion. Passed two rounds of an adversarial subagent review (1 critical + 3 major + 4 minor → all repaired → PASS).Evidence infrastructure (the differentiator)
.github/workflows/gates.yml) runs the deterministic gate suite on every push.evals/run-scenario.sh) — builds a scratch, installs/strips skills, de-leaks the baseline'sCLAUDE.md, runsclaude -p, captures the transcript.evals/RESULTS.md, blind- or programmatically graded against a no-plugin baseline (sonnet-5):The finding, predictively tested (W1 forecast correct; D1 and SA falsified attempts that sharpened it): the plugin's differentiated value is not in eliciting reflection a capable model already does when asked — it is in forcing the disciplined action at the decision point where the model would otherwise proceed (fail the test first, question the ask, halt at the gate). Two methodology fixes came out of the runs (a
CLAUDE.mdbaseline leak; a work-file scaffold leak), both documented.Verification
bash evals/gates.test.sh→ 53/53 green.RESULTS.mdrow that states its n; the 11× full-flow cost carries itsn=1bound wherever it appears.Conflict-free with
main(fast-forward). 23 files changed.🤖 Generated with Claude Code
Generated by Claude Code