xhard flow evals + design skills: project-specific, general, eval-proven

PROGRESS.md

@@ -1,6 +1,8 @@
 # Progress
 
 ## Now
+- DONE (2026-10-02) design eval goal: hard exam (5 hard fixtures x {design, vary}) 5.81 -> 6.55 -> 4.74 (generality overshoot) -> 5.80 -> 5.65 -> **7.97** (Opus builder; min 6.1; 10/10 no blocking fold rows); breadth exam (6 non-illustrated genres, genre-anchored judges) 6.38 -> 5.85 -> 5.88 -> **7.79**, obeys_brief 48/48. Biggest lever was the builder model (memory: design-eval-builder-model). Demo sites (/design:design, Opus): Spói https://claude.ai/artifact/KGQNEUu5ypejFhsygbbmeH, Tally https://claude.ai/artifact/TJ4GjyQUv94ruCzS2geooS, Hrímgangan https://claude.ai/artifact/4yYZdfzdkDNffk2J5ySKPm. ELI5 (flow + design + demos): https://claude.ai/artifact/C7AvjnyfHRqEHDP1HSb7oM. Harness: ~/.claude/jobs/905f7787/tmp/design/ (trial.sh on Opus, judge-hard.js, judge-breadth.js, score.py). Open: fold-check `plates` counts only inline svg/canvas, not `<img src=*.svg>` (non-blocking false FAIL on Hrímgangan); hitataekni benchmark needs its store backend for bench-fold.sh.
+- xhard eval loop (branch `worktree-hard-evals-and-next-finish`, pushed): 32 `xhard-*` pipeline cases, `scripts/case-selfcheck` (static proof; `--live` = headless `claude -p` outside the eval sandbox, which masks /bin/git). `/flow:next --finish` = draft PR #13. Live: L1 9/26 (judge noise), L2 30/52 runs mean 0.86; landed F1-F7, K1-K7, `flow check --since <Base>`. L3 (32x2) → `/tmp/flow-live-L3`. Loop: live → triage (one agent per failing case) → fix flow + cases → harder cases. Resume: `python3 plugins/flow/scripts/case-selfcheck --live -j 6 --runs 2 --ledger --out /tmp/flow-live-L<n>`
 - /claude-api prompt-audit applied to all 69 skills on branch `worktree-prompt-audit-skills`: caps emphasis and load-gates at normal volume (constraints kept), dated prompting advice rewritten (prompt-engineer, agent-architecture, agent-evals, claude-md), old-model fossils dropped. Skill-text tests 3715/35, no new failure. Report + 9 open items needing a human: `docs/research/21-prompt-audit-skills-2026-09-27.md`. Next: read the PR, decide the open items
 - spec 009 (ECC borrows, from a 48-agent 3-lens comparison with affaan-m/ECC) on branch `flow/ecc-borrows`, draft PR: loop child wall-clock cap, review lenses fail closed (`failedLenses`/`incomplete`), `context-pressure.sh` PreToolUse nudge, injection-as-data rule, hidden-unicode scan in tamper-notice, doctor `worktree-hygiene`/`personal-paths`/`reference-docs`, skill-forge reuse check, install/stealth write-confinement tests; plus both suites repaired to green on macOS (TMPDIR slash, shellcheck 0.11) and `flow lint` fixed to join `files:` by pathspec like `flow tick` (every ticked gate with `files: .` read as `lying`). Gates: hooks 1333/0, 9 touched scripts files green, full scripts suite 20 fixed / 0 new (`.specs/009-*/verify/`). Next: human read of the PR, then mark ready; I-001 parks ~87 unrelated scripts-suite failures, I-002 a garbled `started_at` silently disabling the loop's max-minutes cap
 - spec 007 (plugin eval suite + no-slop) on branch `flow/plugin-eval-suite-v2` (worktree `../code-worktrees/flow/plugin-eval-suite-and-optimisation-loop`, draft PR #5): `flow eval` (ledger, tags, -j, Bash grant, post-checks), 31-case suite under `plugins/flow/evals/`, `no-slop` skill (rubric NS-01..29, `slop-check`, developer block in every `task-brief`, `slop` lens per task and at gating), `search-first.sh` hook. v1.1 certification 3 runs/case: quality **1.00** (post-checks 35/36, delta +0.13); pipeline **0.99** (post-checks 9/9); routing **0.93**; invariant **1.00** (`.specs/007-*/loop-log.md`). Next: PR review

docs/reference/scripts.md

@@ -286,3 +286,27 @@ the file is now over 60 lines, a warning to stderr (exit stays 0).
 
 Exit codes: 0 written · 2 usage · 3 exact duplicate · 4 cannot write
 ```
+
+## case-selfcheck
+
+```
+Usage: case-selfcheck [--root <evals-dir>] [--keep] [<case-name>...]
+
+Proves each eval case can tell a no-op from a fix. Default: every
+evals/xhard-* case. Per case: shape (case.yaml, tags, executable scaffold),
+then scaffold a pristine tree — its file graders + postcheck.sh must NOT all
+pass (unless the case ships selfcheck-pristine-passes) — then scaffold again,
+run solution.sh, and every file grader + postcheck.sh must pass. llm,
+tool_used and trace graders are skipped and counted.
+Prints PASS/FAIL per case; exit 0 iff no FAIL.
+
+case-selfcheck --live [--model M] [--judge-model J] [--runs N] [-j N]
+                      [--out <dir>] [--keep] [--ledger] [<case-name>...]
+
+Runs the cases headless with `claude -p` outside the plugin-eval sandbox, in
+throwaway trees, with this checkout's plugin bin first on PATH. Grades regex,
+tool_used, llm (headless judge), file graders and postcheck.sh. One
+`LIVE <case> score=.. post=.. turns=.. cost=$.. <secs>s` line per run, results in
+<out>/results.json. --ledger appends one line to evals/ledger.jsonl.
+Exit 0 iff every run passed, 1 otherwise, 2 on setup errors. Spends money.
+```

docs/reference/workflows-and-cli.md

@@ -107,7 +107,7 @@ Commands:
   install [--dry-run] [--dotfiles <path>] [--marketplace <path>] [--force]
                                                      Deploy the harness on this machine
                                                      (idempotent); ends by running doctor
-  check [--fix]                                     Run project quality gates (check-all)
+  check [--fix] [--since <ref>]                     Run project quality gates (check-all; --since scopes lint/format)
   skills-lint                                       Run ~/.claude/scripts/skills-lint
   next [--json] [--force] [--peek]                  Print the one thing to do next, from
                                                      .specs/ and git alone (14 states; --json