feat(skill): run-assert-eval smoke-run 3 real test cases before full inference

.claude/skills/run-assert-eval/README.md

@@ -17,7 +17,8 @@ measures risks the user names directly when discovery isn't what they need.
 | `workflows/govern-and-remeasure.md` | The ACS governance workflow: turn a measured failure into a deployable ACS policy (`assert-ai acs generate`), wrap the agent, and re-run the same eval to prove the failure rate dropped. |
 | `workflows/diagnose-acs-delta.md` | Fallback reference manual for when a governed run's delta comes out wrong (no drop, or over-gating rose) — symptom-indexed, 15 rules. Most are prevented by the pre-flight classification in `govern-and-remeasure.md` Step 1a. |
 | `clarity_intake.py` | Dependency-free parser: Clarity failure docs → ASSERT candidate behaviors. |
-| `tests/` | Pytest suite + real Clarity fixtures for the parser. |
+| `smoke_slice.py` | Slices N real rows out of a generated test set so a config can be validated before the full suite. |
+| `tests/` | Pytest suite + real Clarity fixtures for the parser and the slicer. |
 | `SETUP-CHECKLIST.md` | One-time in-IDE MCP setup + end-to-end verification. |
 
 Keep the three skill surfaces (`SKILL.md`, the Copilot prompt, the Cursor rule)
@@ -80,6 +81,41 @@ Run the tests:
 python -m pytest .claude/skills/run-assert-eval/tests/test_clarity_intake.py
 ```
 
+## The smoke slicer (`smoke_slice.py`)
+
+```
+python .claude/skills/run-assert-eval/smoke_slice.py \
+  --config evals/<atomic_behavior>.yaml --count 3
+```
+
+Carves the first N rows of a given kind out of a suite's **already generated**
+test set and writes them to `artifacts/smoke/<suite>-<kind>-<n>.jsonl`, so
+`pipeline.inference.test_set_path` can point at a handful of real cases. Emits a
+JSON summary (`source`, `resolved_via`, `out`, `written`, `available`,
+`test_case_ids`). Use `--suite` instead of `--config` to skip the PyYAML import.
+
+- **Resolves through `latest.json`**, the pointer ASSERT itself maintains.
+  Version dirs (`v0001`, `v0002`, …) are allocated fresh on every cache miss, so
+  they are never assumed; a stale published copy is only a fallback.
+- **Copies raw lines**, so the slice is byte-identical to the source rows —
+  the smoke run scores cases the full run will also score.
+- **Refuses to write inside the suite root**, which could clobber the published
+  `test_set.jsonl` and invalidate the cache the smoke run exists to protect.
+- **Treats `--suite` as an identifier, not a path** — same slug rule ASSERT
+  applies to `suite`, and the resolved suite root must stay under the results
+  directory. `--config` resolves `results_dir` exactly as `assert_ai.config`
+  does, artifact-root prefix included, so both flags read the tree ASSERT wrote.
+- **Why not just lower `sample_size`**: that block feeds the test_set stage's
+  `config_hash`, so changing it invalidates the cached test set and cascades
+  downstream — and under `pairwise` sampling it yields a different design, not a
+  subset. See `workflows/measure-clarity-failures.md` Step 5a.
+
+Run the tests:
+
+```
+python -m pytest .claude/skills/run-assert-eval/tests/test_smoke_slice.py
+```
+
 ## Worked example
 
 A full end-to-end walkthrough (one P1 — `user_disengagement` — from parse through

.claude/skills/run-assert-eval/SETUP-CHECKLIST.md

@@ -58,6 +58,15 @@ once per workspace, then the `run-assert-eval` skill's recommended discovery pat
       ```
       python -m pytest .claude/skills/run-assert-eval/tests/test_clarity_intake.py
       ```
+- [ ] **Smoke slicer**: after any run that generated a test set,
+      `python .claude/skills/run-assert-eval/smoke_slice.py --suite <suite> --count 3`
+      writes a 3-row slice and reports `resolved_via: latest.json`; run the unit tests:
+      ```
+      python -m pytest .claude/skills/run-assert-eval/tests/test_smoke_slice.py
+      ```
+- [ ] **Smoke gate**: the workflow offers a smoke run before the full suite, and a
+      deliberately broken `target.callable` fails at the smoke step rather than
+      after a full 25+25 run.
 - [ ] **Single-P1 run**: from an existing `failures.md`, the workflow presents
       triage, you pick one P1, **exactly one** config is generated with a
       variants-derived dimension, `assert-ai run` completes, and the results table

.claude/skills/run-assert-eval/SKILL.md

@@ -364,13 +364,39 @@ single helper call at the top of the callable module — see `docs/targets/calla
 
 ### 5. Run the pipeline
 
+**Offer a smoke run first.** A suite is 25 prompt + 25 scenario cases, and
+plumbing errors (wrong `callable`, missing credentials, a callable that raises
+on its first tool call, tool-schema mismatch, undeployed judge model) surface
+only once inference starts. Validate on 3 real cases first:
+
+```
+# 1. artifacts only, no inference cost
+assert-ai run --config evals/<atomic_behavior>.yaml \
+  --override inference.enabled=false --override judge.enabled=false
+
+# 2. slice 3 real rows out of the generated test set
+python .claude/skills/run-assert-eval/smoke_slice.py \
+  --config evals/<atomic_behavior>.yaml --count 3
+
+# 3. inference + judge on those rows only
+assert-ai run --config evals/<atomic_behavior>.yaml \
+  --override run=<run>-smoke \
+  --override inference.test_set_path=<out path from step 2>
+```
+
+If it fails, stop and report — do not start the full run. Three cases is not a
+measurement, so never report a rate from a smoke run. Never lower
+`test_set.sample_size` instead: it invalidates the cached test set and does not
+produce a subset. Full detail in `workflows/measure-clarity-failures.md`
+Step 5a.
+
 ```
 assert-ai run --config evals/<atomic_behavior>.yaml --output json
 ```
 
 This is long-running (systematize -> test_set -> inference -> judge). Stream status
 to the user as each stage completes. For N configs, run them sequentially and track
-each `suite`/`run`.
+each `suite`/`run`. After a smoke run the first two stages report CACHED.
 
 - To re-run from a specific stage: `--force-stage <stage>`
 - Note the `suite` and `run` names from the config for Step 6.