feat(skill): run-assert-eval smoke-run 3 real test cases before full inference

[alex ngo (ango10)]

Summary

Adds a smoke-run capability to the run-assert-eval skill: validate a config end-to-end on 3 real test cases before committing to a full 25 prompt + 25 scenario run. Implemented entirely with existing ASSERT primitives. No changes to assert_ai/.

Motivation / linked issue

Resolves #316.

Plumbing errors: a wrong target.callable, missing credentials, a callable that raises on its first tool call, a tool-schema mismatch, an undeployed judge model, etc. only surface once inference starts, after systematize and test_set have already run. Today the cheapest way to find out is to pay for the whole suite.

Changes

• smoke_slice.py (new): slices the first N rows of a given kind out of a suite's already-generated test set to artifacts/smoke/<suite>-<kind>-<n>.jsonl, and prints a JSON summary. Stdlib-only (PyYAML only for the optional --config form). Three deliberate safety properties:
  • resolves the test set through latest.json, the pointer ASSERT itself maintains, rather than assuming v0001. Version dirs are allocated fresh on every cache miss.
  • copies raw line text, so sliced rows are byte-identical to rows the full run will score.
  • refuses to write inside the suite root, which could flaw the published test_set.jsonl and invalidate the very cache the smoke run exists to protect.
• tests/test_smoke_slice.py (new): 27 tests; all build isolated suite trees under tmp_path and never touch the real artifacts/results/.
• workflows/measure-clarity-failures.md: new Step 5a, offered by default: generate artifacts with inference.enabled=false --override judge.enabled=false, slice 3 rows, run them under run=<run>-smoke, then gate. Documents where to read the result (results status, viewer URL), that a smoke run is a run inside the suite and not a suite card, and a "never substitute these" block covering the sample_size trap, the separate run: label, and the suite-root write ban.
• workflows/govern-and-remeasure.md: smoke the governed callable before the governed run, since that's where a wrong agent_guarded path is most likely and a failure wastes half an A/B.
• SKILL.md, .github/prompts/run-assert-eval.prompt.md, .cursor/rules/assert.mdc: the run step in all three surfaces the README requires kept aligned.
• README.md, SETUP-CHECKLIST.md: slicer reference section and two verification bullets.

Documentation-only outside the two new files; no assert_ai/ code changed.

Testing

• pytest .claude/skills/run-assert-eval/tests/ → 56 passed (29 existing + 27 new). This is exactly how CI invokes it; regression.yml already globs .claude/skills/** and that test directory, so the new suite is picked up with no workflow change.
• pytest .claude/skills/run-assert-eval/tests/ tests/test_init_command.py → 68 passed after merging main, since the incoming fix(init): emit YAML block scalars for multi-line strings and non-ASCII #314 touches config emission.
• Real end-to-end run on travel-langgraph-budget-overrun: 3 prompt cases, inference 38.6s + judge 14.9s = 53.6s, 3 judge calls, judge failure rate 0.0%. Artifacts, scores.jsonl, metrics.json and all five .viewer/ read-model files were produced, and the run is discoverable via both assert-ai results status <suite> baseline-smoke and the viewer run page.
• Both documented --override commands validated against the real config loader (_apply_config_overrides + load_runtime_context, no LLM calls): step 1 disables inference/judge as genuine booleans; step 3 keeps suite_root unchanged so systematize/test_set stay cache hits, routes the slice to inference.test_set_path, and leaves test_set.prompt.sample_size at 25.

Checklist

• Tests pass locally (pytest and/or viewer checks as applicable).
• Docs updated if behavior or public API changed.
• No secrets, credentials, or customer data committed.
• No breaking change, or a CHANGELOG.md entry is included.

[alex ngo (ango10)]