feat(skill): add run-assert-eval skill and eight worked example domains

.claude/skills/run-assert-eval/README.md

@@ -0,0 +1,107 @@
+# run-assert-eval skill
+
+Take a developer from **"I don't know my risks"** to a **measured violation rate
+per risk** — without leaving the coding assistant. Risk discovery is owned by
+**Clarity** (microsoft/clarity-agent); measurement is owned by **ASSERT**
+(responsibleai/ASSERT). This skill wires the two together.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `SKILL.md` | Claude Code skill entry (the canonical instructions). |
+| `../../.github/prompts/run-assert-eval.prompt.md` | GitHub Copilot mirror. |
+| `../../.cursor/rules/assert.mdc` | Cursor mirror. |
+| `workflows/measure-clarity-failures.md` | The 9-step measurement workflow (parse → triage → configs → run → report → close loop → archive protocol). |
+| `workflows/govern-and-remeasure.md` | The ACS governance workflow: turn a measured failure into a deployable ACS policy (`assert-ai acs generate`), wrap the agent, and re-run the same eval to prove the failure rate dropped. |
+| `workflows/diagnose-acs-delta.md` | Fallback reference manual for when a governed run's delta comes out wrong (no drop, or over-gating rose) — symptom-indexed, 15 rules. Most are prevented by the pre-flight classification in `govern-and-remeasure.md` Step 1a. |
+| `clarity_intake.py` | Dependency-free parser: Clarity failure docs → ASSERT candidate behaviors. |
+| `tests/` | Pytest suite + real Clarity fixtures for the parser. |
+| `SETUP-CHECKLIST.md` | One-time in-IDE MCP setup + end-to-end verification. |
+
+Keep the three skill surfaces (`SKILL.md`, the Copilot prompt, the Cursor rule)
+methodologically aligned when changing the flow.
+
+## Architecture
+
+1. **Discovery (Clarity, shipped):** the Clarity **MCP server** exposes tools —
+   `run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`,
+   and others. `run_clarity` returns Clarity's real process guide inlined; the host
+   agent conducts the clarifying conversation and persists findings. See
+   `SETUP-CHECKLIST.md` to wire it up.
+2. **Handoff (files, not JSON):** Clarity writes `.clarity-protocol/`. The
+   measurement side reads `failures/failures.md` (index) and `failure-NN-*.md`
+   (individual docs). Those files are the **source of truth**; the parser's JSON is
+   a disposable cache. Note it is **gitignored, single-domain scratch** — the next
+   `run_clarity` overwrites it, so each domain's protocol is archived to
+   `examples/<domain>/Clarity Protocol/` at the end of its run (Step 9), guarded by
+   a blocking check before any fresh discovery.
+3. **Measurement (this skill):** `clarity_intake.py` turns failure docs into
+   candidate behaviors; `workflows/measure-clarity-failures.md` runs a **mandatory
+   human triage gate**, generates **one atomic `eval_config.yaml` per selected
+   failure**, runs them sequentially, and reports one behavior per column.
+4. **Governance (ACS, optional):** when a run surfaces a real failure the user wants
+   to *fix and prove*, `workflows/govern-and-remeasure.md` first **classifies the
+   failure against the baseline** (Step 1a — semantic `output` gate vs. structural
+   tool gate, and whether the harm actually routes through the tool being gated),
+   then derives a deployable **ACS** policy from the findings
+   (`assert-ai acs generate`), wraps the agent's high-risk tools (or its output),
+   and re-runs the **same** eval against the governed target to show the
+   failure-rate delta (baseline → governed). If that delta comes out wrong,
+   `workflows/diagnose-acs-delta.md` is the symptom-indexed fallback.
+
+## The parser (`clarity_intake.py`)
+
+```
+python .claude/skills/run-assert-eval/clarity_intake.py .clarity-protocol
+```
+
+Per failure mode it emits a `CandidateBehavior`:
+`{name, description, severity, priority, source_doc, candidate_dimensions,
+multi_behavior, suggested_splits, warnings}`.
+
+- **Severity → priority**: Critical→P1, High→P2, Medium→P3, Low→P4. Ranges (e.g.
+  `Medium–Critical`) collapse to the **maximum** severity.
+- **Dimensions**: the doc's **Variants** list → an `elicitation_variant` stratify
+  dimension (highest value — each variant is a distinct route to the failure);
+  **Failure Chain** conditions → an `interaction_condition` dimension.
+- **Atomicity**: docs that bundle several independently testable behaviors are
+  flagged `multi_behavior` with `suggested_splits` so triage can surface the split.
+- **Tolerant**: unknown severity labels or missing headers degrade to a **flagged**
+  candidate (`warnings` populated) — never a crash, never a silent drop.
+
+Run the tests:
+
+```
+python -m pytest .claude/skills/run-assert-eval/tests/test_clarity_intake.py
+```
+
+## Worked example
+
+A full end-to-end walkthrough (one P1 — `user_disengagement` — from parse through
+triage, config generation, run, headline metrics, and closing the loop) lives in
+`workflows/measure-clarity-failures.md` under **Worked example (one P1)**. The ACS
+governance counterpart is in `workflows/govern-and-remeasure.md`.
+
+## Related ASSERT docs
+
+Product behavior is documented under `docs/` (team-maintained, on `main`); the skill
+**links** rather than restates it — `guides/create-evaluation.md` + `config/schema.md`
+(config authoring), `targets/callable.md` (callable signature, return types, OTel
+auto-instrumentation) + `targets/model-and-tools.md` (target shapes),
+`guides/troubleshooting.md`, `guides/results.md`, `guides/use-local-viewer.md`, and
+`guides/securing-agents-with-acs.md` (the ACS loop). This skill owns the *methodology*;
+those own *product behavior*. The exceptions the skill documents itself are the two
+callable traps those docs omit: `history` is detected by parameter **name** (misnaming it
+silently degrades multi-turn to single-turn), and module resolution falls back
+`sys.path` → config dir → cwd → direct file load.
+
+## Guarantees the skill enforces
+
+- One atomic behavior per config — never bundle.
+- The triage gate and the pre-run confirmation are **human** decisions; declining
+  writes nothing and runs nothing.
+- `.clarity-protocol/` files are authoritative; derived JSON is a cache.
+- Discovery goes through Clarity's real MCP tools — no plain-language fallback, no
+  shelling out to a `clarity cli` process, no separate app.
+- Never read/print/commit `.env`, credential values, or `artifacts/`.

.claude/skills/run-assert-eval/SETUP-CHECKLIST.md

@@ -0,0 +1,79 @@
+# Setup checklist — Clarity MCP ⇄ ASSERT (in-IDE only)
+
+These steps require a real IDE with MCP support (VS Code + Copilot agent mode,
+Claude Code, or Cursor) and cannot be completed from a headless terminal. Do them
+once per workspace, then the `run-assert-eval` skill's discovery front door
+(`run_clarity`) becomes callable.
+
+## Phase 1 — Environment setup
+
+- [ ] **Install Clarity with the MCP extra** from your clarity-agent checkout
+      (Python 3.12+):
+      ```
+      pip install -e ".[mcp]"      # or: uv pip install -e ".[mcp]"
+      ```
+- [ ] **Embed Clarity into this project** (generates `.vscode/mcp.json`, the
+      `.clarity-protocol/` scaffold, and the Clarity-managed block in `AGENTS.md`):
+      ```
+      clarity embed .
+      ```
+      - Verify `.vscode/mcp.json` has a `clarity-agent` stdio entry with
+        `CLARITY_PROJECT_DIR` set to this workspace folder.
+      - If your Clarity checkout is **uv-managed**, verify the entry uses
+        `uv run --extra mcp --directory <checkout> python -m clarity_agent.mcp`.
+- [ ] **Confirm the server starts**: `python -m clarity_agent.mcp --help`.
+- [ ] **Confirm an LLM provider is configured**: `clarity doctor`. Clarity
+      supports GitHub Copilot, Anthropic, OpenAI, Azure AI, and Gemini. Surface any
+      failure with its fix — do not silently continue.
+- [ ] **Verify ASSERT**: `assert-ai --help`, and a smallest-sample dry run of one
+      repo example config is invocable.
+- [ ] **Reload MCP servers** in the IDE so the `clarity-agent` tools appear, then
+      confirm you can call `run_clarity`.
+- [ ] **Do _not_ commit `.vscode/mcp.json`.** It is generated by `clarity embed .`
+      and its `uv --directory` arg holds an **absolute, machine-specific path** to
+      *your* clarity-agent checkout — committing it would hand teammates a broken
+      path. It is gitignored; each developer runs `clarity embed .` to generate their
+      own. (If your team pip-installs clarity-agent as a package, the generated entry
+      is `python -m clarity_agent.mcp` with no absolute path and could be committed —
+      but the default uv-checkout form must stay local.)
+
+## Phase 2 — End-to-end verification (definition of done)
+
+- [ ] **Fresh discovery**: with no `.clarity-protocol/failures/failures.md`, call
+      `run_clarity`, conduct a short clarifying conversation, and confirm
+      `failures.md` gets written.
+- [ ] **Parser**: `python .claude/skills/run-assert-eval/clarity_intake.py .clarity-protocol`
+      emits candidate behaviors; run the unit tests:
+      ```
+      python -m pytest .claude/skills/run-assert-eval/tests/test_clarity_intake.py
+      ```
+- [ ] **Single-P1 run**: from an existing `failures.md`, the workflow presents
+      triage, you pick one P1, **exactly one** config is generated with a
+      variants-derived dimension, `assert-ai run` completes, and the results table
+      renders with the behavior as a column.
+- [ ] **Two failures → two configs**: selecting two failures produces two separate
+      configs and two sequential runs — never one merged config.
+- [ ] **Decline at triage → zero writes**: declining at the triage gate results in
+      zero files written and zero runs.
+- [ ] **Loop-close**: a `record_suggestion` round-trip lands in the Clarity mailbox
+      after a completed run.
+
+## Notes
+
+- Copilot agent mode supports MCP **tools** only (not `clarity://…` resources).
+  `read_protocol_document` covers the same ground as the resource endpoints.
+- Never read/print/commit `.env` or credential values — reference env var **NAMES**
+  only (AZURE_API_KEY, AZURE_API_BASE, OPENAI_API_KEY, GITHUB_TOKEN,
+  ANTHROPIC_API_KEY, azure_ad_token).
+- Do not edit inside the Clarity-managed block in `AGENTS.md`
+  (between `<!-- clarity-begin -->` and `<!-- clarity-end -->`).
+- **Committing `.clarity-protocol/`**: this repo gitignores it because the protocol
+  describes a *system-under-test*, not this framework — it's per-target runtime
+  output. In **your own product's repo**, the protocol describes your product, so
+  prefer committing the durable docs (`goal/`, `solution/`, `failures/`) and
+  ignoring only `transcripts/` (and optionally `mailboxes/`). When you finish a
+  domain here, archive its protocol into `examples/<domain>/Clarity Protocol/` and
+  **commit it** so it is preserved alongside that domain's `evals/` and `acs/` —
+  this is Step 9 of `workflows/measure-clarity-failures.md`, and a blocking gate
+  before any fresh `run_clarity` enforces it (the source dir is gitignored, so an
+  overwrite is unrecoverable). See the per-example replication package in `SKILL.md`.