Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions .claude/skills/run-assert-eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

Take a developer from **"I don't know my risks"** to a **measured violation rate
per risk** — without leaving the coding assistant. Risk discovery is owned by
**Clarity** (microsoft/clarity-agent); measurement is owned by **ASSERT**
(responsibleai/ASSERT). This skill wires the two together.
**Clarity** (microsoft/clarity-agent) when the user wants it; measurement is owned
by **ASSERT** (responsibleai/ASSERT). This skill wires the two together, and also
measures risks the user names directly when discovery isn't what they need.

## Files

Expand All @@ -24,11 +25,14 @@ methodologically aligned when changing the flow.

## Architecture

1. **Discovery (Clarity, shipped):** the Clarity **MCP server** exposes tools —
1. **Discovery (Clarity, shipped — recommended, not required):** the Clarity **MCP server** exposes tools —
`run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`,
and others. `run_clarity` returns Clarity's real process guide inlined; the host
agent conducts the clarifying conversation and persists findings. See
`SETUP-CHECKLIST.md` to wire it up.
`SETUP-CHECKLIST.md` to wire it up. When the user would rather name the risk
themselves — or Clarity isn't set up — the skill takes a user-supplied risk
(prose, PRD, design doc, threat model) through a structured intake instead
(`SKILL.md` Step 1b) and everything downstream is identical.
2. **Handoff (files, not JSON):** Clarity writes `.clarity-protocol/`. The
measurement side reads `failures/failures.md` (index) and `failure-NN-*.md`
(individual docs). Those files are the **source of truth**; the parser's JSON is
Expand Down Expand Up @@ -102,6 +106,9 @@ silently degrades multi-turn to single-turn), and module resolution falls back
- The triage gate and the pre-run confirmation are **human** decisions; declining
writes nothing and runs nothing.
- `.clarity-protocol/` files are authoritative; derived JSON is a cache.
- Discovery goes through Clarity's real MCP tools — no plain-language fallback, no
shelling out to a `clarity cli` process, no separate app.
- Clarity discovery is **recommended, never a gate** — the user picks the risk
source, and a missing `.clarity-protocol/` never blocks a measurement.
- When the user chooses Clarity, discovery goes through its real MCP tools — never
an imitation of Clarity's interview, no shelling out to a `clarity cli` process,
no separate app.
- Never read/print/commit `.env`, credential values, or `artifacts/`.
15 changes: 13 additions & 2 deletions .claude/skills/run-assert-eval/SETUP-CHECKLIST.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
# Setup checklist — Clarity MCP ⇄ ASSERT (in-IDE only)

**Optional.** Clarity is the *recommended* risk source for `run-assert-eval`, not a
prerequisite for it. The skill runs fine without any of this — the user can supply
risks directly as prose or a PRD / design doc / threat model (SKILL.md Step 1b).
Work through this checklist when you want Clarity's discovery to surface failure
modes you haven't thought of.

These steps require a real IDE with MCP support (VS Code + Copilot agent mode,
Claude Code, or Cursor) and cannot be completed from a headless terminal. Do them
once per workspace, then the `run-assert-eval` skill's discovery front door
once per workspace, then the `run-assert-eval` skill's recommended discovery path
(`run_clarity`) becomes callable.

## Phase 1 — Environment setup
Expand Down Expand Up @@ -39,9 +45,14 @@ once per workspace, then the `run-assert-eval` skill's discovery front door

## Phase 2 — End-to-end verification (definition of done)

- [ ] **Fresh discovery**: with no `.clarity-protocol/failures/failures.md`, call
- [ ] **Fresh discovery** (Path A): with no `.clarity-protocol/failures/failures.md`, call
`run_clarity`, conduct a short clarifying conversation, and confirm
`failures.md` gets written.
- [ ] **User-supplied risks work without Clarity** (Path B): with the Clarity MCP
tools unavailable *or* declined, describe one risk in plain language and
confirm the skill offers the choice, does **not** stop on the missing
protocol, and still produces a single atomic config with an explicit
permissible boundary and a variants-derived dimension.
- [ ] **Parser**: `python .claude/skills/run-assert-eval/clarity_intake.py .clarity-protocol`
emits candidate behaviors; run the unit tests:
```
Expand Down
168 changes: 132 additions & 36 deletions .claude/skills/run-assert-eval/SKILL.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@ Specification) policy, then re-run the same eval against the governed agent to
**prove the failure rate dropped** — the ACS delta.

This is the governance half of the story and picks up where
`measure-clarity-failures.md` leaves off: Clarity discovered the risk,
ASSERT measured a baseline violation rate, and now ACS governs the failure at
runtime. It uses ASSERT's **native** ASSERT to ACS adapter (`assert-ai acs …`),
which derives the policy straight from the run's findings — no external `acs`
CLI and no separate checkout of the agent-governance-toolkit are needed.
`measure-clarity-failures.md` leaves off: a risk was established (via Clarity or
supplied by the user), ASSERT measured a baseline violation rate, and now ACS
governs the failure at runtime. It uses ASSERT's **native** ASSERT to ACS adapter
(`assert-ai acs …`), which derives the policy straight from the run's findings —
no external `acs` CLI and no separate checkout of the agent-governance-toolkit are
needed.

> **Everything stays in-IDE.** ACS has no MCP server; the `assert-ai acs`
> subcommands are the in-IDE surface, driven the same way ASSERT already drives
Expand Down Expand Up @@ -659,6 +660,10 @@ exported HTML — it is per-run output.)

## Step 7 — Close the loop in Clarity

**Only when a `.clarity-protocol/` exists.** If the risk came from the user rather
than Clarity, skip this step — mention once that you're skipping it, and go
straight to the regression-check offer below.

Offer to write the outcome back into `.clarity-protocol/` via the Clarity MCP
tool `record_suggestion` (or `record_decision`): the failure mode was measured
against the reviewed ACS policy under `artifacts/acs/<suite>/`, and baseline `X%`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,19 @@
# Workflow: measure-clarity-failures

Turn Clarity-discovered failure modes into measured ASSERT violation rates —
one atomic behavior at a time, with a human in the loop at every gate.
Turn discovered or user-supplied failure modes into measured ASSERT violation
rates — one atomic behavior at a time, with a human in the loop at every gate.

This workflow is the measurement half of the Clarity → ASSERT story. Discovery
is owned by the **Clarity MCP server** (`clarity-agent`, shipped by
microsoft/clarity-agent); measurement is owned by this skill. The handoff is
**files, not JSON**: Clarity writes `.clarity-protocol/failures/`, and this
workflow reads it.

Clarity is the **recommended** risk source, not a required one. When the user
supplies risks directly (SKILL.md Step 1b), skip Step 1 (Parse) and start at
Step 2 (Triage) — Steps 2-9 never touch `.clarity-protocol/` except at the
optional close-the-loop step.

> **Discovery is agent-driven, not scripted.** The Clarity MCP `run_clarity`
> tool returns the relevant process guide inlined as text; **you** (the host
> agent) ask the user the clarifying questions in chat and persist what you learn
Expand All @@ -21,18 +26,25 @@ Trigger this workflow when the user asks to **measure / test / quantify** risks
or failures for their agent, model, or app.

1. **If `.clarity-protocol/failures/failures.md` exists** → go to **Step 1 (Parse)**.
2. **If it does not exist** → run discovery first:
- **Run the preservation gate below first** — a fresh discovery run destroys
any protocol from a previous domain.
- Call the Clarity MCP tool **`run_clarity`**. Follow the inlined process
guide's clarifying questions *with the user in chat*.
- Persist findings via **`write_protocol_document`** and **`record_failure`**.
- Continue until the failure-analysis process has produced
`failures/failures.md`, then proceed to Step 1.
- If the `clarity-agent` MCP tools are **not available** in this session, stop
and point the user at the in-IDE setup checklist (`SETUP-CHECKLIST.md`):
`clarity embed`, reload MCP servers, confirm `run_clarity` is callable. Do
**not** substitute a plain-language risk guess — that produces low-signal evals.
2. **If it does not exist** → ask which risk source the user wants (see SKILL.md,
"Choosing a risk source"). Recommend Clarity, but take their answer:
- **Clarity discovery (recommended)** —
**run the preservation gate below first**; a fresh discovery run destroys
any protocol from a previous domain. Call the Clarity MCP tool
**`run_clarity`** and follow the inlined process guide's clarifying questions
*with the user in chat*. Persist findings via **`write_protocol_document`**
and **`record_failure`**. Continue until the failure-analysis process has
produced `failures/failures.md`, then proceed to Step 1.
- **User-supplied risks** — the user names the risk, or points at a PRD,
design doc, threat model, incident report, or test plan. Follow **SKILL.md
Step 1b** to build the candidate-behavior list by hand, then **skip Step 1
(Parse)** — there is no `failures.md` to parse — and join at **Step 2
(Triage)**. Everything from Step 2 onward is risk-source agnostic.
- If the `clarity-agent` MCP tools are **not available** in this session, say
so, offer the in-IDE setup checklist (`SETUP-CHECKLIST.md`: `clarity embed`,
reload MCP servers, confirm `run_clarity` is callable), and let the user
choose. If they'd rather not set it up now, continue with user-supplied
risks — do not strand them on MCP setup.

### Preservation gate (blocking — check before any fresh `run_clarity`)

Expand All @@ -56,6 +68,10 @@ re-scaffolds a clean one on the next `run_clarity`.

## Step 1 — Parse

**Clarity-sourced risks only.** If the user supplied risks directly (SKILL.md
Step 1b), there is no `failures.md` to parse — skip to Step 2 with the candidate
list you built there.

Run the intake parser (`clarity_intake.py`) on the protocol directory:

```
Expand Down Expand Up @@ -278,6 +294,11 @@ the failing checkpoint, or **re-measure after a fix** to prove the rate dropped.

## Step 8 — Close the loop in Clarity

**Only when a `.clarity-protocol/` exists.** With user-supplied risks there is
nothing to write back to — skip this step, mention once that you're skipping it,
and instead offer Clarity discovery as a way to find failure modes this pass
didn't cover.

After a run, offer to write the outcome back into `.clarity-protocol/` via the
Clarity MCP tool **`record_suggestion`** (or **`record_decision`**): note that the
failure mode now has a **measured baseline** and where the eval lives
Expand Down
Loading
Loading