diff --git a/.claude/skills/run-assert-eval/README.md b/.claude/skills/run-assert-eval/README.md index f0a7d9bf..931189f4 100644 --- a/.claude/skills/run-assert-eval/README.md +++ b/.claude/skills/run-assert-eval/README.md @@ -2,8 +2,9 @@ Take a developer from **"I don't know my risks"** to a **measured violation rate per risk** — without leaving the coding assistant. Risk discovery is owned by -**Clarity** (microsoft/clarity-agent); measurement is owned by **ASSERT** -(responsibleai/ASSERT). This skill wires the two together. +**Clarity** (microsoft/clarity-agent) when the user wants it; measurement is owned +by **ASSERT** (responsibleai/ASSERT). This skill wires the two together, and also +measures risks the user names directly when discovery isn't what they need. ## Files @@ -24,11 +25,14 @@ methodologically aligned when changing the flow. ## Architecture -1. **Discovery (Clarity, shipped):** the Clarity **MCP server** exposes tools — +1. **Discovery (Clarity, shipped — recommended, not required):** the Clarity **MCP server** exposes tools — `run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`, and others. `run_clarity` returns Clarity's real process guide inlined; the host agent conducts the clarifying conversation and persists findings. See - `SETUP-CHECKLIST.md` to wire it up. + `SETUP-CHECKLIST.md` to wire it up. When the user would rather name the risk + themselves — or Clarity isn't set up — the skill takes a user-supplied risk + (prose, PRD, design doc, threat model) through a structured intake instead + (`SKILL.md` Step 1b) and everything downstream is identical. 2. **Handoff (files, not JSON):** Clarity writes `.clarity-protocol/`. The measurement side reads `failures/failures.md` (index) and `failure-NN-*.md` (individual docs). Those files are the **source of truth**; the parser's JSON is @@ -102,6 +106,9 @@ silently degrades multi-turn to single-turn), and module resolution falls back - The triage gate and the pre-run confirmation are **human** decisions; declining writes nothing and runs nothing. - `.clarity-protocol/` files are authoritative; derived JSON is a cache. -- Discovery goes through Clarity's real MCP tools — no plain-language fallback, no - shelling out to a `clarity cli` process, no separate app. +- Clarity discovery is **recommended, never a gate** — the user picks the risk + source, and a missing `.clarity-protocol/` never blocks a measurement. +- When the user chooses Clarity, discovery goes through its real MCP tools — never + an imitation of Clarity's interview, no shelling out to a `clarity cli` process, + no separate app. - Never read/print/commit `.env`, credential values, or `artifacts/`. diff --git a/.claude/skills/run-assert-eval/SETUP-CHECKLIST.md b/.claude/skills/run-assert-eval/SETUP-CHECKLIST.md index d8664baa..2e6165ae 100644 --- a/.claude/skills/run-assert-eval/SETUP-CHECKLIST.md +++ b/.claude/skills/run-assert-eval/SETUP-CHECKLIST.md @@ -1,8 +1,14 @@ # Setup checklist — Clarity MCP ⇄ ASSERT (in-IDE only) +**Optional.** Clarity is the *recommended* risk source for `run-assert-eval`, not a +prerequisite for it. The skill runs fine without any of this — the user can supply +risks directly as prose or a PRD / design doc / threat model (SKILL.md Step 1b). +Work through this checklist when you want Clarity's discovery to surface failure +modes you haven't thought of. + These steps require a real IDE with MCP support (VS Code + Copilot agent mode, Claude Code, or Cursor) and cannot be completed from a headless terminal. Do them -once per workspace, then the `run-assert-eval` skill's discovery front door +once per workspace, then the `run-assert-eval` skill's recommended discovery path (`run_clarity`) becomes callable. ## Phase 1 — Environment setup @@ -39,9 +45,14 @@ once per workspace, then the `run-assert-eval` skill's discovery front door ## Phase 2 — End-to-end verification (definition of done) -- [ ] **Fresh discovery**: with no `.clarity-protocol/failures/failures.md`, call +- [ ] **Fresh discovery** (Path A): with no `.clarity-protocol/failures/failures.md`, call `run_clarity`, conduct a short clarifying conversation, and confirm `failures.md` gets written. +- [ ] **User-supplied risks work without Clarity** (Path B): with the Clarity MCP + tools unavailable *or* declined, describe one risk in plain language and + confirm the skill offers the choice, does **not** stop on the missing + protocol, and still produces a single atomic config with an explicit + permissible boundary and a variants-derived dimension. - [ ] **Parser**: `python .claude/skills/run-assert-eval/clarity_intake.py .clarity-protocol` emits candidate behaviors; run the unit tests: ``` diff --git a/.claude/skills/run-assert-eval/SKILL.md b/.claude/skills/run-assert-eval/SKILL.md index 1eb7ed1b..ef226b26 100644 --- a/.claude/skills/run-assert-eval/SKILL.md +++ b/.claude/skills/run-assert-eval/SKILL.md @@ -1,13 +1,15 @@ --- name: run-assert-eval description: > - Run an ASSERT evaluation starting from Clarity-discovered risks. Use when the - user wants to evaluate, test, or check an AI agent, LLM app, or model against + Run an ASSERT evaluation against a described risk. Use when the user wants to + evaluate, test, or check an AI agent, LLM app, or model against requirements/policies (e.g. "evaluate my agent for budget violations", "test - that the support bot never gives legal advice"). Drives the real Clarity MCP - tools (run_clarity) in-IDE to discover risks, then generates one flat - evals/.yaml per selected risk, runs the pipeline, and reports - pass/violation rates with trace-cited failure examples. + that the support bot never gives legal advice"). Risks come either from + Clarity — recommended, driving the real Clarity MCP tools (run_clarity) in-IDE + to discover failure modes the user has not considered — or directly from the + user as a description, PRD, design doc, threat model, or test plan. Then + generates one flat evals/.yaml per selected risk, runs the + pipeline, and reports pass/violation rates with trace-cited failure examples. --- # Run an ASSERT evaluation @@ -19,26 +21,69 @@ fixing the agent — this skill finds and reports failures. This skill has two entry modes: -- **Run mode** — no usable results exist yet. Risks come from **Clarity** (Steps 1-2): - either an existing `.clarity-protocol/` directory or a fresh discovery run driven - through the **Clarity MCP server** (`run_clarity`), in-IDE. Then turn each selected - risk into an atomic config, run the pipeline (Steps 3-5), and report (Step 6). +- **Run mode** — no usable results exist yet. Establish a **risk source** (Steps 1-2): + **Clarity** (recommended) — an existing `.clarity-protocol/` directory or a fresh + discovery run driven through the **Clarity MCP server** (`run_clarity`), in-IDE — + **or risks the user supplies directly**. Then turn each selected risk into an + atomic config, run the pipeline (Steps 3-5), and report (Step 6). - **Results Q&A mode** — judged artifacts already exist under `artifacts/results///` and the user asks a *question* about them ("what are the highlights?", "top 3 examples of the worst failure mode?", "why did case X fail?"). Skip to Step 6 and answer THAT question from the artifacts — do not re-run, and do not fall back to the full canned report unless asked. -### Clarity is required for Run mode — no non-Clarity fallback - -Risks that seed an eval MUST come from Clarity (an existing `.clarity-protocol/` -or a fresh discovery run via the Clarity MCP `run_clarity` tool). Do **not** -substitute a plain-language description, and do **not** imitate Clarity's -questioning from your own head — instead, `run_clarity` returns Clarity's real -process guide inlined, and you follow *that* to conduct the clarifying loop. An -eval spec that skips Clarity's captured risks produces inaccurate, low-signal -results. If the Clarity MCP tools are not available, STOP and help the user set -them up (see `SETUP-CHECKLIST.md`) rather than proceeding. +### Choosing a risk source (Clarity recommended, never required) + +Every eval starts from a risk. There are two supported sources, and **the user +chooses** — never decide for them and never block on Clarity. + +**Path A — Clarity discovery (recommended — present it first, but never alone).** Use an existing +`.clarity-protocol/` or a fresh run via the Clarity MCP `run_clarity` tool. +Clarity's value is finding failure modes the user has *not* thought of, along +with severity and causal chains. Recommend it whenever the user is unsure what +to measure, is new to the agent, or wants coverage rather than one known bug. + +**Path B — user-supplied risks.** The user names the risk themselves, as prose +or by pointing at a PRD, design doc, threat model, incident report, or test +plan. This is the right path when they already know what they want measured. + +**Whenever you need a new risk to measure**, and the user has not already named +one, **offer the choice**: + +> I can discover risks with Clarity — it interviews you and surfaces failure +> modes you may not have considered (recommended if you're not sure what to +> measure) — or you can tell me the risk directly, in your own words or by +> pointing me at a PRD or design doc. Which do you prefer? + +An existing `.clarity-protocol/` changes the **default**, never the **choice**. +Offer it as the recommended option ("I found an existing Clarity protocol with +these risks — measure one of those, or is there a different risk you have in +mind?"), then take the user's answer. + +Rules that hold on both paths: + +- **An explicit user-supplied risk always wins.** If the user names a risk — in + prose, or by pointing at a document — measure *that*, whether or not a + `.clarity-protocol/` exists. Never substitute the protocol's risks for one the + user just stated. If you think the protocol covers the same ground, say so and + let them decide; do not decide for them. +- **Never silently pick a path**, and never stall the user on Clarity setup. If + the Clarity MCP tools are missing and the user wants Path A, offer + `SETUP-CHECKLIST.md` — but if they'd rather not set it up now, take Path B. +- **Do not imitate Clarity's interview from your own head.** This is the real + prohibition: if the user picked Path A, drive the actual `run_clarity` tool, + which returns Clarity's genuine process guide inlined. Path B is not a + degraded impression of Clarity — it is a distinct, structured intake (Step 1b). +- **Path B meets the same quality bar.** One atomic behavior per config, + variant-derived stratify dimensions, pinned systematize/judge models, an + explicit `sample_size`. Steps 3-6 are risk-source agnostic; nothing about the + config, run, or report changes. +- **Offer Clarity again later.** Declining once is not a permanent opt-out — + after a run, it's a natural next step for finding what they *didn't* think to + measure. +- **Clarity write-backs degrade to no-ops.** `record_failure` / + `record_suggestion` apply only when a protocol exists. On Path B, skip them and + say so once; never treat their absence as an error. ### Copilot vs. the local viewer @@ -64,18 +109,20 @@ runs*, or *watch a live run*. clone of the ASSERT repo itself; inside a customer repo it installs the wrong package. -2. **Clarity MCP server available** (required for Run mode): the `clarity-agent` +2. **Clarity MCP server available** (needed only for Path A): the `clarity-agent` MCP tools (`run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`, …) are callable in this session. Clarity is the risk-discovery engine — the skill drives its real MCP tools, it does not - reimplement it. If the tools are missing, the server is not wired up yet: guide - the user through `SETUP-CHECKLIST.md` (install `clarity-agent` with the `[mcp]` - extra, run `clarity embed .` to generate `.vscode/mcp.json`, reload MCP servers) - and confirm the LLM provider is configured (`clarity doctor` — Clarity supports + reimplement it. If the tools are missing, the server is not wired up yet: offer + `SETUP-CHECKLIST.md` (install `clarity-agent` with the `[mcp]` extra, run + `clarity embed .` to generate `.vscode/mcp.json`, reload MCP servers) and + confirm the LLM provider is configured (`clarity doctor` — Clarity supports GitHub Copilot, Anthropic, OpenAI, Azure AI, and Gemini). - If the Clarity MCP tools cannot be made available, STOP and help the user - resolve it. Do not proceed with a non-Clarity path. + **This is not a blocker.** If the tools can't be made available, or the user + would rather not set them up now, say so plainly and continue on Path B + (Step 1b). Never strand the user on MCP setup when they came to measure + something. 3. **Provider creds exist** in `.env`. NEVER read or print `.env`. If a run fails with an auth error, tell the user which variable NAMES are required @@ -84,13 +131,31 @@ runs*, or *watch a live run*. ## Steps -### 1. Discover risks with Clarity (required front door) +### 1. Establish the risk source + +Ask which path the user wants (see "Choosing a risk source" above), then follow +**1a** or **1b**. + +- **The user already named a risk** (prose, PRD, design doc, threat model, + incident report, test plan) → that is an explicit Path B choice. Go to **1b**, + even if a `.clarity-protocol/` exists. Do not silently switch to the + protocol's risks. +- **Intent is ambiguous and a `.clarity-protocol/` exists** → offer it as the + default and say what's in it, but still ask before selecting it: *"I found an + existing Clarity protocol covering X and Y — want to measure one of those, or + is there a different risk you have in mind?"* +- **Intent is ambiguous and no protocol exists** → offer the choice as written + above. + +#### 1a. Clarity discovery (recommended) Risks come from Clarity's real engine, driven through the **Clarity MCP server** — -never from a plain-language guess and never by imitating Clarity from your own head. +never by imitating Clarity's interview from your own head. -- **If a `.clarity-protocol/` directory already exists** in the workspace, use it - directly as the risk source — skip straight to reading its output below. +- **If a `.clarity-protocol/` directory already exists** in the workspace, and the + user has chosen Path A for this risk, use it directly as the risk source — skip + straight to reading its output below. (Selecting Path A is the user's decision, + made in Step 1; the protocol's presence alone does not make it.) - **Otherwise run discovery via the Clarity MCP tools:** 1. Call **`run_clarity`**. It returns Clarity's real process guide inlined as text. 2. Follow that guide to ask the user the clarifying questions **in chat** — this @@ -125,13 +190,57 @@ Clarity records severity/management-plan signal (the parser maps Critical→P1, High→P2, Medium→P3, ranges→max). Order and annotate by what Clarity actually captured; do not fabricate priorities. +#### 1b. User-supplied risks + +The user already knows what to measure. Your job is to turn their input into the +**same candidate-behavior shape** `clarity_intake.py` produces on Path A — +`{name, description, severity, priority, source_doc, candidate_dimensions, +multi_behavior, suggested_splits}` — so Steps 2-6 are identical either way. + +1. **Take the input as given.** If they point at a PRD, design doc, threat model, + incident report, or test plan, read it and extract candidate risks from it. + Quote what you extracted so they can correct you. If they describe it in prose, + start from their words — do not paraphrase the risk into something broader. +2. **Ask only what the config actually needs**, in one batch, and skip anything + already answered by their input or the repo: + - *What does the agent do, and who uses it?* → `context` + - *What must it never do?* → `behavior.name` + `behavior.description` + - *What is it explicitly allowed to do that looks similar?* → the permissible + boundary. **Do not skip this**: without it the judge cannot separate real harm + from over-refusal, and both rates become uninterpretable. + - *How would a user trigger this — innocently, and deliberately?* → + the `elicitation_variant` stratify dimension, the highest-value one + - *How bad is it when it happens?* → severity → `priority` +3. **Enforce atomicity now.** If their description bundles several independently + testable behaviors ("it leaks data and hallucinates prices"), say so and propose + the split — one config each. This is the `multi_behavior` / `suggested_splits` + check, applied by hand. +4. **Play it back for confirmation** as an explicit candidate list before + generating anything, exactly as triage does on Path A. + +Set `source_doc` to the file you read, or `user-described` when it came from chat. +Record severity as the user rated it; do not invent a priority they didn't give. + +**For the full measurement path** — triage → one atomic config per selected risk → +sequential runs → report → curate the example — follow +`workflows/measure-clarity-failures.md`, the same workflow Path A uses. **Skip its +Step 1 (Parse)**: there is no `failures.md` to parse, so join at Step 2 with the +candidate list you just built. Skip its Step 8 (close the loop in Clarity) too, +unless a `.clarity-protocol/` exists. + +Then continue to Step 2. Everything downstream is unchanged. + ### 2. Triage — choose which risks to measure now Clarity intentionally over-produces (whole-lifecycle threat modeling). Do NOT auto-generate an eval for every failure mode. Surface the enumerated list (ordered -by Clarity's severity signal) and ask the user which to measure now (e.g. +by severity signal) and ask the user which to measure now (e.g. "top-severity only?", or named picks). Carry only the selected risks forward. +On Path B the list is usually short and already chosen — still play it back and +confirm scope before generating configs, rather than assuming every risk they +mentioned should be measured in this pass. + ### 3. Turn each selected risk into an atomic config ASSERT performs best with **one atomic behavior per eval**. Never bundle multiple @@ -142,7 +251,7 @@ hides per-behavior signal. - **N selected risks** → generate N flat `evals/.yaml` files and run them sequentially, one per behavior. -For each selected risk, map the Clarity failure mode → `behavior.name` + +For each selected risk, map the failure mode → `behavior.name` + `behavior.description`, and use its context for `context`: ``` @@ -150,8 +259,9 @@ assert-ai init --default-model --describe-file --non-inte ``` - **Write the description to a file and pass `--describe-file`.** The text is - Clarity-derived prose you did not author, so it can contain quotes, backticks, - or `$(...)`. Interpolating it into `--describe ""` would break the + prose you did not author — Clarity output on Path A, the user's own words or a + PRD excerpt on Path B — so it can contain quotes, backticks, or `$(...)`. + Interpolating it into `--describe ""` would break the command or inject into the user's shell. `--describe` stays available for short text you typed yourself; the two are mutually exclusive. - `--default-model` seeds the generated config's `pipeline.default_model` — the @@ -399,9 +509,22 @@ when they disagree with this skill on *product behavior*, they win; this skill o ## Guardrails -- **Clarity is the required risk source** — for Run mode, risks come from Clarity (existing `.clarity-protocol/` or a fresh discovery run via the `run_clarity` MCP tool). Never substitute a plain-language guess or imitate Clarity's questioning from your own head; if the MCP tools can't be made available, stop and help fix it (`SETUP-CHECKLIST.md`). +- **Clarity is the recommended risk source, not a gate** — present **both** + options together whenever the user needs a new risk: Clarity discovery + (existing `.clarity-protocol/` or a fresh `run_clarity` run) *and* risks they + supply themselves. Recommend Clarity, because it surfaces failure modes they + haven't considered — but never present it as the only route. Any menu, list, or + question you offer that includes a Clarity option must carry the user-supplied + option beside it; a user who doesn't know Path B exists cannot ask for it. Hold + the user-supplied path (Step 1b) to the same bar: atomic behaviors, an explicit + permissible boundary, variant-derived dimensions. Never block a measurement on + Clarity setup. +- **Never imitate Clarity's interview from your own head** — if the user chose + Clarity, drive the real MCP tools (`run_clarity` returns its genuine process + guide inlined). Step 1b is a distinct structured intake, not a hand-rolled + impression of Clarity. - **Drive the real Clarity MCP tools in-IDE** — use `run_clarity` / `write_protocol_document` / `record_failure` for discovery and `record_suggestion` to close the loop; never hand the user off to a separate Clarity app and never shell out to a `clarity cli` process. -- **Close the loop** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives, so Clarity's staleness tracking stays aware of it. +- **Close the loop when a protocol exists** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives, so Clarity's staleness tracking stays aware of it. With no protocol, skip it silently — and consider offering Clarity as a next step for finding risks this pass didn't cover. - **Govern with ACS, don't just prompt-tweak** — to fix and *prove* it, generate an ACS policy from the findings (`assert-ai acs generate`), review it (scope the gated tools, tighten conditions), and re-run the same eval against the governed callable to show the delta; needs a wrappable callable target (`workflows/govern-and-remeasure.md`). Generated policies, guarded targets, and governed configs are local run output by default. Commit them only in the user's own product repo when the user wants a reviewed policy deployed; do not automatically add them to ASSERT's worked examples. Whenever a gate needs a value the model doesn't put in the tool args — a trusted session flag (verification), a trusted comparison value (the caller's own id), a trusted numeric cap, or a running total / prior-call fact — the governed agent must surface that scalar from its **session state** into the tool-call **policy_target** so the generated `input.policy_target.value.*` rule actually fires. ACS evaluates each call in isolation, so multi-call constraints (running totals, ordering, rate limits) are handled by that same injection, not by encoding history in Rego. Free-form content failures (unsafe advice, PII in prose, a verbal-only high-risk promise) and inbound prompt-injection instead use an **annotator-based** gate at the `output`/`input` point, proven by the remeasure delta since offline `validate` can't run annotators. Never hand-drive an external `acs` CLI for this loop. - **Organize by domain across runs** — prefix every eval **suite name** with a domain slug (`-`, e.g. `billing-cross-customer-data-exposure`, `science-`), so `artifacts/results//` and `artifacts/acs//` do not collide. Treat `.clarity-protocol/` as uncommitted single-domain scratch; preserve it outside `examples/` only when the user asks. - **Per-example package** — every worked example must be a small, self-contained folder under `examples//` containing only what a customer needs to understand and reproduce the ASSERT run: @@ -414,7 +537,7 @@ when they disagree with this skill on *product behavior*, they win; this skill o output. Do not commit generated taxonomies, test sets, result artifacts, discovery mailboxes, snapshots, protocol archives, or automatic skill output. - **One atomic behavior per config** — split N selected risks into N configs run sequentially; never bundle. -- **Triage before running** — never auto-generate an eval for every Clarity failure mode; ask which to measure now. +- **Triage before running** — never auto-generate an eval for every enumerated risk; ask which to measure now. - **Don't invent metrics** — only report what's in the artifacts. - **Don't trawl raw traces to answer questions** — answer from `results status`, `scores.jsonl`, and `metrics.json`; hand off to the viewer for visual trace/transcript exploration. - **Hand off, don't reimplement the viewer** — for visual drill-down, baseline compare, or live monitoring, point to the local viewer rather than reproducing it in chat. diff --git a/.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md b/.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md index 54a5b1d0..45330510 100644 --- a/.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md +++ b/.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md @@ -5,11 +5,12 @@ Specification) policy, then re-run the same eval against the governed agent to **prove the failure rate dropped** — the ACS delta. This is the governance half of the story and picks up where -`measure-clarity-failures.md` leaves off: Clarity discovered the risk, -ASSERT measured a baseline violation rate, and now ACS governs the failure at -runtime. It uses ASSERT's **native** ASSERT to ACS adapter (`assert-ai acs …`), -which derives the policy straight from the run's findings — no external `acs` -CLI and no separate checkout of the agent-governance-toolkit are needed. +`measure-clarity-failures.md` leaves off: a risk was established (via Clarity or +supplied by the user), ASSERT measured a baseline violation rate, and now ACS +governs the failure at runtime. It uses ASSERT's **native** ASSERT to ACS adapter +(`assert-ai acs …`), which derives the policy straight from the run's findings — +no external `acs` CLI and no separate checkout of the agent-governance-toolkit are +needed. > **Everything stays in-IDE.** ACS has no MCP server; the `assert-ai acs` > subcommands are the in-IDE surface, driven the same way ASSERT already drives @@ -659,6 +660,10 @@ exported HTML — it is per-run output.) ## Step 7 — Close the loop in Clarity +**Only when a `.clarity-protocol/` exists.** If the risk came from the user rather +than Clarity, skip this step — mention once that you're skipping it, and go +straight to the regression-check offer below. + Offer to write the outcome back into `.clarity-protocol/` via the Clarity MCP tool `record_suggestion` (or `record_decision`): the failure mode was measured against the reviewed ACS policy under `artifacts/acs//`, and baseline `X%` diff --git a/.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md b/.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md index 7f24aa10..489f47a8 100644 --- a/.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md +++ b/.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md @@ -1,7 +1,7 @@ # Workflow: measure-clarity-failures -Turn Clarity-discovered failure modes into measured ASSERT violation rates — -one atomic behavior at a time, with a human in the loop at every gate. +Turn discovered or user-supplied failure modes into measured ASSERT violation +rates — one atomic behavior at a time, with a human in the loop at every gate. This workflow is the measurement half of the Clarity → ASSERT story. Discovery is owned by the **Clarity MCP server** (`clarity-agent`, shipped by @@ -9,6 +9,11 @@ microsoft/clarity-agent); measurement is owned by this skill. The handoff is **files, not JSON**: Clarity writes `.clarity-protocol/failures/`, and this workflow reads it. +Clarity is the **recommended** risk source, not a required one. When the user +supplies risks directly (SKILL.md Step 1b), skip Step 1 (Parse) and start at +Step 2 (Triage) — Steps 2-9 never touch `.clarity-protocol/` except at the +optional close-the-loop step. + > **Discovery is agent-driven, not scripted.** The Clarity MCP `run_clarity` > tool returns the relevant process guide inlined as text; **you** (the host > agent) ask the user the clarifying questions in chat and persist what you learn @@ -20,19 +25,35 @@ workflow reads it. Trigger this workflow when the user asks to **measure / test / quantify** risks or failures for their agent, model, or app. -1. **If `.clarity-protocol/failures/failures.md` exists** → go to **Step 1 (Parse)**. -2. **If it does not exist** → run discovery first: - - **Run the preservation gate below first** — a fresh discovery run destroys - any protocol from a previous domain. - - Call the Clarity MCP tool **`run_clarity`**. Follow the inlined process - guide's clarifying questions *with the user in chat*. - - Persist findings via **`write_protocol_document`** and **`record_failure`**. - - Continue until the failure-analysis process has produced - `failures/failures.md`, then proceed to Step 1. - - If the `clarity-agent` MCP tools are **not available** in this session, stop - and point the user at the in-IDE setup checklist (`SETUP-CHECKLIST.md`): - `clarity embed`, reload MCP servers, confirm `run_clarity` is callable. Do - **not** substitute a plain-language risk guess — that produces low-signal evals. +1. **If the user already named a risk** (prose, PRD, design doc, threat model, + incident report, test plan) → that is an explicit user-supplied choice. + Follow **SKILL.md Step 1b**, then **skip Step 1 (Parse)** and join at **Step 2 + (Triage)**. Do this even when `.clarity-protocol/failures/failures.md` exists — + never substitute the protocol's risks for the one the user just stated. +2. **Otherwise, if `.clarity-protocol/failures/failures.md` exists** → offer it as + the default, naming what it covers, and ask before selecting it: *"I found an + existing Clarity protocol covering X and Y — want to measure one of those, or is + there a different risk you have in mind?"* On confirmation, go to + **Step 1 (Parse)**. If they name something else instead, treat it as case 1. +3. **If no protocol exists** → ask which risk source the user wants (see SKILL.md, + "Choosing a risk source"). Recommend Clarity, but take their answer: + - **Clarity discovery (recommended)** — + **run the preservation gate below first**; a fresh discovery run destroys + any protocol from a previous domain. Call the Clarity MCP tool + **`run_clarity`** and follow the inlined process guide's clarifying questions + *with the user in chat*. Persist findings via **`write_protocol_document`** + and **`record_failure`**. Continue until the failure-analysis process has + produced `failures/failures.md`, then proceed to Step 1. + - **User-supplied risks** — the user names the risk, or points at a PRD, + design doc, threat model, incident report, or test plan. Follow **SKILL.md + Step 1b** to build the candidate-behavior list by hand, then **skip Step 1 + (Parse)** — there is no `failures.md` to parse — and join at **Step 2 + (Triage)**. Everything from Step 2 onward is risk-source agnostic. + - If the `clarity-agent` MCP tools are **not available** in this session, say + so, offer the in-IDE setup checklist (`SETUP-CHECKLIST.md`: `clarity embed`, + reload MCP servers, confirm `run_clarity` is callable), and let the user + choose. If they'd rather not set it up now, continue with user-supplied + risks — do not strand them on MCP setup. ### Preservation gate (blocking — check before any fresh `run_clarity`) @@ -56,6 +77,10 @@ re-scaffolds a clean one on the next `run_clarity`. ## Step 1 — Parse +**Clarity-sourced risks only.** If the user supplied risks directly (SKILL.md +Step 1b), there is no `failures.md` to parse — skip to Step 2 with the candidate +list you built there. + Run the intake parser (`clarity_intake.py`) on the protocol directory: ``` @@ -278,6 +303,11 @@ the failing checkpoint, or **re-measure after a fix** to prove the rate dropped. ## Step 8 — Close the loop in Clarity +**Only when a `.clarity-protocol/` exists.** With user-supplied risks there is +nothing to write back to — skip this step, mention once that you're skipping it, +and instead offer Clarity discovery as a way to find failure modes this pass +didn't cover. + After a run, offer to write the outcome back into `.clarity-protocol/` via the Clarity MCP tool **`record_suggestion`** (or **`record_decision`**): note that the failure mode now has a **measured baseline** and where the eval lives diff --git a/.cursor/rules/assert.mdc b/.cursor/rules/assert.mdc index 9b140470..8b12b4aa 100644 --- a/.cursor/rules/assert.mdc +++ b/.cursor/rules/assert.mdc @@ -1,5 +1,5 @@ --- -description: ASSERT repo orientation pointer plus the Clarity-driven run-assert-eval evaluation workflow. +description: ASSERT repo orientation pointer plus the run-assert-eval evaluation workflow (Clarity-recommended risk discovery, or user-supplied risks). globs: alwaysApply: true --- @@ -17,28 +17,44 @@ azure_ad_token_provider, GITHUB_TOKEN, ANTHROPIC_API_KEY) — never their values ## Run an ASSERT evaluation When the user wants evidence of how their agent or model actually behaves, run an end-to-end -evaluation whose risks are discovered with Clarity. Drive the existing Clarity **MCP tools** and -`assert-ai` CLI — do not reimplement Clarity's questioning or any pipeline logic. This finds and reports -failures; it is not for fixing the agent. +evaluation. Risks come from Clarity (recommended) or directly from the user. Drive the existing +Clarity **MCP tools** and `assert-ai` CLI — do not reimplement Clarity's questioning or any pipeline +logic. This finds and reports failures; it is not for fixing the agent. Two entry modes: -- **Run mode** — no usable results exist yet. Risks come from **Clarity** (Steps 1-2): an existing - `.clarity-protocol/` directory or a fresh discovery run via the Clarity MCP `run_clarity` tool, driven - in-IDE. Then turn each selected risk into an atomic config, run the pipeline (Steps 3-5), then report - (Step 6). +- **Run mode** — no usable results exist yet. Establish a **risk source** (Steps 1-2): **Clarity** + (recommended) — an existing `.clarity-protocol/` directory or a fresh discovery run via the Clarity + MCP `run_clarity` tool, driven in-IDE — **or risks the user supplies directly**. Then turn each + selected risk into an atomic config, run the pipeline (Steps 3-5), then report (Step 6). - **Results Q&A mode** — judged artifacts already exist under `artifacts/results///` and the user asks a *question* about them ("what are the highlights?", "top 3 examples of the worst failure mode?", "why did case X fail?"). Skip to Step 6 and answer THAT question from the artifacts — do not re-run, and do not emit the full canned report unless asked. -**Clarity is required for Run mode — no non-Clarity fallback.** Risks that seed an eval MUST come from -Clarity (an existing `.clarity-protocol/` or a fresh discovery run via the Clarity MCP `run_clarity` -tool). Do not substitute a plain-language description, and do not imitate Clarity's questioning from -your own head — `run_clarity` returns Clarity's real process guide inlined, and you follow *that* to -conduct the clarifying loop. Skipping Clarity's captured risks produces inaccurate, low-signal results. -If the Clarity MCP tools are not available, STOP and help the user set them up (`SETUP-CHECKLIST.md`) -rather than proceeding. +**Choosing a risk source (Clarity recommended, never required).** Every eval starts from a risk, and +**the user chooses** the source — never decide for them and never block on Clarity. **Path A — +Clarity discovery (recommended — present it first, but never alone):** an existing `.clarity-protocol/` or a fresh +`run_clarity` run; its value is finding failure modes the user has *not* thought of, plus severity and +causal chains. Recommend it whenever they're unsure what to measure or want coverage rather than one +known bug. **Path B — user-supplied risks:** the user names the risk themselves, as prose or by +pointing at a PRD, design doc, threat model, incident report, or test plan. Right when they already +know what they want measured. **Whenever you need a new risk to measure**, and the user has not +already named one, offer the choice explicitly. An existing `.clarity-protocol/` changes the +**default**, never the **choice** — offer it as the recommended option ("I found an existing Clarity +protocol with these risks — measure one of those, or is there a different risk you have in mind?"), +then take the user's answer. **An explicit user-supplied risk always wins:** if the user names a +risk, in prose or by pointing at a document, measure *that*, whether or not a `.clarity-protocol/` +exists; never substitute the protocol's risks for one the user just stated, and if you think the +protocol covers the same ground, say so and let them decide. Never silently pick a path and never +stall the user on Clarity setup — if the MCP tools are missing and they'd +rather not set them up now, take Path B. Do not imitate Clarity's questioning from your own head: +on Path A drive the real `run_clarity` tool (it returns Clarity's genuine process guide inlined); +Path B is a distinct structured intake (Step 1b), not a hand-rolled impression of Clarity. Path B meets +the same quality bar (atomic behaviors, explicit permissible boundary, variant-derived dimensions, +pinned models, explicit `sample_size`) — Steps 3-6 are risk-source agnostic. Offer Clarity again later; +declining once is not a permanent opt-out. Clarity write-backs degrade to no-ops when no protocol +exists — skip them and say so once, never treat their absence as an error. **Cursor vs. the local viewer**: Cursor is for *answering questions* and *synthesis* (direct answers, failure-mode clustering, cited examples, next actions — no clicking). The bundled local @@ -50,24 +66,33 @@ stepping with citations highlighted). Answer in chat for "what / why / which"; h 1. **ASSERT installed**: verify `assert-ai --help` succeeds. If not, guide install: `python -m pip install -e ".[otel,langgraph]"`. -2. **Clarity MCP server available** (required for Run mode): the `clarity-agent` MCP tools +2. **Clarity MCP server available** (needed only for Path A): the `clarity-agent` MCP tools (`run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`, …) are callable. Clarity is the risk-discovery engine — the skill drives its real MCP tools, it does not reimplement - it. If the tools are missing, the server is not wired up yet: guide the user through - `SETUP-CHECKLIST.md` (install `clarity-agent` with the `[mcp]` extra, run `clarity embed .` to - generate `.vscode/mcp.json`, reload MCP servers) and confirm the LLM provider is configured - (`clarity doctor`). If the MCP tools cannot be made available, STOP and help resolve it — do not - proceed with a non-Clarity path. + it. If the tools are missing, the server is not wired up yet: offer `SETUP-CHECKLIST.md` (install + `clarity-agent` with the `[mcp]` extra, run `clarity embed .` to generate `.vscode/mcp.json`, reload + MCP servers) and confirm the LLM provider is configured (`clarity doctor`). **This is not a + blocker** — if the tools can't be made available, or the user would rather not set them up now, say + so plainly and continue on Path B (Step 1b). 3. **Provider creds exist** in `.env`. NEVER read or print `.env`. On an auth error, tell the user which variable NAMES are required (AZURE_API_KEY, AZURE_API_BASE, OPENAI_API_KEY, GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.) — never values. -### 1. Discover risks with Clarity (required front door) +### 1. Establish the risk source -Risks come from Clarity's real engine, driven through the **Clarity MCP server** — never from a -plain-language guess and never by imitating Clarity from your own head. +Ask which path the user wants (see "Choosing a risk source" above), then follow **1a** or **1b**. If +the user already named a risk (prose, PRD, design doc, threat model, incident report, test plan), that +is an explicit Path B choice — go to **1b** even if a `.clarity-protocol/` exists, and do not silently +switch to the protocol's risks. If intent is ambiguous and a protocol exists, offer it as the default +and say what's in it, but still ask before selecting it: *"I found an existing Clarity protocol +covering X and Y — want to measure one of those, or is there a different risk you have in mind?"* If +intent is ambiguous and no protocol exists, offer the choice as written above. -- **Existing `.clarity-protocol/`** — use it directly as the risk source. +**1a. Clarity discovery (recommended).** Risks come from Clarity's real engine, driven through the +**Clarity MCP server** — never by imitating Clarity's interview from your own head. + +- **Existing `.clarity-protocol/`** — once the user has chosen Path A for this risk, use it directly + as the risk source. Its presence alone is not that choice; Step 1 is. - **Otherwise run discovery via the Clarity MCP tools:** call `run_clarity` (it returns Clarity's real process guide inlined as text), follow that guide to ask the user the clarifying questions in chat, and persist findings with `write_protocol_document` and `record_failure` until @@ -85,11 +110,35 @@ single-domain scratch and `run_clarity` **overwrites** it, destroying the prior `goal/`, and `solution/` with no git recovery. If another domain's protocol is present, STOP and let the user export it to a user-owned location or explicitly discard it. Never commit it into `examples/`. +**1b. User-supplied risks.** Turn the user's input into the same candidate-behavior shape +`clarity_intake.py` produces on Path A — `{name, description, severity, priority, source_doc, +candidate_dimensions, multi_behavior, suggested_splits}` — so Steps 2-6 are identical either way. +Take the input as given: if they point at a PRD, design doc, threat model, incident report, or test +plan, read it and extract candidate risks, quoting what you extracted so they can correct you; if they +describe it in prose, start from their words rather than paraphrasing the risk into something broader. +Then ask only what the config actually needs, in one batch, skipping anything their input or the repo +already answers: *what the agent does and who uses it* (→ `context`); *what it must never do* (→ +`behavior.name` + `behavior.description`); *what it is explicitly allowed to do that looks similar* (→ +the permissible boundary — **do not skip this**, without it the judge cannot separate real harm from +over-refusal and both rates become uninterpretable); *how a user would trigger it, innocently and +deliberately* (→ the `elicitation_variant` stratify dimension, the highest-value one); *how bad it is +when it happens* (→ severity → `priority`). Enforce atomicity now — if the description bundles several +independently testable behaviors ("it leaks data and hallucinates prices"), say so and propose the +split, one config each. Play the candidate list back for confirmation before generating anything, just +as triage does on Path A. Set `source_doc` to the file you read, or `user-described` when it came from +chat; record severity as the user rated it and do not invent a priority they didn't give. For the +full measurement path (triage → one atomic config per selected risk → sequential runs → report → +curate the example) follow `../../.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md`, +the same workflow Path A uses: **skip its Step 1 (Parse)** since there is no `failures.md` — join at +Step 2 with the candidate list you just built — and skip its Step 8 (close the loop in Clarity) unless +a `.clarity-protocol/` exists. Everything else downstream is unchanged. + ### 2. Triage — choose which risks to measure now Clarity intentionally over-produces. Do NOT auto-generate an eval for every failure mode. Surface the -list (ordered by Clarity's severity signal) and ask the user which to measure now (e.g. "top-severity -only?", or named picks). Carry only the selected risks forward. +list (ordered by severity signal) and ask the user which to measure now (e.g. "top-severity +only?", or named picks). Carry only the selected risks forward. On Path B the list is usually short and +already chosen — still play it back and confirm scope before generating configs. ### 3. Turn each selected risk into an atomic config @@ -99,8 +148,8 @@ ASSERT performs best with **one atomic behavior per eval**. Never bundle multipl - **1 selected risk** → one config, run once. - **N selected risks** → N flat `evals/.yaml` files, run sequentially, one per behavior. -Map each Clarity failure mode → `behavior.name` + `behavior.description`, use its context for `context`: -`assert-ai init --default-model --describe-file --non-interactive -o evals/.yaml`. Write the failure-mode text (failure mode + how it arises + target context) to a file and pass `--describe-file` rather than interpolating Clarity-derived prose into `--describe ""`, where a quote, backtick, or `$(...)` would break the command or inject into the shell. `--default-model` seeds the generated config's `pipeline.default_model` (the model the **eval** runs against); `--model` is only the init assistant's own conversation model (default `azure/gpt-5.4-mini`) and does not affect the eval. **Then pin the two ground-truth stages to the strong model by hand** — `init` has no `--systematize-model` / `--judge-model` flag, so everything inherits `default_model` unless you edit the config: keep `default_model.name: azure/gpt-5.4-mini` for target/test-set/tester, but set `pipeline.systematize.model: azure/gpt-5.4` and `pipeline.judge.model: azure/gpt-5.4` (the convention in the repo's own `examples/` configs). `systematize` authors the behavior tree and the permissible / non-permissible split every metric is computed against, and `judge` decides applicability *and* violation per row on a single sample (`judge.n` defaults to `1`, temperature unpinned) — a weak model there moves the target rather than adding noise around it, and inflates run-to-run applicability drift. Verify via `assert-ai results status --json` → `prompt_metrics.judge_model`. +Map each selected risk → `behavior.name` + `behavior.description`, use its context for `context`: +`assert-ai init --default-model --describe-file --non-interactive -o evals/.yaml`. Write the failure-mode text (failure mode + how it arises + target context) to a file and pass `--describe-file` rather than interpolating prose you did not author (Clarity-derived, or the user's own words / a doc excerpt) into `--describe ""`, where a quote, backtick, or `$(...)` would break the command or inject into the shell. `--default-model` seeds the generated config's `pipeline.default_model` (the model the **eval** runs against); `--model` is only the init assistant's own conversation model (default `azure/gpt-5.4-mini`) and does not affect the eval. **Then pin the two ground-truth stages to the strong model by hand** — `init` has no `--systematize-model` / `--judge-model` flag, so everything inherits `default_model` unless you edit the config: keep `default_model.name: azure/gpt-5.4-mini` for target/test-set/tester, but set `pipeline.systematize.model: azure/gpt-5.4` and `pipeline.judge.model: azure/gpt-5.4` (the convention in the repo's own `examples/` configs). `systematize` authors the behavior tree and the permissible / non-permissible split every metric is computed against, and `judge` decides applicability *and* violation per row on a single sample (`judge.n` defaults to `1`, temperature unpinned) — a weak model there moves the target rather than adding noise around it, and inflates run-to-run applicability drift. Verify via `assert-ai results status --json` → `prompt_metrics.judge_model`. To extend an existing config, use `--from `. **Check the built-in presets first** — `assert-ai library list` shows bundled behavior and judge presets (`prompt_injection`, `doxxing`, `stereotyping`, `sycophancy`, `harmful_medical_advice`, `tool_orchestration_errors`, …) and `assert-ai library show ` prints one; if one matches the risk, seed with `--behavior ` / `--judge-preset ` rather than generating from scratch. **Ask the user for the `sample_size` — do not pick it silently:** each rate is `violations / sample_size`, so at `10` one flipped case = ±10pp of noise, and inference is non-deterministic (agent temperature 1.0; gpt-5 can't be pinned lower) so two runs of the same config drift by chance. Before generating, ask e.g. *"How many cases per behavior? `10` = fast/noisy, `25` = stable (recommended), `50`+ = tightest — same size for prompt and scenario."* Recommend `25`, and **`≥25` for any run headed to an ACS before/after A/B** (the governed config is a byte-identical copy that inherits this size — see `govern-and-remeasure.md`); default to `25` if the user has no preference. After generation, show the user the generated `behavior.description`, `context`, and `pipeline.judge` settings, plus the resolved `systematize` / `judge` models. Confirm before running. **Do not author judge `dimensions`:** `policy_violation` and `overrefusal` are `BUILT_IN_DIMENSIONS` (`assert_ai/core/judge.py`) and are always judged unless explicitly disabled, so no `dimensions` block is needed. Config dimensions merge over the built-ins **by name**, so declaring one with a built-in name silently replaces that built-in's rubric. Add one only for a genuinely new metric, never reusing a built-in name. @@ -219,12 +268,13 @@ product behavior. ### Guardrails -- **Clarity is the required risk source** — for Run mode, risks come from Clarity (existing `.clarity-protocol/` or a fresh discovery run via the `run_clarity` MCP tool). Never substitute a plain-language guess or imitate Clarity's questioning from your own head; if the MCP tools can't be made available, stop and help fix it (`SETUP-CHECKLIST.md`). +- **Clarity is the recommended risk source, not a gate** — present **both** options together whenever the user needs a new risk: Clarity discovery (existing `.clarity-protocol/` or a fresh `run_clarity` run) *and* risks they supply themselves. Recommend Clarity, because it surfaces failure modes they haven't considered — but never present it as the only route. Any menu, list, or question you offer that includes a Clarity option must carry the user-supplied option beside it; a user who doesn't know Path B exists cannot ask for it. Hold the user-supplied path (Step 1b) to the same bar: atomic behaviors, an explicit permissible boundary, variant-derived dimensions. Never block a measurement on Clarity setup. +- **Never imitate Clarity's interview from your own head** — if the user chose Clarity, drive the real MCP tools (`run_clarity` returns its genuine process guide inlined). Step 1b is a distinct structured intake, not a hand-rolled impression of Clarity. - **Drive the real Clarity MCP tools in-IDE** — use `run_clarity` / `write_protocol_document` / `record_failure` for discovery and `record_suggestion` to close the loop; never hand the user off to a separate Clarity app and never shell out to a `clarity cli` process. -- **Close the loop** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives. +- **Close the loop when a protocol exists** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives. With no protocol, skip it silently — and consider offering Clarity as a next step for finding risks this pass didn't cover. - **Govern with ACS, don't just prompt-tweak** — to fix and *prove* it, generate an ACS policy from the findings (`assert-ai acs generate`), **review and commit** it (scope the gated tools, tighten conditions), and re-run the same eval against the governed callable to show the delta; needs a wrappable callable target (`../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). Whenever a gate needs a value the model doesn't put in the tool args — a trusted session flag (verification), a trusted comparison value (the caller's own id), a trusted numeric cap, or a running total / prior-call fact — the governed agent must surface that scalar from its **session state** into the tool-call **policy_target** so the generated `input.policy_target.value.*` rule actually fires. ACS evaluates each call in isolation, so multi-call constraints (running totals, ordering, rate limits) are handled by that same injection, not by encoding history in Rego. Free-form content failures (unsafe advice, PII in prose, a verbal-only high-risk promise) and inbound prompt-injection instead use an **annotator-based** gate at the `output`/`input` point, proven by the remeasure delta since offline `validate` can't run annotators. Never hand-drive an external `acs` CLI for this loop. - **One atomic behavior per config** — split N selected risks into N configs run sequentially; never bundle. -- **Triage before running** — never auto-generate an eval for every Clarity failure mode; ask which to measure now. +- **Triage before running** — never auto-generate an eval for every enumerated risk; ask which to measure now. - **Don't invent metrics** — only report what's in the artifacts. - **Don't trawl raw traces to answer questions** — answer from `results status`, `scores.jsonl`, and `metrics.json`; hand off to the viewer for visual trace/transcript exploration. - **Hand off, don't reimplement the viewer** — for visual drill-down, baseline compare, or live monitoring, point to the local viewer rather than reproducing it in chat. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 16ce88b8..5a246ff6 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -10,7 +10,7 @@ Never read, print, commit, or infer secrets from `.env` or other local environme Use the matching prompt file when the user's request matches: -- **run-assert-eval** (`.github/prompts/run-assert-eval.prompt.md`): Run an end-to-end ASSERT evaluation whose risks are discovered with Clarity. Drives the Clarity MCP tools (`run_clarity`) in-IDE to surface risks, then follows `workflows/measure-clarity-failures.md` — human triage, splits the selected risks into one atomic config per behavior, runs the pipeline, and summarizes scored results with cited failures. Reports `policy_violation` and `overrefusal` separately. To fix and *prove* a failure, `workflows/govern-and-remeasure.md` generates an ACS policy from the findings (`assert-ai acs generate`) and re-runs the same eval against the governed agent to measure the failure-rate delta. +- **run-assert-eval** (`.github/prompts/run-assert-eval.prompt.md`): Run an end-to-end ASSERT evaluation against a described risk. Risks come from Clarity (recommended — drives the Clarity MCP tools (`run_clarity`) in-IDE to surface failure modes the user hasn't considered) or directly from the user as prose, a PRD, design doc, or threat model; Clarity is never required. Then follows `workflows/measure-clarity-failures.md` — human triage, splits the selected risks into one atomic config per behavior, runs the pipeline, and summarizes scored results with cited failures. Reports `policy_violation` and `overrefusal` separately. To fix and *prove* a failure, `workflows/govern-and-remeasure.md` generates an ACS policy from the findings (`assert-ai acs generate`) and re-runs the same eval against the governed agent to measure the failure-rate delta. Equivalent guidance for other assistants lives in `.claude/skills/run-assert-eval/SKILL.md` (Claude Code) and `.cursor/rules/assert.mdc` (Cursor). Keep all three aligned when you change the methodology. diff --git a/.github/prompts/run-assert-eval.prompt.md b/.github/prompts/run-assert-eval.prompt.md index 516a5aa0..cbedea77 100644 --- a/.github/prompts/run-assert-eval.prompt.md +++ b/.github/prompts/run-assert-eval.prompt.md @@ -1,11 +1,11 @@ --- agent: agent -description: 'Run an ASSERT evaluation starting from Clarity-discovered risks. Drives the real Clarity MCP tools (run_clarity) in-IDE to discover risks, generates one flat evals/.yaml per selected risk, runs the assert-ai pipeline, and reports per-dimension pass/violation rates with trace-cited failure examples.' +description: 'Run an ASSERT evaluation against a described risk. Risks come from Clarity (recommended — drives the real Clarity MCP tools (run_clarity) in-IDE to discover failure modes the user has not considered) or directly from the user as a description, PRD, design doc, or threat model. Generates one flat evals/.yaml per selected risk, runs the assert-ai pipeline, and reports per-dimension pass/violation rates with trace-cited failure examples.' --- # Run an ASSERT evaluation -You help the user run an end-to-end ASSERT evaluation whose risks are discovered with Clarity. You drive the existing Clarity **MCP tools** and `assert-ai` CLI — you do not reimplement Clarity's questioning or any pipeline logic. +You help the user run an end-to-end ASSERT evaluation. Risks come from Clarity (recommended) or directly from the user. You drive the existing Clarity **MCP tools** and `assert-ai` CLI — you do not reimplement Clarity's questioning or any pipeline logic. Read `AGENTS.md` at the repository root for full orientation on the ASSERT project, terminology, and target selection. @@ -15,12 +15,19 @@ The user wants evidence of how their agent or model actually behaves. This skill This skill has two entry modes: -- **Run mode** — no usable results exist yet. Risks come from **Clarity** (Steps 1-2): an existing `.clarity-protocol/` directory or a fresh discovery run via the Clarity MCP `run_clarity` tool, driven in-IDE. Then turn each selected risk into an atomic config, run the pipeline (Steps 3-5), then report (Step 6). +- **Run mode** — no usable results exist yet. Establish a **risk source** (Steps 1-2): **Clarity** (recommended) — an existing `.clarity-protocol/` directory or a fresh discovery run via the Clarity MCP `run_clarity` tool, driven in-IDE — **or risks the user supplies directly**. Then turn each selected risk into an atomic config, run the pipeline (Steps 3-5), then report (Step 6). - **Results Q&A mode** — judged artifacts already exist under `artifacts/results///` and the user asks a *question* about them ("what are the highlights?", "top 3 examples of the worst failure mode?", "why did case X fail?"). Skip to Step 6 and answer THAT question from the artifacts — do not re-run, and do not fall back to the full canned report unless asked. -### Clarity is required for Run mode — no non-Clarity fallback +### Choosing a risk source (Clarity recommended, never required) -Risks that seed an eval MUST come from Clarity (an existing `.clarity-protocol/` or a fresh discovery run via the Clarity MCP `run_clarity` tool). Do **not** substitute a plain-language description, and do **not** imitate Clarity's questioning from your own head — `run_clarity` returns Clarity's real process guide inlined, and you follow *that* to conduct the clarifying loop. An eval spec that skips Clarity's captured risks produces inaccurate, low-signal results. If the Clarity MCP tools are not available, STOP and help the user set them up (see `SETUP-CHECKLIST.md`) rather than proceeding. +Every eval starts from a risk. There are two supported sources, and **the user chooses** — never decide for them and never block on Clarity. + +- **Path A — Clarity discovery (recommended — present it first, but never alone).** An existing `.clarity-protocol/` or a fresh run via `run_clarity`. Clarity's value is finding failure modes the user has *not* thought of, plus severity and causal chains. Recommend it whenever the user is unsure what to measure or wants coverage rather than one known bug. +- **Path B — user-supplied risks.** The user names the risk themselves, as prose or by pointing at a PRD, design doc, threat model, incident report, or test plan. Right when they already know what they want measured. + +**Whenever you need a new risk to measure**, and the user has not already named one, **offer the choice**: "I can discover risks with Clarity — it interviews you and surfaces failure modes you may not have considered (recommended if you're unsure what to measure) — or you can tell me the risk directly, in your own words or by pointing me at a PRD or design doc. Which do you prefer?" An existing `.clarity-protocol/` changes the **default**, never the **choice** — offer it as the recommended option ("I found an existing Clarity protocol with these risks — measure one of those, or is there a different risk you have in mind?"), then take the user's answer. + +Rules on both paths: **an explicit user-supplied risk always wins** — if the user names a risk, in prose or by pointing at a document, measure *that*, whether or not a `.clarity-protocol/` exists; never substitute the protocol's risks for one the user just stated, and if you think the protocol covers the same ground, say so and let them decide. Never silently pick a path, and never stall the user on Clarity setup — if the MCP tools are missing and they'd rather not set them up now, take Path B. **Do not imitate Clarity's interview from your own head**: if they picked Path A, drive the real `run_clarity` tool; Path B is a distinct structured intake (Step 1b), not a hand-rolled impression of Clarity. Path B meets the same quality bar (atomic behaviors, an explicit permissible boundary, variant-derived dimensions, pinned systematize/judge models, explicit `sample_size`) — Steps 3-6 are risk-source agnostic. Offer Clarity again later; declining once is not a permanent opt-out. Clarity write-backs (`record_failure` / `record_suggestion`) degrade to no-ops when no protocol exists — skip them and say so once, never treat their absence as an error. ### Copilot vs. the local viewer @@ -34,17 +41,21 @@ Copilot is for *answering questions* and *synthesis* — direct answers, failure ``` Add route-specific extras as needed, for example `assert-ai[otel,langgraph]` for LangGraph. `target.endpoint` needs `aiohttp`, which ships transitively via `litellm`'s own dependency — no separate extra to install. Use `pip install -e ".[otel,langgraph]"` **only** when the working directory is a clone of the ASSERT repo itself; inside a customer repo it installs the wrong package. -2. **Clarity MCP server available** (required for Run mode): the `clarity-agent` MCP tools (`run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`, …) are callable in this session. Clarity is the risk-discovery engine — the skill drives its real MCP tools, it does not reimplement it. If the tools are missing, the server is not wired up yet: guide the user through `SETUP-CHECKLIST.md` (install `clarity-agent` with the `[mcp]` extra, run `clarity embed .` to generate `.vscode/mcp.json`, reload MCP servers) and confirm the LLM provider is configured (`clarity doctor` — Clarity supports GitHub Copilot, Anthropic, OpenAI, Azure AI, and Gemini). If the Clarity MCP tools cannot be made available, STOP and help the user resolve it. Do not proceed with a non-Clarity path. +2. **Clarity MCP server available** (needed only for Path A): the `clarity-agent` MCP tools (`run_clarity`, `write_protocol_document`, `record_failure`, `record_suggestion`, …) are callable in this session. Clarity is the risk-discovery engine — the skill drives its real MCP tools, it does not reimplement it. If the tools are missing, the server is not wired up yet: offer `SETUP-CHECKLIST.md` (install `clarity-agent` with the `[mcp]` extra, run `clarity embed .` to generate `.vscode/mcp.json`, reload MCP servers) and confirm the LLM provider is configured (`clarity doctor` — Clarity supports GitHub Copilot, Anthropic, OpenAI, Azure AI, and Gemini). **This is not a blocker**: if the tools can't be made available, or the user would rather not set them up now, say so plainly and continue on Path B (Step 1b). Never strand the user on MCP setup when they came to measure something. 3. **Provider creds exist** in `.env`. NEVER read or print `.env`. If a run fails with an auth error, tell the user which variable NAMES are required (AZURE_API_KEY, AZURE_API_BASE, OPENAI_API_KEY, GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.) — never their values. ## Steps -### 1. Discover risks with Clarity (required front door) +### 1. Establish the risk source + +Ask which path the user wants (see "Choosing a risk source" above), then follow **1a** or **1b**. If the user already named a risk (prose, PRD, design doc, threat model, incident report, test plan), that is an explicit Path B choice — go to **1b** even if a `.clarity-protocol/` exists, and do not silently switch to the protocol's risks. If intent is ambiguous and a protocol exists, offer it as the default and say what's in it, but still ask before selecting it: *"I found an existing Clarity protocol covering X and Y — want to measure one of those, or is there a different risk you have in mind?"* If intent is ambiguous and no protocol exists, offer the choice as written above. + +#### 1a. Clarity discovery (recommended) -Risks come from Clarity's real engine, driven through the **Clarity MCP server** — never from a plain-language guess and never by imitating Clarity from your own head. +Risks come from Clarity's real engine, driven through the **Clarity MCP server** — never by imitating Clarity's interview from your own head. -- **If a `.clarity-protocol/` directory already exists** in the workspace, use it directly as the risk source — skip straight to reading its output below. +- **If a `.clarity-protocol/` directory already exists** in the workspace, and the user has chosen Path A for this risk, use it directly as the risk source — skip straight to reading its output below. (Selecting Path A is the user's decision, made in Step 1; the protocol's presence alone does not make it.) - **Otherwise run discovery via the Clarity MCP tools:** call **`run_clarity`** (it returns Clarity's real process guide inlined as text), follow that guide to ask the user the clarifying questions **in chat**, and persist findings with **`write_protocol_document`** and **`record_failure`** until `.clarity-protocol/failures/failures.md` is written. (Copilot agent mode supports MCP *tools*, so drive the loop yourself rather than expecting a separate chat UI.) Read Clarity's output to enumerate risks: @@ -58,9 +69,27 @@ Read Clarity's output to enumerate risks: Clarity records severity/management-plan signal (the parser maps Critical→P1, High→P2, Medium→P3, ranges→max) — order and annotate by what Clarity actually captured; do not fabricate priorities. +#### 1b. User-supplied risks + +Turn the user's input into the **same candidate-behavior shape** `clarity_intake.py` produces on Path A — `{name, description, severity, priority, source_doc, candidate_dimensions, multi_behavior, suggested_splits}` — so Steps 2-6 are identical either way. + +1. **Take the input as given.** If they point at a PRD, design doc, threat model, incident report, or test plan, read it and extract candidate risks from it, quoting what you extracted so they can correct you. If they describe it in prose, start from their words — do not paraphrase the risk into something broader. +2. **Ask only what the config actually needs**, in one batch, skipping anything already answered by their input or the repo: + - *What does the agent do, and who uses it?* → `context` + - *What must it never do?* → `behavior.name` + `behavior.description` + - *What is it explicitly allowed to do that looks similar?* → the permissible boundary. **Do not skip this**: without it the judge cannot separate real harm from over-refusal, and both rates become uninterpretable. + - *How would a user trigger this — innocently, and deliberately?* → the `elicitation_variant` stratify dimension, the highest-value one + - *How bad is it when it happens?* → severity → `priority` +3. **Enforce atomicity now.** If their description bundles several independently testable behaviors ("it leaks data and hallucinates prices"), say so and propose the split — one config each. This is the `multi_behavior` / `suggested_splits` check, applied by hand. +4. **Play it back for confirmation** as an explicit candidate list before generating anything, exactly as triage does on Path A. + +Set `source_doc` to the file you read, or `user-described` when it came from chat. Record severity as the user rated it; do not invent a priority they didn't give. + +**For the full measurement path** — triage → one atomic config per selected risk → sequential runs → report → curate the example — follow `../../.claude/skills/run-assert-eval/workflows/measure-clarity-failures.md`, the same workflow Path A uses. **Skip its Step 1 (Parse)** — there is no `failures.md`, so join at Step 2 with the candidate list you just built — and skip its Step 8 (close the loop in Clarity) unless a `.clarity-protocol/` exists. Everything else downstream is unchanged. + ### 2. Triage — choose which risks to measure now -Clarity intentionally over-produces (whole-lifecycle threat modeling). Do NOT auto-generate an eval for every failure mode. Surface the enumerated list (ordered by Clarity's severity signal) and ask the user which to measure now (e.g. "top-severity only?", or named picks). Carry only the selected risks forward. +Clarity intentionally over-produces (whole-lifecycle threat modeling). Do NOT auto-generate an eval for every failure mode. Surface the enumerated list (ordered by severity signal) and ask the user which to measure now (e.g. "top-severity only?", or named picks). Carry only the selected risks forward. On Path B the list is usually short and already chosen — still play it back and confirm scope before generating configs. ### 3. Turn each selected risk into an atomic config @@ -69,13 +98,13 @@ ASSERT performs best with **one atomic behavior per eval**. Never bundle multipl - **1 selected risk** → generate one config and run once. - **N selected risks** → generate N flat `evals/.yaml` files and run them sequentially, one per behavior. -For each selected risk, map the Clarity failure mode → `behavior.name` + `behavior.description`, and use its context for `context`: +For each selected risk, map the failure mode → `behavior.name` + `behavior.description`, and use its context for `context`: ``` assert-ai init --default-model --describe-file --non-interactive -o evals/.yaml ``` -- **Write the description to a file and pass `--describe-file`.** The text is Clarity-derived prose you did not author, so it can contain quotes, backticks, or `$(...)`; interpolating it into `--describe ""` would break the command or inject into the user's shell. `--describe` stays available for short text you typed yourself; the two are mutually exclusive. +- **Write the description to a file and pass `--describe-file`.** The text is prose you did not author — Clarity-derived on Path A, the user's own words or a doc excerpt on Path B — so it can contain quotes, backticks, or `$(...)`; interpolating it into `--describe ""` would break the command or inject into the user's shell. `--describe` stays available for short text you typed yourself; the two are mutually exclusive. - `--default-model` seeds the generated config's `pipeline.default_model` — the model the **eval** runs against. Do **not** use `--model` for this: that is the init assistant's own conversation model (default `azure/gpt-5.4-mini`) and has no effect on the eval. Note `--default-model` is a prompt-level hint the design agent is asked to *confirm*, not a deterministic write — verify the value actually landed in the generated YAML. - **Pin `systematize` and `judge` to the strong model by hand after init.** `init` has no `--systematize-model` / `--judge-model` flag, so every stage inherits `default_model` unless you edit the config. Run the eval cheap and the two ground-truth stages strong — `default_model.name: azure/gpt-5.4-mini` (target, test-set, tester) plus `pipeline.systematize.model: azure/gpt-5.4` and `pipeline.judge.model: azure/gpt-5.4`. This is the convention in the repo's own `examples/` configs. `systematize` authors the behavior tree and the permissible / non-permissible split that **every** metric is computed against, and `judge` decides both applicability and violation per row on a single sample (`judge.n` defaults to `1`, judge temperature unpinned) — a weak model there moves the target rather than adding noise around it, and inflates run-to-run applicability drift. Verify after the run with `assert-ai results status --json` → `prompt_metrics.judge_model` / `scenario_metrics.judge_model`. - **Check the built-in presets first** — `assert-ai library list` shows bundled behavior and judge presets (`prompt_injection`, `doxxing`, `stereotyping`, `sycophancy`, `harmful_medical_advice`, `tool_orchestration_errors`, …); `assert-ai library show ` prints one. If one matches the risk, seed with `--behavior ` / `--judge-preset ` instead of generating from scratch. @@ -177,12 +206,13 @@ Team-maintained docs under `docs/` on `main` — prefer them over restating prod ## Guardrails -- **Clarity is the required risk source** — for Run mode, risks come from Clarity (existing `.clarity-protocol/` or a fresh discovery run via the `run_clarity` MCP tool). Never substitute a plain-language guess or imitate Clarity's questioning from your own head; if the MCP tools can't be made available, stop and help fix it (`SETUP-CHECKLIST.md`). +- **Clarity is the recommended risk source, not a gate** — present **both** options together whenever the user needs a new risk: Clarity discovery (existing `.clarity-protocol/` or a fresh `run_clarity` run) *and* risks they supply themselves. Recommend Clarity, because it surfaces failure modes they haven't considered — but never present it as the only route. Any menu, list, or question you offer that includes a Clarity option must carry the user-supplied option beside it; a user who doesn't know Path B exists cannot ask for it. Hold the user-supplied path (Step 1b) to the same bar: atomic behaviors, an explicit permissible boundary, variant-derived dimensions. Never block a measurement on Clarity setup. +- **Never imitate Clarity's interview from your own head** — if the user chose Clarity, drive the real MCP tools (`run_clarity` returns its genuine process guide inlined). Step 1b is a distinct structured intake, not a hand-rolled impression of Clarity. - **Drive the real Clarity MCP tools in-IDE** — use `run_clarity` / `write_protocol_document` / `record_failure` for discovery and `record_suggestion` to close the loop; never hand the user off to a separate Clarity app and never shell out to a `clarity cli` process. -- **Close the loop** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives. +- **Close the loop when a protocol exists** — after a run, offer `record_suggestion` (or `record_decision`) back into `.clarity-protocol/` noting the failure mode now has a measured baseline and where the eval lives. With no protocol, skip it silently — and consider offering Clarity as a next step for finding risks this pass didn't cover. - **Govern with ACS, don't just prompt-tweak** — to fix and *prove* it, generate an ACS policy from the findings (`assert-ai acs generate`), **review and commit** it (scope the gated tools, tighten conditions), and re-run the same eval against the governed callable to show the delta; needs a wrappable callable target (`../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). Whenever a gate needs a value the model doesn't put in the tool args — a trusted session flag (verification), a trusted comparison value (the caller's own id), a trusted numeric cap, or a running total / prior-call fact — the governed agent must surface that scalar from its **session state** into the tool-call **policy_target** so the generated `input.policy_target.value.*` rule actually fires. ACS evaluates each call in isolation, so multi-call constraints (running totals, ordering, rate limits) are handled by that same injection, not by encoding history in Rego. Free-form content failures (unsafe advice, PII in prose, a verbal-only high-risk promise) and inbound prompt-injection instead use an **annotator-based** gate at the `output`/`input` point, proven by the remeasure delta since offline `validate` can't run annotators. Never hand-drive an external `acs` CLI for this loop. - **One atomic behavior per config** — split N selected risks into N configs run sequentially; never bundle. -- **Triage before running** — never auto-generate an eval for every Clarity failure mode; ask which to measure now. +- **Triage before running** — never auto-generate an eval for every enumerated risk; ask which to measure now. - **Don't invent metrics** — only report what's in the artifacts. - **Don't trawl raw traces to answer questions** — answer from `results status`, `scores.jsonl`, and `metrics.json`; hand off to the viewer for visual trace/transcript exploration. - **Hand off, don't reimplement the viewer** — for visual drill-down, baseline compare, or live monitoring, point to the local viewer rather than reproducing it in chat. diff --git a/AGENTS.md b/AGENTS.md index 00e39b1e..360a49c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,7 +166,7 @@ These skills are for end users running evaluations, not for repository maintenan | Skill | Claude Code | GitHub Copilot | Cursor | What it does | |---|---|---|---|---| -| `run-assert-eval` | `.claude/skills/run-assert-eval/SKILL.md` | `.github/prompts/run-assert-eval.prompt.md` | `.cursor/rules/assert.mdc` | Discover risks via the Clarity MCP tools (`run_clarity`) in-IDE, then follow `workflows/measure-clarity-failures.md`: triage, split selected risks into one atomic config per behavior, run the pipeline, summarize results with cited failures. Reports policy violation and overrefusal separately. To fix and prove a failure, `workflows/govern-and-remeasure.md` generates an ACS policy from the findings and re-runs the same eval against the governed agent to measure the failure-rate delta. | +| `run-assert-eval` | `.claude/skills/run-assert-eval/SKILL.md` | `.github/prompts/run-assert-eval.prompt.md` | `.cursor/rules/assert.mdc` | Establish a risk source — discover risks via the Clarity MCP tools (`run_clarity`) in-IDE (recommended), or take risks the user supplies directly as prose or a PRD / design doc / threat model — then follow `workflows/measure-clarity-failures.md`: triage, split selected risks into one atomic config per behavior, run the pipeline, summarize results with cited failures. Reports policy violation and overrefusal separately. To fix and prove a failure, `workflows/govern-and-remeasure.md` generates an ACS policy from the findings and re-runs the same eval against the governed agent to measure the failure-rate delta. | ## Output style for coding agents diff --git a/README.md b/README.md index cc02a668..8c5b997b 100644 --- a/README.md +++ b/README.md @@ -57,21 +57,25 @@ The skill turns "I think my agent might do something bad" into measured evidence | | | | |---|---|---| -| **Clarity** | *discovery* | An interviewing agent that walks you through what your system is for and where it could fail, and writes the risks down. | +| **Clarity** | *discovery* **(recommended)** | An interviewing agent that walks you through what your system is for and where it could fail, and writes the risks down. | | **ASSERT** | *measurement* | Turns each risk into a generated test suite, runs it against your agent, and judges the transcripts. | | **ACS** | *governance* | Generates an Agent Control Specification from the real failures, then re-runs the same eval against the governed agent to prove the rate dropped. | -Risks always come from Clarity — the skill won't let you seed an eval from an off-the-cuff description, because that is what produces low-signal results. +Every eval starts from a risk, and you choose where it comes from. Clarity is recommended — it surfaces failure modes you haven't thought of, plus severity and causal chains — but it is never required. If you already know what you want measured, tell the skill directly, in your own words or by pointing it at a PRD, design doc, threat model, or incident report. Either way the skill holds the eval to the same bar: one atomic behavior per config, an explicit permissible boundary, and an explicit sample size. #### 1. Onboard (once per workspace) -You need **Python 3.12+** (ASSERT itself runs on 3.11+, but Clarity requires 3.12) and an IDE with MCP support — VS Code + Copilot agent mode, Claude Code, or Cursor. Clarity's discovery step runs as an MCP server, so this part can't be done from a bare terminal. +ASSERT needs **Python 3.11+**. If you want Clarity's discovery step, you also need **Python 3.12+** and an IDE with MCP support — VS Code + Copilot agent mode, Claude Code, or Cursor — because Clarity runs as an MCP server and can't be driven from a bare terminal. ```bash pip install -e ".[otel,langgraph]" # install ASSERT cp .env.example .env # add your provider key assert-ai --help # verify +``` + +That is enough to run the skill with risks you describe yourself. To add Clarity's discovery front end: +```bash pip install -e ".[mcp]" # from your clarity-agent checkout clarity embed . # wires Clarity into this workspace clarity doctor # verify an LLM provider is configured @@ -110,11 +114,11 @@ Describe your agent in chat — what it does, what it can touch, and what it mus > legal/tax/financial advice, must not expose another customer's data, and must verify > identity before high-risk actions (plan changes, cancellations, refunds).* -That description is the shipped [`billing_support_agent`](examples/billing_support_agent/) example. The more precisely you state the boundaries, the sharper the risks Clarity comes back with. +That description is the shipped [`billing_support_agent`](examples/billing_support_agent/) example. The more precisely you state the boundaries, the sharper the risks — whether Clarity discovers them or you name them yourself. The skill then, with you in the loop: -1. **Discovers** risks via Clarity, or reuses an existing `.clarity-protocol/`. +1. **Establishes the risk source** — discovers risks via Clarity, reuses an existing `.clarity-protocol/`, or takes the risks you supply directly. 2. **Stops at a triage gate** and shows you the candidate risks. You pick which to measure. Declining here writes nothing and runs nothing. 3. **Generates one atomic config per selected risk** — never one merged config, so each result is attributable to a single behavior. 4. **Confirms**, then runs the suites sequentially.