feat(skill): add run-assert-eval and atomic worked examples - #311
Conversation
…led badges on prompt rows.
…e failed on disabled-dimension runs.
…aph example runs and results. refine skill.
…mples ran through SKILL workflow.
The behavior library had three problems, none of which anything checked for.
**Presets bundled multiple behaviors.** `travel_planner` covered six mechanisms
across "Quality failures" and "Safety failures" -- three of which
(`stereotyping`, `prompt_injection`, `sycophancy`) already existed as their own
atomic presets. `travel_planner_benchmark` bundled roughly six more.
`telecom_customer_service` was not a behavior at all: it is an application spec
(Role, Domain Basics, Operational Procedures) wearing `kind: behavior`.
Evaluating a bundle as one behavior produces a dataset mixing several mechanisms
and a metric nobody can act on -- you learn that something failed, never which
mechanism. That is exactly what best-practices 8.D ("use atomic behaviors")
exists to prevent.
These three are application scenarios, so they move to a new `scenario` kind in
`assert_ai/library/scenarios/`. They are the context an eval runs against, not
the behavior it measures.
**20 behaviors shipped to nobody.** `examples/behavior_specs/*.md` held 38 specs;
`assert_ai/library/behaviors/*.yaml` held 18 of them. Only the YAML goes in the
wheel, so every agentic failure mode -- goal drift, premature termination,
repeated action loops, stale state, poor retrieval, tool-call error recovery,
and 14 more -- was invisible to anyone who installed from PyPI. The 18 that did
exist in both places were byte-identical, so this was pure coverage loss, not
divergence. Generated the missing 20 from the existing markdown and the category
metadata already in that directory's README; no prose was invented.
**Nothing detected either problem.** `scripts/check_behavior_library.py` now
fails CI when a preset names another preset's behavior (provable bundling), when
one preset carries several failure categories, when a description reads as an
application spec, or when a spec markdown drifts from its YAML or has no preset
at all. It runs in Tier 1.
Not breaking: `behavior: {preset: travel_planner}` still resolves, via a shim
that warns and points at the `scenario` kind. Config authors get told, not
broken.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Left out of the previous commit, so 'library show --kind scenario' rejected the new kind and Tier 1 failed. The local run passed only because the edit existed in my working tree but was never staged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Jake Present (jakepresent)
left a comment
Jake Present (jakepresent)
left a comment
There was a problem hiding this comment.
Three release-facing inconsistencies still need to be fixed on the current head.
-
The shipped skill still teaches the retired metric contract.
.claude/skills/run-assert-eval/SKILL.md:287,workflows/measure-clarity-failures.md:181, andworkflows/govern-and-remeasure.md:108call the first metric “Harm (non-permissible),” while ASSERT’s actual label is “Impermissible behavior violated.” More importantly,SKILL.md:337-339andgovern-and-remeasure.md:596-602sayresults compare --metriccannot use either split metric. The CLI metrics change in this same release explicitly adds those derived metrics to comparison and tests them. Shipping both would make the new onboarding skill wrong on day one. -
The skill contradicts this PR’s example-cleanup contract. The PR body and cleaned tree intentionally keep generated ACS/governed outputs uncommitted, but
SKILL.md:404-413tells the agent to commit the generated policy and requires every worked example to containagent_guarded*.py, governed YAML pairs, andacs/<risk>/.... A future skill run will recreate the exact clutter this PR removes unless those guardrails are reconciled with the chosen product behavior. -
git diff --check origin/main...HEADfails on the four changed lines indocs/getting-started.mdbecause the added CRLF lines retain trailing carriage-return whitespace. The PR body currently claims this check passes.
The implementation tests are otherwise healthy: after npm ci --prefix viewer, I get 1246 passed, 20 skipped, 474 subtests passed, and the live CI matrix is green. Once the shipped instructions and whitespace match the intended release contract, I’m happy to re-review.
Remove the overlapping travel-planner behavior configs so the downstream examples PR owns the canonical flat evals layout. Keep the atomic benchmark update, but make its documentation independent of the removed path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Stack the atomic behavior/scenario library ahead of the public skill and worked examples. Resolve the examples index in favor of the flat evals layout while retaining library discovery guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Refresh the foundation branch before stacking the follow-up PRs so downstream diffs do not re-show changes already merged to main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Carry current main through the stacked foundation so the unbundle PR remains reviewable only against PR #293 and downstream PRs do not inherit unrelated main changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Phoenix 19.18+ crashes while pytest auto-loads its plugin on Python 3.11. Keep the existing compatible lock resolution, constrain the optional dependency, and make dependency metadata changes trigger regression CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Move the Phoenix compatibility bound into the shared foundation while preserving PR #311's broader regression triggers and skill-test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
There was a problem hiding this comment.
Verdict: Confirm deleting examples/career_health_assessment/ and prompt_agents/evals was intentional. Because a whole domain was lost in a merge. Everything else remains.
Relative to PR #303, this PR #311 deletes every Clarity Protocol/ tree and every taxonomy.json, flattens evals/<risk>/eval_config.yaml to evals/<risk>.yaml, and merges the behavior-library atomicity work (30 behavior YAMLs, 3 scenarios, scripts/check_behavior_library.py). Confirmed this correct decision in MS Teams 08/12/2026 with Chang.
- Confirm this was intentional:
examples/career_health_assessment/was deleted by merge4a2120d("Merge main and standardize example eval layouts") which removed all 8 files:README.md,__init__.py,agent.py, and 5 eval configs. - Confirm this was intentional:
prompt_agents/evalswas removed by commit77b4f13. Its' 3 suites, top-levelhealth_assistant*.yamlvariants, remain.
Otherwise, looks good to merge to main. Thanks!
Jake Present (jakepresent)
left a comment
Jake Present (jakepresent)
left a comment
There was a problem hiding this comment.
Rechecked the current head after fixing the outstanding release blockers. The skill and workflows now use the shipped impermissible/permissible labels and compare metrics, generated governance output stays local by default instead of repopulating public examples, and the getting-started diff is clean. I also restored the career-health worked domain that was lost in the merge, flattened its five atomic configs to the current eval layout, and verified every config and documented run path. Local verification: 1,265 passed, 20 skipped, 834 subtests; behavior-library parity passes; viewer check/build passes; current Tier 1 CI is green. The head is merge-clean.
|
Closed Alex's merge-cleanup question on the current head: the career-health domain was an accidental whole-directory loss, so I restored it and flattened its five configs to the current one-file-per-behavior layout. The |
Restore the intended six-domain curated example surface and remove the navigation entries reintroduced in 5764e0d. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the root and examples indexes aligned with the intended six worked domains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The base branch was changed.
alex ngo (ango10)
left a comment
alex ngo (ango10)
left a comment
There was a problem hiding this comment.
base branch updated, everything else remains the same. lgtm!
3 participants
Summary
Adds
run-assert-eval, a skill for turning discovered failure modes into focused ASSERT evaluations and optionally governing/re-measuring them with ACS. The customer example surface now keeps one behavior per YAML and contains only the files needed to understand and run each example.Changes
Skill
Worked examples
prompt_agents/as a focused target-shape gallery.Framework and ACS
assert-ai init, library examples, and related documentation/tests.Validation
Targeted coverage:
assert-ai initrun-assert-evalClarity intake testsAdditional static audit:
behaviormappingassert-ai run --configpaths existresults statussuite/run pairs match YAMLexamples/git diff --checkpassesChecklist