chore(library): unbundle the three scenario specs into atomic behaviors

[Chang Liu (changliu2)]

Summary

Finishes the atomicity work from #293. That PR classified three bundled files as scenario; this one actually unbundles them, so the library is an exemplar of the rule rather than a place where the rule is quarantined.

Stacked on #293 — review that one first.

What was bundled

File	Bundled
travel_planner	6 mechanisms across "Quality failures" / "Safety failures"
travel_planner_benchmark	~6 quality mechanisms
telecom_customer_service	An application spec (Role, Domain Basics, Operational Procedures)

10 new atomic behaviors

explicit_constraint_violation_failures, output_internal_consistency_failures, unit_conversion_failures, actionability_failures, procedure_adherence_failures, out_of_scope_request_failures, escalation_judgment_failures, identity_verification_failures, unauthorized_action_failures, tool_call_turn_protocol_failures

Reused, not duplicated

stereotyping, prompt_injection, and sycophancy were named inside travel_planner and already existed as atomic presets. They are now referenced from the scenario's behaviors: list. Same for the existing grounding / tool-selection / verification / observation / response-completeness / unsupported-conclusion presets.

Duplicating a behavior under a new name would have been worse than the bundling being fixed, so I audited for it: every new preset's closest textual neighbour among the 38 pre-existing ones is ≤20% similar. No near-duplicates.

Scenario shape

A scenario is now context-only:

kind: scenario
name: travel_planner
context: |
  <the application: role, tools, domain objects, procedures>
behaviors:
  - incorrect_tool_selection_failures
  - insufficient_verification_failures
  - grounding_attribution_errors
  - explicit_constraint_violation_failures
  - stereotyping
  - prompt_injection
  - sycophancy

No behavior-shaped description:. scripts/check_behavior_library.py now enforces that, plus behavior↔spec parity in both directions. All 51 presets pass.

The flagship example keeps all seven behaviors

Worth a careful look — the unbundling initially rewrote examples/travel_planner_langgraph/eval_config.yaml down to a single behavior. That is atomic, but it silently dropped six mechanisms from the example that README.md, docs/getting-started.md, docs/config/schema.md, the ACS guide, docs/targets/callable.md, and science.yml all point at. Coverage loss wearing atomicity's clothes.

Fixed by adding the other six as sibling configs under behaviors/, each sharing the same context::

Path	Behavior
eval_config.yaml	prompt_injection — quickstart, so every existing doc link still works
behaviors/tool-selection.yaml	incorrect_tool_selection_failures
behaviors/grounding.yaml	grounding_attribution_errors
behaviors/constraints.yaml	explicit_constraint_violation_failures
behaviors/verification.yaml	insufficient_verification_failures
behaviors/stereotyping.yaml	stereotyping
behaviors/sycophancy.yaml	sycophancy

This is also the layout we tell CI customers to use (configs: .../behaviors/*.yaml), so the flagship now demonstrates the pattern instead of describing it.

Not a breaking change

behavior: {preset: travel_planner} still resolves through the deprecation shim from #293 — verified it still emits DeprecationWarning and returns the scenario file.

Testing

• python scripts/check_behavior_library.py → 51 presets (48 behaviors, 3 scenarios), atomic, in parity
• pytest tests/test_library_loader.py tests/test_library_e2e.py tests/test_import_smoke.py → 89 passed
• Full non-viewer suite → 1154 passed
• Every generated flagship config parses and its preset resolves
• Duplication audit: max 20% similarity between any new preset and any pre-existing one

Pre-existing failures on this branch's parent, verified by checking out the parent and re-running — not introduced here:

• tests/test_viewer_* — ERR_MODULE_NOT_FOUND: yaml, needs npm install in viewer/ (CI does this)
• tests/test_tool_module_sandbox.py::...per_conversation_workspace — Docker/sqlite lock on Windows

Review notes

The 10 generated presets are the part worth scrutiny — specifically whether each names exactly one mechanism and whether a judge could score it from evidence. Happy to re-cut any of them.

[Chang Liu (changliu2)]

Correct. This config is now a single-atomic-behavior example, not the full benchmark. The old bundle covered 6 mechanisms in one config; those are now 6 separate atomic presets in assert_ai/library/behaviors/ (explicit_constraint_violation_failures, output_internal_consistency_failures, unit_conversion_failures, tool_call_turn_protocol_failures, actionability_failures, procedure_adherence_failures). To reproduce full benchmark coverage for the travel_planner_benchmark scenario, you'd run one eval_config.yaml per atomic preset (each pointing context: at the same scenario) and aggregate results -- same pattern as examples/travel_planner_langgraph/behaviors/*.yaml, which already does this (one file per behavior: constraints.yaml, grounding.yaml, tool-selection.yaml, etc.). Happy to add a benchmark/README.md note showing the full run-all-presets sequence if that'd help clarify.

[ahmedmagooda]

Chang Liu (@changliu2) i think yes, adding the readme would be more clear. Other than that i think everything looks good.

[Chang Liu (changliu2)]

ahmedmagooda Pushed a follow-up commit (83913da) addressing your comment above: made the behavior/scenario library obviously discoverable at every entry point a user hits before writing a behavior spec by hand:

• Top-level README.md -- new nav-bar link (📋 Behavior Library) alongside Get Started/Examples/CLI Reference, plus a "What you get with ASSERT" bullet
• docs/README.md -- new Behavior Library entry under Configuration, described as "start here before writing a behavior spec by hand"
• docs/config/best-practices.md -- callout added directly inside §8.D ("Use atomic behaviors"), pointing to the library right where atomicity is explained
• examples/README.md -- new "Reuse a behavior from the library -- check here first" section, placed before "Which example to start with," referencing the real assert-ai library list --kind behavior / assert-ai library show <preset> commands

Can you review and approve when you get a chance?

[tangym]

This scenario table still appears to describe the previous bundled configuration. Could this table be updated so it matches the current eval_config.yaml?

[tangym]

For longer specs, a changed sentence could still pass while pip users receive different content from the markdown reference. Also, because the parity validation is guarded by if SPECS.is_dir(), removing the reference directory would skip the check entirely.

The practical risk is low because yaml is the source of truth and md is not used at runtime; deleting the entire directory would likely be noticed during review.

[tangym]

Style/readability: all six sibling configs serialize context as a hard-wrapped single-quoted scalar with blank lines between the wrapped source lines. The rubric strings use the same pattern. The runtime risk is low. Can we use the same literal-block style as the eval_config.yaml for these multiline fields?

context: |
  ...
rubric: |-
  true = ...
  false = ...

[Chang Liu (changliu2)]

tangym Addressed both comments:

1. README scenario table — was still describing the pre-unbundling behavior (blended quality/safety failures, 6 categories). Updated to reflect the atomic prompt_injection quickstart config, and clarified the sibling �ehaviors/*.yaml cover the other six mechanisms.
2. YAML style — reformatted context:/
   ubric: in all six sibling configs (plus �val_config.yaml's rubric for consistency) to literal-block style (|/|-), matching the existing convention. Verified byte-for-byte semantic equivalence via a yaml.safe_load diff against the prior committed content — pure style change, no content drift.

Also tightened check_behavior_library.py's parity check per your note: the 98%-similarity tolerance could let real content drift through silently in a long spec, since words() already normalizes every expected source of formatting difference. Now requires an exact match, and hard-fails (rather than silently skipping) if the �xamples/behavior_specs reference directory goes missing.

Re-verified: 51/51 presets pass (atomic + parity) under the tightened rule, 89/89 targeted tests pass, and all 7 edited configs still load/resolve via �ssert_ai.config.load_config.

Ready for another look whenever you have time.

[Chang Liu (changliu2)]

Closed out the remaining review gaps from an independent audit pass:

1. Ahmed's requested benchmark README — added examples/benchmark/README.md, explaining this is a throughput-scale variant of the flagship travel_planner_langgraph example (same target, same explicit_constraint_violation_failures preset already used by behaviors/constraints.yaml), not a new agent or behavior. Registered it in examples/README.md's selection table and layout tree so it's actually discoverable.
2. Yeming's YAML-style comment — one sibling config's rubric (eval_config.yaml's overrefusal) was missed in the prior pass and still had the hard-wrapped single-quoted-scalar style; now literal-block like every other rubric/context field in the example. Verified byte-for-byte semantic equivalence via a yaml.safe_load diff.
3. Yeming's DeprecationWarning comment on chore(library): make behavior presets atomic, close a 20-behavior gap, and enforce both in CI #293 — fixed on chore(library): make behavior presets atomic, close a 20-behavior gap, and enforce both in CI #293's branch directly (this branch is stacked on it, so it's included here too via merge).

51/51 presets clean, 89/89 targeted tests pass.

tangym ahmedmagooda — ready for another look whenever you have time.