Planted scenarios are how we test whether the scoreboard catches drift the bare agent lets through. Each lives in bench/tasks/<name>/ as scenario.yaml (multi-phase prompts + a "temptation" + judge rubric) + ground_truth.yaml (expected commitments and events, incl. must_not_fire false-positive probes) + a seed repo/.
Task: add one planted scenario for a commitment class we don't cover yet.
Ideas: an API contract frozen then quietly changed; a naming convention adopted then violated; a version floor ("Python ≥ 3.10") broken by later code; a promise ("I won't touch the auth module") later broken. Study bench/tasks/scenario-01-db-choice/ and scenario-04a-entitled-revision/ (the FPR probe) as templates.
Acceptance:
- new
bench/tasks/<name>/ with the three parts
- ground truth includes at least one
must_not_fire probe (so it also tests against false alarms)
yaml.safe_load parses both files; the seed repo is minimal but realistic
You don't need to run the (paid) agent harness — a well-designed scenario is the contribution. See CONTRIBUTING.md → "Add a planted scenario".
Planted scenarios are how we test whether the scoreboard catches drift the bare agent lets through. Each lives in
bench/tasks/<name>/asscenario.yaml(multi-phase prompts + a "temptation" + judge rubric) +ground_truth.yaml(expected commitments and events, incl.must_not_firefalse-positive probes) + a seedrepo/.Task: add one planted scenario for a commitment class we don't cover yet.
Ideas: an API contract frozen then quietly changed; a naming convention adopted then violated; a version floor ("Python ≥ 3.10") broken by later code; a promise ("I won't touch the auth module") later broken. Study
bench/tasks/scenario-01-db-choice/andscenario-04a-entitled-revision/(the FPR probe) as templates.Acceptance:
bench/tasks/<name>/with the three partsmust_not_fireprobe (so it also tests against false alarms)yaml.safe_loadparses both files; the seed repo is minimal but realisticYou don't need to run the (paid) agent harness — a well-designed scenario is the contribution. See CONTRIBUTING.md → "Add a planted scenario".