You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #566 adds the durability eval runner and its fail-closed unit tests. Those tests are intentionally not wired into CI by that same PR because AGENTS.md forbids editing a gate that judges your own work. A separate, independently reviewed change should own the gate.
Scope
Run node --test scripts/run-evals.test.mjs scripts/benchmark-protocol.test.mjs in CI.
Confirm failures in timeout classification, execution witnesses, dirty provenance, output confinement, or competitive-protocol drift fail the job.
Avoid running the 210-trial publication benchmark on every PR unless runtime/cost is explicitly accepted; the fast harness tests are the required gate.
Current evidence
At PR #566 head c4092182870fe148afe673da38358ad7dd20c1e3, the literal local command reports 17 tests passed, 0 failed. The publication eval separately reports 210/210 at the same clean head.
Acceptance
A PR independent from #566 lands a required CI job that runs the two Node test files and demonstrates the job fails when one of their assertions is intentionally broken.
Why
PR #566 adds the durability eval runner and its fail-closed unit tests. Those tests are intentionally not wired into CI by that same PR because
AGENTS.mdforbids editing a gate that judges your own work. A separate, independently reviewed change should own the gate.Scope
node --test scripts/run-evals.test.mjs scripts/benchmark-protocol.test.mjsin CI.Current evidence
At PR #566 head
c4092182870fe148afe673da38358ad7dd20c1e3, the literal local command reports 17 tests passed, 0 failed. The publication eval separately reports 210/210 at the same clean head.Acceptance
A PR independent from #566 lands a required CI job that runs the two Node test files and demonstrates the job fails when one of their assertions is intentionally broken.