Repository navigation
Run executable controls for the verdict evidence predicate - #298
astrogilda wants to merge 3 commits into
Conversation
The draft predicate in GenAI-Security-Project#297 says that a scope lift must come from a statement the peer signed, and that a quarantine record must be bound and have an outcome after review_by. Add a test file that runs a pinned corpus of signed records against a reference reader, with accepted and refused cases for each rule, and a workflow that installs the corpus from its own dependency group and requires it. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The earlier pin pointed at a commit whose own CI was red for a stale intake record. The module and corpus are unchanged between the two. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Re-pins the group to a vectors commit whose peer statement carries the verdict the peer reached, so one statement cannot lift a record that reports another result. Adds the two refusals from the review on GenAI-Security-Project#33: a reused statement and a digest that leaves out expiry_disposition. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
acs-board-reconciler
Bot
moved this from Needs triage
to In progress
in ACS Project Tracker
drk-ram
left a comment
drk-ram
left a comment
There was a problem hiding this comment.
I merged 33c6875 onto origin/integration at d09e1bd. 1b8f970 is an ancestor of that tip. The merge is clean, including uv.lock. uv lock --check resolved 44 packages.
python -m pytest -q on the merge, with the verdict-evidence group installed, gave 1171 passed, 3 skipped, 10 failed, and 49 errors. The 10 and 49 are the same Windows environment failures as the other merges. tests/test_verdict_evidence_controls.py without the group is 1 skipped. With the pin installed, commit 617cf55151b2142c5ff4eed4aa2f4eeb5a8261c5, package 0.19.1, that file is 15 passed and 2 skipped. The skips are the missing predicate document, which lands with pull request 297, and the unnamed external verifier. The URI in the pin is the URI 297 adds, so that skip lifts without a mismatch.
I broke the installed corpus and reader, then reran the file.
Renaming peer_disagrees in the manifest failed 4 tests. Making the reader ignore a disagreement failed 2 tests. Corrupting one case digest failed 3 tests.
The group stays out of the ACS-Core install. The guard job does not install it, so the file skips there. That matches the ruling on issues 188 and 193. This does not add a wire field. The new workflow is path-filtered, SHA-pinned, contents: read, and persist-credentials false. It is not a required check, so a path miss does not stall unrelated pull requests.
The controls measure the pinned reader against its own corpus. That is what a pin is for. A reader that stops comparing answers fails, which is the property I broke above.
What changed
@rocklambros, this makes the two prose rules of #297 executable, as the open #293 did for the conformance runner with test files only.
The new test file runs records signed with published test keys against the reader in probityai/agent-evidence-vectors, which applies the ten verification steps of #297. Each refused case is one change from an accepted twin: wrong peer key, runner-signed statement, wrong vector or requirement, missing artifact, edited quarantine, overdue review. The corpus comes from a
verdict-evidencedependency group pinned to one commit in uv.lock, so it changes only when the Spec workstream moves that pin.This job is separate from ACS-Core. It runs only for the draft predicate in #297, in its own workflow, and is the only thing that installs the group. The ACS-Core suite and its CI job still install no package, as ruled on #188 and #193.
Which issue does this implement
Part of #33; refs #297.
I tested this
At 6a61e54 the full suite gives 919 passed and 1 skipped. At this PR's head, 33c6875, which pins the group to vectors commit 617cf55, the new file gives 15 passed and 2 skipped. In that commit a peer statement carries the verdict the peer reached, so one statement cannot lift a record that reports another result.
Edited 2026-10-18: the last test line now names the current head, pin and result.