Repository navigation
Add negative controls for the conformance runner - #293
Open
astrogilda wants to merge 5 commits into
Open
astrogilda wants to merge 5 commits into
astrogilda wants to merge 5 commits into
Conversation
Issue 53 lists what a conformance suite has to get right. Requirements 2, 3, 4, 5, 7, 9 and 11 are about the gate rather than any one Guardian, so each gets a control here: a stub Guardian or a corpus the runner must refuse, and seven runner mutants that at least one control must catch. The runner is standard library only and runs as a child process. It posts signed envelopes over HTTP and scores by section 17.1 code, never by message text. The same vectors run against the reference Guardian, where the four known gaps (signature, missing signature, repeated request_id and the skew window) are strict expected failures that cite issue 184. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The test job in deploy-pages.yml has no Bun, so the reference Guardian group in tests/test_negative_controls.py skips there. The agt job already installs Bun and the tree's dependencies, so it now runs that file too, with ACS_REQUIRE_REFERENCE_GUARDIAN set so the group fails rather than skips when the Guardian cannot start. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The runner's key derivation matches the parameters pull request 272 adds to section 10, so the docstring now says so instead of describing them as this runner's own choice. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Each strict xfail reason now names the section whose MUST the reference Guardian fails, so the marker reads as a known failure rather than as an optional case. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
astrogilda
requested review from
GangGreenTemperTatum,
afogel,
bar-capsule,
mamicidal and
rocklambros
as code owners
October 17, 2026 15:06
acs-board-reconciler
Bot
moved this from Needs triage
to In progress
in ACS Project Tracker
An agent that loses the response to a non-idempotent call resends the same request, signature and timestamp included. A Guardian that answers that resend with its first decision, as an idempotent retry, lets the action run twice, and it passes every single-request replay vector. Add a replay vector that resends the previous request unchanged and expects REPLAY_DETECTED under section 10.3, a stub Guardian that treats the resend as idempotent and must fail exactly that vector, a rule that refuses a suite with no resend, and a runner mutant the conformant control catches. The reference Guardian does not track request_id, so the new vector joins its strict expected failures. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
@rocklambros, this is the tests/ file promised on #53, for the suite that #19 gave the Spec workstream of @bar-capsule and @afogel.
tests/test_negative_controls.pycovers requirements 2, 3, 4, 5, 7, 9 and 11 with subjects a small runner must refuse, like an allow-everything Guardian. Its MUTANTS list holds eight single-line weakenings of that runner, one per guard, and a control catches each.The replay group now covers the lost-response retry from #53, a byte-identical resend that section 10.3 refuses as a repeated request ID. A Guardian that answers the resend as an idempotent retry fails
request-resent-after-lost-response, and the runner refuses a suite that never resends.The reference Guardian fails four section 10 MUSTs today, on signatures, replay and the skew window, and those vectors are strict xfails citing #184.
Which issue does this implement
Part of #33; refs #53.
I tested this
At df234a9, pytest gives 946 passed and 8 skipped, and the strict mkdocs build passes. The skips are the eight reference Guardian cases, and the Bun job agt in reference-implementation.yml runs them: 3 passed, 5 xfailed.
Edited 2026-10-24: added the lost-response resend vector (df234a9) and its test run.