Skip to content

test: conformance vectors for seven unexercised receipt rules - #122

Open
lywinged wants to merge 1 commit into
agentrust-io:mainfrom
lywinged:conformance/rule-coverage-vectors
Open

test: conformance vectors for seven unexercised receipt rules#122
lywinged wants to merge 1 commit into
agentrust-io:mainfrom
lywinged:conformance/rule-coverage-vectors

Conversation

@lywinged

@lywinged lywinged commented Aug 5, 2026

Copy link
Copy Markdown

What

Seven conformance fixtures, one per receipt rule that the verifier applies and that no
fixture previously exercised.

Rule What an implementation can omit today and still pass
action_ref_invalid Recomputing the action reference instead of trusting the declared value
call_id_mismatch Checking the receipt is bound to this call
session_id_mismatch Checking it is bound to this session
evidence_hash_mismatch Recomputing the evidence digest
issuer_key_untrusted Checking the signing key against a pinned set at all
receipt_from_future Rejecting a receipt issued after the verification time
decision_invalid Refusing to read an unknown verb as accept or reject

Why these two matter beyond tidiness

Without issuer_key_untrusted, a receipt authenticates itself. A signature verifies
against whatever key it names, and only a pinned set decides whether that key was ever
entitled to speak.

Without evidence_hash_mismatch, the signature covers a digest whose document can be
swapped. The receipt signs evidence_hash, not the evidence body, so only recomputation
catches a substituted body.

The other five are ordinary binding and freshness checks, and the same argument applies
in weaker form: a suite that never exercises a rule certifies implementations that skip
it.

How they were found

By walking the verifier's source for every failure code it can emit and comparing that
against the codes the fixtures expect. The seven above were emitted by the verifier and
expected by nothing.

I am happy to contribute the check itself as a follow-up if it is wanted — it recovers
the rule inventory from source with ast rather than from a hand-maintained list, so a
rule added without a fixture fails rather than passing quietly. Keeping it out of this PR
so the fixtures can be judged on their own.

Shape

One fixture per rule, each triggering exactly that rule and nothing else, so a failure
names the check that broke rather than a set of them.

They pin their own deterministic test key. The private half of the key behind 0109
is not published, and each fixture already carries its own trusted_issuer_keys, so a
separate key changes nothing about how the set is verified.
gen_rule_coverage_vectors.py regenerates the set byte-for-byte; only public JWKs appear
in the files. Worth stating plainly that the keys are deliberately deterministic test
keys rather than leaving a reviewer to notice the pattern.

Scope

Additive. No existing fixture changes, no verifier changes, no schema or spec changes.
The only edits to existing files are the filename list in test_fixture_set_is_complete
and a table in the examples README.

Under GOVERNANCE.md this is a conformance-test addition: no normative text, so no
sponsor and no comment window. Please tell me if you read it differently.

Checks

  • 112 tests pass (105 before, plus 7)
  • ruff check src tests and mypy src/agentrust_trace clean
  • Verified on Python 3.11, 3.12, 3.13 and 3.14
  • DCO signed

The receipt verifier applies rules that no fixture exercises. An implementation
can omit each of these checks entirely and still pass the published set, which is
the one thing a conformance suite is supposed to prevent.

  action_ref_invalid       recomputing the action reference rather than trusting
                           the declared value
  call_id_mismatch         that the receipt is bound to this call
  session_id_mismatch      that it is bound to this session
  evidence_hash_mismatch   recomputing the evidence digest
  issuer_key_untrusted     the signing key against a pinned set at all
  receipt_from_future      a receipt issued after the verification time
  decision_invalid         refusing to read an unknown verb as accept or reject

Two are load-bearing for the trust model rather than tidiness. Without
issuer_key_untrusted a receipt authenticates itself, since a signature verifies
against whatever key it names and only a pinned set decides whether that key was
entitled to speak. Without evidence_hash_mismatch the signature covers a digest
whose document can be swapped, because the receipt signs the digest and not the
evidence body.

One fixture per rule, each triggering exactly that rule and nothing else, so a
failure names the check that broke. They pin their own deterministic test key:
the private half of the key behind 01-09 is not published, and each fixture
already carries its own trusted_issuer_keys. gen_rule_coverage_vectors.py
regenerates the set byte-for-byte and only public JWKs appear in the files.

Found by walking the verifier's source for every failure code it can emit and
comparing that against the codes the fixtures expect, rather than by reading the
set and guessing what was missing.

Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant