test: ten of thirty-three conformance failure paths are unguarded against regression

[lywinged]

None of these checks is wrong. They all behave correctly today. What follows is a
measurement of something narrower: whether this repository's own suite would notice if one
of them silently stopped working.

For 23 of the 33 failure paths, it would. For ten, it would not.

Why the question

trace-tests is what an implementer runs to establish conformance, so its output is a
certification claim. That makes a regression inside a conformance module different in kind
from a regression in ordinary code: it does not produce a wrong answer, it produces a
missing question, and implementations keep being stamped without the property being
checked.

Measurement

For every Finding(..., Status.FAIL, ...) construction in
src/trace_tests/modules/tr_*.py, rewrite the site so the check can never fail, run the
whole suite, count how many tests notice, restore. Measured against 7fc189c (0.4.1).

modules   7      checks 18 (TR-xxx-nnn)      sites 33

by check   15 of 18 verified,  3 unverified
by site    23 of 33 verified, 10 unverified

site margins   0→10   1→12   2→6   3→1   4→3   5→1

Counting by check understates it: a code emitted from three places can have one unguarded
failure path while its siblings are covered, and still count as verified.

Check	Enforces	Unguarded sites
TR-SIG-002	cnf.jwk is OKP/Ed25519 and carries x	2 of 2
TR-TXN-001	tool_transcript present at Level 2, is an object, hash is a well-formed digest	3 of 3
TR-TXN-002	call_count is a non-negative integer	1 of 1

TR-ANC-001, TR-SCA-001, TR-SIG-004 and TR-SIG-005 each have one unguarded site
alongside covered ones.

TR-TXN-001 is the one I would look at first

Three things stack:

1. It is the sole enforcement. tool_transcript is not in the schema's required
   array, so nothing else rejects a Level 2 record that omits it.
2. All three of its failure paths are unguarded.
3. The coverage looks present. Five files mention tool_transcript, and none of them
   exercises tr_txn.py:
   • test_level0.py::test_transcript_digest_when_present re-implements the digest check
     inline against the fixture rather than calling the module
   • two cases in test_level0_negative.py assert jsonschema.ValidationError, so they
     test the schema
   • unit/test_runner.py::test_level2_trace_format_includes_txn_and_anc asserts
     "TR-TXN" in results, which is that the module ran, not what it concluded

That third point is the reason this is a mutation run rather than a reading. Someone
scanning the test files would reasonably conclude the transcript requirement is well
covered.

I also checked whether these were known and deferred: the five xpass results are all
Level 2 hardware-TEE cases, unrelated to any of the above.

Separately: a hand-written enum has drifted

tests/test_level0.py:7 restates the schema's runtime.platform enum as a set literal.
It holds nine values; the schema holds ten. software-only is missing.

tr_rte.py:_VALID_PLATFORMS has all ten, so this is only in the self-test. It is latent —
no vector uses that platform — but on a Level 0 record carrying it:

tr_rte.check(record, level=0)   PASS      (what an implementer runs)
normative schema                accepts
tests/test_level0.py            FAIL

It would reject behaviour #16/#17 deliberately added. Small, and I mention it mostly
because it is the same shape as the rest: a copy that was correct when it was written.

Reproducing

Two scripts and the full report, in a fork of this repository so they run in place:
https://github.com/lywinged/trace-tests/tree/main/measurement

pip install -e ".[dev]" && pytest -q          # 118 passed, 5 xpassed
python measurement/scripts/mutate_modules.py  # exits 1 if any check is unguarded
python measurement/scripts/enum_drift.py      # exits 1 on any drifted enum

No dependencies beyond what the suite already needs. About a minute — 33 sites, one full
suite run each. Both scripts refuse to start rather than report success over a corpus they
cannot find, refuse a rewrite that did not change the file, and refuse to proceed if the
baseline is not green; each of those guards exists because its absence produced a wrong
answer while the scripts were being written.

Nothing in this repository was modified while measuring. Every mutation is reverted,
restoration is verified before the next site, and the checkout was re-confirmed green
afterwards.

I can open a PR with the scripts, or with fixtures closing the unguarded paths, or
neither — happy to be told this is not worth the weight.

What this does not establish

• Not that the checks are wrong. They are unguarded, which is a statement about the
  suite rather than about them.
• Not that implementations can skip them. An implementation still faces whatever the
  module does today.
• Only Status.FAIL sites were mutated, so a check passing for the wrong reason is
  outside what this sees.
• The margin counts failing tests, not independent ones. Five tests failing together
  because they share a fixture count as five here and are arguably one.

[github-actions]

❔ Contributor Check: UNKNOWN

Check	Result
Profile	UNKNOWN
Credential	LOW
Overall	UNKNOWN

Automated check by AgenTrust Contributor Check.