evalmut/vac is pending in the VAC registry as of vac-protocol@98fa81d. The structural verifier refuses it with:
evidence-unchecked: dogfood_fixtures.json, promptfoo_fixtures.json:
listed in evidence but read by no check
Both files are declared as evidence in the bundle manifest, and no check in the bundle reads either one. This is an evidence-closure failure rather than a wrong number: nothing in the bundle is contradicted, but two artifacts are shipped as if they support the claim while nothing actually consults them.
That distinction matters for the fix. Unlike the model-drift bundle, there is no reason to believe any published figure is wrong. The claim is under-supported in a specific, named way, not false.
Two honest repairs, and they are not equivalent:
- Add a check that actually reads them, if the fixtures are meant to be load-bearing. Then the evidence list and the checks agree because the check consults the file.
- Remove them from the evidence list, if they are inputs the checks do not need to re-read. Then the bundle stops claiming support it does not draw on.
Pick whichever is true of the fixtures. What must not happen is relaxing the verifier so an unread evidence file passes: evidence-unchecked exists precisely to stop a bundle from padding its evidence list with files nobody reads, which is a close cousin of the vacuous-pass problem evalmut itself was built to detect.
Worth noting while this is open. The portfolio currently states evalmut figures that have moved since they were written: 18 mutation operators (now 25), 32/35 with three holes (now 42/46 with four), 5/11 with six holes (now 9/17 with eight), and 97 tests (now 284). Those are separate from this issue and are being handled in the portfolio claim audit, but a regenerated bundle is the artifact those corrections should be sourced from, so the two are worth sequencing together.
Until a regenerated bundle verifies, the registry stays at 9 accepted and 2 pending.
Related: the registry authority gap is tracked separately in egnaro9/vac-protocol#10.
evalmut/vacis pending in the VAC registry as ofvac-protocol@98fa81d. The structural verifier refuses it with:Both files are declared as evidence in the bundle manifest, and no check in the bundle reads either one. This is an evidence-closure failure rather than a wrong number: nothing in the bundle is contradicted, but two artifacts are shipped as if they support the claim while nothing actually consults them.
That distinction matters for the fix. Unlike the model-drift bundle, there is no reason to believe any published figure is wrong. The claim is under-supported in a specific, named way, not false.
Two honest repairs, and they are not equivalent:
Pick whichever is true of the fixtures. What must not happen is relaxing the verifier so an unread evidence file passes:
evidence-uncheckedexists precisely to stop a bundle from padding its evidence list with files nobody reads, which is a close cousin of the vacuous-pass problem evalmut itself was built to detect.Worth noting while this is open. The portfolio currently states evalmut figures that have moved since they were written:
18 mutation operators(now 25),32/35 with three holes(now 42/46 with four),5/11 with six holes(now 9/17 with eight), and97 tests(now 284). Those are separate from this issue and are being handled in the portfolio claim audit, but a regenerated bundle is the artifact those corrections should be sourced from, so the two are worth sequencing together.Until a regenerated bundle verifies, the registry stays at 9 accepted and 2 pending.
Related: the registry authority gap is tracked separately in egnaro9/vac-protocol#10.