Externally-sourced operators: liveness reporting, shape probe, cue-token insert

[egnaro9]

Nine commits. The catalog goes 18 to 20 operators, and three of them now cite a source outside my own repositories where yesterday none did.

Why

An audit of the operator catalog found that 16 of the 18 shipped operators trace their real_origin to graders in repos I own. The citation gate proves each operator reproduces something documented; it proves almost nothing about whether the population is representative of faults anyone else would care about missing. Two rounds of mining external sources produced the cards below. This PR ships the ones that run against today's EvalCase.

What landed

OperatorType.LIVENESS and score.forged_verdicts. A survival where the mutant writes the grader's own verdict channel does not mean this check missed this defect. It means the check has no liveness: its verdict can be authored by the thing it grades, so every result it has ever reported is unfalsifiable, including its passes. Reported apart from blind_spots because "fix the check" is the wrong instruction when the channel is the bug.

verdict_channel, a declared field on EvalCase. Folds BenchJack's payload class into inject_denylisted_tool rather than shipping a tenth operator. The denylist path probes the grader first and declines if it accepts, since the case's expected may not match the real denylist. The verdict channel cannot work that way: a grader that accepts a verdict write cannot tell you whether the write was legitimate output or a forged result. Wrongness comes from the author's declaration, like tolerates. Undeclared means decline.

garbage_answer gains a shape probe. The old battery only asked whether a grader accepts opaque nonsense. A grader that rejects bare tokens but accepts the same tokens capitalized and punctuated read as discriminating, when in fact it checks formatting. _GARBAGE3 is _GARBAGE's exact tokens wearing answer shape, so it adds no lexical surface and carries no digit.

spurious_cue_token_insert. Inverse of keyword_present_but_negated: there the cue survives in a sentence meaning the opposite, here in text meaning nothing. Both rest on the presence_check tag, where the author has declared the grep is a proxy for a property it cannot see.

append_grader_directed_suffix. Appends text arguing for the answer to a base that is already wrong, so polarity never depends on the suffix.

The result worth reading

The dogfood score went down: 91.4% (32/35, 3 holes) to 90.9% (40/44, 4 holes).

The denominator grew faster than the catches, which is the honest direction. An operator that only ever confirmed existing catches would be worth nothing. The new blind spot is a contains grader needing all of ['capital'] on a presence_check task: it passes opaque garbage carrying the token capital and nothing else, so the grep is the entire signal. That is the ARCT result reproduced against my own code.

append_grader_directed_suffix found nothing new, which is what a deliberately degraded static suffix should do against lexical graders. Recorded as a true negative rather than dressed up.

Decisions that could reasonably go the other way

garbage_answer stays SANITY. The null-model paper argues it should be KILL, since "asserts nothing about the answer" would be a false report against a judge that was genuinely persuaded. True for a judge, but this operator gates on _requires_content, which admits only lexical families and can never reach one. For every grader it can touch, passing the full disjoint battery does mean vacuity, and retyping it would mis-report genuine vacuity as a broken check.

append_grader_directed_suffix is degraded on purpose, and that asymmetry is written into its real_origin rather than a comment. A static sentence is not an optimized adversarial sequence and will not reproduce 90.8% ASR. A survival is meaningful; a catch proves very little.

Fixtures are not optional here. The parametrized defect test skips any operator without one, and a skip reads green. Shipping without them would have left both new operators never exercised while the suite still reported 97 passed.

Provenance

• BenchJack, arXiv:2605.12673 (Wang, Li, Mang, Cheung, Sen, Song; Berkeley): 219 distinct flaws across ten agent benchmarks
• Null models, arXiv:2410.07137: 86.5% LC win rate on AlpacaEval 2.0 from a constant reply shaped like the evaluator's scaffolding, against 0.6% naive
• ARCT, arXiv:1907.07355 (Niven & Kao, ACL 2019): 77% off the unigram "not"; the symmetric-cue control drops the identical model to chance
• Judge injection, arXiv:2403.17710 (Shi et al., ACM CCS 2024): 10.8% ASR-B to 90.8% ASR on MT-Bench

Every arXiv ID above was fetched and checked by hand, not taken from a summary.

Verification

99/99 tests pass. VAC bundle regenerated by emit_vac.py and byte-identical on re-emission. bundle-freshness green in CI.