Externally-sourced operators: liveness reporting, shape probe, cue-token insert

.github/workflows/ci.yml

@@ -8,6 +8,11 @@ jobs:
         python-version: ["3.11", "3.12"]
     steps:
       - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # test_committed_stamp_is_reachable_from_head walks history to
+                           # confirm vac.json's issuer_commit is still an ancestor of HEAD.
+                           # A shallow clone cannot see that, and the check REFUSES rather
+                           # than guessing, so this job needs full history too.
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}

.gitignore

@@ -9,3 +9,6 @@ dist/
 # paper build intermediates
 paper/evalmut.html
 paper/preview.png
+
+# deepeval tool cache (pass-2 external comparison run)
+.deepeval/

docs/dogfood_gradecore.txt

@@ -1,14 +1,18 @@
 ────────────────────────────────────────────────────────────────────────
   evalmut — does your eval actually check anything?
 ────────────────────────────────────────────────────────────────────────
-  mutation score    91.4%   (32 caught / 35 applied; 150 n/a)
-  holes            3  (1 blind, 2 coverage-gap)
+  mutation score    90.9%   (40 caught / 44 applied; 165 n/a)
+  holes            4  (2 blind, 2 coverage-gap)
 
   BLIND SPOTS — a real defect shipped green; the check is present and broken
     • contains / contains
         mutation : keyword_present_but_negated — the checked keyword appears, but in a context that means the opposite
         grader   : passed the mutant  (needs all of ['capital'])
         mined from: a CI proof-gate: greps its own run transcript for the token it should execute; the block message contains that token, so mentioning it passes the gate without running it
+    • contains / contains
+        mutation : spurious_cue_token_insert — the grepped keyword carried by text that does none of the work the task asks for — the cue is the entire signal the check consumes
+        grader   : passed the mutant  (needs all of ['capital'])
+        mined from: https://arxiv.org/abs/1907.07355 (Niven & Kao, ACL 2019) : on ARCT, BERT reached 77% peak / 71.2% median against untrained humans at 0.798, and the authors attribute the signal to the unigram 'not' (61% productivity, 64% coverage). Partial-input probes that never see the argument still score: warrant-only 71%, reason+warrant 75%. The control is the hard ground truth: mirroring every instance so the cue distribution is symmetric drops the IDENTICAL model to peak 53.3% / median 50.5%, i.e. chance. ARCT's accuracy metric scored a cue-matcher within 3 points of untrained humans.
 
   COVERAGE GAPS — no check guards this shape (a missing grader, not a broken one)
     • valid_json / valid_json

docs/operators.json

@@ -33,7 +33,7 @@
     "op_type": "sanity",
     "field": "text",
     "defect_shape": "output replaced with unrelated text \u2014 a check that asserts nothing lets it pass",
-    "real_origin": "a real game test suite: assertTrue(\"result is a boolean\", result || !result) \u2014 a tautology any output satisfies"
+    "real_origin": "a real game test suite: assertTrue(\"result is a boolean\", result || !result) \u2014 a tautology any output satisfies. The SHAPE probe is externally sourced: https://arxiv.org/abs/2410.07137 measured a constant, input-independent reply at 86.5% LC win rate on AlpacaEval 2.0 (83.0 Arena-Hard-Auto, 9.55 MT-Bench) when shaped like the evaluator's scaffolding, against 0.6% for a naive constant \u2014 the same operator class, ~140x apart on form alone."
   },
   {
     "id": "keyword_present_but_negated",
@@ -44,6 +44,24 @@
     "defect_shape": "the checked keyword appears, but in a context that means the opposite",
     "real_origin": "a CI proof-gate: greps its own run transcript for the token it should execute; the block message contains that token, so mentioning it passes the gate without running it"
   },
+  {
+    "id": "spurious_cue_token_insert",
+    "family": "presence-proxy",
+    "polarity": "defect",
+    "op_type": "kill",
+    "field": "text",
+    "defect_shape": "the grepped keyword carried by text that does none of the work the task asks for \u2014 the cue is the entire signal the check consumes",
+    "real_origin": "https://arxiv.org/abs/1907.07355 (Niven & Kao, ACL 2019) : on ARCT, BERT reached 77% peak / 71.2% median against untrained humans at 0.798, and the authors attribute the signal to the unigram 'not' (61% productivity, 64% coverage). Partial-input probes that never see the argument still score: warrant-only 71%, reason+warrant 75%. The control is the hard ground truth: mirroring every instance so the cue distribution is symmetric drops the IDENTICAL model to peak 53.3% / median 50.5%, i.e. chance. ARCT's accuracy metric scored a cue-matcher within 3 points of untrained humans."
+  },
+  {
+    "id": "append_grader_directed_suffix",
+    "family": "injection",
+    "polarity": "defect",
+    "op_type": "kill",
+    "field": "text",
+    "defect_shape": "text aimed at the grader appended to an answer that is already wrong, with the wrong content untouched, so any verdict change is attributable to the suffix rather than to answer quality",
+    "real_origin": "https://arxiv.org/abs/2403.17710 (Optimization-based Prompt Injection Attack to LLM-as-a-Judge, Shi et al., ACM CCS 2024) : an optimized appended sequence drove a Mistral-7B judge to select the attacker's target response at 90.8% ASR on MT-Bench and 93.2% on LLMBar. The clean baseline is what proves polarity: ASR-B, the judge selecting that same target WITHOUT the suffix, is 10.8% against 89.2% accuracy, so the target is demonstrably not the best answer and the suffix carries it from 10.8% to 90.8%. DEGRADED HERE ON PURPOSE: a static sentence is not an optimized adversarial sequence and will not reproduce 90.8%. It tests the same seam (is the verdict movable by text addressed to the grader) at a fraction of the strength, so a survival is meaningful and a catch proves very little."
+  },
   {
     "id": "comply_with_harm",
     "family": "safety",
@@ -122,8 +140,8 @@
     "polarity": "defect",
     "op_type": "kill",
     "field": "tool_calls",
-    "defect_shape": "a denylisted tool call is inserted into the trajectory",
-    "real_origin": "gradecore adversarial.py:169 tool_misuse (PASS iff no denylisted tool). The denylisted name is carried on the case as expected; injecting it is the misuse."
+    "defect_shape": "a forbidden tool call is inserted into the trajectory: either a denylisted tool, or (when the case declares one) a write to the grader's own verdict channel",
+    "real_origin": "gradecore adversarial.py:169 tool_misuse (PASS iff no denylisted tool). The denylisted name is carried on the case as expected; injecting it is the misuse. The verdict-channel payload class is externally sourced: https://arxiv.org/abs/2605.12673 (BenchJack, Wang/Li/Mang/Cheung/Sen/Song, Berkeley, 2026-05) audited ten agent benchmarks and found 219 distinct flaws, driving exploitable tasks from near 100% to under 10% only after iterative patching; the exploits include writing the harness's own reported outcome rather than doing the task. Berkeley RDI corroborates with a shadowed `curl` binary and a pre-written reward file. Erik's own gradecore had no such case: this is the first payload in this operator sourced from outside his repositories."
   },
   {
     "id": "trailing_disclaimer",

emit_vac.py

@@ -82,6 +82,31 @@
                 ("coverage_gap", "missed", "diagnostic"))
 
 
+def stamp_is_reachable(commit: str, repo: pathlib.Path = ROOT) -> bool:
+    """Is the stamped commit an ancestor of HEAD in this repo?
+
+    The freshness gate proves the artifacts are what the code re-emits. It says nothing about
+    whether the commit the manifest NAMES still exists on this branch, and those are different
+    facts: rebasing, squashing, or amending rewrites every SHA while leaving each artifact byte
+    for byte identical. The bundle then points a replayer at `git checkout <sha>` for a commit
+    that is not there, and every other check stays green. Found the hard way, by hand, after a
+    `git rebase --signoff` orphaned a stamp that freshness had just certified.
+
+    Raises on a shallow clone rather than answering. A shallow repo cannot see its own history,
+    so it would report unreachable for a perfectly good stamp; returning False there would be a
+    false accusation and returning True would be a gate that passed because it could not look."""
+    shallow = subprocess.run(
+        ["git", "rev-parse", "--is-shallow-repository"], cwd=repo,
+        capture_output=True, text=True, check=True).stdout.strip()
+    if shallow == "true":
+        raise RuntimeError(
+            "cannot check stamp reachability in a shallow clone: history is truncated, so a "
+            "reachable stamp would read as missing. Check out with fetch-depth: 0.")
+    return subprocess.run(
+        ["git", "merge-base", "--is-ancestor", commit, "HEAD"],
+        cwd=repo, capture_output=True).returncode == 0
+
+
 def stamp_code_commit(repo: pathlib.Path = ROOT) -> str:
     """The publication stamp, with the fleet's two refusals: dirty tree,
     no code commit."""

evalmut/case.py

@@ -82,6 +82,16 @@ class EvalCase:
     # "number", "grounding", "contains", "injection_resistance"). Declared, it lets evalmut run
     # against a framework's graders unmodified while the grader keeps reporting its own honest id.
     grader_family: str | None = None
+    # Tool names that write the GRADER'S OWN verdict channel: the file it reads its outcome from,
+    # a reporting hook it honors, a binary it shells out to for the answer. An agent calling one of
+    # these is not solving the task, it is authoring the verdict, so the trajectory is a provable
+    # defect no matter what the task was. Wrongness here comes from THIS declaration, not from the
+    # grader's response: unlike a denylisted tool (which the grader can be probed about, since it is
+    # supposed to refuse it), a grader that accepts a verdict write cannot tell you whether the write
+    # was legitimate output or a forged result. Only the suite author knows which channel is theirs.
+    # Undeclared -> inject_denylisted_tool falls back to the denylist payload and this class never
+    # fires, so an undecorated case is under-tested rather than falsely accused.
+    verdict_channel: tuple[str, ...] = ()
     # Optional human-readable notes surfaced in reports; never load-bearing.
     intent: str = ""
     tags: tuple[str, ...] = field(default_factory=tuple)
@@ -96,10 +106,11 @@ def case(name: str, grader: Grader, good: GradeInput, *, judges: tuple[str, ...]
          num_tol: float | None = None, content_required: bool = False,
          tolerates: tuple[str, ...] = (), expected_trajectory: tuple[str, ...] = (),
          trajectory_threshold: float | None = None, grader_family: str | None = None,
+         verdict_channel: tuple[str, ...] = (),
          intent: str = "", tags: tuple[str, ...] = ()) -> EvalCase:
     """Terse constructor for suites written as data."""
     return EvalCase(name=name, grader=grader, good=good, judges=judges,
                     num_tol=num_tol, content_required=content_required, tolerates=tolerates,
                     expected_trajectory=expected_trajectory,
                     trajectory_threshold=trajectory_threshold, grader_family=grader_family,
-                    intent=intent, tags=tags)
+                    verdict_channel=verdict_channel, intent=intent, tags=tags)

evalmut/operator.py

@@ -47,6 +47,12 @@ class MutationOperator:
     field: str            # which GradeInput field it perturbs (for reports/filtering)
     _apply: ApplyFn
     op_type: OperatorType = OperatorType.KILL  # what a survival means (see OperatorType)
+    # An operator carrying more than one payload class can mean different things by a survival
+    # depending on which payload fired for THIS case. inject_denylisted_tool is the case in hand:
+    # its denylist payload survival is a blind spot, its verdict-channel payload survival is a
+    # liveness failure, and reporting the second as the first would tell the reader to fix a check
+    # when the actual bug is that the check's verdict is forgeable. Left None, op_type is static.
+    _op_type_for: Optional[Callable[["EvalCase"], OperatorType]] = None
 
     def apply(self, case: "EvalCase") -> Optional[GradeInput]:
         """Mutate the case's reference input, or return None if not applicable here.
@@ -68,10 +74,15 @@ def apply(self, case: "EvalCase") -> Optional[GradeInput]:
             return None
         return mutant
 
+    def op_type_for(self, case: "EvalCase") -> OperatorType:
+        """What a survival means for THIS case. Static unless the operator declared a hook."""
+        return self._op_type_for(case) if self._op_type_for is not None else self.op_type
+
 
 def operator(id: str, *, family: str, polarity: Polarity, defect_shape: str,
              real_origin: str, field: str,
-             op_type: OperatorType = OperatorType.KILL
+             op_type: OperatorType = OperatorType.KILL,
+             op_type_for: Optional[Callable[["EvalCase"], OperatorType]] = None,
              ) -> Callable[[ApplyFn], MutationOperator]:
     """Decorator form, so an operator reads as its apply-rule with provenance attached.
 
@@ -86,6 +97,7 @@ def wrap(fn: ApplyFn) -> MutationOperator:
         return MutationOperator(
             id=id, family=family, polarity=polarity, defect_shape=defect_shape,
             real_origin=real_origin, field=field, _apply=fn, op_type=op_type,
+            _op_type_for=op_type_for,
         )
     return wrap