Correct mutation-score explanation with a reduced-suite regression

README.md

@@ -50,9 +50,9 @@ configurable candidate commands:
 | `robot-evidence-review` | Report on attributed recording data | Coordinate and clock transforms, missing observations, source attribution | 6 | 1 |
 
 Every pack detects every declared fault: 21 faults, 21 detected. Six of the 21 are detected by a single case each, so the
-suite would lose them if that one case were weakened — which a mutation score of
-1.0 does not tell you. Every audit now reports these detection margins beside
-the score.
+suite would lose coverage if that case were removed or stopped detecting its
+fault. A fresh audit would then lower the mutation score. Detection margins
+identify these dependencies before a change, alongside the current score.
 [How this relates to hack-verifiable environments](docs/methodology.md#relation-to-hack-verifiable-environments).
 
 v0.6 adds **Python and JavaScript workspace templates for both tasks**.

README.zh-CN.md

@@ -40,7 +40,7 @@ EvalArc 关注智能体实际完成的结果，以及支撑评分结论的证据
 | `support-routing` | 策略通过工具操作模拟工单 | 路由、精确备注、条件关闭、无关数据保护与协议完成 | 7 | 2 |
 | `robot-evidence-review` | 基于有出处的记录数据出报告 | 坐标与时钟换算、缺失观测、来源归属 | 6 | 1 |
 
-三个任务包都检出了全部声明缺陷：21 个缺陷，21 个检出。但其中有六个各自只靠一个用例检出，那个用例一旦被弱化，对应缺陷就会失去覆盖——而 1.0 的变异分数不会告诉你这件事。现在每次审计都会在分数旁报告这些检出余量。[与 hack-verifiable environments 的关系](docs/methodology.md#relation-to-hack-verifiable-environments)。
+三个任务包都检出了全部声明缺陷：21 个缺陷，21 个检出。其中六个各自只靠一个用例检出；删除该用例，或使其无法再检出对应缺陷，就会失去这部分覆盖，重新审计的变异分数也会下降。检出余量在改动之前就能指出这些依赖，与当前分数一起报告。[与 hack-verifiable environments 的关系](docs/methodology.md#relation-to-hack-verifiable-environments)。
 
 v0.6 为两个任务都提供 **Python 和 JavaScript 工作区模板**：
 `init --language javascript` 生成起步代码，添加 `--reference` 生成脚本对照；

docs/methodology.md

@@ -71,11 +71,11 @@ model is being ranked.
 
 A mutation score of 1.0 says every declared fault was caught. It does not say
 how narrowly. Each mutant therefore reports `detection_margin`, the number of
-cases that independently failed on it, and the audit reports the weakest margin
+distinct cases that failed on its target dimension, and the audit reports the weakest margin
 across the pack, the faults caught by exactly one case, and the cases that are
 the sole detector of some fault.
 
-The measured margins for the two shipped packs, from real runs:
+The measured margins for the three shipped packs, from real runs:
 
 | Pack | Declared faults | Detected | Weakest margin | Caught by a single case |
 | --- | ---: | ---: | ---: | --- |
@@ -86,15 +86,24 @@ The measured margins for the two shipped packs, from real runs:
 All three packs score 1.0. Six of the twenty-one declared faults rest on a
 single case each, and their sole detectors are `cas-type-sensitivity`,
 `reject-and-continue`, `rollback-batch`, `retry-after-commit`,
-`retry-before-commit` and `incomplete-recording`. Weaken or drop any one of
-those six cases and the corresponding fault becomes invisible while the mutation
-score still reads 1.0. That is the number worth publishing next to a perfect
-score, and it is a statement about this suite's own coverage, not about any
-candidate.
+`retry-before-commit` and `incomplete-recording`. Removing a sole detector loses
+that fault's recorded coverage. A weakening loses coverage only if it stops
+detecting the fault. Re-running the same declared controls against such a changed,
+valid suite lowers its mutation score; only an old, unrecomputed report would
+still say 1.0. Margins expose this dependency before the change, while the current
+score is still perfect.
+
+For example, removing `cas-type-sensitivity` from `durable-kv` and re-running
+the controls with seed 17 leaves `boolean-equals-one` undetected: the score
+changes from 8/8 to 7/8 (0.875), and the weakest margin becomes zero. The
+regression in `tests/test_grading.py` executes this reduced suite. This is a
+check of the declared fault models, not a model or reward-hacking result.
 
 A margin counts distinct cases, not case runs. The same case failing under two
 seeds is one detector; counting runs would double every margin per added seed
 and make a suite look more robust for changing nothing.
+Distinct case IDs do not establish statistical independence or different
+underlying failure mechanisms.
 
 The margins above are identical under the Python and the JavaScript reference
 implementations, for all three packs. That is what should happen if a margin

docs/outreach/hve-correspondence.md

@@ -1,100 +1,79 @@
-# Draft note to the hack-verifiable environments authors
+# Research note: sensitivity checks for deterministic hack labels
 
-**Status: submitted 2026-09-15** as [issue #4](https://github.com/MajoRoth/hack-verifiable-environments/issues/4), on the maintainer's instruction. This is a research
-note about a correspondence between two lines of work, not a request for a link or a mention.
-Nothing here claims review, agreement or endorsement by the HVE authors.
+**Submitted:** [HVE issue #4](https://github.com/MajoRoth/hack-verifiable-environments/issues/4).
+The existing issue was corrected after review, including the mutation-score
+explanation and acknowledgement of upstream onset tests. The text below records
+the reviewed question; no author agreement or endorsement is claimed.
+[Publication log](status.md).
 
-Target: <https://github.com/MajoRoth/hack-verifiable-environments> (issues enabled). Papers:
-[arXiv:2605.20744v1](https://arxiv.org/abs/2605.20744v1) introducing HVE, and
-[arXiv:2608.22103v1](https://arxiv.org/abs/2608.22103v1) applying it to Terminal Bench.
-
----
+Target: [Hack-Verifiable Environments](https://github.com/MajoRoth/hack-verifiable-environments).
+Background: [HVE, arXiv:2605.20744v1](https://arxiv.org/abs/2605.20744v1) and
+[HVTB, arXiv:2608.22103v1](https://arxiv.org/abs/2608.22103v1).
+The repository has issues enabled and no Discussions at the time of review.
+Check for an existing submission before sending; do not post the same note in
+several channels.
 
 ## Suggested title
 
-A grader-side dual of HVE: planting faults in submissions to measure detector sensitivity
+Question: mutation-checking the existing hack-onset regression tests
 
 ## Suggested body
 
-Maintainer disclosure: I maintain EvalArc, an independent MIT-licensed research preview
-developed with AI assistance. This note was prepared by an assistant on the maintainer's
-behalf. I am not affiliated with the HVE work and claim no agreement from its authors. I am
-raising a correspondence and one measurement that may be useful, and I want to state the
-difference precisely rather than imply an equivalence.
-
-### The correspondence, and where it stops
-
-HVE embeds a detectable hacking opportunity in the environment and measures whether the agent
-exploits it. EvalArc embeds a declared fault in the submission and measures whether the
-evaluation's own checks catch it.
-
-|  | HVE / HVTB | EvalArc audit |
-| --- | --- | --- |
-| What is planted | A detectable hacking opportunity, in the environment | A declared fault, in the submission |
-| Subject measured | The agent | The grader and its checks |
-| Question | Does the agent exploit it? | Does the evaluation catch it, and by how much? |
-| Reported | Reward-hacking rate across models | Detection of every declared fault, with margins |
-
-EvalArc is not an HVE implementation and does not measure reward hacking. Both lines plant
-things so that measurement can be automatic and deterministic rather than resting on trajectory
-inspection or an LLM judge, which is the unreliability your 2608.22103 abstract names directly.
-They probe opposite directions of the same failure: an evaluation signal that has come apart
-from intent.
-
-### Why the grader side may matter to the agent side
-
-HVE takes for granted that a planted hack is detectable by construction, which is what makes an
-exploitation rate meaningful. That assumption is exactly what the grader-side measurement puts a
-number on, and the number turned out to be less comfortable than a headline suggests.
-
-Across three task packs, every declared fault is detected: 21 of 21, mutation score 1.0. That
-reads like a grader with nothing to worry about. Aggregating which cases caught each fault
-shows something else: **six of the twenty-one are caught by exactly one case each.** Remove or
-loosen that single case and the fault becomes undetected while the score still reads 1.0.
-
-In HVE terms, a fault whose detectability rests on one check is a fault that would become an
-*undetected* hack if that check were absent or weakened. So a margin is a measure of how much
-of an evaluation's integrity depends on individual checks, and a suite can be at 1.0 while
-sitting one edit away from blindness in six places.
-
-One implementation detail that cost us a wrong number first: margins have to count distinct
-cases, not case runs. With two seeds the raw failing list for one mutant is six entries over
-three cases, so counting runs would double every margin per added seed and a suite would look
-twice as robust for changing nothing. A single-seed test cannot tell the two apart, which is
-how it got past us.
-
-### If any of it is useful
-
-The margin computation is about forty lines and reads the audit output that already exists; it
-is MIT licensed and reusable. The comparable numbers are in
-[the methodology](https://github.com/noteflowai/evalarc/blob/main/docs/methodology.md#relation-to-hack-verifiable-environments),
-which cites both papers with the conservative venue handling the project's citation record
-uses, and states the difference above rather than blurring it.
-
-What I would find genuinely useful in return, if you have a view: whether the detection margin
-of a planted hack is something HVE could report alongside the hacking rate. A hack that only
-one check would have caught is arguably a weaker instrument than one that several would, and if
-so the two measurements are complements in a stronger sense than a table of differences.
-
-Scope, stated plainly: EvalArc runs scripted controls, not models. It ranks nothing, bounds
-nothing about reward hacking, and its denominator is a pack's declared fault models rather than
-the space of possible exploits. No model results, RL results or benchmark scores are claimed.
-
----
-
-## Verification behind the claims above
-
-| Claim | How it was checked |
+Maintainer disclosure: I maintain EvalArc, an independent MIT-licensed research
+preview developed with AI assistance. This question was prepared by an
+assistant on my behalf. I am not affiliated with the HVE authors.
+
+I read the onset tests merged in
+[#2](https://github.com/MajoRoth/hack-verifiable-environments/pull/2).
+They already cover clean episodes, onset at step zero, first-versus-last firing,
+and reset behavior. The question is about measuring the sensitivity of those
+existing assertions to deliberate changes in the instrumentation.
+
+Would a small mutation check be useful alongside those tests? Two concrete
+fault models would be replacing a missing onset with zero, and overwriting the
+first onset when the hook fires again. The existing clean-episode/step-zero
+tests and first-firing test appear intended to catch these respectively.
+Recording which distinct tests reject each perturbation could make that
+relationship explicit. I have inspected the tests, but have not run those
+perturbations in your environment.
+
+The motivation comes from a different setup. EvalArc's audit plants declared
+faults in scripted submissions and checks whether the grader detects them.
+It does not plant hacking opportunities in an environment, and is not an HVE
+implementation or a measurement of model reward hacking.
+
+Our three development task packs detect all 21 declared faults (8 + 7 + 6),
+but six are each detected by exactly one distinct case. This led us to report
+detecting case IDs beside the mutation score. The margin counts distinct
+cases, not repeated seed executions, and makes no statistical-independence
+claim.
+
+We also corrected an erroneous explanation of this result: removing the sole
+detector does **not** preserve a freshly computed mutation score of 1.0. In a
+real reduced-suite run at seed 17, removing `cas-type-sensitivity` leaves the
+`boolean-equals-one` fault undetected, changing the durable-kv result from 8/8
+to 7/8 (0.875), with weakest margin zero. An old report could remain stale;
+a fresh audit exposes the regression.
+
+For HVE, the units would be instrumentation faults and the existing labeled
+trajectory tests, rather than directly transplanting our grader-case margin.
+It would be useful to know whether there is already such a check or whether
+these fault models are the wrong abstraction for the instrumentation.
+
+This is a methodological question, not evidence that HVE has a detector bug:
+we have not evaluated your instrumentation or run HVE model experiments.
+The small reproduction and its limits are in
+[our methodology](https://github.com/noteflowai/evalarc/blob/main/docs/methodology.md#detection-margins)
+and [the reduced-suite test](https://github.com/noteflowai/evalarc/blob/main/tests/test_grading.py).
+If there is already a preferred test or discussion location, a pointer would
+be useful; there is no request for a listing, citation or endorsement.
+
+## Verification
+
+| Claim | Evidence |
 | --- | --- |
-| 21 of 21 declared faults detected across three packs | `audit` on `durable-kv`, `support-routing` and `robot-evidence-review`, 8 + 7 + 6, each reporting a mutation score of 1.0 |
-| Six rest on a single case each | The `detection.single_case_detections` list across the three packs: three, two and one entry |
-| Removing one case would lose the fault at an unchanged score | The sole detectors are named per pack in `detection.sole_detector_cases` |
-| Margins count distinct cases, not runs | A two-seed audit reproduces identical margins, and the raw failing list for `ack-without-work` is six entries over three cases |
-| Identical under both reference implementations | The same margins under the Python and JavaScript references, for all three packs |
-
-## What this deliberately does not do
-
-- It does not ask for a link, a citation or a mention.
-- It does not describe EvalArc as measuring reward hacking, and says the opposite.
-- It opens one issue. If a discussion thread or an email suits the authors better, that is where
-  this should go instead of being duplicated.
+| Three packs detect 21 declared faults | Saved audits: 8 + 7 + 6; original JSON retained |
+| Six faults have one detecting case each | Three, two and one in `detection.single_case_detections` |
+| A reduced suite lowers the recomputed score | `test_removing_sole_detector_lowers_the_recomputed_mutation_score` runs real scripted controls |
+| Repeated seeds must not inflate margins | Existing two-seed regression counts distinct failing case IDs |
+| This is not a reward-hacking measurement | No HVE trajectories, model calls or detector-performance claims in this reproduction |

docs/outreach/status.md

@@ -6,23 +6,21 @@ Submitted [issue #4](https://github.com/MajoRoth/hack-verifiable-environments/is
 maintainer's explicit instruction, and included EvalArc in a finding-led thread from
 [@glay_oneai](https://x.com/glay_oneai/status/2099745359224578381).
 
-The note states the correspondence and where it stops. HVE plants a hack in the environment and
-measures whether the agent exploits it; an audit here plants a fault in the submission and measures
-whether the checks catch it. It says twice that EvalArc is not an HVE implementation and does not
-measure reward hacking.
-
-The substance is the assumption their measurement rests on. An exploitation rate is meaningful
-because a planted hack is detectable by construction, and the grader-side measurement puts a number
-on that: 21 of 21 declared faults detected at a mutation score of 1.0, with six of the twenty-one
-caught by exactly one case each. It asks a question back, whether a planted hack's own detection
-margin is worth reporting beside the hacking rate, and volunteers the implementation error that
-produced a wrong number first, namely counting case runs instead of distinct cases.
-
-Numbers were re-verified against the current main before submission rather than copied from an
-earlier session: 8/8, 7/7 and 6/6, three plus two plus one single-case detections, weakest margin
-one in every pack. No link, citation or mention was requested; the issue offers to move to a
-discussion thread or email rather than duplicate itself. Nothing is claimed about the authors'
-agreement.
+The existing issue was corrected after review. A fresh audit does not keep a
+mutation score of 1.0 after its sole detector is removed: the reduced durable-kv
+suite at seed 17 detects 7/8 faults (0.875), with weakest margin zero. A real
+regression test executes that case, and the methodology states the scope.
+
+The current question acknowledges the onset tests already merged in upstream #2
+and asks about checking those tests with deliberate instrumentation mutations.
+EvalArc's 21 declared faults and six single-case dependencies motivate the
+question; they are not evidence about HVE's detector. No HVE model experiment,
+detector-performance result or author agreement is claimed.
+
+The [reviewed note](hve-correspondence.md) preserves the distinction between
+scripted grader audits and reward-hacking measurements. The X thread above is
+an external publication; this correction edits the GitHub issue and repository
+record, and does not claim that X posts were edited.
 
 
 ## 0.8.0 suite handoff publication · 2026-09-14

src/evalarc/audit.py

@@ -113,9 +113,9 @@ def observer(control: str) -> EventCallback | None:
                     "name": name,
                     "target_dimension": target,
                     "killed": result["valid"] and bool(failures),
-                    # How many cases caught this fault independently. A detected
-                    # fault with a margin of one is a deleted case away from
-                    # undetected, while the mutation score still reads 1.0.
+                    # Distinct cases detecting the target fault. Removing its
+                    # sole detector lowers a recomputed mutation score; the
+                    # margin identifies that dependency before the change.
                     "detection_margin": len(set(failures)) if result["valid"] else None,
                     "valid": result["valid"],
                     "score": result["score"],
@@ -127,8 +127,7 @@ def observer(control: str) -> EventCallback | None:
     killed = sum(row["killed"] for row in rows)
     valid = reference["valid"] and all(row["valid"] for row in rows)
     margins = {row["name"]: row["detection_margin"] for row in rows if row["valid"]}
-    # A case that is the only detector of some fault cannot be removed or
-    # loosened without losing coverage the mutation score still claims.
+    # Identify cases whose removal would lose coverage of a declared fault.
     sole_detectors = sorted(
         {row["failing_cases"][0] for row in rows if row["killed"] and row["detection_margin"] == 1}
     )