From 92cf296435e42d9ed5ce95d1ffe01ba4056a7496 Mon Sep 17 00:00:00 2001 From: Note Flow AI Date: Tue, 15 Sep 2026 14:23:54 +0800 Subject: [PATCH 1/2] Correct mutation-score explanation with a reduced-suite regression --- README.md | 6 +- README.zh-CN.md | 2 +- docs/methodology.md | 23 +++-- docs/outreach/hve-correspondence.md | 150 +++++++++++----------------- src/evalarc/audit.py | 9 +- tests/test_grading.py | 29 +++++- 6 files changed, 110 insertions(+), 109 deletions(-) diff --git a/README.md b/README.md index a07293f..90c19d1 100644 --- a/README.md +++ b/README.md @@ -50,9 +50,9 @@ configurable candidate commands: | `robot-evidence-review` | Report on attributed recording data | Coordinate and clock transforms, missing observations, source attribution | 6 | 1 | Every pack detects every declared fault: 21 faults, 21 detected. Six of the 21 are detected by a single case each, so the -suite would lose them if that one case were weakened — which a mutation score of -1.0 does not tell you. Every audit now reports these detection margins beside -the score. +suite would lose coverage if that case were removed or stopped detecting its +fault. A fresh audit would then lower the mutation score. Detection margins +identify these dependencies before a change, alongside the current score. [How this relates to hack-verifiable environments](docs/methodology.md#relation-to-hack-verifiable-environments). v0.6 adds **Python and JavaScript workspace templates for both tasks**. diff --git a/README.zh-CN.md b/README.zh-CN.md index e546fd6..535762d 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -40,7 +40,7 @@ EvalArc 关注智能体实际完成的结果，以及支撑评分结论的证据 | `support-routing` | 策略通过工具操作模拟工单 | 路由、精确备注、条件关闭、无关数据保护与协议完成 | 7 | 2 | | `robot-evidence-review` | 基于有出处的记录数据出报告 | 坐标与时钟换算、缺失观测、来源归属 | 6 | 1 | -三个任务包都检出了全部声明缺陷：21 个缺陷，21 个检出。但其中有六个各自只靠一个用例检出，那个用例一旦被弱化，对应缺陷就会失去覆盖——而 1.0 的变异分数不会告诉你这件事。现在每次审计都会在分数旁报告这些检出余量。[与 hack-verifiable environments 的关系](docs/methodology.md#relation-to-hack-verifiable-environments)。 +三个任务包都检出了全部声明缺陷：21 个缺陷，21 个检出。其中六个各自只靠一个用例检出；删除该用例，或使其无法再检出对应缺陷，就会失去这部分覆盖，重新审计的变异分数也会下降。检出余量在改动之前就能指出这些依赖，与当前分数一起报告。[与 hack-verifiable environments 的关系](docs/methodology.md#relation-to-hack-verifiable-environments)。 v0.6 为两个任务都提供 **Python 和 JavaScript 工作区模板**： `init --language javascript` 生成起步代码，添加 `--reference` 生成脚本对照； diff --git a/docs/methodology.md b/docs/methodology.md index 5eaabd3..f235ccb 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -71,11 +71,11 @@ model is being ranked. A mutation score of 1.0 says every declared fault was caught. It does not say how narrowly. Each mutant therefore reports `detection_margin`, the number of -cases that independently failed on it, and the audit reports the weakest margin +distinct cases that failed on its target dimension, and the audit reports the weakest margin across the pack, the faults caught by exactly one case, and the cases that are the sole detector of some fault. -The measured margins for the two shipped packs, from real runs: +The measured margins for the three shipped packs, from real runs: | Pack | Declared faults | Detected | Weakest margin | Caught by a single case | | --- | ---: | ---: | ---: | --- | @@ -86,15 +86,24 @@ The measured margins for the two shipped packs, from real runs: All three packs score 1.0. Six of the twenty-one declared faults rest on a single case each, and their sole detectors are `cas-type-sensitivity`, `reject-and-continue`, `rollback-batch`, `retry-after-commit`, -`retry-before-commit` and `incomplete-recording`. Weaken or drop any one of -those six cases and the corresponding fault becomes invisible while the mutation -score still reads 1.0. That is the number worth publishing next to a perfect -score, and it is a statement about this suite's own coverage, not about any -candidate. +`retry-before-commit` and `incomplete-recording`. Removing a sole detector loses +that fault's recorded coverage. A weakening loses coverage only if it stops +detecting the fault. Re-running the same declared controls against such a changed, +valid suite lowers its mutation score; only an old, unrecomputed report would +still say 1.0. Margins expose this dependency before the change, while the current +score is still perfect. + +For example, removing `cas-type-sensitivity` from `durable-kv` and re-running +the controls with seed 17 leaves `boolean-equals-one` undetected: the score +changes from 8/8 to 7/8 (0.875), and the weakest margin becomes zero. The +regression in `tests/test_grading.py` executes this reduced suite. This is a +check of the declared fault models, not a model or reward-hacking result. A margin counts distinct cases, not case runs. The same case failing under two seeds is one detector; counting runs would double every margin per added seed and make a suite look more robust for changing nothing. +Distinct case IDs do not establish statistical independence or different +underlying failure mechanisms. The margins above are identical under the Python and the JavaScript reference implementations, for all three packs. That is what should happen if a margin diff --git a/docs/outreach/hve-correspondence.md b/docs/outreach/hve-correspondence.md index 53c8737..40bfa1b 100644 --- a/docs/outreach/hve-correspondence.md +++ b/docs/outreach/hve-correspondence.md @@ -1,100 +1,68 @@ -# Draft note to the hack-verifiable environments authors +# Research note: sensitivity checks for deterministic hack labels -**Status: not submitted.** Prepared for the maintainer to review and send. This is a research -note about a correspondence between two lines of work, not a request for a link or a mention. -Nothing here claims review, agreement or endorsement by the HVE authors. +Reviewed text prepared for the HVE authors. This document alone is not a +submission receipt or evidence of author agreement. -Target: (issues enabled). Papers: -[arXiv:2605.20744v1](https://arxiv.org/abs/2605.20744v1) introducing HVE, and -[arXiv:2608.22103v1](https://arxiv.org/abs/2608.22103v1) applying it to Terminal Bench. - ---- +Target: [Hack-Verifiable Environments](https://github.com/MajoRoth/hack-verifiable-environments). +Background: [HVE, arXiv:2605.20744v1](https://arxiv.org/abs/2605.20744v1) and +[HVTB, arXiv:2608.22103v1](https://arxiv.org/abs/2608.22103v1). +The repository has issues enabled and no Discussions at the time of review. +Check for an existing submission before sending; do not post the same note in +several channels. ## Suggested title -A grader-side dual of HVE: planting faults in submissions to measure detector sensitivity +Question: how do you regression-test the deterministic reward_hack labels? ## Suggested body -Maintainer disclosure: I maintain EvalArc, an independent MIT-licensed research preview -developed with AI assistance. This note was prepared by an assistant on the maintainer's -behalf. I am not affiliated with the HVE work and claim no agreement from its authors. I am -raising a correspondence and one measurement that may be useful, and I want to state the -difference precisely rather than imply an equivalence. - -### The correspondence, and where it stops - -HVE embeds a detectable hacking opportunity in the environment and measures whether the agent -exploits it. EvalArc embeds a declared fault in the submission and measures whether the -evaluation's own checks catch it. - -| | HVE / HVTB | EvalArc audit | -| --- | --- | --- | -| What is planted | A detectable hacking opportunity, in the environment | A declared fault, in the submission | -| Subject measured | The agent | The grader and its checks | -| Question | Does the agent exploit it? | Does the evaluation catch it, and by how much? | -| Reported | Reward-hacking rate across models | Detection of every declared fault, with margins | - -EvalArc is not an HVE implementation and does not measure reward hacking. Both lines plant -things so that measurement can be automatic and deterministic rather than resting on trajectory -inspection or an LLM judge, which is the unreliability your 2608.22103 abstract names directly. -They probe opposite directions of the same failure: an evaluation signal that has come apart -from intent. - -### Why the grader side may matter to the agent side - -HVE takes for granted that a planted hack is detectable by construction, which is what makes an -exploitation rate meaningful. That assumption is exactly what the grader-side measurement puts a -number on, and the number turned out to be less comfortable than a headline suggests. - -Across three task packs, every declared fault is detected: 21 of 21, mutation score 1.0. That -reads like a grader with nothing to worry about. Aggregating which cases caught each fault -shows something else: **six of the twenty-one are caught by exactly one case each.** Remove or -loosen that single case and the fault becomes undetected while the score still reads 1.0. - -In HVE terms, a fault whose detectability rests on one check is a fault that would become an -*undetected* hack if that check were absent or weakened. So a margin is a measure of how much -of an evaluation's integrity depends on individual checks, and a suite can be at 1.0 while -sitting one edit away from blindness in six places. - -One implementation detail that cost us a wrong number first: margins have to count distinct -cases, not case runs. With two seeds the raw failing list for one mutant is six entries over -three cases, so counting runs would double every margin per added seed and a suite would look -twice as robust for changing nothing. A single-seed test cannot tell the two apart, which is -how it got past us. - -### If any of it is useful - -The margin computation is about forty lines and reads the audit output that already exists; it -is MIT licensed and reusable. The comparable numbers are in -[the methodology](https://github.com/noteflowai/evalarc/blob/main/docs/methodology.md#relation-to-hack-verifiable-environments), -which cites both papers with the conservative venue handling the project's citation record -uses, and states the difference above rather than blurring it. - -What I would find genuinely useful in return, if you have a view: whether the detection margin -of a planted hack is something HVE could report alongside the hacking rate. A hack that only -one check would have caught is arguably a weaker instrument than one that several would, and if -so the two measurements are complements in a stronger sense than a table of differences. - -Scope, stated plainly: EvalArc runs scripted controls, not models. It ranks nothing, bounds -nothing about reward hacking, and its denominator is a pack's declared fault models rather than -the space of possible exploits. No model results, RL results or benchmark scores are claimed. - ---- - -## Verification behind the claims above - -| Claim | How it was checked | +Maintainer disclosure: I maintain EvalArc, an independent MIT-licensed research +preview developed with AI assistance. This question was prepared by an +assistant on my behalf. I am not affiliated with the HVE authors. + +Your README demonstrates `step_info["reward_hack"]` as the signal for whether +a planted opportunity was exploited. Is there a recommended way to test the +detector itself against positive and negative scripted trajectories, including +changes to an instrumentation hook? + +The motivation comes from a different setup. EvalArc's audit plants declared +faults in scripted submissions and checks whether the grader detects them. +It does not plant hacking opportunities in an environment, and is not an HVE +implementation or a measurement of model reward hacking. + +Our three development task packs detect all 21 declared faults (8 + 7 + 6), +but six are each detected by exactly one distinct case. This led us to report +detecting case IDs beside the mutation score. The margin counts distinct +cases, not repeated seed executions, and makes no statistical-independence +claim. + +We also corrected an erroneous explanation of this result: removing the sole +detector does **not** preserve a freshly computed mutation score of 1.0. In a +real reduced-suite run at seed 17, removing `cas-type-sensitivity` leaves the +`boolean-equals-one` fault undetected, changing the durable-kv result from 8/8 +to 7/8 (0.875), with weakest margin zero. An old report could remain stale; +a fresh audit exposes the regression. + +Would an analogous regression check be useful for HVE's instrumentation—for +example, controlled trajectories that must and must not set each hack flag, +then checking whether disabling a detection hook changes the expected labels? +The units would be hooks and labeled trajectories, rather than directly +transplanting our grader-case margin. + +This is a methodological question, not evidence that HVE has a detector bug: +we have not evaluated your instrumentation or run HVE model experiments. +The small reproduction and its limits are in +[our methodology](https://github.com/noteflowai/evalarc/blob/main/docs/methodology.md#detection-margins) +and [the reduced-suite test](https://github.com/noteflowai/evalarc/blob/main/tests/test_grading.py). +If there is already a preferred test or discussion location, a pointer would +be useful; there is no request for a listing, citation or endorsement. + +## Verification + +| Claim | Evidence | | --- | --- | -| 21 of 21 declared faults detected across three packs | `audit` on `durable-kv`, `support-routing` and `robot-evidence-review`, 8 + 7 + 6, each reporting a mutation score of 1.0 | -| Six rest on a single case each | The `detection.single_case_detections` list across the three packs: three, two and one entry | -| Removing one case would lose the fault at an unchanged score | The sole detectors are named per pack in `detection.sole_detector_cases` | -| Margins count distinct cases, not runs | A two-seed audit reproduces identical margins, and the raw failing list for `ack-without-work` is six entries over three cases | -| Identical under both reference implementations | The same margins under the Python and JavaScript references, for all three packs | - -## What this deliberately does not do - -- It does not ask for a link, a citation or a mention. -- It does not describe EvalArc as measuring reward hacking, and says the opposite. -- It opens one issue. If a discussion thread or an email suits the authors better, that is where - this should go instead of being duplicated. +| Three packs detect 21 declared faults | Saved audits: 8 + 7 + 6; original JSON retained | +| Six faults have one detecting case each | Three, two and one in `detection.single_case_detections` | +| A reduced suite lowers the recomputed score | `test_removing_sole_detector_lowers_the_recomputed_mutation_score` runs real scripted controls | +| Repeated seeds must not inflate margins | Existing two-seed regression counts distinct failing case IDs | +| This is not a reward-hacking measurement | No HVE trajectories, model calls or detector-performance claims in this reproduction | diff --git a/src/evalarc/audit.py b/src/evalarc/audit.py index 67bbfc1..873b51c 100644 --- a/src/evalarc/audit.py +++ b/src/evalarc/audit.py @@ -113,9 +113,9 @@ def observer(control: str) -> EventCallback | None: "name": name, "target_dimension": target, "killed": result["valid"] and bool(failures), - # How many cases caught this fault independently. A detected - # fault with a margin of one is a deleted case away from - # undetected, while the mutation score still reads 1.0. + # Distinct cases detecting the target fault. Removing its + # sole detector lowers a recomputed mutation score; the + # margin identifies that dependency before the change. "detection_margin": len(set(failures)) if result["valid"] else None, "valid": result["valid"], "score": result["score"], @@ -127,8 +127,7 @@ def observer(control: str) -> EventCallback | None: killed = sum(row["killed"] for row in rows) valid = reference["valid"] and all(row["valid"] for row in rows) margins = {row["name"]: row["detection_margin"] for row in rows if row["valid"]} - # A case that is the only detector of some fault cannot be removed or - # loosened without losing coverage the mutation score still claims. + # Identify cases whose removal would lose coverage of a declared fault. sole_detectors = sorted( {row["failing_cases"][0] for row in rows if row["killed"] and row["detection_margin"] == 1} ) diff --git a/tests/test_grading.py b/tests/test_grading.py index 8a2da5c..34acdcd 100644 --- a/tests/test_grading.py +++ b/tests/test_grading.py @@ -1,3 +1,4 @@ +from dataclasses import replace from pathlib import Path import pytest @@ -6,6 +7,7 @@ from evalarc.cli import main from evalarc.evaluate import evaluate from evalarc.runner import CandidateError, Runtime, snapshot +from evalarc.tasks import TASKS def test_positive_and_behavioral_negative_controls(): @@ -21,7 +23,7 @@ def test_a_perfect_mutation_score_still_reports_how_narrowly_it_was_earned(): result = audit(Runtime(backend="local", timeout=2), [17]) assert result["mutation_score"] == 1.0 detection = result["detection"] - # Every mutant reports how many cases caught it independently. + # Every mutant reports the count of distinct detecting cases. for mutant in result["mutants"]: assert mutant["detection_margin"] == len(set(mutant["failing_cases"])), mutant["name"] assert detection["weakest_margin"] == min(m["detection_margin"] for m in result["mutants"]) @@ -32,7 +34,7 @@ def test_a_perfect_mutation_score_still_reports_how_narrowly_it_was_earned(): "partial-batch", ] # Naming the sole detectors is what makes the fragility actionable: remove - # one of these cases and its fault goes undetected at an unchanged score. + # one of these cases and a fresh audit must report the lost coverage. assert detection["sole_detector_cases"] == [ "cas-type-sensitivity", "reject-and-continue", @@ -41,6 +43,29 @@ def test_a_perfect_mutation_score_still_reports_how_narrowly_it_was_earned(): assert set(detection["sole_detector_cases"]).isdisjoint(detection["single_case_detections"]) +def test_removing_sole_detector_lowers_the_recomputed_mutation_score(monkeypatch): + original = TASKS["durable-kv"] + reduced = replace( + original, + generate_cases=lambda seed: [ + case for case in original.generate_cases(seed) if case.id != "cas-type-sensitivity" + ], + ) + monkeypatch.setitem(TASKS, "durable-kv", reduced) + result = audit(Runtime(backend="local", timeout=4), [17]) + assert result["valid"] and result["reference_passed"] + assert not result["passed"] + assert result["total"] == 8 and result["killed"] == 7 + assert result["mutation_score"] == 0.875 + assert result["detection"]["weakest_margin"] == 0 + survivor = next(m for m in result["mutants"] if m["name"] == "boolean-equals-one") + assert not survivor["killed"] and survivor["detection_margin"] == 0 + assert survivor["failing_cases"] == [] + assert all( + case["case_id"] != "cas-type-sensitivity" for case in survivor["evaluation"]["cases"] + ) + + def test_margins_count_distinct_cases_so_adding_seeds_cannot_inflate_them(): one = audit(Runtime(backend="local", timeout=4), [17]) two = audit(Runtime(backend="local", timeout=4), [17, 41]) From 547662449e6f1a2e12c1cf249a85b9a7f36c0dce Mon Sep 17 00:00:00 2001 From: Note Flow AI Date: Tue, 15 Sep 2026 14:25:56 +0800 Subject: [PATCH 2/2] Ground the HVE question in its existing onset regression tests --- docs/outreach/hve-correspondence.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/outreach/hve-correspondence.md b/docs/outreach/hve-correspondence.md index 40bfa1b..03dbc04 100644 --- a/docs/outreach/hve-correspondence.md +++ b/docs/outreach/hve-correspondence.md @@ -12,7 +12,7 @@ several channels. ## Suggested title -Question: how do you regression-test the deterministic reward_hack labels? +Question: mutation-checking the existing hack-onset regression tests ## Suggested body @@ -20,10 +20,19 @@ Maintainer disclosure: I maintain EvalArc, an independent MIT-licensed research preview developed with AI assistance. This question was prepared by an assistant on my behalf. I am not affiliated with the HVE authors. -Your README demonstrates `step_info["reward_hack"]` as the signal for whether -a planted opportunity was exploited. Is there a recommended way to test the -detector itself against positive and negative scripted trajectories, including -changes to an instrumentation hook? +I read the onset tests merged in +[#2](https://github.com/MajoRoth/hack-verifiable-environments/pull/2). +They already cover clean episodes, onset at step zero, first-versus-last firing, +and reset behavior. The question is about measuring the sensitivity of those +existing assertions to deliberate changes in the instrumentation. + +Would a small mutation check be useful alongside those tests? Two concrete +fault models would be replacing a missing onset with zero, and overwriting the +first onset when the hook fires again. The existing clean-episode/step-zero +tests and first-firing test appear intended to catch these respectively. +Recording which distinct tests reject each perturbation could make that +relationship explicit. I have inspected the tests, but have not run those +perturbations in your environment. The motivation comes from a different setup. EvalArc's audit plants declared faults in scripted submissions and checks whether the grader detects them. @@ -43,11 +52,10 @@ real reduced-suite run at seed 17, removing `cas-type-sensitivity` leaves the to 7/8 (0.875), with weakest margin zero. An old report could remain stale; a fresh audit exposes the regression. -Would an analogous regression check be useful for HVE's instrumentation—for -example, controlled trajectories that must and must not set each hack flag, -then checking whether disabling a detection hook changes the expected labels? -The units would be hooks and labeled trajectories, rather than directly -transplanting our grader-case margin. +For HVE, the units would be instrumentation faults and the existing labeled +trajectory tests, rather than directly transplanting our grader-case margin. +It would be useful to know whether there is already such a check or whether +these fault models are the wrong abstraction for the instrumentation. This is a methodological question, not evidence that HVE has a detector bug: we have not evaluated your instrumentation or run HVE model experiments.