Connect audit coverage to case evidence across three task packs

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## 0.10.0 — 2026-09-14
 
+- Add a three-task audit coverage section with direct links from single-case dependencies to recorded fault evidence. Re-render all three reports from unchanged saved JSON, with accessible disclosures and layouts for narrow screens.
+- Include assessed surviving faults as zero in the weakest detection margin; environment failures remain unassessed.
+- Extend the Hugging Face casebook to all three task packs: 251 case rows, with the parent control's distinct-case detection margin. Preserve the original source records and distinguish reference controls with a null margin.
+
 - Report detection margins beside the mutation score: how many cases caught each declared fault, the weakest margin in the pack, the faults caught by exactly one case, and the cases that are the sole detector of some fault. All three packs score 1.0, and six of their 21 declared faults rest on a single case each; that fragility was previously invisible.
 - Count margins over distinct cases rather than case runs, so adding a seed cannot inflate them. Measured margins are identical under the Python and JavaScript references.
 - State the relation to the hack-verifiable environments methodology and record both papers as verified primary sources. HVE plants a hack in the environment to measure whether an agent exploits it; an audit here plants a fault in the submission to measure whether the checks catch it. Opposite directions, no equivalence claimed.

README.md

@@ -107,6 +107,11 @@ integrations remain planned. No frontier-model benchmark result is claimed.
 evidence and fingerprints every input. Use `--require-resolved` when your handoff
 also requires all checks to pass. [Verification and limits](docs/verification.md).
 
+
+[![Three task packs with direct evidence links for six single-case dependencies.](docs/coverage-review.png)](https://noteflowai.github.io/evalarc/#coverage)
+
+**Inspect coverage before trusting a perfect score.** The website now lists all three task packs and links each single-case dependency directly to its recorded checks and seeds. Offline audit reports provide the same disclosures without scripts or remote assets. These are new views of the original records, not new model runs.
+
 ## New in 0.9.0: research you can inspect
 
 [Explore all 27 real GPU skill trials](https://noteflowai.github.io/evalarc/skill-impact/) and [the research pilots](docs/research-pilots.md). Robot Reel's [captured-scene editor](https://noteflowai.github.io/robot-reel/scene-lab/) and [official LIBERO-Plus replay](https://noteflowai.github.io/robot-reel/libero-plus/) connect real source records with portable skill delivery and independent grading. Every failed attempt stays visible; no skill efficacy, full-benchmark or real-hardware result is implied.

README.zh-CN.md

@@ -91,6 +91,11 @@ Docker 评测：参考策略 3/3 轮完全通过，重复写入策略虽然平
 需要全部任务通过时加 `--require-resolved`。
 [使用说明与校验范围](docs/verification.md)。
 
+
+[![三个任务包的覆盖薄弱点与原始证据入口](docs/coverage-review.png)](https://noteflowai.github.io/evalarc/#coverage)
+
+**在信任满分前，先检查覆盖薄弱点。** 网页现展示全部三个任务包，可从仅靠一个用例检出的缺陷直接定位到原始检查与种子记录。离线审计报告也提供相同的证据展开入口，不依赖脚本或远程资源；新增界面沿用原始数据，不冒充新模型运行。
+
 ## 0.9.0：有原始证据的研究场景
 
 [查看 27 次真实 GPU 技能评测](https://noteflowai.github.io/evalarc/skill-impact/)，并阅读[完整方法与限制](docs/research-pilots.md)。新增[实景 Blender 编辑](https://noteflowai.github.io/robot-reel/scene-lab/)与[官方 LIBERO-Plus 子集回放](https://noteflowai.github.io/robot-reel/libero-plus/)，把原始记录、技能交付与独立验收连接起来。失败尝试全部保留；不宣称技能提分、完整基准成绩或真机效果。

huggingface/DATASET.md

@@ -47,14 +47,17 @@ and `fully_resolved` are separate columns.
 | Configuration | Rows | Unit and purpose |
 | --- | ---: | --- |
 | `suite_jobs` | 3 | One configured gate per job, backed by five Docker attempts; compare acceptance and full resolution. |
-| `audit_cases` | 167 | One case execution per scripted control: 135 coding and 32 support cases across two references and 15 declared faults. |
+| `audit_cases` | 251 | One case execution per scripted control: 135 coding, 32 support and 84 robot-evidence cases across three references and 21 declared faults. |
 | `repetition_attempts` | 6 | One recorded attempt of a frozen support policy; three reference and three faulty attempts. |
 
 Each configuration has a single **`development`** split. These are different
 units, so their row counts must not be summed into a number of independent
 trials or benchmark examples. `evaluation_score` in the case table is the
 parent evaluation's score, repeated for navigation; averaging that column
 across case rows would reweight evaluations incorrectly.
+`control_detection_margin` counts distinct detecting case IDs for the parent
+faulty control; references have null in this field. It is repeated for filtering,
+and must not be summed across case rows or interpreted as independent trials.
 
 These are saved **scripted controls on public development tasks**, not runs of
 a trained language model. Support tickets, identifiers and messages are
@@ -89,7 +92,7 @@ The suite configuration, plan, all five attempts and JUnit are included under
 `evidence/examples/suite/`. Repetition summaries and all six attempts are included
 under their original example directories.
 
-The original records come from EvalArc 0.2, 0.4 and 0.5. Their
+The original records come from EvalArc 0.2, 0.4, 0.5 and 0.9. Their
 `recorded_evalarc_version` and grader fingerprints are preserved. Do not treat
 the three configurations as matched version comparisons; use the lab's separate
 matched comparison for that question.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "evalarc-evidence-site",
-  "version": "0.9.0",
+  "version": "0.10.0",
   "private": true,
   "description": "Browser checks for the static EvalArc evidence explorer",
   "scripts": {

scripts/build_dataset.py

@@ -12,7 +12,7 @@
 
 ROOT = build_site.ROOT
 SCHEMA = "evalarc.casebook.v1"
-COUNTS = {"audit_cases": 167, "repetition_attempts": 6, "suite_jobs": 3}
+COUNTS = {"audit_cases": 251, "repetition_attempts": 6, "suite_jobs": 3}
 
 
 def compact(value: object) -> str:
@@ -62,9 +62,13 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict:
             "runtime_json": compact(evaluation["runtime"]),
         }
 
-    for directory in ("audit", "support-audit"):
+    for directory in ("audit", "support-audit", "research/robot-audit-python"):
         path = ROOT / "examples" / directory / "audit.json"
         audit = json.loads(path.read_text())
+        margins = {
+            item["name"]: len(set(item["failing_cases"])) if item["valid"] else None
+            for item in audit["mutants"]
+        }
         controls = [("reference", "/reference", audit["reference"], "reference")]
         controls.extend(
             (item["name"], f"/mutants/{i}/evaluation", item["evaluation"], "declared-fault")
@@ -78,6 +82,7 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict:
                         "id": f"{evaluation['task']['id']}/{name}/{case['seed']}/{case['case_id']}",
                         "control": name,
                         "control_kind": kind,
+                        "control_detection_margin": margins.get(name),
                         "case_id": case["case_id"],
                         "seed": case["seed"],
                         "case_passed": case["passed"],

scripts/build_site.py

@@ -4,6 +4,7 @@
 
 import argparse
 import hashlib
+import html
 import json
 import re
 import shutil
@@ -242,6 +243,31 @@ def verify(folder: Path) -> dict:
     return record
 
 
+def coverage_cards(audits: dict) -> str:
+    cards = []
+    for name, audit in audits.items():
+        fragile = [
+            (i, row)
+            for i, row in enumerate(audit["mutants"])
+            if row["valid"] and row["killed"] and len(set(row["failing_cases"])) == 1
+        ]
+        links = "".join(
+            f'<li><a href="{name}/index.html#fault-{index}">'
+            f"{html.escape(row['name'])}"
+            f"</a><span> Sole case: {html.escape(row['failing_cases'][0])}</span></li>"
+            for index, row in fragile
+        )
+        cards.append(
+            '<article class="coverage-card">'
+            f"<h3>{html.escape(audit['reference']['task']['id'])}</h3>"
+            f"<p><strong>{audit['killed']}/{audit['total']}</strong> declared faults detected</p>"
+            f'<p class="coverage-fragile">{len(fragile)} single-case dependencies</p>'
+            f'<ul>{links}</ul><a href="{name}/index.html">'
+            "Inspect all controls &#8599;</a></article>"
+        )
+    return "".join(cards)
+
+
 def build(destination: Path) -> dict:
     destination = destination.resolve()
     if destination.exists():
@@ -255,8 +281,20 @@ def build(destination: Path) -> dict:
         ("coding", "audit", "durable-kv", 8, "boolean-equals-one", 0.925),
         ("support", "support-audit", "support-routing", 7, "new-key-on-retry", 0.9375),
     ]
-    for _, directory, task, count, spotlight, score in specifications:
+    specifications.append(
+        (
+            "robot",
+            "research/robot-audit-python",
+            "robot-evidence-review",
+            6,
+            "invent-source",
+            0.9,
+        )
+    )
+    audits = {}
+    for name, directory, task, count, spotlight, score in specifications:
         audit = json.loads((ROOT / "examples" / directory / "audit.json").read_text())
+        audits[name] = audit
         if (
             not audit["valid"]
             or not audit["reference_passed"]
@@ -271,6 +309,16 @@ def build(destination: Path) -> dict:
         highlighted = next(row for row in audit["mutants"] if row["name"] == spotlight)
         if highlighted["score"] != score or highlighted["evaluation"]["resolved"]:
             raise ValueError(f"Spotlight outcome changed: {spotlight}")
+        for row in audit["mutants"]:
+            observed = sorted(
+                {
+                    case["case_id"]
+                    for case in row["evaluation"]["cases"]
+                    if case["checks"].get(row["target_dimension"]) is False
+                }
+            )
+            if observed != sorted(set(row["failing_cases"])):
+                raise ValueError(f"Audit detection evidence disagrees: {row['name']}")
     verify_comparison()
     verify_repetitions()
     verify_suite()
@@ -280,22 +328,22 @@ def build(destination: Path) -> dict:
             if path.name == "index.html":
                 version = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["version"]
                 (destination / path.name).write_text(
-                    path.read_text().replace("__EVALARC_VERSION__", version)
+                    path.read_text()
+                    .replace("__EVALARC_VERSION__", version)
+                    .replace("__AUDIT_COVERAGE__", coverage_cards(audits))
                 )
             else:
                 shutil.copyfile(path, destination / path.name)
     for name, directory, *_ in specifications:
         target = destination / name
         target.mkdir()
-        for filename in ("audit.json", "index.html"):
-            source = ROOT / "examples" / directory / filename
-            if filename.endswith(".html"):
-                # Avoid multibyte HTML corruption by the static Space injector.
-                (target / filename).write_bytes(
-                    source.read_text().encode("ascii", errors="xmlcharrefreplace")
-                )
-            else:
-                shutil.copyfile(source, target / filename)
+        # Render current, accessible UI from unchanged recorded evidence.
+        from evalarc.report import render_audit
+
+        shutil.copyfile(ROOT / "examples" / directory / "audit.json", target / "audit.json")
+        render_audit(audits[name], target / "index.html")
+        page = target / "index.html"
+        page.write_bytes(page.read_text().encode("ascii", errors="xmlcharrefreplace"))
     for directory, filenames in (
         ("comparison", ("index.html", "comparison.json", "baseline.json", "current.json")),
         ("evaluation", ("index.html", "evaluation.json")),

scripts/check_site.cjs

@@ -40,6 +40,8 @@ async function main() {
       await app.locator("#comparison-workspace").waitFor({state:"visible"});
       await app.locator("#repeat-workspace").waitFor({state:"visible"});
       await app.locator("#suite-workspace").waitFor({state:"visible"});
+      assert.equal(await app.locator(".coverage-card").count(),3);
+      assert.equal(await app.locator(".coverage-card li").count(),6);
       await page.waitForLoadState("networkidle");
       for (const name of ["partial", "protected"]) {
         assert.equal(await app.locator("#suite-" + name + "-score").innerText(), "93.75%");
@@ -248,6 +250,16 @@ async function main() {
       await app.getByRole("heading",{name:"Skill composition: 12 attempts, 3 accepted"}).waitFor();
       assert.equal(await app.locator("tbody tr").count(),18);
       assert.equal(await app.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false);
+      const coveragePage = await browser.newPage({viewport:{width,height:1000}});
+      try {
+        await coveragePage.goto(new URL("robot/index.html#fault-3", suiteBase).href);
+        await coveragePage.locator("#fault-3").waitFor({state:"visible"});
+        assert.match(await coveragePage.locator("#fault-3").innerText(), /incomplete-recording/);
+        assert.equal(await coveragePage.locator(".audit-control").count(),6);
+        assert(await coveragePage.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
+        await coveragePage.locator("#fault-3 details summary").first().click();
+        assert.match(await coveragePage.locator("#fault-3 pre").first().innerText(), /"completeness": false/);
+      } finally { await coveragePage.close(); }
       assert.deepEqual(errors, []);
       results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, suiteDownloadVerified:true, sharedTraceRestored:true, keyboardCaseReturn:true, realModelTrials:27, supplementaryTrials:18, errors});
       await page.close();

site/index.html

@@ -44,7 +44,14 @@ <h1>Look past<br>the <em>score.</em></h1>
       <div><strong>JSON</strong><span>downloadable evidence</span></div>
       <div><strong>CPU</strong><span>reproduce without a model API</span></div>
     </div>
-    <nav class="section-nav" aria-label="Evidence sections"><a href="#suite">Acceptance gates</a><a href="#repeat">Repeated attempts</a><a href="#regression">Revision comparison</a><a href="#explorer">Case explorer</a></nav>
+    <nav class="section-nav" aria-label="Evidence sections"><a href="#coverage">Audit coverage</a><a href="#suite">Acceptance gates</a><a href="#repeat">Repeated attempts</a><a href="#regression">Revision comparison</a><a href="#explorer">Case explorer</a></nav>
+    <section id="coverage" aria-labelledby="coverage-title">
+      <p class="eyebrow">THREE TASK PACKS / DECLARED FAULTS</p>
+      <h2 id="coverage-title">A perfect score.<br>How much coverage remains?</h2>
+      <p class="muted">A fault detected by one case loses its coverage if that case is removed or weakened. Open each dependency to inspect the recorded checks, status and seed. Margins count distinct case IDs, not repeated runs.</p>
+      <div class="coverage-grid">__AUDIT_COVERAGE__</div>
+      <p class="muted">These summaries derive from the original scripted audit records. They are not new agent runs or a guarantee against unseen faults.</p>
+    </section>
     <section id="suite" class="suite">
       <div class="section-top"><div><p class="eyebrow">NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES</p><h2>Same score. Different gate.</h2></div><p>One frozen policy.<br>Two explicit acceptance rules.</p></div>
       <p class="muted">Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.</p>

src/evalarc/audit.py

@@ -126,7 +126,7 @@ def observer(control: str) -> EventCallback | None:
             )
     killed = sum(row["killed"] for row in rows)
     valid = reference["valid"] and all(row["valid"] for row in rows)
-    margins = {row["name"]: row["detection_margin"] for row in rows if row["killed"]}
+    margins = {row["name"]: row["detection_margin"] for row in rows if row["valid"]}
     # A case that is the only detector of some fault cannot be removed or
     # loosened without losing coverage the mutation score still claims.
     sole_detectors = sorted(
@@ -144,7 +144,7 @@ def observer(control: str) -> EventCallback | None:
         # Reported next to the score because a perfect score says nothing about
         # how much of the suite has to survive for it to stay perfect.
         "detection": {
-            "weakest_margin": min(margins.values()) if margins else None,
+            "weakest_margin": min(margins.values()) if valid and margins else None,
             "single_case_detections": sorted(name for name, n in margins.items() if n == 1),
             "sole_detector_cases": sole_detectors,
         },