{html.escape(audit['reference']['task']['id'])}
" + f"{audit['killed']}/{audit['total']} declared faults detected
" + f'{len(fragile)} single-case dependencies
' + f'- {links}
diff --git a/CHANGELOG.md b/CHANGELOG.md index a631e58..dc94284 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## 0.10.0 — 2026-09-14 +- Add a three-task audit coverage section with direct links from single-case dependencies to recorded fault evidence. Re-render all three reports from unchanged saved JSON, with accessible disclosures and layouts for narrow screens. +- Include assessed surviving faults as zero in the weakest detection margin; environment failures remain unassessed. +- Extend the Hugging Face casebook to all three task packs: 251 case rows, with the parent control's distinct-case detection margin. Preserve the original source records and distinguish reference controls with a null margin. + - Report detection margins beside the mutation score: how many cases caught each declared fault, the weakest margin in the pack, the faults caught by exactly one case, and the cases that are the sole detector of some fault. All three packs score 1.0, and six of their 21 declared faults rest on a single case each; that fragility was previously invisible. - Count margins over distinct cases rather than case runs, so adding a seed cannot inflate them. Measured margins are identical under the Python and JavaScript references. - State the relation to the hack-verifiable environments methodology and record both papers as verified primary sources. HVE plants a hack in the environment to measure whether an agent exploits it; an audit here plants a fault in the submission to measure whether the checks catch it. Opposite directions, no equivalence claimed. diff --git a/README.md b/README.md index eadac26..a07293f 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,11 @@ integrations remain planned. No frontier-model benchmark result is claimed. evidence and fingerprints every input. Use `--require-resolved` when your handoff also requires all checks to pass. [Verification and limits](docs/verification.md). + +[](https://noteflowai.github.io/evalarc/#coverage) + +**Inspect coverage before trusting a perfect score.** The website now lists all three task packs and links each single-case dependency directly to its recorded checks and seeds. Offline audit reports provide the same disclosures without scripts or remote assets. These are new views of the original records, not new model runs. + ## New in 0.9.0: research you can inspect [Explore all 27 real GPU skill trials](https://noteflowai.github.io/evalarc/skill-impact/) and [the research pilots](docs/research-pilots.md). Robot Reel's [captured-scene editor](https://noteflowai.github.io/robot-reel/scene-lab/) and [official LIBERO-Plus replay](https://noteflowai.github.io/robot-reel/libero-plus/) connect real source records with portable skill delivery and independent grading. Every failed attempt stays visible; no skill efficacy, full-benchmark or real-hardware result is implied. diff --git a/README.zh-CN.md b/README.zh-CN.md index 1b12f59..e546fd6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -91,6 +91,11 @@ Docker 评测:参考策略 3/3 轮完全通过,重复写入策略虽然平 需要全部任务通过时加 `--require-resolved`。 [使用说明与校验范围](docs/verification.md)。 + +[](https://noteflowai.github.io/evalarc/#coverage) + +**在信任满分前,先检查覆盖薄弱点。** 网页现展示全部三个任务包,可从仅靠一个用例检出的缺陷直接定位到原始检查与种子记录。离线审计报告也提供相同的证据展开入口,不依赖脚本或远程资源;新增界面沿用原始数据,不冒充新模型运行。 + ## 0.9.0:有原始证据的研究场景 [查看 27 次真实 GPU 技能评测](https://noteflowai.github.io/evalarc/skill-impact/),并阅读[完整方法与限制](docs/research-pilots.md)。新增[实景 Blender 编辑](https://noteflowai.github.io/robot-reel/scene-lab/)与[官方 LIBERO-Plus 子集回放](https://noteflowai.github.io/robot-reel/libero-plus/),把原始记录、技能交付与独立验收连接起来。失败尝试全部保留;不宣称技能提分、完整基准成绩或真机效果。 diff --git a/docs/coverage-review.png b/docs/coverage-review.png new file mode 100644 index 0000000..b8b3a9d Binary files /dev/null and b/docs/coverage-review.png differ diff --git a/huggingface/DATASET.md b/huggingface/DATASET.md index d81b0c2..00c058c 100644 --- a/huggingface/DATASET.md +++ b/huggingface/DATASET.md @@ -47,7 +47,7 @@ and `fully_resolved` are separate columns. | Configuration | Rows | Unit and purpose | | --- | ---: | --- | | `suite_jobs` | 3 | One configured gate per job, backed by five Docker attempts; compare acceptance and full resolution. | -| `audit_cases` | 167 | One case execution per scripted control: 135 coding and 32 support cases across two references and 15 declared faults. | +| `audit_cases` | 251 | One case execution per scripted control: 135 coding, 32 support and 84 robot-evidence cases across three references and 21 declared faults. | | `repetition_attempts` | 6 | One recorded attempt of a frozen support policy; three reference and three faulty attempts. | Each configuration has a single **`development`** split. These are different @@ -55,6 +55,9 @@ units, so their row counts must not be summed into a number of independent trials or benchmark examples. `evaluation_score` in the case table is the parent evaluation's score, repeated for navigation; averaging that column across case rows would reweight evaluations incorrectly. +`control_detection_margin` counts distinct detecting case IDs for the parent +faulty control; references have null in this field. It is repeated for filtering, +and must not be summed across case rows or interpreted as independent trials. These are saved **scripted controls on public development tasks**, not runs of a trained language model. Support tickets, identifiers and messages are @@ -89,7 +92,7 @@ The suite configuration, plan, all five attempts and JUnit are included under `evidence/examples/suite/`. Repetition summaries and all six attempts are included under their original example directories. -The original records come from EvalArc 0.2, 0.4 and 0.5. Their +The original records come from EvalArc 0.2, 0.4, 0.5 and 0.9. Their `recorded_evalarc_version` and grader fingerprints are preserved. Do not treat the three configurations as matched version comparisons; use the lab's separate matched comparison for that question. diff --git a/package-lock.json b/package-lock.json index 3d099b7..2d2089d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "devDependencies": { "playwright": "1.63.0" } diff --git a/package.json b/package.json index d04d507..d21c9d5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "private": true, "description": "Browser checks for the static EvalArc evidence explorer", "scripts": { diff --git a/scripts/build_dataset.py b/scripts/build_dataset.py index c5b33e0..778d2d6 100644 --- a/scripts/build_dataset.py +++ b/scripts/build_dataset.py @@ -12,7 +12,7 @@ ROOT = build_site.ROOT SCHEMA = "evalarc.casebook.v1" -COUNTS = {"audit_cases": 167, "repetition_attempts": 6, "suite_jobs": 3} +COUNTS = {"audit_cases": 251, "repetition_attempts": 6, "suite_jobs": 3} def compact(value: object) -> str: @@ -62,9 +62,13 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict: "runtime_json": compact(evaluation["runtime"]), } - for directory in ("audit", "support-audit"): + for directory in ("audit", "support-audit", "research/robot-audit-python"): path = ROOT / "examples" / directory / "audit.json" audit = json.loads(path.read_text()) + margins = { + item["name"]: len(set(item["failing_cases"])) if item["valid"] else None + for item in audit["mutants"] + } controls = [("reference", "/reference", audit["reference"], "reference")] controls.extend( (item["name"], f"/mutants/{i}/evaluation", item["evaluation"], "declared-fault") @@ -78,6 +82,7 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict: "id": f"{evaluation['task']['id']}/{name}/{case['seed']}/{case['case_id']}", "control": name, "control_kind": kind, + "control_detection_margin": margins.get(name), "case_id": case["case_id"], "seed": case["seed"], "case_passed": case["passed"], diff --git a/scripts/build_site.py b/scripts/build_site.py index c8df56c..357912d 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -4,6 +4,7 @@ import argparse import hashlib +import html import json import re import shutil @@ -242,6 +243,31 @@ def verify(folder: Path) -> dict: return record +def coverage_cards(audits: dict) -> str: + cards = [] + for name, audit in audits.items(): + fragile = [ + (i, row) + for i, row in enumerate(audit["mutants"]) + if row["valid"] and row["killed"] and len(set(row["failing_cases"])) == 1 + ] + links = "".join( + f'
{audit['killed']}/{audit['total']} declared faults detected
" + f'{len(fragile)} single-case dependencies
' + f'THREE TASK PACKS / DECLARED FAULTS
+A fault detected by one case loses its coverage if that case is removed or weakened. Open each dependency to inspect the recorded checks, status and seed. Margins count distinct case IDs, not repeated runs.
+These summaries derive from the original scripted audit records. They are not new agent runs or a guarantee against unseen faults.
+NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES
One frozen policy.
Two explicit acceptance rules.
Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.
diff --git a/site/style.css b/site/style.css index 2675fb6..a7d318e 100644 --- a/site/style.css +++ b/site/style.css @@ -35,3 +35,5 @@ button:disabled{opacity:.5;cursor:default} .proof-top{flex-wrap:wrap} .receipt{grid-template-columns:minmax(0,1.2fr) minmax(0,1fr);overflow-wrap:anywhere;font-size:11px} } + +#coverage{padding:48px 0;border-top:1px solid var(--line)}.coverage-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:18px}.coverage-card{padding:22px;border:1px solid var(--line);border-radius:12px;min-width:0}.coverage-card h3{font-size:15px;overflow-wrap:anywhere}.coverage-card strong{font-size:32px;color:var(--mint)}.coverage-card ul{padding-left:18px}.coverage-card li{margin:14px 0;overflow-wrap:anywhere;font-size:13px}.coverage-card li span{display:block;color:var(--muted);font-size:12px}.coverage-fragile{color:#efa6a0}@media(max-width:850px){.coverage-grid{grid-template-columns:minmax(0,1fr)}} diff --git a/src/evalarc/audit.py b/src/evalarc/audit.py index 569e9f6..67bbfc1 100644 --- a/src/evalarc/audit.py +++ b/src/evalarc/audit.py @@ -126,7 +126,7 @@ def observer(control: str) -> EventCallback | None: ) killed = sum(row["killed"] for row in rows) valid = reference["valid"] and all(row["valid"] for row in rows) - margins = {row["name"]: row["detection_margin"] for row in rows if row["killed"]} + margins = {row["name"]: row["detection_margin"] for row in rows if row["valid"]} # A case that is the only detector of some fault cannot be removed or # loosened without losing coverage the mutation score still claims. sole_detectors = sorted( @@ -144,7 +144,7 @@ def observer(control: str) -> EventCallback | None: # Reported next to the score because a perfect score says nothing about # how much of the suite has to survive for it to stay perfect. "detection": { - "weakest_margin": min(margins.values()) if margins else None, + "weakest_margin": min(margins.values()) if valid and margins else None, "single_case_detections": sorted(name for name, n in margins.items() if n == 1), "sole_detector_cases": sole_detectors, }, diff --git a/src/evalarc/report.py b/src/evalarc/report.py index 5f5343c..7b29f25 100644 --- a/src/evalarc/report.py +++ b/src/evalarc/report.py @@ -23,6 +23,13 @@ .metadata{overflow-wrap:anywhere;font-size:13px} pre{overflow:auto;padding:18px;background:#17221c;font-size:12px;max-height:480px} summary{cursor:pointer;color:#96e8b9}details{margin:16px 0} +summary{min-height:44px;padding:10px 0} +a:focus-visible,summary:focus-visible{outline:3px solid #96e8b9;outline-offset:4px} +.audit-control{border:1px solid #33463d;border-radius:12px;padding:12px 18px} +.audit-control summary{overflow-wrap:anywhere}.audit-control p{overflow-wrap:anywhere} +.audit-coverage{border-left:3px solid #edb68d;padding:12px 20px;background:#17221c} +@media(max-width:640px){body{margin:28px auto;padding:0 16px}.cards{gap:12px} +.card{min-width:0;flex-basis:100%}.dimension{flex-wrap:wrap}.audit-control{padding:10px 14px}} a{color:#96e8b9}.failed,.agent_error,.environment_error{color:#edb68d} .passed{color:#96e8b9}td{overflow-wrap:anywhere} .scroll table{min-width:680px}td:first-child{min-width:170px;overflow-wrap:normal} @@ -37,7 +44,13 @@ def render_audit(data: dict, destination: Path) -> None: score_text = lambda value: "unassessed" if value is None else f"{value:.3f}" # noqa: E731 margin_text = lambda value: "—" if value is None else str(value) # noqa: E731 rows = [] - for row in data["mutants"]: + fragile = [ + row + for row in data["mutants"] + if row.get("valid") is not False and row["killed"] and len(set(row["failing_cases"])) == 1 + ] + for index, row in enumerate(data["mutants"]): + margin = None if row.get("valid") is False else len(set(row["failing_cases"])) state = ( "UNASSESSED" if row.get("valid") is False @@ -47,8 +60,8 @@ def render_audit(data: dict, destination: Path) -> None: f"{esc(row['name'])}{len(fragile)} of {data['total']} declared faults have " + "exactly one distinct detecting case in the assessed records. Repeating a " + "case under more seeds does not increase this margin. A surviving fault has " + "margin zero; an environment failure remains unassessed.
" + "Margins below are derived from the saved failing case IDs. " + "Rendering an older audit does not rerun its controls.
| Negative control | Target | " "Candidate score | Result | " 'Margin | ' "Evidence |
|---|
Target: {esc(row["target_dimension"])}. ' + f"Detecting case IDs: {esc(case_ids)}.
" + ) + cases = [ + case + for case in row.get("evaluation", {}).get("cases", []) + if case.get("checks", {}).get(row["target_dimension"]) is False + ] + if cases: + for case in cases: + document += ( + f""
+ f"{esc(json.dumps(case, ensure_ascii=False, indent=2))}No detecting-case trace is embedded for this control. " + "Consult the complete JSON evidence for its recorded validity and outcome.
" + ) + document += "