diff --git a/CHANGELOG.md b/CHANGELOG.md index a631e58..dc94284 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## 0.10.0 — 2026-09-14 +- Add a three-task audit coverage section with direct links from single-case dependencies to recorded fault evidence. Re-render all three reports from unchanged saved JSON, with accessible disclosures and layouts for narrow screens. +- Include assessed surviving faults as zero in the weakest detection margin; environment failures remain unassessed. +- Extend the Hugging Face casebook to all three task packs: 251 case rows, with the parent control's distinct-case detection margin. Preserve the original source records and distinguish reference controls with a null margin. + - Report detection margins beside the mutation score: how many cases caught each declared fault, the weakest margin in the pack, the faults caught by exactly one case, and the cases that are the sole detector of some fault. All three packs score 1.0, and six of their 21 declared faults rest on a single case each; that fragility was previously invisible. - Count margins over distinct cases rather than case runs, so adding a seed cannot inflate them. Measured margins are identical under the Python and JavaScript references. - State the relation to the hack-verifiable environments methodology and record both papers as verified primary sources. HVE plants a hack in the environment to measure whether an agent exploits it; an audit here plants a fault in the submission to measure whether the checks catch it. Opposite directions, no equivalence claimed. diff --git a/README.md b/README.md index eadac26..a07293f 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,11 @@ integrations remain planned. No frontier-model benchmark result is claimed. evidence and fingerprints every input. Use `--require-resolved` when your handoff also requires all checks to pass. [Verification and limits](docs/verification.md). + +[![Three task packs with direct evidence links for six single-case dependencies.](docs/coverage-review.png)](https://noteflowai.github.io/evalarc/#coverage) + +**Inspect coverage before trusting a perfect score.** The website now lists all three task packs and links each single-case dependency directly to its recorded checks and seeds. Offline audit reports provide the same disclosures without scripts or remote assets. These are new views of the original records, not new model runs. + ## New in 0.9.0: research you can inspect [Explore all 27 real GPU skill trials](https://noteflowai.github.io/evalarc/skill-impact/) and [the research pilots](docs/research-pilots.md). Robot Reel's [captured-scene editor](https://noteflowai.github.io/robot-reel/scene-lab/) and [official LIBERO-Plus replay](https://noteflowai.github.io/robot-reel/libero-plus/) connect real source records with portable skill delivery and independent grading. Every failed attempt stays visible; no skill efficacy, full-benchmark or real-hardware result is implied. diff --git a/README.zh-CN.md b/README.zh-CN.md index 1b12f59..e546fd6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -91,6 +91,11 @@ Docker 评测:参考策略 3/3 轮完全通过,重复写入策略虽然平 需要全部任务通过时加 `--require-resolved`。 [使用说明与校验范围](docs/verification.md)。 + +[![三个任务包的覆盖薄弱点与原始证据入口](docs/coverage-review.png)](https://noteflowai.github.io/evalarc/#coverage) + +**在信任满分前,先检查覆盖薄弱点。** 网页现展示全部三个任务包,可从仅靠一个用例检出的缺陷直接定位到原始检查与种子记录。离线审计报告也提供相同的证据展开入口,不依赖脚本或远程资源;新增界面沿用原始数据,不冒充新模型运行。 + ## 0.9.0:有原始证据的研究场景 [查看 27 次真实 GPU 技能评测](https://noteflowai.github.io/evalarc/skill-impact/),并阅读[完整方法与限制](docs/research-pilots.md)。新增[实景 Blender 编辑](https://noteflowai.github.io/robot-reel/scene-lab/)与[官方 LIBERO-Plus 子集回放](https://noteflowai.github.io/robot-reel/libero-plus/),把原始记录、技能交付与独立验收连接起来。失败尝试全部保留;不宣称技能提分、完整基准成绩或真机效果。 diff --git a/docs/coverage-review.png b/docs/coverage-review.png new file mode 100644 index 0000000..b8b3a9d Binary files /dev/null and b/docs/coverage-review.png differ diff --git a/huggingface/DATASET.md b/huggingface/DATASET.md index d81b0c2..00c058c 100644 --- a/huggingface/DATASET.md +++ b/huggingface/DATASET.md @@ -47,7 +47,7 @@ and `fully_resolved` are separate columns. | Configuration | Rows | Unit and purpose | | --- | ---: | --- | | `suite_jobs` | 3 | One configured gate per job, backed by five Docker attempts; compare acceptance and full resolution. | -| `audit_cases` | 167 | One case execution per scripted control: 135 coding and 32 support cases across two references and 15 declared faults. | +| `audit_cases` | 251 | One case execution per scripted control: 135 coding, 32 support and 84 robot-evidence cases across three references and 21 declared faults. | | `repetition_attempts` | 6 | One recorded attempt of a frozen support policy; three reference and three faulty attempts. | Each configuration has a single **`development`** split. These are different @@ -55,6 +55,9 @@ units, so their row counts must not be summed into a number of independent trials or benchmark examples. `evaluation_score` in the case table is the parent evaluation's score, repeated for navigation; averaging that column across case rows would reweight evaluations incorrectly. +`control_detection_margin` counts distinct detecting case IDs for the parent +faulty control; references have null in this field. It is repeated for filtering, +and must not be summed across case rows or interpreted as independent trials. These are saved **scripted controls on public development tasks**, not runs of a trained language model. Support tickets, identifiers and messages are @@ -89,7 +92,7 @@ The suite configuration, plan, all five attempts and JUnit are included under `evidence/examples/suite/`. Repetition summaries and all six attempts are included under their original example directories. -The original records come from EvalArc 0.2, 0.4 and 0.5. Their +The original records come from EvalArc 0.2, 0.4, 0.5 and 0.9. Their `recorded_evalarc_version` and grader fingerprints are preserved. Do not treat the three configurations as matched version comparisons; use the lab's separate matched comparison for that question. diff --git a/package-lock.json b/package-lock.json index 3d099b7..2d2089d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "devDependencies": { "playwright": "1.63.0" } diff --git a/package.json b/package.json index d04d507..d21c9d5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.9.0", + "version": "0.10.0", "private": true, "description": "Browser checks for the static EvalArc evidence explorer", "scripts": { diff --git a/scripts/build_dataset.py b/scripts/build_dataset.py index c5b33e0..778d2d6 100644 --- a/scripts/build_dataset.py +++ b/scripts/build_dataset.py @@ -12,7 +12,7 @@ ROOT = build_site.ROOT SCHEMA = "evalarc.casebook.v1" -COUNTS = {"audit_cases": 167, "repetition_attempts": 6, "suite_jobs": 3} +COUNTS = {"audit_cases": 251, "repetition_attempts": 6, "suite_jobs": 3} def compact(value: object) -> str: @@ -62,9 +62,13 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict: "runtime_json": compact(evaluation["runtime"]), } - for directory in ("audit", "support-audit"): + for directory in ("audit", "support-audit", "research/robot-audit-python"): path = ROOT / "examples" / directory / "audit.json" audit = json.loads(path.read_text()) + margins = { + item["name"]: len(set(item["failing_cases"])) if item["valid"] else None + for item in audit["mutants"] + } controls = [("reference", "/reference", audit["reference"], "reference")] controls.extend( (item["name"], f"/mutants/{i}/evaluation", item["evaluation"], "declared-fault") @@ -78,6 +82,7 @@ def provenance(path: Path, pointer: str, evaluation: dict) -> dict: "id": f"{evaluation['task']['id']}/{name}/{case['seed']}/{case['case_id']}", "control": name, "control_kind": kind, + "control_detection_margin": margins.get(name), "case_id": case["case_id"], "seed": case["seed"], "case_passed": case["passed"], diff --git a/scripts/build_site.py b/scripts/build_site.py index c8df56c..357912d 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -4,6 +4,7 @@ import argparse import hashlib +import html import json import re import shutil @@ -242,6 +243,31 @@ def verify(folder: Path) -> dict: return record +def coverage_cards(audits: dict) -> str: + cards = [] + for name, audit in audits.items(): + fragile = [ + (i, row) + for i, row in enumerate(audit["mutants"]) + if row["valid"] and row["killed"] and len(set(row["failing_cases"])) == 1 + ] + links = "".join( + f'
  • ' + f"{html.escape(row['name'])}" + f" Sole case: {html.escape(row['failing_cases'][0])}
  • " + for index, row in fragile + ) + cards.append( + '
    ' + f"

    {html.escape(audit['reference']['task']['id'])}

    " + f"

    {audit['killed']}/{audit['total']} declared faults detected

    " + f'

    {len(fragile)} single-case dependencies

    ' + f'' + "Inspect all controls ↗
    " + ) + return "".join(cards) + + def build(destination: Path) -> dict: destination = destination.resolve() if destination.exists(): @@ -255,8 +281,20 @@ def build(destination: Path) -> dict: ("coding", "audit", "durable-kv", 8, "boolean-equals-one", 0.925), ("support", "support-audit", "support-routing", 7, "new-key-on-retry", 0.9375), ] - for _, directory, task, count, spotlight, score in specifications: + specifications.append( + ( + "robot", + "research/robot-audit-python", + "robot-evidence-review", + 6, + "invent-source", + 0.9, + ) + ) + audits = {} + for name, directory, task, count, spotlight, score in specifications: audit = json.loads((ROOT / "examples" / directory / "audit.json").read_text()) + audits[name] = audit if ( not audit["valid"] or not audit["reference_passed"] @@ -271,6 +309,16 @@ def build(destination: Path) -> dict: highlighted = next(row for row in audit["mutants"] if row["name"] == spotlight) if highlighted["score"] != score or highlighted["evaluation"]["resolved"]: raise ValueError(f"Spotlight outcome changed: {spotlight}") + for row in audit["mutants"]: + observed = sorted( + { + case["case_id"] + for case in row["evaluation"]["cases"] + if case["checks"].get(row["target_dimension"]) is False + } + ) + if observed != sorted(set(row["failing_cases"])): + raise ValueError(f"Audit detection evidence disagrees: {row['name']}") verify_comparison() verify_repetitions() verify_suite() @@ -280,22 +328,22 @@ def build(destination: Path) -> dict: if path.name == "index.html": version = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["version"] (destination / path.name).write_text( - path.read_text().replace("__EVALARC_VERSION__", version) + path.read_text() + .replace("__EVALARC_VERSION__", version) + .replace("__AUDIT_COVERAGE__", coverage_cards(audits)) ) else: shutil.copyfile(path, destination / path.name) for name, directory, *_ in specifications: target = destination / name target.mkdir() - for filename in ("audit.json", "index.html"): - source = ROOT / "examples" / directory / filename - if filename.endswith(".html"): - # Avoid multibyte HTML corruption by the static Space injector. - (target / filename).write_bytes( - source.read_text().encode("ascii", errors="xmlcharrefreplace") - ) - else: - shutil.copyfile(source, target / filename) + # Render current, accessible UI from unchanged recorded evidence. + from evalarc.report import render_audit + + shutil.copyfile(ROOT / "examples" / directory / "audit.json", target / "audit.json") + render_audit(audits[name], target / "index.html") + page = target / "index.html" + page.write_bytes(page.read_text().encode("ascii", errors="xmlcharrefreplace")) for directory, filenames in ( ("comparison", ("index.html", "comparison.json", "baseline.json", "current.json")), ("evaluation", ("index.html", "evaluation.json")), diff --git a/scripts/check_site.cjs b/scripts/check_site.cjs index d1c9c07..1540ec8 100644 --- a/scripts/check_site.cjs +++ b/scripts/check_site.cjs @@ -40,6 +40,8 @@ async function main() { await app.locator("#comparison-workspace").waitFor({state:"visible"}); await app.locator("#repeat-workspace").waitFor({state:"visible"}); await app.locator("#suite-workspace").waitFor({state:"visible"}); + assert.equal(await app.locator(".coverage-card").count(),3); + assert.equal(await app.locator(".coverage-card li").count(),6); await page.waitForLoadState("networkidle"); for (const name of ["partial", "protected"]) { assert.equal(await app.locator("#suite-" + name + "-score").innerText(), "93.75%"); @@ -248,6 +250,16 @@ async function main() { await app.getByRole("heading",{name:"Skill composition: 12 attempts, 3 accepted"}).waitFor(); assert.equal(await app.locator("tbody tr").count(),18); assert.equal(await app.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false); + const coveragePage = await browser.newPage({viewport:{width,height:1000}}); + try { + await coveragePage.goto(new URL("robot/index.html#fault-3", suiteBase).href); + await coveragePage.locator("#fault-3").waitFor({state:"visible"}); + assert.match(await coveragePage.locator("#fault-3").innerText(), /incomplete-recording/); + assert.equal(await coveragePage.locator(".audit-control").count(),6); + assert(await coveragePage.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1)); + await coveragePage.locator("#fault-3 details summary").first().click(); + assert.match(await coveragePage.locator("#fault-3 pre").first().innerText(), /"completeness": false/); + } finally { await coveragePage.close(); } assert.deepEqual(errors, []); results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, suiteDownloadVerified:true, sharedTraceRestored:true, keyboardCaseReturn:true, realModelTrials:27, supplementaryTrials:18, errors}); await page.close(); diff --git a/site/index.html b/site/index.html index 9ad81a2..0e212a0 100644 --- a/site/index.html +++ b/site/index.html @@ -44,7 +44,14 @@

    Look past
    the score.

    JSONdownloadable evidence
    CPUreproduce without a model API
    - + +
    +

    THREE TASK PACKS / DECLARED FAULTS

    +

    A perfect score.
    How much coverage remains?

    +

    A fault detected by one case loses its coverage if that case is removed or weakened. Open each dependency to inspect the recorded checks, status and seed. Margins count distinct case IDs, not repeated runs.

    +
    __AUDIT_COVERAGE__
    +

    These summaries derive from the original scripted audit records. They are not new agent runs or a guarantee against unseen faults.

    +

    NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES

    Same score. Different gate.

    One frozen policy.
    Two explicit acceptance rules.

    Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.

    diff --git a/site/style.css b/site/style.css index 2675fb6..a7d318e 100644 --- a/site/style.css +++ b/site/style.css @@ -35,3 +35,5 @@ button:disabled{opacity:.5;cursor:default} .proof-top{flex-wrap:wrap} .receipt{grid-template-columns:minmax(0,1.2fr) minmax(0,1fr);overflow-wrap:anywhere;font-size:11px} } + +#coverage{padding:48px 0;border-top:1px solid var(--line)}.coverage-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:18px}.coverage-card{padding:22px;border:1px solid var(--line);border-radius:12px;min-width:0}.coverage-card h3{font-size:15px;overflow-wrap:anywhere}.coverage-card strong{font-size:32px;color:var(--mint)}.coverage-card ul{padding-left:18px}.coverage-card li{margin:14px 0;overflow-wrap:anywhere;font-size:13px}.coverage-card li span{display:block;color:var(--muted);font-size:12px}.coverage-fragile{color:#efa6a0}@media(max-width:850px){.coverage-grid{grid-template-columns:minmax(0,1fr)}} diff --git a/src/evalarc/audit.py b/src/evalarc/audit.py index 569e9f6..67bbfc1 100644 --- a/src/evalarc/audit.py +++ b/src/evalarc/audit.py @@ -126,7 +126,7 @@ def observer(control: str) -> EventCallback | None: ) killed = sum(row["killed"] for row in rows) valid = reference["valid"] and all(row["valid"] for row in rows) - margins = {row["name"]: row["detection_margin"] for row in rows if row["killed"]} + margins = {row["name"]: row["detection_margin"] for row in rows if row["valid"]} # A case that is the only detector of some fault cannot be removed or # loosened without losing coverage the mutation score still claims. sole_detectors = sorted( @@ -144,7 +144,7 @@ def observer(control: str) -> EventCallback | None: # Reported next to the score because a perfect score says nothing about # how much of the suite has to survive for it to stay perfect. "detection": { - "weakest_margin": min(margins.values()) if margins else None, + "weakest_margin": min(margins.values()) if valid and margins else None, "single_case_detections": sorted(name for name, n in margins.items() if n == 1), "sole_detector_cases": sole_detectors, }, diff --git a/src/evalarc/report.py b/src/evalarc/report.py index 5f5343c..7b29f25 100644 --- a/src/evalarc/report.py +++ b/src/evalarc/report.py @@ -23,6 +23,13 @@ .metadata{overflow-wrap:anywhere;font-size:13px} pre{overflow:auto;padding:18px;background:#17221c;font-size:12px;max-height:480px} summary{cursor:pointer;color:#96e8b9}details{margin:16px 0} +summary{min-height:44px;padding:10px 0} +a:focus-visible,summary:focus-visible{outline:3px solid #96e8b9;outline-offset:4px} +.audit-control{border:1px solid #33463d;border-radius:12px;padding:12px 18px} +.audit-control summary{overflow-wrap:anywhere}.audit-control p{overflow-wrap:anywhere} +.audit-coverage{border-left:3px solid #edb68d;padding:12px 20px;background:#17221c} +@media(max-width:640px){body{margin:28px auto;padding:0 16px}.cards{gap:12px} +.card{min-width:0;flex-basis:100%}.dimension{flex-wrap:wrap}.audit-control{padding:10px 14px}} a{color:#96e8b9}.failed,.agent_error,.environment_error{color:#edb68d} .passed{color:#96e8b9}td{overflow-wrap:anywhere} .scroll table{min-width:680px}td:first-child{min-width:170px;overflow-wrap:normal} @@ -37,7 +44,13 @@ def render_audit(data: dict, destination: Path) -> None: score_text = lambda value: "unassessed" if value is None else f"{value:.3f}" # noqa: E731 margin_text = lambda value: "—" if value is None else str(value) # noqa: E731 rows = [] - for row in data["mutants"]: + fragile = [ + row + for row in data["mutants"] + if row.get("valid") is not False and row["killed"] and len(set(row["failing_cases"])) == 1 + ] + for index, row in enumerate(data["mutants"]): + margin = None if row.get("valid") is False else len(set(row["failing_cases"])) state = ( "UNASSESSED" if row.get("valid") is False @@ -47,8 +60,8 @@ def render_audit(data: dict, destination: Path) -> None: f"{esc(row['name'])}" f"{esc(row['target_dimension'])}" f"{score_text(row['score'])}{state}" - f"{esc(margin_text(row.get('detection_margin')))}" - f"{esc(', '.join(row['failing_cases']))}" + f"{esc(margin_text(margin))}" + f'Inspect recorded cases' ) reference = data["reference"] bars = "".join( @@ -88,14 +101,55 @@ def render_audit(data: dict, destination: Path) -> None: '
    reference correctness
    ' f'
    {esc(reference["runtime"]["backend"])}
    ' '
    execution backend
    ' - '

    Does the grader detect plausible defects?

    ' + '
    ' + '

    Which faults depend on one case?

    ' + f"

    {len(fragile)} of {data['total']} declared faults have " + "exactly one distinct detecting case in the assessed records. Repeating a " + "case under more seeds does not increase this margin. A surviving fault has " + "margin zero; an environment failure remains unassessed.

    " + "

    Margins below are derived from the saved failing case IDs. " + "Rendering an older audit does not rerun its controls.

    " + "

    Does the grader detect plausible defects?

    " + '
    ' "" "" '' "" + "".join(rows) + "
    Negative controlTargetCandidate scoreResultMarginEvidence
    " - "

    Positive control

    " + bars ) + document += "

    Inspect the detecting cases

    " + for index, row in enumerate(data["mutants"]): + margin = None if row.get("valid") is False else len(set(row["failing_cases"])) + label = ( + "unassessed" + if margin is None + else ("single-case dependency" if margin == 1 else f"{margin} detecting cases") + ) + case_ids = ", ".join(sorted(set(row["failing_cases"]))) or "none" + document += ( + f'
    {esc(row["name"])} · {label}' + f'

    Target: {esc(row["target_dimension"])}. ' + f"Detecting case IDs: {esc(case_ids)}.

    " + ) + cases = [ + case + for case in row.get("evaluation", {}).get("cases", []) + if case.get("checks", {}).get(row["target_dimension"]) is False + ] + if cases: + for case in cases: + document += ( + f"
    {esc(case['case_id'])} · seed {esc(case['seed'])}" + f" · {esc(case['status'])}
    "
    +                    f"{esc(json.dumps(case, ensure_ascii=False, indent=2))}
    " + ) + else: + document += ( + "

    No detecting-case trace is embedded for this control. " + "Consult the complete JSON evidence for its recorded validity and outcome.

    " + ) + document += "
    " + document += "

    Positive control

    " + bars traces = [case for case in reference.get("cases", []) if "trace" in case] if traces: document += "

    Reference execution evidence

    " diff --git a/tests/test_dataset.py b/tests/test_dataset.py index 0b93fdd..2ca87cc 100644 --- a/tests/test_dataset.py +++ b/tests/test_dataset.py @@ -31,11 +31,22 @@ def test_casebook_preserves_source_objects_and_distinct_gate_outcomes(tmp_path): folder = tmp_path / "casebook" manifest = build_dataset.build(folder) assert manifest["row_counts"] == { - "audit_cases": 167, + "audit_cases": 251, "repetition_attempts": 6, "suite_jobs": 3, } cases = table(folder, "audit_cases") + assert {row["task"] for row in cases} == { + "durable-kv", + "support-routing", + "robot-evidence-review", + } + assert len([row for row in cases if row["task"] == "robot-evidence-review"]) == 84 + assert all( + row["control_detection_margin"] is None + for row in cases + if row["control_kind"] == "reference" + ) for row in cases: case = original(folder, row) assert json.loads(row["case_json"]) == case diff --git a/tests/test_detection_margin.py b/tests/test_detection_margin.py new file mode 100644 index 0000000..b654826 --- /dev/null +++ b/tests/test_detection_margin.py @@ -0,0 +1,25 @@ +"""An assessed surviving control is zero coverage, not an absent measurement.""" + +import importlib + +from evalarc.runner import Runtime + + +def test_surviving_controls_are_included_in_the_weakest_margin(monkeypatch): + module = importlib.import_module("evalarc.audit") + monkeypatch.setattr( + module, + "evaluate", + lambda *args, **kwargs: { + "valid": True, + "resolved": True, + "score": 1.0, + "candidate_sha256": "0" * 64, + "cases": [{"case_id": "permissive", "checks": {}}], + }, + ) + result = module.audit(Runtime(backend="local"), [17]) + assert result["valid"] and not result["passed"] + assert result["mutation_score"] == 0 + assert result["detection"]["weakest_margin"] == 0 + assert result["detection"]["single_case_detections"] == [] diff --git a/tests/test_report.py b/tests/test_report.py index 007f00d..732ec12 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -32,3 +32,27 @@ def test_report_escapes_external_content(tmp_path): assert "