From d46a45cc0cd857c03652b9cd708b939608636e62 Mon Sep 17 00:00:00 2001 From: Note Flow AI Date: Mon, 14 Sep 2026 14:26:41 +0800 Subject: [PATCH] Share precise evidence and recover failed explorer sections --- CHANGELOG.md | 14 +++ CITATION.cff | 2 +- README.md | 5 +- README.zh-CN.md | 4 +- docs/explorer.md | 25 +++++ huggingface/README.md | 9 +- package-lock.json | 4 +- package.json | 2 +- pyproject.toml | 2 +- scripts/build_site.py | 8 +- scripts/check_site.cjs | 70 +++++++++++++- site/app.js | 198 +++++++++++++++++++++++++++------------- site/index.html | 36 +++++--- site/style.css | 25 +++++ src/evalarc/__init__.py | 2 +- tests/test_site.py | 4 + 16 files changed, 319 insertions(+), 91 deletions(-) create mode 100644 docs/explorer.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 080ca05..bb2a073 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## 0.7.1 · 2026-09-14 · Research preview + +- Share and restore a specific task pack, control, seed, case and trace step in + the evidence explorer, including a selectable link when clipboard access is + denied. Out-of-range links show an explicit fallback. +- Retry failed suite, repetition, comparison or task-pack requests independently. + Bound network waits and keep a successfully loaded task pack usable when the + other fails; retrying can restore the originally linked evidence. +- Add section navigation, case-detail focus and return controls, keyboard-readable + JSON panels, larger buttons and readable mobile evidence text. +- Render the site's preview version from package metadata. + +Original reports, task contracts, scoring and CLI behavior are unchanged. + ## 0.7.0 · 2026-09-14 · Research preview - Add `evalarc verify` for received evaluation, repetition and comparison diff --git a/CITATION.cff b/CITATION.cff index 3aec58c..b1af528 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -4,7 +4,7 @@ title: "EvalArc: Auditable Evaluations for AI Agents" type: software authors: - name: EvalArc contributors -version: 0.7.0 +version: 0.7.1 date-released: 2026-09-14 license: MIT repository-code: "https://github.com/noteflowai/evalarc" diff --git a/README.md b/README.md index 621d7b6..6de5a8f 100644 --- a/README.md +++ b/README.md @@ -27,10 +27,13 @@ The [interactive evidence lab](https://huggingface.co/spaces/glayguo/evalarc) lets you compare revisions side by side, switch between correct and faulty implementations, and step through the tool call that changed the state. It replays the committed Docker audits without a model API or installation. +Share the exact case and trace step with **Copy evidence link**, return from +details to the case list, and retry failed sections independently. +[Explorer guide](docs/explorer.md). [![EvalArc v0.3: score rises from 90% to 93.75% while a check regresses](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc) -**v0.6 includes two working task packs**, a shared evidence format, and +**Two working task packs** share an evidence format and configurable candidate commands: | Task | Interaction | Host verification | Declared faults | diff --git a/README.zh-CN.md b/README.zh-CN.md index d1456ba..8757f34 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -21,10 +21,12 @@ EvalArc 关注智能体实际完成的结果，以及支撑评分结论的证据 [交互证据实验室](https://huggingface.co/spaces/glayguo/evalarc) 可以并排比较两个版本，查看一处退步、两处改进，再逐步检查工具调用和状态变化。 页面读取仓库保存的 Docker 审计记录，无需安装，也不调用模型。 +使用 **Copy evidence link** 分享具体案例和步骤；打开详情后可返回原案例， +某一分区加载失败时可单独重试。[交互使用说明](docs/explorer.md)。 [![EvalArc v0.3：分数上升，一项检查却退步](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc) -**v0.6 已实现 coding 和业务工具两个场景。** +**已实现 coding 和业务工具两个场景。** | 任务 | 交互方式 | 验证内容 | | --- | --- | --- | diff --git a/docs/explorer.md b/docs/explorer.md new file mode 100644 index 0000000..fe65b60 --- /dev/null +++ b/docs/explorer.md @@ -0,0 +1,25 @@ +# Reviewing and sharing recorded evidence + +Use the section navigation to move between acceptance gates, repeated attempts, +revision comparisons and the case explorer. These views replay saved reports; +they do not start candidates. + +In the case explorer, choose a task and implementation, open a recorded case, +and step through its action and state changes. **Copy evidence link** includes +the task, control, case, seed and zero-based trace-step index. The visible step +label starts at one. Coding cases have no tool trace and use step zero. + +The recipient opens the same observation, with focus on its case heading. +**Back to cases** returns focus to the selected row. The URL contains only +selection values, not uploaded evidence or a signed claim about its producer. +Unsupported or out-of-range links display a recovery message. Existing section +anchors continue to navigate without resetting the current case. + +Requests time out after 15 seconds. Retry only the failed section, or open its +standalone report. When one audit task pack loads, it stays usable even if the +other fails. Retrying a missing pack can restore a pending shared link. + +The browser checks exercise desktop and mobile layouts, clipboard denial, +reload/history restoration, keyboard focus, partial loading and retries. +They do not constitute a full accessibility certification or a browser support +matrix. The original casebook and audit bytes retain their recorded provenance. diff --git a/huggingface/README.md b/huggingface/README.md index 07aced0..4ed19c0 100644 --- a/huggingface/README.md +++ b/huggingface/README.md @@ -18,7 +18,12 @@ tags: # EvalArc — Look past the score. -**New in v0.6: Python and JavaScript candidates.** Generate starters or independent +**Share the exact evidence.** Select a case and trace step, then **Copy evidence +link**. Recipients reopen the same recorded observation. Keyboard users can +inspect a case and return to its list; failed sections can be retried separately. +If one task pack fails to load, the other remains inspectable. + +**Python and JavaScript candidates.** Generate starters or independent references for both task packs, audit the declared faults, and combine runtimes in one suite. The task contracts and graders are unchanged. See the [language guide](https://github.com/noteflowai/evalarc/blob/main/docs/languages.md) @@ -76,7 +81,7 @@ dependencies; the bundled trusted controls can run on a CPU. ## Scope -Research preview 0.6.0. These are scripted controls and public development +Research preview 0.7.1. These are scripted controls and public development tasks, not held-out frontier-model results. Detection applies only to the declared faults. No arbitrary reward-hack resistance, human time horizon, hardware-agent validation or RL improvement is established. Repeated fixed diff --git a/package-lock.json b/package-lock.json index 6a24985..e87ef9f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.7.0", + "version": "0.7.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.7.0", + "version": "0.7.1", "devDependencies": { "playwright": "1.63.0" } diff --git a/package.json b/package.json index 3db3fa6..aef4d1a 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.7.0", + "version": "0.7.1", "private": true, "description": "Browser checks for the static EvalArc evidence explorer", "scripts": { diff --git a/pyproject.toml b/pyproject.toml index cff40e3..656b1fd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "evalarc" -version = "0.7.0" +version = "0.7.1" description = "Auditable task environments and evaluations for coding and tool-using agents." readme = "README.md" requires-python = ">=3.11" diff --git a/scripts/build_site.py b/scripts/build_site.py index 4394a2a..f1d58de 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -257,7 +257,13 @@ def build(destination: Path) -> dict: destination.mkdir(parents=True) for path in (ROOT / "site").iterdir(): if path.is_file(): - shutil.copyfile(path, destination / path.name) + if path.name == "index.html": + version = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["version"] + (destination / path.name).write_text( + path.read_text().replace("__EVALARC_VERSION__", version) + ) + else: + shutil.copyfile(path, destination / path.name) for name, directory, *_ in specifications: target = destination / name target.mkdir() diff --git a/scripts/check_site.cjs b/scripts/check_site.cjs index 759fb9c..08cd1ef 100644 --- a/scripts/check_site.cjs +++ b/scripts/check_site.cjs @@ -26,8 +26,12 @@ async function main() { const browser = await chromium.launch({headless:true}); try { const results = []; - for (const width of [1440, 390]) { + for (const width of [1440, 390, 320]) { const page = await browser.newPage({viewport:{width, height:1000}}); + await page.emulateMedia({reducedMotion:"reduce"}); + await page.addInitScript(() => { + Object.defineProperty(navigator, "clipboard", {configurable:true, value:{writeText:async () => { throw new Error("Clipboard denied"); }}}); + }); const errors = []; page.on("pageerror", error => errors.push(error.message)); await page.goto(base, {waitUntil:"networkidle", timeout:60000}); @@ -118,6 +122,31 @@ async function main() { await app.locator("#next-step").click(); const faulty = JSON.parse(await app.locator("#trace-changes").innerText()); assert.equal(Object.values(faulty)[0].after.notes.length, 3); + const detailBounds = await app.locator("#case-detail").boundingBox(); + const traceBounds = await app.locator("#trace-action").boundingBox(); + assert(traceBounds.x + traceBounds.width <= detailBounds.x + detailBounds.width, "Trace panel must scroll within its card"); + assert.equal(await app.locator("#trace-step").getAttribute("aria-valuetext"), "Step 4 of 6"); + await app.locator("#share-case").click(); + const shared = await app.locator("#share-url").inputValue(); + const sharedParams = new URLSearchParams(new URL(shared).hash.slice(1)); + assert.equal(sharedParams.get("case"), "retry-after-commit"); + assert.equal(sharedParams.get("step"), "3"); + assert.equal(await app.locator("body").evaluate(() => document.activeElement.id), "share-url"); + await app.locator("body").evaluate(() => location.reload()); + await app.locator("#share-status").filter({hasText:"Shared evidence restored"}).waitFor(); + assert.equal(await app.locator("#trace-step").inputValue(), "3"); + assert.deepEqual(JSON.parse(await app.locator("#trace-changes").innerText()), faulty); + assert.equal(await app.locator("body").evaluate(() => document.activeElement.id), "case-title"); + await app.locator("#back-to-cases").click(); + assert.equal(await app.locator("body").evaluate(() => document.activeElement.getAttribute("aria-pressed")), "true"); + await app.locator("body").evaluate(() => { location.hash = "v=1&pack=support&control=missing&case=missing&seed=NaN&step=99999"; }); + await app.locator("#share-status").filter({hasText:"Some link values were outside"}).waitFor(); + assert(await app.locator("#next-step").isDisabled()); + await app.locator("body").evaluate(() => history.back()); + await app.locator("#share-status").filter({hasText:"Shared evidence restored"}).waitFor(); + assert.equal(await app.locator("#trace-step").inputValue(), "3"); + await app.locator('nav[aria-label="Evidence sections"] a[href="#suite"]').click(); + assert.equal(await app.locator("#trace-step").inputValue(), "3"); await app.locator("#control").selectOption("reference"); assert.equal(await app.locator("#score").innerText(), "100%"); assert.equal(await app.locator("#verdict").innerText(), "FULLY RESOLVED"); @@ -128,6 +157,8 @@ async function main() { assert.equal(await app.locator("#score").innerText(), "92.5%"); assert.match(await app.locator("#case-title").innerText(), /cas-type-sensitivity/); assert.match(await app.locator("#coding-evidence").innerText(), /response mismatch/); + await app.locator("#share-case").click(); + assert.equal(new URLSearchParams(new URL(await app.locator("#share-url").inputValue()).hash.slice(1)).get("step"), "0"); for (const task of ["support", "coding"]) { await app.locator(`#${task}-task`).click(); const values = await app.locator("#control option").evaluateAll(options => options.map(o => o.value)); @@ -137,6 +168,7 @@ async function main() { for (let i=0; i document.activeElement.id), "case-title"); } } } @@ -181,7 +213,41 @@ async function main() { assert.match(await app.locator("body").innerText(), /retry-after-commit/); } assert.deepEqual(errors, []); - results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, errors}); + results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, sharedTraceRestored:true, keyboardCaseReturn:true, errors}); + await page.close(); + } + if (!process.env.SITE_URL) { + const page = await browser.newPage({viewport:{width:390,height:1000}, reducedMotion:"reduce"}); + const errors = []; + page.on("pageerror", error => errors.push(error.message)); + const failures = new Set(["suite/suite.json", "repeat/faulty/repetition.json", "comparison/current.json", "coding/audit.json"]); + await page.route("**/*.json", route => { + const url = new URL(route.request().url()); + return [...failures].some(file => url.pathname.endsWith("/" + file)) ? route.abort() : route.continue(); + }); + await page.goto(base + "#v=1&pack=coding&control=boolean-equals-one&case=cas-type-sensitivity&seed=17&step=0"); + for (const name of ["suite", "repeat", "comparison", "audit"]) { + await page.locator("#" + name + "-recovery").waitFor({state:"visible"}); + assert(await page.locator("#" + name + "-retry").isEnabled()); + } + assert(await page.locator("#workspace").isVisible()); + assert.equal(await page.locator("#score").innerText(), "93.75%"); + assert(await page.locator("#coding-task").isDisabled()); + assert.match(await page.locator("#share-status").innerText(), /linked coding evidence is unavailable/); + failures.clear(); + for (const name of ["suite", "repeat", "comparison", "audit"]) { + await page.locator("#" + name + "-retry").click(); + await page.locator("#" + name + "-recovery").waitFor({state:"hidden"}); + } + assert(await page.locator("#coding-task").isEnabled()); + assert.equal(await page.locator("#score").innerText(), "92.5%"); + assert.equal(await page.locator("#changed-cases button").count(), 3); + assert.equal(await page.locator("#repeat-attempts a").count(), 3); + assert.equal(await page.locator("#cases button").count(), 15); + assert.match(await page.locator("#share-status").innerText(), /Shared evidence restored/); + assert.equal(await page.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false); + assert.deepEqual(errors, []); + results.push({independentSectionRetry:true, partialTaskPack:true, pendingLinkRestored:true, errors}); await page.close(); } console.log(JSON.stringify({url:base, checks:results}, null, 2)); diff --git a/site/app.js b/site/app.js index 1e0b581..2f206fd 100644 --- a/site/app.js +++ b/site/app.js @@ -14,10 +14,43 @@ let comparison, baseline, current; const pretty = (data) => JSON.stringify(data, null, 2); const percent = (value) => value === null ? "Unassessed" : `${Number((value * 100).toFixed(4))}%`; const repetitions = {}; +async function fetchRecord(path) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), 15000); + try { + const response = await fetch(path, {signal: controller.signal}); + if (!response.ok) throw new Error(`Evidence request failed (${response.status})`); + return await response.json(); + } finally { + clearTimeout(timer); + } +} +async function loadSection(name, load, retry = false) { + const status = $(name === "audit" ? "load-status" : name + "-status"); + const workspace = $(name === "audit" ? "workspace" : name + "-workspace"); + const button = $(name + "-retry"), recovery = $(name + "-recovery"); + button.disabled = true; + status.hidden = false; + status.textContent = "Loading recorded evidence…"; + workspace.setAttribute("aria-busy", "true"); + try { + const partial = await load(); + workspace.hidden = false; + status.hidden = !partial; + status.textContent = partial || ""; + recovery.hidden = !partial; + const restored = name === "audit" && restoreView(); + if (retry && !restored) workspace.focus(); + } catch { + status.textContent = "This evidence could not be loaded. Retry this section or open its standalone report; other sections remain available."; + recovery.hidden = false; + } finally { + workspace.setAttribute("aria-busy", "false"); + button.disabled = false; + } +} async function loadSuite() { - const response = await fetch("suite/suite.json"); - if (!response.ok) throw new Error("Suite evidence unavailable"); - const suite = await response.json(); + const suite = await fetchRecord("suite/suite.json"); const jobs = {}; for (const name of ["partial", "protected"]) { const job = suite.jobs.find(row => row.id === "support-" + name); @@ -40,14 +73,7 @@ async function loadSuite() { protected_gate: jobs.protected.decision, interpretation: suite.interpretation, }); - $("suite-status").hidden = true; - $("suite-workspace").hidden = false; } -loadSuite().catch(() => { - $("suite-status").textContent = "Suite evidence could not be loaded. "; - const link = document.createElement("a"); link.href = "suite/index.html"; link.textContent = "Open the standalone suite report"; - $("suite-status").append(link); -}); function chooseRepetition(name) { const record = repetitions[name]; if (!record) return; @@ -99,26 +125,17 @@ function chooseRepetition(name) { }); } for (const name of ["reference", "faulty"]) $("repeat-" + name).addEventListener("click", () => chooseRepetition(name)); -Promise.all(["reference", "faulty"].map(async (name) => { - const response = await fetch(`repeat/${name}/repetition.json`); - if (!response.ok) throw new Error("Repetition evidence unavailable"); - repetitions[name] = await response.json(); -})).then(() => { +async function loadRepetitions() { + await Promise.all(["reference", "faulty"].map(async (name) => { + repetitions[name] = await fetchRecord(`repeat/${name}/repetition.json`); + })); chooseRepetition("faulty"); - $("repeat-status").hidden = true; - $("repeat-workspace").hidden = false; -}).catch(() => { - $("repeat-status").textContent = "Repetition evidence could not be loaded. Open a standalone report:"; - for (const name of ["reference", "faulty"]) { - const link = document.createElement("a"); link.href = `repeat/${name}/index.html`; link.textContent = ` ${name} report`; - $("repeat-status").append(link); - } -}); +} function badge(element, passed, text) { element.className = `badge ${passed ? "pass" : "fail"}`; element.textContent = text; } -function choosePack(nextPack) { +function choosePack(nextPack, view) { pack = nextPack; for (const name of Object.keys(packs)) $(name + "-task").setAttribute("aria-pressed", String(pack === name)); const select = $("control"); @@ -130,11 +147,12 @@ function choosePack(nextPack) { select.append(option); } select.value = packs[pack].defaultControl; + if (view?.control && [...select.options].some(option => option.value === view.control)) select.value = view.control; $("report-link").href = `${packs[pack].path}/index.html`; $("download-link").href = `${packs[pack].path}/audit.json`; - chooseControl(); + chooseControl(view); } -function chooseControl() { +function chooseControl(view) { const name = $("control").value; const audit = audits[pack]; const mutant = audit.mutants.find((row) => row.name === name); @@ -159,7 +177,8 @@ function chooseControl() { const button = document.createElement("button"); button.type = "button"; const status = document.createElement("span"); status.className = row.passed ? "pass" : "fail"; status.textContent = row.passed ? "PASS" : row.status.toUpperCase(); button.append(status, document.createTextNode(row.case_id)); - button.addEventListener("click", () => chooseCase(index)); + button.setAttribute("aria-controls", "case-detail"); + button.addEventListener("click", () => { chooseCase(index); rememberView(); $("case-title").focus(); }); $("cases").append(button); }); $("provenance").textContent = pretty({ @@ -170,9 +189,13 @@ function chooseControl() { }); let index = evaluation.cases.findIndex((row) => !row.passed); if (index < 0) index = evaluation.cases.findIndex((row) => row.case_id === packs[pack].defaultCase); - chooseCase(Math.max(index, 0)); + if (view) { + const requested = evaluation.cases.findIndex(row => row.case_id === view.case && String(row.seed) === view.seed); + if (requested >= 0) index = requested; + } + chooseCase(Math.max(index, 0), view?.step); } -function chooseCase(index) { +function chooseCase(index, step = 0) { selectedCase = evaluation.cases[index]; [...$("cases").children].forEach((button, i) => button.setAttribute("aria-pressed", String(i === index))); $("case-title").textContent = `${selectedCase.case_id} / seed ${selectedCase.seed}`; @@ -188,7 +211,7 @@ function chooseCase(index) { if (hasTrace) { $("states").textContent = pretty({initial_state: selectedCase.initial_state, final_state: selectedCase.final_state}); $("trace-step").max = selectedCase.trace.length - 1; - $("trace-step").value = 0; + $("trace-step").value = Math.max(0, Math.min(selectedCase.trace.length - 1, step)); showStep(); } else { $("coding-evidence").textContent = pretty(selectedCase); @@ -198,11 +221,66 @@ function showStep() { const index = Number($("trace-step").value); const step = selectedCase.trace[index]; $("step-label").textContent = `${index + 1} / ${selectedCase.trace.length}`; + $("trace-step").setAttribute("aria-valuetext", `Step ${index + 1} of ${selectedCase.trace.length}`); $("trace-action").textContent = pretty(Object.fromEntries(Object.entries(step).filter(([key]) => key !== "changes"))); $("trace-changes").textContent = step.changes && Object.keys(step.changes).length ? pretty(step.changes) : "No state change in this step."; $("previous-step").disabled = index === 0; $("next-step").disabled = index === selectedCase.trace.length - 1; } +function viewFragment() { + return "#" + new URLSearchParams({ + v: "1", pack, control: $("control").value, case: selectedCase.case_id, + seed: String(selectedCase.seed), step: $("trace-panel").hidden ? "0" : $("trace-step").value, + }); +} +function rememberView() { + $("share-url").hidden = true; + $("share-status").textContent = ""; + try { history.replaceState(null, "", viewFragment()); } catch { /* Restricted embeds can still copy links. */ } +} +function restoreView(focus = true) { + if (!location.hash.includes("=")) return false; + const params = new URLSearchParams(location.hash.slice(1)); + if (location.hash.length > 2048 || params.get("v") !== "1" || !Object.hasOwn(packs, params.get("pack"))) { + $("share-status").textContent = "This evidence link is not supported. Choose a task, control and case below."; + return false; + } + const requestedPack = params.get("pack"); + if (!audits[requestedPack]) { + $("share-status").textContent = `The linked ${requestedPack} evidence is unavailable. Retry the missing task pack to open this view.`; + return false; + } + const step = Number(params.get("step")); + const view = {control: params.get("control"), case: params.get("case"), seed: params.get("seed"), + step: Number.isSafeInteger(step) && step >= 0 ? step : 0}; + choosePack(requestedPack, view); + const exact = $("control").value === view.control && selectedCase.case_id === view.case && + String(selectedCase.seed) === view.seed && params.has("step") && String(view.step) === params.get("step") && + (selectedCase.trace?.length ? Number($("trace-step").value) === view.step : view.step === 0); + $("share-url").hidden = true; + $("share-status").textContent = exact + ? "Shared evidence restored. The link selects a recorded case and trace step." + : "Some link values were outside this recording. Showing the nearest available view; check the selected case and step."; + if (focus) $("case-title").focus(); + return true; +} +$("share-case").addEventListener("click", async () => { + const url = new URL(location.href); + url.hash = viewFragment(); + try { + await navigator.clipboard.writeText(url.href); + $("share-url").hidden = true; + $("share-status").textContent = "Evidence link copied: task, control, seed, case and trace step."; + } catch { + $("share-url").hidden = false; + $("share-url").value = url.href; + $("share-url").focus(); + $("share-url").select(); + $("share-status").textContent = "Clipboard unavailable here. Select and copy the evidence link."; + } +}); +$("back-to-cases").addEventListener("click", () => $("cases").querySelector('[aria-pressed="true"]')?.focus()); +window.addEventListener("hashchange", () => { if (evaluation) restoreView(); }); function chooseTransition(index) { const transition = comparison.case_transitions[index]; [...$("changed-cases").children].forEach((button, i) => button.setAttribute("aria-pressed", String(i === index))); @@ -224,15 +302,12 @@ function chooseTransition(index) { } } async function loadComparison() { - [comparison, baseline, current] = await Promise.all(["comparison", "baseline", "current"].map(async (name) => { - const response = await fetch(`comparison/${name}.json`); - if (!response.ok) throw new Error("Comparison evidence unavailable"); - return response.json(); - })); + [comparison, baseline, current] = await Promise.all(["comparison", "baseline", "current"].map(name => fetchRecord(`comparison/${name}.json`))); $("before-score").textContent = percent(comparison.baseline.score); $("after-score").textContent = percent(comparison.current.score); $("regression-count").textContent = `${comparison.regressions.length} REGRESSED CHECK`; $("compare-delta").textContent = `+${Number((comparison.score_delta * 100).toFixed(4))} percentage points`; + $("changed-cases").replaceChildren(); comparison.case_transitions.forEach((transition, index) => { const button = document.createElement("button"); button.type = "button"; const tag = document.createElement("span"); tag.className = transition.regressed_checks.length ? "fail" : "pass"; @@ -242,33 +317,28 @@ async function loadComparison() { $("changed-cases").append(button); }); chooseTransition(0); - $("comparison-status").hidden = true; - $("comparison-workspace").hidden = false; } -loadComparison().catch(() => { - $("comparison-status").textContent = "Comparison evidence could not be loaded. Reload the page or open the standalone report."; - const link = document.createElement("a"); link.href = "comparison/index.html"; link.textContent = "Open comparison report"; - $("comparison-status").append(document.createTextNode(" "), link); -}); -$("control").addEventListener("change", chooseControl); -$("trace-step").addEventListener("input", showStep); -$("previous-step").addEventListener("click", () => { $("trace-step").stepDown(); showStep(); }); -$("next-step").addEventListener("click", () => { $("trace-step").stepUp(); showStep(); }); -for (const name of Object.keys(packs)) $(name + "-task").addEventListener("click", () => { if (audits[name]) choosePack(name); }); -Promise.all(Object.entries(packs).map(async ([name, config]) => { - const response = await fetch(`${config.path}/audit.json`); - if (!response.ok) throw new Error(`Evidence request failed (${response.status})`); - audits[name] = await response.json(); -})).then(() => { - choosePack("support"); - $("load-status").hidden = true; - $("workspace").hidden = false; -}).catch(() => { - $("load-status").textContent = "Evidence could not be loaded. Reload this page, or open the standalone support and coding reports linked below."; - const links = document.createElement("p"); - for (const [name, config] of Object.entries(packs)) { - const link = document.createElement("a"); link.href = `${config.path}/index.html`; link.textContent = `Open ${name} report`; - links.append(link, document.createTextNode(" ")); - } - $("load-status").after(links); +$("control").addEventListener("change", () => { chooseControl(); rememberView(); }); +$("trace-step").addEventListener("input", () => { showStep(); rememberView(); }); +$("previous-step").addEventListener("click", () => { $("trace-step").stepDown(); showStep(); rememberView(); }); +$("next-step").addEventListener("click", () => { $("trace-step").stepUp(); showStep(); rememberView(); }); +for (const name of Object.keys(packs)) $(name + "-task").addEventListener("click", () => { + if (audits[name]) { choosePack(name); rememberView(); } }); +async function loadAudits() { + await Promise.allSettled(Object.entries(packs).filter(([name]) => !audits[name]).map(async ([name, config]) => { + const record = await fetchRecord(`${config.path}/audit.json`); + if (!Array.isArray(record.reference?.cases) || !Array.isArray(record.mutants) || + record.mutants.some(row => !Array.isArray(row.evaluation?.cases))) throw new Error("Incomplete audit evidence"); + audits[name] = record; + $(name + "-task").disabled = false; + })); + const available = Object.keys(audits), missing = Object.keys(packs).filter(name => !audits[name]); + if (!available.length) throw new Error("No task pack available"); + if (!evaluation) choosePack(audits.support ? "support" : available[0]); + return missing.length ? `Could not load ${missing.join(" and ")} evidence. The other task pack is ready; retry to load the missing pack.` : ""; +} +for (const [name, load] of [["suite", loadSuite], ["repeat", loadRepetitions], ["comparison", loadComparison], ["audit", loadAudits]]) { + $(name + "-retry").addEventListener("click", () => void loadSection(name, load, true)); + void loadSection(name, load); +} diff --git a/site/index.html b/site/index.html index 11fe91b..ee3d05d 100644 --- a/site/index.html +++ b/site/index.html @@ -19,7 +19,7 @@
-

OPEN SOURCE   /   RESEARCH PREVIEW 0.6

+

OPEN SOURCE   /   RESEARCH PREVIEW __EVALARC_VERSION__

Look past
the score.

A tool returned an error. The write already happened.
A retry added the same note twice.

EvalArc tests whether an agent's grader catches the difference between plausible progress and a completed task.

@@ -36,18 +36,20 @@

Look past
the score.

Scripted negative control · public seed 17
No model call or live customer data.

-

New in v0.6: generate Python or JavaScript candidates and audit independent controls against the same task contracts. Try both runtimes ↗. The recorded showcases below retain their original versions and fingerprints.

+

Generate Python or JavaScript candidates and audit independent controls against the same task contracts. Try both runtimes ↗. The recorded showcases below retain their original versions and fingerprints.

2task packs
15declared faults caught
JSONdownloadable evidence
CPUreproduce without a model API
+

NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES

Same score. Different gate.

One frozen policy.
Two explicit acceptance rules.

Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.

Loading the recorded suite…

-