Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
# Changelog

## 0.7.1 · 2026-09-14 · Research preview

- Share and restore a specific task pack, control, seed, case and trace step in
the evidence explorer, including a selectable link when clipboard access is
denied. Out-of-range links show an explicit fallback.
- Retry failed suite, repetition, comparison or task-pack requests independently.
Bound network waits and keep a successfully loaded task pack usable when the
other fails; retrying can restore the originally linked evidence.
- Add section navigation, case-detail focus and return controls, keyboard-readable
JSON panels, larger buttons and readable mobile evidence text.
- Render the site's preview version from package metadata.

Original reports, task contracts, scoring and CLI behavior are unchanged.

## 0.7.0 · 2026-09-14 · Research preview

- Add `evalarc verify` for received evaluation, repetition and comparison
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
type: software
authors:
- name: EvalArc contributors
version: 0.7.0
version: 0.7.1
date-released: 2026-09-14
license: MIT
repository-code: "https://github.com/noteflowai/evalarc"
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,13 @@ The [interactive evidence lab](https://huggingface.co/spaces/glayguo/evalarc)
lets you compare revisions side by side, switch between correct and faulty
implementations, and step through the tool call that changed the state. It replays the
committed Docker audits without a model API or installation.
Share the exact case and trace step with **Copy evidence link**, return from
details to the case list, and retry failed sections independently.
[Explorer guide](docs/explorer.md).

[![EvalArc v0.3: score rises from 90% to 93.75% while a check regresses](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.6 includes two working task packs**, a shared evidence format, and
**Two working task packs** share an evidence format and
configurable candidate commands:

| Task | Interaction | Host verification | Declared faults |
Expand Down
4 changes: 3 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@ EvalArc 关注智能体实际完成的结果,以及支撑评分结论的证据
[交互证据实验室](https://huggingface.co/spaces/glayguo/evalarc)
可以并排比较两个版本,查看一处退步、两处改进,再逐步检查工具调用和状态变化。
页面读取仓库保存的 Docker 审计记录,无需安装,也不调用模型。
使用 **Copy evidence link** 分享具体案例和步骤;打开详情后可返回原案例,
某一分区加载失败时可单独重试。[交互使用说明](docs/explorer.md)。

[![EvalArc v0.3:分数上升,一项检查却退步](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.6 已实现 coding 和业务工具两个场景。**
**已实现 coding 和业务工具两个场景。**

| 任务 | 交互方式 | 验证内容 |
| --- | --- | --- |
Expand Down
25 changes: 25 additions & 0 deletions docs/explorer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Reviewing and sharing recorded evidence

Use the section navigation to move between acceptance gates, repeated attempts,
revision comparisons and the case explorer. These views replay saved reports;
they do not start candidates.

In the case explorer, choose a task and implementation, open a recorded case,
and step through its action and state changes. **Copy evidence link** includes
the task, control, case, seed and zero-based trace-step index. The visible step
label starts at one. Coding cases have no tool trace and use step zero.

The recipient opens the same observation, with focus on its case heading.
**Back to cases** returns focus to the selected row. The URL contains only
selection values, not uploaded evidence or a signed claim about its producer.
Unsupported or out-of-range links display a recovery message. Existing section
anchors continue to navigate without resetting the current case.

Requests time out after 15 seconds. Retry only the failed section, or open its
standalone report. When one audit task pack loads, it stays usable even if the
other fails. Retrying a missing pack can restore a pending shared link.

The browser checks exercise desktop and mobile layouts, clipboard denial,
reload/history restoration, keyboard focus, partial loading and retries.
They do not constitute a full accessibility certification or a browser support
matrix. The original casebook and audit bytes retain their recorded provenance.
9 changes: 7 additions & 2 deletions huggingface/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@ tags:

# EvalArc — Look past the score.

**New in v0.6: Python and JavaScript candidates.** Generate starters or independent
**Share the exact evidence.** Select a case and trace step, then **Copy evidence
link**. Recipients reopen the same recorded observation. Keyboard users can
inspect a case and return to its list; failed sections can be retried separately.
If one task pack fails to load, the other remains inspectable.

**Python and JavaScript candidates.** Generate starters or independent
references for both task packs, audit the declared faults, and combine runtimes
in one suite. The task contracts and graders are unchanged. See the
[language guide](https://github.com/noteflowai/evalarc/blob/main/docs/languages.md)
Expand Down Expand Up @@ -76,7 +81,7 @@ dependencies; the bundled trusted controls can run on a CPU.

## Scope

Research preview 0.6.0. These are scripted controls and public development
Research preview 0.7.1. These are scripted controls and public development
tasks, not held-out frontier-model results. Detection applies only to the
declared faults. No arbitrary reward-hack resistance, human time horizon,
hardware-agent validation or RL improvement is established. Repeated fixed
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "evalarc-evidence-site",
"version": "0.7.0",
"version": "0.7.1",
"private": true,
"description": "Browser checks for the static EvalArc evidence explorer",
"scripts": {
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "evalarc"
version = "0.7.0"
version = "0.7.1"
description = "Auditable task environments and evaluations for coding and tool-using agents."
readme = "README.md"
requires-python = ">=3.11"
Expand Down
8 changes: 7 additions & 1 deletion scripts/build_site.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,13 @@ def build(destination: Path) -> dict:
destination.mkdir(parents=True)
for path in (ROOT / "site").iterdir():
if path.is_file():
shutil.copyfile(path, destination / path.name)
if path.name == "index.html":
version = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["version"]
(destination / path.name).write_text(
path.read_text().replace("__EVALARC_VERSION__", version)
)
else:
shutil.copyfile(path, destination / path.name)
for name, directory, *_ in specifications:
target = destination / name
target.mkdir()
Expand Down
70 changes: 68 additions & 2 deletions scripts/check_site.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,12 @@ async function main() {
const browser = await chromium.launch({headless:true});
try {
const results = [];
for (const width of [1440, 390]) {
for (const width of [1440, 390, 320]) {
const page = await browser.newPage({viewport:{width, height:1000}});
await page.emulateMedia({reducedMotion:"reduce"});
await page.addInitScript(() => {
Object.defineProperty(navigator, "clipboard", {configurable:true, value:{writeText:async () => { throw new Error("Clipboard denied"); }}});
});
const errors = [];
page.on("pageerror", error => errors.push(error.message));
await page.goto(base, {waitUntil:"networkidle", timeout:60000});
Expand Down Expand Up @@ -118,6 +122,31 @@ async function main() {
await app.locator("#next-step").click();
const faulty = JSON.parse(await app.locator("#trace-changes").innerText());
assert.equal(Object.values(faulty)[0].after.notes.length, 3);
const detailBounds = await app.locator("#case-detail").boundingBox();
const traceBounds = await app.locator("#trace-action").boundingBox();
assert(traceBounds.x + traceBounds.width <= detailBounds.x + detailBounds.width, "Trace panel must scroll within its card");
assert.equal(await app.locator("#trace-step").getAttribute("aria-valuetext"), "Step 4 of 6");
await app.locator("#share-case").click();
const shared = await app.locator("#share-url").inputValue();
const sharedParams = new URLSearchParams(new URL(shared).hash.slice(1));
assert.equal(sharedParams.get("case"), "retry-after-commit");
assert.equal(sharedParams.get("step"), "3");
assert.equal(await app.locator("body").evaluate(() => document.activeElement.id), "share-url");
await app.locator("body").evaluate(() => location.reload());
await app.locator("#share-status").filter({hasText:"Shared evidence restored"}).waitFor();
assert.equal(await app.locator("#trace-step").inputValue(), "3");
assert.deepEqual(JSON.parse(await app.locator("#trace-changes").innerText()), faulty);
assert.equal(await app.locator("body").evaluate(() => document.activeElement.id), "case-title");
await app.locator("#back-to-cases").click();
assert.equal(await app.locator("body").evaluate(() => document.activeElement.getAttribute("aria-pressed")), "true");
await app.locator("body").evaluate(() => { location.hash = "v=1&pack=support&control=missing&case=missing&seed=NaN&step=99999"; });
await app.locator("#share-status").filter({hasText:"Some link values were outside"}).waitFor();
assert(await app.locator("#next-step").isDisabled());
await app.locator("body").evaluate(() => history.back());
await app.locator("#share-status").filter({hasText:"Shared evidence restored"}).waitFor();
assert.equal(await app.locator("#trace-step").inputValue(), "3");
await app.locator('nav[aria-label="Evidence sections"] a[href="#suite"]').click();
assert.equal(await app.locator("#trace-step").inputValue(), "3");
await app.locator("#control").selectOption("reference");
assert.equal(await app.locator("#score").innerText(), "100%");
assert.equal(await app.locator("#verdict").innerText(), "FULLY RESOLVED");
Expand All @@ -128,6 +157,8 @@ async function main() {
assert.equal(await app.locator("#score").innerText(), "92.5%");
assert.match(await app.locator("#case-title").innerText(), /cas-type-sensitivity/);
assert.match(await app.locator("#coding-evidence").innerText(), /response mismatch/);
await app.locator("#share-case").click();
assert.equal(new URLSearchParams(new URL(await app.locator("#share-url").inputValue()).hash.slice(1)).get("step"), "0");
for (const task of ["support", "coding"]) {
await app.locator(`#${task}-task`).click();
const values = await app.locator("#control option").evaluateAll(options => options.map(o => o.value));
Expand All @@ -137,6 +168,7 @@ async function main() {
for (let i=0; i<await app.locator("#cases button").count();i++) {
await app.locator("#cases button").nth(i).click();
assert.ok((await app.locator("#case-title").innerText()).includes("seed 17"));
assert.equal(await app.locator("body").evaluate(() => document.activeElement.id), "case-title");
}
}
}
Expand Down Expand Up @@ -181,7 +213,41 @@ async function main() {
assert.match(await app.locator("body").innerText(), /retry-after-commit/);
}
assert.deepEqual(errors, []);
results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, errors});
results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, sharedTraceRestored:true, keyboardCaseReturn:true, errors});
await page.close();
}
if (!process.env.SITE_URL) {
const page = await browser.newPage({viewport:{width:390,height:1000}, reducedMotion:"reduce"});
const errors = [];
page.on("pageerror", error => errors.push(error.message));
const failures = new Set(["suite/suite.json", "repeat/faulty/repetition.json", "comparison/current.json", "coding/audit.json"]);
await page.route("**/*.json", route => {
const url = new URL(route.request().url());
return [...failures].some(file => url.pathname.endsWith("/" + file)) ? route.abort() : route.continue();
});
await page.goto(base + "#v=1&pack=coding&control=boolean-equals-one&case=cas-type-sensitivity&seed=17&step=0");
for (const name of ["suite", "repeat", "comparison", "audit"]) {
await page.locator("#" + name + "-recovery").waitFor({state:"visible"});
assert(await page.locator("#" + name + "-retry").isEnabled());
}
assert(await page.locator("#workspace").isVisible());
assert.equal(await page.locator("#score").innerText(), "93.75%");
assert(await page.locator("#coding-task").isDisabled());
assert.match(await page.locator("#share-status").innerText(), /linked coding evidence is unavailable/);
failures.clear();
for (const name of ["suite", "repeat", "comparison", "audit"]) {
await page.locator("#" + name + "-retry").click();
await page.locator("#" + name + "-recovery").waitFor({state:"hidden"});
}
assert(await page.locator("#coding-task").isEnabled());
assert.equal(await page.locator("#score").innerText(), "92.5%");
assert.equal(await page.locator("#changed-cases button").count(), 3);
assert.equal(await page.locator("#repeat-attempts a").count(), 3);
assert.equal(await page.locator("#cases button").count(), 15);
assert.match(await page.locator("#share-status").innerText(), /Shared evidence restored/);
assert.equal(await page.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false);
assert.deepEqual(errors, []);
results.push({independentSectionRetry:true, partialTaskPack:true, pendingLinkRestored:true, errors});
await page.close();
}
console.log(JSON.stringify({url:base, checks:results}, null, 2));
Expand Down
Loading
Loading