Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## 0.11.1 — 2026-09-15

- Use explicit HTML filenames for published lab navigation. Hugging Face redirects bare directory paths to Hub routes rather than serving each directory index. Cover homepage entries and research/skill-lab return links.
- Reseal only the published lab presentation files; recorded trials, raw JSON and original research archives stay byte-identical. Test navigation with a static server that rejects implicit directory indexes.

## 0.11.0 — 2026-09-15

- Add offline AgentCore Evaluate import, golden-case/rubric comparison and recomputable trace review with responsive standalone reports.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,11 +117,11 @@ also requires all checks to pass. [Verification and limits](docs/verification.md

## Trace Workbench — 0.11.0

Import saved AgentCore Evaluate responses, versioned golden cases and Skills Anywhere delivery receipts. Inspect zero scores, skipped judges, missing results and missed skills separately. Compare matching datasets/rubrics and verify preserved input bytes offline. [Try the authored controls](https://noteflowai.github.io/evalarc/trace-workbench/) · [Actual local MCP delivery](https://noteflowai.github.io/evalarc/trace-mcp/) · [Input contract](docs/trace-workbench.md). No live AWS evaluation is claimed.
Import saved AgentCore Evaluate responses, versioned golden cases and Skills Anywhere delivery receipts. Inspect zero scores, skipped judges, missing results and missed skills separately. Compare matching datasets/rubrics and verify preserved input bytes offline. [Try the authored controls](https://noteflowai.github.io/evalarc/trace-workbench/index.html) · [Actual local MCP delivery](https://noteflowai.github.io/evalarc/trace-mcp/index.html) · [Input contract](docs/trace-workbench.md). No live AWS evaluation is claimed.

## New in 0.9.0: research you can inspect

[Explore all 27 real GPU skill trials](https://noteflowai.github.io/evalarc/skill-impact/) and [the research pilots](docs/research-pilots.md). Robot Reel's [captured-scene editor](https://noteflowai.github.io/robot-reel/scene-lab/) and [official LIBERO-Plus replay](https://noteflowai.github.io/robot-reel/libero-plus/) connect real source records with portable skill delivery and independent grading. Every failed attempt stays visible; no skill efficacy, full-benchmark or real-hardware result is implied.
[Explore all 27 real GPU skill trials](https://noteflowai.github.io/evalarc/skill-impact/index.html) and [the research pilots](docs/research-pilots.md). Robot Reel's [captured-scene editor](https://noteflowai.github.io/robot-reel/scene-lab/) and [official LIBERO-Plus replay](https://noteflowai.github.io/robot-reel/libero-plus/) connect real source records with portable skill delivery and independent grading. Every failed attempt stays visible; no skill efficacy, full-benchmark or real-hardware result is implied.


## Run an audit
Expand Down
4 changes: 2 additions & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ Docker 评测:参考策略 3/3 轮完全通过,重复写入策略虽然平

## 0.11.0:运行记录评估工作台

导入已保存的 AgentCore Evaluate 结果、版本化黄金案例和 Skills Anywhere 加载回执,分别查看有效零分、评估跳过、缺少结果及漏调用技能。支持相同测试集与评分规则下的对比,以及原始输入的离线复核。[交互示例](https://noteflowai.github.io/evalarc/trace-workbench/) · [真实本地 MCP 加载](https://noteflowai.github.io/evalarc/trace-mcp/) · [数据契约](docs/trace-workbench.md)。示例明确区分合成评分与真实加载记录,未运行云端评估。
导入已保存的 AgentCore Evaluate 结果、版本化黄金案例和 Skills Anywhere 加载回执,分别查看有效零分、评估跳过、缺少结果及漏调用技能。支持相同测试集与评分规则下的对比,以及原始输入的离线复核。[交互示例](https://noteflowai.github.io/evalarc/trace-workbench/index.html) · [真实本地 MCP 加载](https://noteflowai.github.io/evalarc/trace-mcp/index.html) · [数据契约](docs/trace-workbench.md)。示例明确区分合成评分与真实加载记录,未运行云端评估。

## 0.9.0:有原始证据的研究场景

[查看 27 次真实 GPU 技能评测](https://noteflowai.github.io/evalarc/skill-impact/),并阅读[完整方法与限制](docs/research-pilots.md)。新增[实景 Blender 编辑](https://noteflowai.github.io/robot-reel/scene-lab/)与[官方 LIBERO-Plus 子集回放](https://noteflowai.github.io/robot-reel/libero-plus/),把原始记录、技能交付与独立验收连接起来。失败尝试全部保留;不宣称技能提分、完整基准成绩或真机效果。
[查看 27 次真实 GPU 技能评测](https://noteflowai.github.io/evalarc/skill-impact/index.html),并阅读[完整方法与限制](docs/research-pilots.md)。新增[实景 Blender 编辑](https://noteflowai.github.io/robot-reel/scene-lab/)与[官方 LIBERO-Plus 子集回放](https://noteflowai.github.io/robot-reel/libero-plus/),把原始记录、技能交付与独立验收连接起来。失败尝试全部保留;不宣称技能提分、完整基准成绩或真机效果。


## 直接运行
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "evalarc-evidence-site",
"version": "0.11.0",
"version": "0.11.1",
"private": true,
"description": "Browser checks for the static EvalArc evidence explorer",
"scripts": {
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "evalarc"
version = "0.11.0"
version = "0.11.1"
description = "Auditable task environments and evaluations for coding and tool-using agents."
readme = "README.md"
requires-python = ">=3.11"
Expand Down
28 changes: 28 additions & 0 deletions scripts/build_site.py
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,30 @@ def sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()


def explicit_lab_navigation(folder: Path, commit: str) -> None:
"""Update copied presentation links, preserving every original experiment byte."""
page = folder / "index.html"
original = page.read_bytes()
updated = original.replace(b'href="../"', b'href="../index.html"').replace(
b'href="../skill-impact/"', b'href="../skill-impact/index.html"'
)
if updated == original:
return
page.write_bytes(updated)
manifest_path = folder / "manifest.json"
manifest = json.loads(manifest_path.read_bytes())
manifest["presentation"] = {
"source_commit": commit,
"original_index_sha256": hashlib.sha256(original).hexdigest(),
"change": "Explicit index.html navigation for static hosting",
}
manifest["files"]["index.html"] = {
"sha256": hashlib.sha256(updated).hexdigest(),
"bytes": len(updated),
}
manifest_path.write_text(json.dumps(manifest, indent=2) + "\n")


def verify(folder: Path) -> dict:
record = json.loads((folder / MANIFEST).read_text())
if record["source_repository"] != SOURCE or not re.fullmatch(
Expand Down Expand Up @@ -394,6 +418,10 @@ def build(destination: Path) -> dict:
verify_records(ROOT / "examples/research")
shutil.copytree(ROOT / "examples/skill-impact", destination / "skill-impact")
shutil.copytree(ROOT / "examples/research", destination / "research")
for lab in ("skill-impact", "research"):
explicit_lab_navigation(destination / lab, commit)
verify_lab(destination / "skill-impact")
verify_records(destination / "research")
from evalarc.trace_review import import_trace

trace_examples = ROOT / "examples/trace-workbench"
Expand Down
23 changes: 17 additions & 6 deletions scripts/check_site.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ async function main() {
const mime = {".html":"text/html", ".js":"text/javascript", ".json":"application/json", ".css":"text/css", ".svg":"image/svg+xml", ".zip":"application/zip"};
server = http.createServer((req, res) => {
const route = decodeURIComponent(new URL(req.url, "http://localhost").pathname);
const filename = path.resolve(root, "." + (route.endsWith("/") ? route + "index.html" : route));
// HF static hosting does not resolve subdirectory indexes. Keep that
// production constraint in the local browser fixture.
if (route !== "/" && route.endsWith("/")) {
res.writeHead(404).end(); return;
}
const filename = path.resolve(root, "." + (route === "/" ? "/index.html" : route));
if (!filename.startsWith(root + path.sep) || !fs.existsSync(filename) || !fs.statSync(filename).isFile()) {
res.writeHead(404).end(); return;
}
Expand Down Expand Up @@ -226,7 +231,7 @@ async function main() {
await app.getByRole("heading", {name:"support-routing", exact:true}).waitFor();
assert.match(await app.locator("body").innerText(), /retry-after-commit/);
}
await app.locator("body").evaluate((element, url) => { location.href = new URL("skill-impact/", url).href; }, appUrl);
await app.locator("body").evaluate((element, url) => { location.href = new URL("skill-impact/index.html", url).href; }, appUrl);
await app.locator("#workspace").waitFor({state:"visible"});
assert.equal(await app.locator(".trial").count(), 9);
assert.match(await app.locator("#profile-totals").innerText(), /2\/9 fully resolved/);
Expand All @@ -248,10 +253,14 @@ async function main() {
fs.mkdirSync(process.env.RESEARCH_SCREENSHOTS,{recursive:true});
await page.screenshot({path:path.join(process.env.RESEARCH_SCREENSHOTS,"skill-impact.png"),fullPage:true});
}
await app.locator("body").evaluate((element, url) => { location.href = new URL("research/", url).href; }, appUrl);
await app.locator("body").evaluate((element, url) => { location.href = new URL("research/index.html", url).href; }, appUrl);
await app.getByRole("heading",{name:"Skill composition: 12 attempts, 3 accepted"}).waitFor();
assert.equal(await app.locator("tbody tr").count(),18);
assert.equal(await app.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false);
await app.getByRole("link",{name:"27-trial Skill Impact Lab"}).click();
await app.locator("#profile").waitFor();
await app.getByRole("link",{name:"Playground"}).click();
await app.locator("#workspace").waitFor({state:"visible"});
const coveragePage = await browser.newPage({viewport:{width,height:1000}});
try {
await coveragePage.goto(new URL("robot/index.html#fault-3", suiteBase).href);
Expand Down Expand Up @@ -298,7 +307,7 @@ async function main() {
assert.equal(await page.locator("body").evaluate(() => document.documentElement.scrollWidth > innerWidth), false);
assert.deepEqual(errors, []);
failures.add("skill-impact/lab.json");
await page.goto(new URL("skill-impact/",base).href);
await page.goto(new URL("skill-impact/index.html",base).href);
await page.locator("#retry").waitFor({state:"visible"});
failures.clear();
await page.locator("#retry").click();
Expand Down Expand Up @@ -361,7 +370,8 @@ async function main() {
const errors = [];
page.on("pageerror", error => errors.push(error.message));
try {
await page.goto(new URL("trace-workbench/",base).href);
await page.goto(base);
await page.getByRole("link",{name:"Explore five authored review controls"}).click();
await page.locator("#filter-status").filter({hasText:"5 cases shown"}).waitFor();
assert.equal(await page.locator("article[data-gate=accepted]").count(),1);
assert.equal(await page.locator("article[data-gate=rejected]").count(),2);
Expand All @@ -387,7 +397,8 @@ async function main() {
assert.equal(await download.failure(),null);
const original = fs.readFileSync(path.join(root,"trace-workbench/input.json"));
assert.deepEqual(fs.readFileSync(await download.path()),original);
await page.goto(new URL("trace-mcp/",base).href);
await page.goto(base);
await page.getByRole("link",{name:"Inspect an actual MCP delivery"}).click();
await page.locator("#filter-status").filter({hasText:"1 cases shown"}).waitFor();
assert.match(await page.locator("body").innerText(),/Actual local stdio MCP/);
assert.match(await page.locator("article").innerText(),/MATCHED/);
Expand Down
4 changes: 2 additions & 2 deletions site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ <h1>Look past<br>the <em>score.</em></h1>
</a>
</section>
<p class="muted">Generate Python or JavaScript candidates and audit independent controls against the same task contracts. <a href="https://github.com/noteflowai/evalarc/blob/main/docs/languages.md">Try both runtimes &#8599;</a>. The recorded showcases below retain their original versions and fingerprints.</p>
<section class="suite" aria-label="Real GPU research pilots"><p class="eyebrow">NEW / RECORDED MODEL EVIDENCE</p><h2>A skill loaded. Did the task pass?</h2><p>27 real Qwen3-8B trials compare no skill, direct loading and MCP delivery on attributed robot recordings. Inspect every tool receipt, candidate, independent score and ATIF trajectory. Three engineering profiles, including negative results.</p><div class="comparison-links"><a href="skill-impact/">Explore the Skill Impact Lab &#8599;</a><a href="research/">Composition, handoff &amp; Harbor evidence &#8599;</a></div><p class="caption">L40S recordings; one public development task. No skill accuracy gain or general model ranking is claimed.</p></section>
<section class="suite" aria-label="Trace Workbench"><p class="eyebrow">NEW / BRING YOUR AGENT RECORDS</p><h2>Zero, skipped, or missing?</h2><p>A zero can be a valid judgment. A skipped evaluator needs context. A required skill may never have loaded. Inspect each against a versioned golden case, with recording identity and explicit acceptance rules.</p><div class="comparison-links"><a href="trace-workbench/">Explore five authored review controls &#8599;</a><a href="trace-mcp/">Inspect an actual MCP delivery &#8599;</a></div><p class="caption">The five controls are synthetic. The separate MCP record contains a real local load and no evaluator scores. Review your own saved AgentCore Evaluate export offline with <code>evalarc trace-import</code>.</p></section>
<section class="suite" aria-label="Real GPU research pilots"><p class="eyebrow">NEW / RECORDED MODEL EVIDENCE</p><h2>A skill loaded. Did the task pass?</h2><p>27 real Qwen3-8B trials compare no skill, direct loading and MCP delivery on attributed robot recordings. Inspect every tool receipt, candidate, independent score and ATIF trajectory. Three engineering profiles, including negative results.</p><div class="comparison-links"><a href="skill-impact/index.html">Explore the Skill Impact Lab &#8599;</a><a href="research/index.html">Composition, handoff &amp; Harbor evidence &#8599;</a></div><p class="caption">L40S recordings; one public development task. No skill accuracy gain or general model ranking is claimed.</p></section>
<section class="suite" aria-label="Trace Workbench"><p class="eyebrow">NEW / BRING YOUR AGENT RECORDS</p><h2>Zero, skipped, or missing?</h2><p>A zero can be a valid judgment. A skipped evaluator needs context. A required skill may never have loaded. Inspect each against a versioned golden case, with recording identity and explicit acceptance rules.</p><div class="comparison-links"><a href="trace-workbench/index.html">Explore five authored review controls &#8599;</a><a href="trace-mcp/index.html">Inspect an actual MCP delivery &#8599;</a></div><p class="caption">The five controls are synthetic. The separate MCP record contains a real local load and no evaluator scores. Review your own saved AgentCore Evaluate export offline with <code>evalarc trace-import</code>.</p></section>
<div class="facts" aria-label="Audit scope">
<div><strong>__TASK_PACK_COUNT__</strong><span>task packs</span></div>
<div><strong>__FAULT_COUNT__</strong><span>declared faults caught</span></div>
Expand Down
2 changes: 1 addition & 1 deletion site/labs/research.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Research records &#183; EvalArc</title><style>
:root{color-scheme:dark}*{box-sizing:border-box}body{margin:auto;max-width:1050px;padding:24px;background:#10191c;color:#edf5ec;font:16px/1.6 system-ui}a{color:#b2f4ca}h1{font-size:clamp(2.4rem,6vw,4rem);line-height:1.1}h2{margin-top:45px}table{border-collapse:collapse;width:100%;margin:20px 0}td,th{padding:10px;text-align:left;border-bottom:1px solid #48625e;overflow-wrap:anywhere}section{background:#172428;padding:20px;border-radius:12px;margin:20px 0}code{overflow-wrap:anywhere}.scope{border-left:3px solid #ffb7a8;padding-left:18px}:focus-visible{outline:3px solid #ffb7a8} @media(max-width:480px){body{padding:15px}section{padding:12px}td,th{padding:6px;font-size:14px}}</style>
<a href="../">&#8592; EvalArc</a> &#183; <a href="../skill-impact/">27-trial Skill Impact Lab</a>
<a href="../index.html">&#8592; EvalArc</a> &#183; <a href="../skill-impact/index.html">27-trial Skill Impact Lab</a>
<h1>Keep the failed attempts.<br>Check the artifact.</h1>
<p>Recorded public development experiments, 14 September 2026. Qwen3-8B/Qwen3-4B on an NVIDIA L40S. No private customer data or personal agent history.</p>
<p class="scope">These pilots test specific integration paths and inspectable outcomes. They do not establish skill or memory efficacy, a general leakage detector, or native commercial-agent interoperability.</p>
Expand Down
2 changes: 1 addition & 1 deletion site/labs/skill-impact.html
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
</head>
<body>
<a href="#explorer" class="skip">Skip to trial explorer</a>
<header><a href="../">&#8592; Playground</a><span>Robot Reel &#215; Skills Anywhere &#215; EvalArc</span><a href="METHODS.md">Methods &amp; limits</a></header>
<header><a href="../index.html">&#8592; Playground</a><span>Robot Reel &#215; Skills Anywhere &#215; EvalArc</span><a href="METHODS.md">Methods &amp; limits</a></header>
<main>
<p class="eyebrow">Real model calls / NVIDIA L40S / Public development evidence</p>
<h1>A skill loaded.<br><em>Did the task pass?</em></h1>
Expand Down
2 changes: 1 addition & 1 deletion src/evalarc/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Auditable evaluations for AI agents."""

__version__ = "0.11.0"
__version__ = "0.11.1"
18 changes: 18 additions & 0 deletions tests/test_site.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,24 @@ def test_bundle_rejects_changed_evidence_and_extra_files(tmp_path):
assert "__AUDIT_COVERAGE__" not in (folder / "index.html").read_text()
assert (folder / "index.html").read_text().count("single-case dependencies") == 3
assert (folder / "robot/index.html").is_file()
from scripts.verify_research import verify_lab, verify_records

for name, verifier in (("skill-impact", verify_lab), ("research", verify_records)):
verifier(folder / name)
original = builder.ROOT / "examples" / name
for source in original.rglob("*"):
if source.is_file() and source.relative_to(original).as_posix() not in (
"index.html",
"manifest.json",
):
assert (folder / name / source.relative_to(original)).read_bytes() == (
source.read_bytes()
)
page = (folder / name / "index.html").read_text()
assert 'href="../index.html"' in page
assert 'href="../"' not in page
for name in ("skill-impact", "research", "trace-workbench", "trace-mcp"):
assert f'href="{name}/index.html"' in (folder / "index.html").read_text()
assert "single-case dependency" in (folder / "robot/index.html").read_text()
audit = folder / "support" / "audit.json"
original = audit.read_bytes()
Expand Down
Loading