diff --git a/CHANGELOG.md b/CHANGELOG.md index bb2a073..1dacf13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## 0.8.0 · 2026-09-14 · Research preview + +- Verify whole suite handoffs from the original TOML, plan, every repetition and + attempt, recomputed custom gates and JUnit failure/error records. +- Add `--require-accepted` for configured suite gates, distinct from record + consistency and full task resolution. Original candidate paths and timings + remain reported metadata; no candidate or grader is executed. +- Download the featured suite as a deterministic ZIP from the evidence lab. + Preserve all 12 original input files and verify the extracted archive offline. +- Check received suites from the installed wheel outside the checkout with an + empty PATH, including rejection of a modified JUnit record. + +Task contracts, scoring rules and historical evidence bytes are unchanged. + + ## 0.7.1 · 2026-09-14 · Research preview - Share and restore a specific task pack, control, seed, case and trace step in diff --git a/README.md b/README.md index 6de5a8f..b653f3a 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,13 @@ details to the case list, and retry failed sections independently. [![EvalArc v0.3: score rises from 90% to 93.75% while a check regresses](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc) +**v0.8: verify the whole handoff.** Download the suite evidence ZIP from the +lab, then run `evalarc verify suite-evidence --json` to recompute its original +TOML, plan, five attempts, custom gates and JUnit. Add `--require-accepted` +for CI acceptance. Consistency, configured acceptance and full resolution are +reported separately. No candidate execution is required. +[Offline verification and limits](docs/verification.md). + **Two working task packs** share an evidence format and configurable candidate commands: diff --git a/README.zh-CN.md b/README.zh-CN.md index 8757f34..4d3ec36 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -26,6 +26,12 @@ EvalArc 关注智能体实际完成的结果，以及支撑评分结论的证据 [![EvalArc v0.3：分数上升，一项检查却退步](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc) +**v0.8：完整验收证据可离线复核。** 从首页下载套件 ZIP，解压后运行 +`evalarc verify suite-evidence --json`，复核原始 TOML、执行计划、五次尝试、 +验收规则与 JUnit。使用 `--require-accepted` 接入 CI 验收；证据一致、规则接受、 +任务完全完成分别报告。整个检查不执行候选程序。 +[离线验证流程与边界](docs/verification.md)。 + **已实现 coding 和业务工具两个场景。** | 任务 | 交互方式 | 验证内容 | diff --git a/docs/outreach/hellogithub-submission.md b/docs/outreach/hellogithub-submission.md index 6e13ede..0bef7a7 100644 --- a/docs/outreach/hellogithub-submission.md +++ b/docs/outreach/hellogithub-submission.md @@ -12,31 +12,31 @@ https://github.com/noteflowai/evalarc ### 项目描述 -EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python／JavaScript 候选、TOML 套件、JUnit 和离线报告复核，保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例；新版可分享到具体任务、用例与调用步骤，并独立重试加载失败的分区。提供可筛选 HF Casebook 和中英文文档，现有演示为脚本对照。 +EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python／JavaScript 候选、TOML 套件、JUnit 和离线报告复核，保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例；0.8.0 可下载完整套件证据，并离线复算原始配置、执行计划、验收规则和 JUnit；浏览器可分享到具体调用步骤。提供可筛选 HF Casebook 和中英文文档，现有演示为脚本对照。 ### 亮点 -- **把讨论定位到同一步**：0.7.1 的 Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点，键盘可从详情返回案例列表；受限剪贴板有手动复制入口。 +- **把讨论定位到同一步**： Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点，键盘可从详情返回案例列表；受限剪贴板有手动复制入口。 - **失败可恢复**：suite、重复尝试、前后比较和任务包可以分别重试；一个任务包不可用时仍可检查另一个。页面经过 320／390／1440 像素与键盘路径验证。 - **高分不代表完成**：工单策略得 93.75% 却重复写备注；代码实现得 92.5% 却混淆 JSON true 与 1。可检查全部 15 种声明缺陷，以及三项 suite 作业、六条重复尝试和 167 条审计用例的数据表。 -- **可复核交付**：`evalarc verify` 不执行候选代码，重算单次、重复和对照报告；`--require-resolved` 另行要求完整通过。Python／JavaScript 参考实现共享任务约定。 +- **可复核交付**：`evalarc verify` 不执行候选代码，重算单次、重复、对照和完整套件报告，包括 TOML、plan、五次尝试与 JUnit；`--require-accepted` 检查配置的验收规则，`--require-resolved` 另行要求任务完整通过。Python／JavaScript 参考实现共享任务约定。 -本账号为维护者，项目与 AI 结对开发，采用 MIT 许可，处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照，不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器；suite 级规则和 JUnit 暂不在 verify 的核验范围。 +本账号为维护者，项目与 AI 结对开发，采用 MIT 许可，处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照，不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器，也不认证报告作者；原始候选路径与计时仍是报告元数据。 ### 示例代码 -安装发布的 Python wheel 后，核对收到的重复评测目录，无需 Docker 或 Node： +安装发布的 Python wheel，从首页下载 suite-evidence.zip 并解压后，无需 Docker 或 Node 即可核验；默认退出 0 表示证据一致，添加 --require-accepted 退出 1 表示严格备注规则拒绝该结果： ```sh -evalarc verify received/repetition --json +evalarc verify suite-evidence --json -# Also require valid, fully resolved results: -evalarc verify received/repetition --json --require-resolved +# Require every configured suite gate: +evalarc verify suite-evidence --json --require-accepted ``` ### 截图或演示视频 在线体验：https://huggingface.co/spaces/glayguo/evalarc -版本：https://github.com/noteflowai/evalarc/releases/tag/v0.7.1 +版本：https://github.com/noteflowai/evalarc/releases/tag/v0.8.0 ![分享具体失败证据，检查智能体评分器的盲点](https://github.com/noteflowai/evalarc/raw/main/docs/assets/suite-lab.png) diff --git a/docs/outreach/huggingface-introduction.md b/docs/outreach/huggingface-introduction.md index ee846e9..c62b144 100644 --- a/docs/outreach/huggingface-introduction.md +++ b/docs/outreach/huggingface-introduction.md @@ -1,9 +1,9 @@ -Select a case and trace step, then use **Copy evidence link**. The recipient reopens the same task, implementation, seed, case and state change. Clipboard-denied embeds offer a selectable link. Keyboard users can move into case details and return to the list. +**Download the evidence. Check every gate.** EvalArc 0.8.0 verifies a whole suite from its original TOML, plan, every repetition and attempt, custom acceptance rules and JUnit. It runs offline without a candidate process, Docker, Node or the original candidate directory. -Failed suite, repetition, comparison and audit requests can be retried independently. One unavailable task pack no longer blocks the other; retrying can restore the originally linked view. The explorer fits 320px screens and keeps JSON scrolling inside its cards. Saved reports and scoring rules are unchanged. +Download **suite-evidence.zip** from the lab and unzip it. `evalarc verify suite-evidence --json` checks all 12 original input files and exits 0 for consistency. Adding `--require-accepted` exits 1: the strict notes gate rejects the faulty policy. Two of three jobs are accepted; only one is fully resolved. Those are separate outcomes. -[Open the Space](https://huggingface.co/spaces/glayguo/evalarc) · [Release 0.7.1](https://github.com/noteflowai/evalarc/releases/tag/v0.7.1) · [Source and methods](https://github.com/noteflowai/evalarc) +[Open the evidence lab](https://huggingface.co/spaces/glayguo/evalarc) · [Release 0.8.0](https://github.com/noteflowai/evalarc/releases/tag/v0.8.0) · [Verification workflow and limits](https://github.com/noteflowai/evalarc/blob/main/docs/verification.md) -The same public casebook still provides 167 audit cases, six repeated attempts and three suite jobs. These are scripted development controls, not frontier-model scores or unseen-task reliability estimates. EvalArc remains a research preview. +The public Casebook still provides 167 audit cases, six repeated attempts and three suite jobs. Historical source bytes and scoring rules are unchanged. These are scripted development controls; offline record consistency does not authenticate the producer or independently rerun the grader. Original candidate paths and durations remain reported metadata. EvalArc remains a research preview. Maintainer update to the existing introduction, developed with AI assistance. Independent community project; no upstream or Hugging Face endorsement is implied. diff --git a/docs/outreach/weekly-submission.md b/docs/outreach/weekly-submission.md index f411865..755c96a 100644 --- a/docs/outreach/weekly-submission.md +++ b/docs/outreach/weekly-submission.md @@ -1,24 +1,24 @@ -维护者自荐：**evalarc**。EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python／JavaScript 候选、TOML 套件、JUnit 和离线报告复核，保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例；新版可分享到具体任务、用例与调用步骤，并独立重试加载失败的分区。提供可筛选 HF Casebook 和中英文文档，现有演示为脚本对照。 +维护者自荐：**evalarc**。EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python／JavaScript 候选、TOML 套件、JUnit 和离线报告复核，保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例；0.8.0 可下载完整套件证据，并离线复算原始配置、执行计划、验收规则和 JUnit；浏览器可分享到具体调用步骤。提供可筛选 HF Casebook 和中英文文档，现有演示为脚本对照。 在线体验：https://huggingface.co/spaces/glayguo/evalarc -- **把讨论定位到同一步**：0.7.1 的 Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点，键盘可从详情返回案例列表；受限剪贴板有手动复制入口。 +- **把讨论定位到同一步**： Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点，键盘可从详情返回案例列表；受限剪贴板有手动复制入口。 - **失败可恢复**：suite、重复尝试、前后比较和任务包可以分别重试；一个任务包不可用时仍可检查另一个。页面经过 320／390／1440 像素与键盘路径验证。 - **高分不代表完成**：工单策略得 93.75% 却重复写备注；代码实现得 92.5% 却混淆 JSON true 与 1。可检查全部 15 种声明缺陷，以及三项 suite 作业、六条重复尝试和 167 条审计用例的数据表。 -- **可复核交付**：`evalarc verify` 不执行候选代码，重算单次、重复和对照报告；`--require-resolved` 另行要求完整通过。Python／JavaScript 参考实现共享任务约定。 +- **可复核交付**：`evalarc verify` 不执行候选代码，重算单次、重复、对照和完整套件报告，包括 TOML、plan、五次尝试与 JUnit；`--require-accepted` 检查配置的验收规则，`--require-resolved` 另行要求任务完整通过。Python／JavaScript 参考实现共享任务约定。 ```sh -evalarc verify received/repetition --json +evalarc verify suite-evidence --json -# Also require valid, fully resolved results: -evalarc verify received/repetition --json --require-resolved +# Require every configured suite gate: +evalarc verify suite-evidence --json --require-accepted ``` -安装发布的 Python wheel 后，核对收到的重复评测目录，无需 Docker 或 Node： +安装发布的 Python wheel，从首页下载 suite-evidence.zip 并解压后，无需 Docker 或 Node 即可核验；默认退出 0 表示证据一致，添加 --require-accepted 退出 1 表示严格备注规则拒绝该结果： 项目：https://github.com/noteflowai/evalarc -版本：https://github.com/noteflowai/evalarc/releases/tag/v0.7.1 +版本：https://github.com/noteflowai/evalarc/releases/tag/v0.8.0 ![分享具体失败证据，检查智能体评分器的盲点](https://github.com/noteflowai/evalarc/raw/main/docs/assets/suite-lab.png) -本账号为维护者，项目与 AI 结对开发，采用 MIT 许可，处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照，不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器；suite 级规则和 JUnit 暂不在 verify 的核验范围。 +本账号为维护者，项目与 AI 结对开发，采用 MIT 许可，处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照，不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器，也不认证报告作者；原始候选路径与计时仍是报告元数据。 diff --git a/docs/suites.md b/docs/suites.md index 4cbdcc9..7df3082 100644 --- a/docs/suites.md +++ b/docs/suites.md @@ -161,3 +161,11 @@ cancellation; keep that file outside candidates and the output directory. The suite stream adds job IDs to the existing `evalarc.event.v1` events. See the [validation record](validation-v0.5.md) for the checks actually run. + +## Verify a received suite without executing it + +With EvalArc 0.8+, keep the original TOML, plan, suite JSON, JUnit and all job +attempts together. Run `evalarc verify received/suite --json --require-accepted` +to recompute configured gates offline. Full resolution is a separate condition, +available through `--require-resolved`. Original candidate paths are metadata; +do not rewrite them when moving the evidence. [Handoff guide](verification.md). diff --git a/docs/suites.zh-CN.md b/docs/suites.zh-CN.md index e76006b..91782e0 100644 --- a/docs/suites.zh-CN.md +++ b/docs/suites.zh-CN.md @@ -107,3 +107,10 @@ JUnit 中一个 testcase 对应一个 **job 验收门槛**： 把外部日志放在候选及输出目录之外。 完整字段与 CI 示例见[英文指南](suites.md)，本轮实测范围见[验证记录](validation-v0.5.md)。 + +## 不执行候选程序，复核收到的套件 + +EvalArc 0.8 起，保留原始 TOML、计划、套件 JSON、JUnit 和各作业全部尝试， +运行 `evalarc verify received/suite --json --require-accepted`，即可离线复算验收规则。 +`--require-resolved` 另行要求任务完全完成。原始候选路径作为元数据保留， +移动证据时不要改写。[完整交付指南](verification.md)。 diff --git a/docs/verification.md b/docs/verification.md index df7f5df..8d30b68 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -1,58 +1,90 @@ # Verify received evaluation evidence -EvalArc 0.7 adds a read-only handoff check. A customer or CI job can recompute a -report's claims without installing Node, contacting Docker, locating the -original candidate or executing its command. +EvalArc provides a read-only handoff check. Recompute recorded claims without +Node, Docker, the original candidate directory or candidate execution. +Version 0.8 also checks complete suite configuration, acceptance gates and JUnit. ```bash evalarc verify received/evaluation.json --json evalarc verify received/repetition --json evalarc verify received/comparison --json +evalarc verify received/suite --json ``` -An evaluation can have any filename. For a repetition, keep `repetition.json` -with `attempts/0001/evaluation.json`, `0002/evaluation.json`, and so on. For a -comparison, keep `comparison.json`, `baseline.json` and `current.json` together. -Pass either the summary file or its directory. A directory must contain exactly -one supported report type. +Pass a summary file or its directory. A directory must contain exactly one +supported report type. An individual evaluation may have any filename. +Keep a repetition with `attempts/0001/evaluation.json`, `0002/evaluation.json`, +and so on. A comparison needs `baseline.json` and `current.json` alongside it. -The result includes a SHA-256 and byte length for every JSON file actually -checked. These hashes identify the handoff bytes, including historical package -versions. It does not rewrite reports or execute commands embedded in metadata. +A suite handoff contains `suite.json`, the original `suite.toml`, `plan.json`, +`junit.xml` and every `jobs//repetition.json` with its attempt evaluations. +Do not edit the TOML to relocate candidate paths: they describe the original +machine and the verifier never resolves or reads them. The TOML byte hash must +match both the saved plan and the suite summary. | Evidence | Recomputed checks | | --- | --- | | Evaluation v2 | Case/check counts, dimensions, weighted score, validity and resolution | -| Repetition v1 | Exact attempt inventory, input identities, all aggregate counts, scores and variance | -| Comparison v1 | Matching task/runtime/grader/cases, score delta, individual regressions and improvements | +| Repetition v1 | Exact attempt inventory, input identities, aggregate counts, scores and variance | +| Comparison v1 | Matching task/runtime/grader/cases, score delta, regressions and improvements | +| Suite v1 | Original configuration/hash, job order and inventory, plan/budgets, task/runtime/seeds, attempts, custom gates, totals and JUnit | -Malformed or contradictory evidence exits **2**, with `verified: false` in JSON -mode. Default exit **0** means the records are internally consistent, even if -the recorded candidate failed or had an environment error. +The result includes SHA-256 and byte length for every input checked, including +TOML and XML. Historical package versions remain readable; verification does +not rewrite reports. JUnit counts, testcase identities, failure versus error, +gate payloads and recorded observations must match the recomputed suite. +XML indentation and attribute order may differ. -To require full resolution as well: +## Consistency, acceptance and resolution + +Default exit **0** means records are internally consistent, even if a gate +rejects the candidate or the run records an environment error. Contradictory, +malformed or incomplete evidence exits **2** with `verified: false` in JSON. ```bash -evalarc verify received/repetition --json --require-resolved +# Require the suite's configured gates: +evalarc verify received/suite --json --require-accepted +# Require full resolution for any supported report: +evalarc verify received/suite --json --require-resolved ``` -This exits **0** for valid, fully resolved records, **1** for valid but unresolved -records, and **2** for invalid records or a verification error. For a comparison, -full resolution refers to its current evaluation; `has_regressions` is also -reported separately. This flag is not a replacement for a suite's custom gates. - -Suite manifests, custom gate decisions, JUnit, audit summaries and trajectories -are not currently standalone supported inputs. For a suite, verify each -`jobs/` repetition directory; this checks its attempts but does not verify -the suite-level gate decision. The published site additionally checks the -featured suite against its original configuration and JUnit. - -Input files must be regular files without symlinks in the evidence path. -Reads are limited to 64 MiB per JSON and 256 MiB per handoff. Repetitions contain -at most 100 attempts. Duplicate JSON keys and non-finite values are rejected. -Original HTML, logs and source programs are outside the verified inventory. - -This is record consistency, not an independent rerun of the business-state -grader or authentication of the report's producer. A coordinated fabrication -can be internally consistent. Re-run trusted grading on the candidate when -execution evidence is required. +`--require-accepted` applies only to suites. Exit **0** requires valid records +and every gate accepting; **1** means valid records with rejected gates; +**2** means invalid records or a verification error. `--require-resolved` uses +the same exit convention for full resolution. In a comparison it refers to the +current evaluation. Using both flags requires both conditions. A permissive +gate can accept a partially resolved result. + +## Try the published handoff + +Download **suite-evidence.zip** from the +[evidence lab](https://huggingface.co/spaces/glayguo/evalarc), unzip it, and run: + +```bash +evalarc verify suite-evidence --json +# Expected: verified=true; accepted=false; accepted_jobs=2; +# fully_resolved_jobs=1; total_jobs=3; 12 input files; exit 0. +evalarc verify suite-evidence --json --require-accepted +# Expected exit 1: the strict notes gate rejects the recorded faulty policy. +``` + +The ZIP retains the original bytes of all 12 verified files from the three-job, +five-attempt Docker suite. Its stable ZIP metadata makes repeated builds +reproducible. HTML and logs remain available separately in the lab. + +## Bounds and trust scope + +Inputs must be regular files without symlinks in the evidence path. Limits are +64 MiB per JSON, 1 MiB for TOML, 4 MiB for JUnit, and 256 MiB per handoff. Suites +have at most 100 jobs, 1,000 planned attempts and 100,000 planned case executions; +each repetition has at most 100 attempts. Duplicate JSON keys, non-finite +numbers, XML DTDs and entities are rejected. + +Candidate path strings and durations remain reported metadata. They are +bounded and checked for internal consistency where applicable, but cannot be +reconstructed from the handoff. Original HTML, logs, audit summaries, +trajectories and source programs are outside the verified inventory. + +This checks record consistency, not independent business-state grading or +producer authentication. A coordinated fabrication can be internally +consistent. Re-run trusted grading when execution evidence is required. diff --git a/huggingface/README.md b/huggingface/README.md index 4ed19c0..ce74fcb 100644 --- a/huggingface/README.md +++ b/huggingface/README.md @@ -81,7 +81,7 @@ dependencies; the bundled trusted controls can run on a CPU. ## Scope -Research preview 0.7.1. These are scripted controls and public development +Research preview 0.8.0. These are scripted controls and public development tasks, not held-out frontier-model results. Detection applies only to the declared faults. No arbitrary reward-hack resistance, human time horizon, hardware-agent validation or RL improvement is established. Repeated fixed @@ -93,7 +93,11 @@ maintainer and does not imply endorsement by Hugging Face. ### Verify a handoff offline -With EvalArc 0.7+, run `evalarc verify path/to/report --json` on an evaluation, -repetition or comparison report. It checks the recorded inputs without executing -a candidate; `--require-resolved` separately requires fully resolved results. +With EvalArc 0.8+, download **suite-evidence.zip** from the lab and unzip it. +Run `evalarc verify suite-evidence --json` to check the original configuration, +plan, all five attempts, custom gates and JUnit without executing a candidate. +The 12 original input files are unchanged: two of three jobs are accepted, one +is fully resolved. Default verification exits 0 for consistency; +`--require-accepted` exits 1 because the strict notes gate rejects the result. +`--require-resolved` separately requires full resolution. [Workflow and limits](https://github.com/noteflowai/evalarc/blob/main/docs/verification.md). diff --git a/package-lock.json b/package-lock.json index e87ef9f..1c0d61f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.7.1", + "version": "0.8.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.7.1", + "version": "0.8.0", "devDependencies": { "playwright": "1.63.0" } diff --git a/package.json b/package.json index aef4d1a..b544cbf 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.7.1", + "version": "0.8.0", "private": true, "description": "Browser checks for the static EvalArc evidence explorer", "scripts": { diff --git a/pyproject.toml b/pyproject.toml index 656b1fd..a356b32 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "evalarc" -version = "0.7.1" +version = "0.8.0" description = "Auditable task environments and evaluations for coding and tool-using agents." readme = "README.md" requires-python = ">=3.11" diff --git a/scripts/build_site.py b/scripts/build_site.py index f1d58de..c655cf7 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -11,6 +11,7 @@ import sys import tempfile import tomllib +import zipfile from pathlib import Path ROOT = Path(__file__).resolve().parents[1] @@ -20,6 +21,24 @@ sys.path.insert(0, str(ROOT / "src")) +def write_suite_bundle(destination: Path) -> None: + """Package only verified input bytes, with stable ZIP metadata.""" + from evalarc.verify import verify as verify_evidence + + source = ROOT / "examples" / "suite" + receipt = verify_evidence(source) + with zipfile.ZipFile(destination, "x", compression=zipfile.ZIP_DEFLATED) as archive: + for name, identity in sorted(receipt["files"].items()): + content = (source / name).read_bytes() + if hashlib.sha256(content).hexdigest() != identity["sha256"]: + raise ValueError("Suite evidence changed while packaging") + member = zipfile.ZipInfo(f"suite-evidence/{name}", date_time=(2020, 1, 1, 0, 0, 0)) + member.compress_type = zipfile.ZIP_DEFLATED + member.create_system = 3 + member.external_attr = 0o100644 << 16 + archive.writestr(member, content) + + def verify_comparison() -> None: from evalarc.compare import compare from evalarc.records import read_evaluation @@ -317,6 +336,7 @@ def build(destination: Path) -> dict: target.write_bytes(source.read_text().encode("ascii", errors="xmlcharrefreplace")) else: shutil.copyfile(source, target) + write_suite_bundle(destination / "suite-evidence.zip") shutil.copyfile(ROOT / "LICENSE", destination / "LICENSE") shutil.copyfile(ROOT / "huggingface" / "README.md", destination / "README.md") (destination / ".nojekyll").touch() diff --git a/scripts/check_installed.py b/scripts/check_installed.py index 880b6ad..12d587b 100644 --- a/scripts/check_installed.py +++ b/scripts/check_installed.py @@ -49,7 +49,7 @@ def hashes() -> dict: if p.is_file() } - for name in ("evaluation", "repetition", "repetition-faulty", "comparison"): + for name in ("evaluation", "repetition", "repetition-faulty", "comparison", "suite"): target = folder / name shutil.copytree(source / "examples" / name, target) before = hashes() @@ -59,6 +59,15 @@ def hashes() -> dict: results[name] = result run(folder / "repetition", 0, "--require-resolved") run(folder / "repetition-faulty", 1, "--require-resolved") + suite = run(folder / "suite", 1, "--require-accepted") + if suite["accepted_jobs"] != 2 or suite["fully_resolved_jobs"] != 1: + raise ValueError("suite acceptance and resolution were conflated") + junit = folder / "suite" / "junit.xml" + junit.write_text( + junit.read_text().replace(" { const route = decodeURIComponent(new URL(req.url, "http://localhost").pathname); const filename = path.resolve(root, "." + (route.endsWith("/") ? route + "index.html" : route)); @@ -54,6 +54,16 @@ async function main() { assert.equal(suiteProvenance.protected_gate.reasons.length, 1); await app.locator("#suite-provenance").locator("..").locator("summary").click(); const suiteBase = await app.locator("body").evaluate(() => location.href); + const downloadLink = app.getByRole("link", {name:"Download suite evidence (ZIP)"}); + assert.equal(await downloadLink.getAttribute("href"), "suite-evidence.zip"); + const received = page.waitForEvent("download"); + await downloadLink.click(); + const download = await received; + assert.equal(await download.failure(), null); + assert.equal(download.suggestedFilename(), "suite-evidence.zip"); + const downloaded = fs.readFileSync(await download.path()); + const manifest = await (await page.request.get(new URL("manifest.json", suiteBase).href)).json(); + assert.equal(require("node:crypto").createHash("sha256").update(downloaded).digest("hex"), manifest.files["suite-evidence.zip"]); const junitResponse = await page.request.get(new URL("suite/junit.xml", suiteBase).href); assert.equal(junitResponse.status(), 200); const junit = await page.evaluate(xml => { @@ -213,7 +223,7 @@ async function main() { assert.match(await app.locator("body").innerText(), /retry-after-commit/); } assert.deepEqual(errors, []); - results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, sharedTraceRestored:true, keyboardCaseReturn:true, errors}); + results.push({width, controls:17, cases:167, comparedCases:3, repeatedControls:2, attempts:6, suiteJobs:3, suiteAttempts:5, junitFailures:1, offlineReports:7, suiteDownloadVerified:true, sharedTraceRestored:true, keyboardCaseReturn:true, errors}); await page.close(); } if (!process.env.SITE_URL) { diff --git a/site/index.html b/site/index.html index ee3d05d..884ff1c 100644 --- a/site/index.html +++ b/site/index.html @@ -158,12 +158,13 @@

Look past
the score.

-

HAND OFF EVIDENCE

Received a report?
Check its claims.

Recompute recorded scores, repeated outcomes and regressions without executing the candidate.

Offline verification guide ↗
-
EVALARC 0.7+ / READ-ONLY
evalarc verify received/repetition --json
+      
HAND OFF EVIDENCE
Download the evidence.
Check every gate.
Recompute the original suite configuration, plan, five attempts, acceptance decisions and JUnit without executing a candidate.
Download suite evidence (ZIP) · Offline verification guide ↗

+      
EVALARC 0.8+ / READ-ONLY
# Unzip the downloaded archive first.
+evalarc verify suite-evidence --json
 
-# Also require valid, fully resolved results:
-evalarc verify received/repetition \
-  --json --require-resolved
Consistency is distinct from task success. No grader rerun or producer authentication is implied.

+# Require all configured gates to accept:
+evalarc verify suite-evidence \
+  --json --require-accepted

This example is consistent (exit 0), but one gate rejects it (exit 1 with --require-accepted). Two jobs are accepted; one is fully resolved. No grader rerun or producer authentication is implied.

FROM THE BROWSER TO YOUR TERMINAL

Make the grader
earn your trust.

Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.

Installation & execution guide ↗
diff --git a/src/evalarc/__init__.py b/src/evalarc/__init__.py index e9a07d7..c05bd82 100644 --- a/src/evalarc/__init__.py +++ b/src/evalarc/__init__.py @@ -1,3 +1,3 @@ """Auditable evaluations for AI agents.""" -__version__ = "0.7.1" +__version__ = "0.8.0" diff --git a/src/evalarc/cli.py b/src/evalarc/cli.py index 1cec3f1..feca9d8 100644 --- a/src/evalarc/cli.py +++ b/src/evalarc/cli.py @@ -102,7 +102,7 @@ def parser() -> argparse.ArgumentParser: suite.add_argument("--progress", action="store_true") suite.add_argument("--output", type=Path, default=Path("runs/suite")) verification = commands.add_parser( - "verify", help="check saved evaluation, repetition or comparison evidence without execution" + "verify", help="check saved evaluation, repetition, comparison or suite evidence" ) verification.add_argument("evidence", type=Path, help="report JSON or its containing directory") verification.add_argument("--json", action="store_true") @@ -111,6 +111,11 @@ def parser() -> argparse.ArgumentParser: action="store_true", help="also require valid, fully resolved results (current result for comparisons)", ) + verification.add_argument( + "--require-accepted", + action="store_true", + help="also require a valid suite whose configured acceptance gates all passed", + ) return root @@ -122,7 +127,21 @@ def main(argv: list[str] | None = None) -> int: code = 0 if args.require_resolved: code = 2 if not result["records_valid"] else (0 if result["fully_resolved"] else 1) - except (OSError, ValueError, KeyError, TypeError, IndexError, OverflowError) as error: + if args.require_accepted: + if result["kind"] != "suite": + raise ValueError("--require-accepted requires suite evidence") + code = max( + code, 2 if not result["records_valid"] else (0 if result["accepted"] else 1) + ) + except ( + OSError, + ValueError, + KeyError, + TypeError, + IndexError, + OverflowError, + RecursionError, + ) as error: result = { "schema_version": "evalarc.verification.v1", "verified": False, @@ -134,10 +153,12 @@ def main(argv: list[str] | None = None) -> int: print(json.dumps(result, indent=2)) elif result["verified"]: print( - f"Verified {result['kind']}: {len(result['files'])} JSON files | " + f"Verified {result['kind']}: {len(result['files'])} evidence files | " f"Valid records: {result['records_valid']} | " - f"Fully resolved: {result['fully_resolved']}\n{SCOPE}" + f"Fully resolved: {result['fully_resolved']}\n{result['scope']}" ) + if result["kind"] == "suite": + print(f"Accepted gates: {result['accepted_jobs']}/{result['total_jobs']}") else: print(f"Verification failed: {result['error']}", file=sys.stderr) return code diff --git a/src/evalarc/junit.py b/src/evalarc/junit.py index 45355af..79cbf7a 100644 --- a/src/evalarc/junit.py +++ b/src/evalarc/junit.py @@ -21,7 +21,8 @@ def _xml_text(value: str) -> str: ) -def render_junit(data: dict, destination: Path) -> None: +def junit_tree(data: dict) -> ET.Element: + """Build the exported gate results without performing filesystem writes.""" jobs = data["jobs"] failures = sum(row["decision"]["valid"] and not row["decision"]["accepted"] for row in jobs) errors = sum(not row["decision"]["valid"] for row in jobs) @@ -70,6 +71,11 @@ def render_junit(data: dict, destination: Path) -> None: allow_nan=False, ) ) + return root + + +def render_junit(data: dict, destination: Path) -> None: + root = junit_tree(data) ET.indent(root, space=" ") destination.parent.mkdir(parents=True, exist_ok=True) ET.ElementTree(root).write(destination, encoding="utf-8", xml_declaration=True) diff --git a/src/evalarc/records.py b/src/evalarc/records.py index 5a2df80..1ad344f 100644 --- a/src/evalarc/records.py +++ b/src/evalarc/records.py @@ -32,18 +32,25 @@ def _object(pairs: list[tuple[str, object]]) -> dict: return result -def read_json(path: Path, *, limit: int = MAX_REPORT_BYTES) -> tuple[dict, bytes]: - """Read finite, unambiguous JSON from a bounded regular file.""" +def read_bytes(path: Path, *, limit: int = MAX_REPORT_BYTES) -> bytes: + """Bound reads of a regular evidence file, including files replaced during a read.""" + oversized = f"evidence exceeds the read limit ({limit} bytes; {limit / 1048576:g} MiB)" fd = os.open(path, os.O_RDONLY | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_NOFOLLOW", 0)) with os.fdopen(fd, "rb") as source: info = os.fstat(source.fileno()) if not stat.S_ISREG(info.st_mode): raise ValueError("report must be a regular file") if info.st_size > limit: - raise ValueError("evaluation report exceeds the 64 MiB read limit") + raise ValueError(oversized) content = source.read(limit + 1) if len(content) > limit: - raise ValueError("evaluation report exceeds the 64 MiB read limit") + raise ValueError(oversized) + return content + + +def read_json(path: Path, *, limit: int = MAX_REPORT_BYTES) -> tuple[dict, bytes]: + """Read finite, unambiguous JSON from a bounded regular file.""" + content = read_bytes(path, limit=limit) try: data = json.loads(content, object_pairs_hook=_object) json.dumps(data, allow_nan=False) diff --git a/src/evalarc/suite.py b/src/evalarc/suite.py index 1a8371d..1c11841 100644 --- a/src/evalarc/suite.py +++ b/src/evalarc/suite.py @@ -16,7 +16,7 @@ from evalarc.evaluate import write_json from evalarc.events import EventCallback, EventLog, emit from evalarc.junit import render_junit -from evalarc.records import numeric +from evalarc.records import numeric, read_bytes from evalarc.repetition import repeat from evalarc.report import render_evaluation, render_repetition, render_suite from evalarc.runner import Runtime, snapshot @@ -87,11 +87,8 @@ def describe(self) -> dict: } -def load_suite(path: Path) -> SuitePlan: - """Parse bounded configuration without starting candidates or contacting Docker.""" - source = path.resolve() - with source.open("rb") as stream: - content = stream.read(MAX_CONFIG_BYTES + 1) +def parse_suite_config(content: bytes) -> dict: + """Normalize schema-v1 settings without inspecting paths, tasks or runtimes.""" if len(content) > MAX_CONFIG_BYTES: raise ValueError("suite configuration exceeds 1 MiB") data = _keys( @@ -128,9 +125,8 @@ def load_suite(path: Path) -> SuitePlan: ) seen.add(identity) label = f"{label} ({identity})" - if not isinstance(raw["task"], str): + if not isinstance(raw["task"], str) or not raw["task"]: raise ValueError(f"{label}: task must be a string") - task = get_task(raw["task"]) candidate = raw["candidate"] if ( not isinstance(candidate, str) @@ -141,9 +137,6 @@ def load_suite(path: Path) -> SuitePlan: raise ValueError( f"{label}: candidate must be a nonempty path of at most 4096 characters" ) - candidate = (source.parent / candidate).resolve() - if not candidate.is_dir(): - raise ValueError(f"{label}: candidate directory does not exist: {candidate}") seeds = raw.get("seeds", [17, 41, 97]) if ( not isinstance(seeds, list) @@ -171,11 +164,15 @@ def load_suite(path: Path) -> SuitePlan: or not runtime.image.isprintable() ): raise ValueError(f"{label}: image must be a nonempty printable string") - # Validate limits without resolving an image or executing a configured command. - try: - replace(runtime, backend="local").prepare() - except ValueError as error: - raise ValueError(f"{label}: {error}") from error + if ( + any( + not numeric(value) or value <= 0 + for value in (runtime.timeout, runtime.case_timeout) + ) + or type(runtime.output_limit) is not int + or runtime.output_limit < 1 + ): + raise ValueError(f"{label}: runtime limits must be positive finite values") acceptance = _keys( raw.get("gate", {}), {"min_mean_score", "min_resolution_rate", "required_dimensions"}, @@ -189,30 +186,62 @@ def load_suite(path: Path) -> SuitePlan: dimensions = acceptance.get("required_dimensions", []) if ( not isinstance(dimensions, list) - or any(not isinstance(item, str) or item not in task.dimensions for item in dimensions) + or any(not isinstance(item, str) or not item for item in dimensions) or len(set(dimensions)) != len(dimensions) ): - raise ValueError( - f"{label}: required_dimensions must name unique dimensions of {task.id}" - ) + raise ValueError(f"{label}: required_dimensions must name unique dimensions") gate = Gate( acceptance.get("min_mean_score", 1.0), acceptance.get("min_resolution_rate", 1.0), tuple(dimensions), ) + jobs.append( + { + "id": identity, + "task": raw["task"], + "candidate": candidate, + "seeds": seeds, + "attempts": attempts, + "runtime": { + key: getattr(runtime, key) + for key in ("backend", "image", "timeout", "case_timeout", "output_limit") + }, + "gate": asdict(gate), + } + ) + if sum(job["attempts"] for job in jobs) > 1000: + raise ValueError("suite exceeds 1000 attempts") + return {"schema_version": data["schema_version"], "name": name, "jobs": jobs} + + +def load_suite(path: Path) -> SuitePlan: + """Plan an execution; candidate and case checks belong only to this path.""" + source = path.resolve() + content = read_bytes(source, limit=MAX_CONFIG_BYTES) + config = parse_suite_config(content) + jobs = [] + for raw in config["jobs"]: + task = get_task(raw["task"]) + candidate = (source.parent / raw["candidate"]).resolve() + if not candidate.is_dir(): + raise ValueError(f"job {raw['id']}: candidate directory does not exist: {candidate}") + if not set(raw["gate"]["required_dimensions"]) <= set(task.dimensions): + raise ValueError( + f"job {raw['id']}: required_dimensions must name dimensions of {task.id}" + ) jobs.append( SuiteJob( - identity, + raw["id"], task.id, candidate, - tuple(seeds), - attempts, - runtime, - gate, - sum(len(task.generate_cases(seed)) for seed in seeds), + tuple(raw["seeds"]), + raw["attempts"], + Runtime(**raw["runtime"]), + Gate(**raw["gate"]), + sum(len(task.generate_cases(seed)) for seed in raw["seeds"]), ) ) - plan = SuitePlan(name, source, content, tuple(jobs)) + plan = SuitePlan(config["name"], source, content, tuple(jobs)) description = plan.describe() if description["planned_attempts"] > 1000 or description["planned_case_executions"] > 100_000: raise ValueError("suite exceeds 1000 attempts or 100000 planned case executions") @@ -273,6 +302,72 @@ def assess_gate(summary: dict, gate: Gate) -> dict: } +def summarize_job(identity: str, repetition: dict, gate: Gate, duration_seconds: float) -> dict: + """Recompute one suite row from its verified repetition and configured gate.""" + decision = assess_gate(repetition, gate) + return { + "id": identity, + "task": repetition["task"], + "candidate_sha256": repetition["candidate_sha256"], + "grader_sha256": repetition["grader_sha256"], + "cases_sha256": repetition["cases_sha256"], + "runtime": repetition["runtime"], + "seeds": repetition["seeds"], + "gate": asdict(gate), + "decision": decision, + "status": ( + "environment_error" + if not decision["valid"] + else ("passed" if decision["accepted"] else "failed") + ), + "fully_resolved": repetition["all_attempts_resolved"], + "observed": { + key: repetition[key] + for key in ( + "requested_attempts", + "completed_attempts", + "assessed_attempts", + "invalid_attempts", + "resolved_attempts", + "mean_score", + "assessed_resolution_rate", + "variable_cases", + "variable_checks", + ) + }, + "duration_seconds": duration_seconds, + } + + +def summarize_suite( + name: str, manifest_sha256: str, rows: list[dict], duration_seconds: float +) -> dict: + """Keep acceptance, validity and full resolution separate at suite level.""" + valid = all(row["decision"]["valid"] for row in rows) + accepted = valid and all(row["decision"]["accepted"] for row in rows) + return { + "schema_version": "evalarc.suite.v1", + "evalarc_version": __version__, + "created_at": datetime.now(timezone.utc).isoformat(), + "name": name, + "manifest_sha256": manifest_sha256, + "valid": valid, + "accepted": accepted, + "status": "environment_error" if not valid else ("passed" if accepted else "failed"), + "total_jobs": len(rows), + "accepted_jobs": sum(row["decision"]["accepted"] for row in rows), + "invalid_jobs": sum(not row["decision"]["valid"] for row in rows), + "fully_resolved_jobs": sum(row["fully_resolved"] for row in rows), + "duration_seconds": duration_seconds, + "jobs": rows, + "interpretation": ( + "Acceptance follows each job's declared gate. No score is averaged across jobs " + "or domains. Gate acceptance does not imply full task resolution. " + "Repeated observations use fixed public cases and are descriptive only." + ), + } + + def run_suite( plan: SuitePlan, destination: Path, @@ -352,71 +447,24 @@ def save(index: int, evaluation: dict) -> None: raise ValueError(f"frozen candidate changed during suite execution: {job.id}") write_json(directory / "repetition.json", repetition) render_repetition(repetition, directory / "index.html") - decision = assess_gate(repetition, job.gate) - row = { - "id": job.id, - "task": repetition["task"], - "candidate_sha256": fingerprint, - "grader_sha256": repetition["grader_sha256"], - "cases_sha256": repetition["cases_sha256"], - "runtime": repetition["runtime"], - "seeds": list(job.seeds), - "gate": asdict(job.gate), - "decision": decision, - "status": ( - "environment_error" - if not decision["valid"] - else ("passed" if decision["accepted"] else "failed") - ), - "fully_resolved": repetition["all_attempts_resolved"], - "observed": { - key: repetition[key] - for key in ( - "requested_attempts", - "completed_attempts", - "assessed_attempts", - "invalid_attempts", - "resolved_attempts", - "mean_score", - "assessed_resolution_rate", - "variable_cases", - "variable_checks", - ) - }, - "duration_seconds": round(time.monotonic() - job_started, 6), - } + row = summarize_job( + job.id, repetition, job.gate, round(time.monotonic() - job_started, 6) + ) rows.append(row) emit( notify, "job_completed", job=job.id, status=row["status"], - accepted=decision["accepted"], + accepted=row["decision"]["accepted"], fully_resolved=row["fully_resolved"], ) - valid = all(row["decision"]["valid"] for row in rows) - accepted = valid and all(row["decision"]["accepted"] for row in rows) - report = { - "schema_version": "evalarc.suite.v1", - "evalarc_version": __version__, - "created_at": datetime.now(timezone.utc).isoformat(), - "name": plan.name, - "manifest_sha256": description["manifest_sha256"], - "valid": valid, - "accepted": accepted, - "status": "environment_error" if not valid else ("passed" if accepted else "failed"), - "total_jobs": len(rows), - "accepted_jobs": sum(row["decision"]["accepted"] for row in rows), - "invalid_jobs": sum(not row["decision"]["valid"] for row in rows), - "fully_resolved_jobs": sum(row["fully_resolved"] for row in rows), - "duration_seconds": round(time.monotonic() - started, 6), - "jobs": rows, - "interpretation": ( - "Acceptance follows each job's declared gate. No score is averaged across jobs " - "or domains. Gate acceptance does not imply full task resolution. " - "Repeated observations use fixed public cases and are descriptive only." - ), - } + report = summarize_suite( + plan.name, + description["manifest_sha256"], + rows, + round(time.monotonic() - started, 6), + ) write_json(output / "suite.json", report) render_suite(report, output / "index.html") render_junit(report, output / "junit.xml") diff --git a/src/evalarc/verify.py b/src/evalarc/verify.py index 06771b1..4d33760 100644 --- a/src/evalarc/verify.py +++ b/src/evalarc/verify.py @@ -4,14 +4,23 @@ import hashlib import json +import xml.etree.ElementTree as ET from pathlib import Path from evalarc.compare import compare -from evalarc.records import read_json, validate_evaluation +from evalarc.junit import junit_tree +from evalarc.records import numeric, read_bytes, read_json, validate_evaluation from evalarc.repetition import summarize_attempts +from evalarc.suite import ( + MAX_CONFIG_BYTES, + Gate, + parse_suite_config, + summarize_job, + summarize_suite, +) MAX_BUNDLE_BYTES = 256 * 1024 * 1024 -REPORTS = ("evaluation.json", "repetition.json", "comparison.json") +REPORTS = ("evaluation.json", "repetition.json", "comparison.json", "suite.json") SCOPE = ( "Consistency of recorded checks, summaries and identities only. " "No candidate execution, grader rerun, producer authentication or HTML verification." @@ -32,49 +41,88 @@ def same_summary(recorded: dict, computed: dict) -> bool: ) +def inventory(directory: Path, expected: set[str], maximum: int) -> None: + if directory.is_symlink(): + raise ValueError("evidence directory must not be a symlink") + found = set() + for child in directory.iterdir(): + found.add(child.name) + if len(found) > maximum: + raise ValueError("too many evidence directory entries") + if found != expected: + raise ValueError("evidence directory inventory differs from the declared records") + + +def duration(row: dict) -> float: + value = row.get("duration_seconds") + if not numeric(value) or value < 0: + raise ValueError("recorded duration must be finite and nonnegative") + return value + + +def xml_signature(node: ET.Element, depth: int = 0) -> tuple: + if depth > 4 or len(node) > 100: + raise ValueError("JUnit structure exceeds suite limits") + text = node.text or "" + if len(node) and text.strip(): + raise ValueError("unexpected text in JUnit container") + if node.tail and node.tail.strip(): + raise ValueError("unexpected trailing text in JUnit") + return ( + node.tag, + sorted(node.attrib.items()), + "" if len(node) else text, + tuple(xml_signature(child, depth + 1) for child in node), + ) + + def verify(path: Path) -> dict: - """Verify evaluation, repetition or comparison JSON and its required inputs.""" + """Verify saved reports and their required inputs without executing a candidate.""" path = path.absolute() if any(item.is_symlink() for item in (path, *path.parents)): raise ValueError("choose an evidence path without symlinks") if path.is_dir(): choices = [path / name for name in REPORTS if (path / name).exists()] - if len(choices) != 1 or (path / "suite.json").exists(): - raise ValueError( - "choose one evaluation, repetition or comparison report; " - "for a suite, verify each jobs/ repetition directory" - ) + if len(choices) != 1: + raise ValueError("choose one evaluation, repetition, comparison or suite report") path = choices[0] root = path.parent files: dict[str, dict] = {} total = 0 - def read(source: Path) -> dict: - nonlocal total + def check_path(source: Path) -> None: if any(item.is_symlink() for item in (source, *source.parents)): raise ValueError("evidence contains a symlink") - document, raw = read_json(source, limit=min(64 * 1024 * 1024, MAX_BUNDLE_BYTES - total)) - total += len(raw) + + def track(source: Path, content: bytes) -> None: + nonlocal total + total += len(content) files[source.relative_to(root).as_posix()] = { - "sha256": hashlib.sha256(raw).hexdigest(), - "bytes": len(raw), + "sha256": hashlib.sha256(content).hexdigest(), + "bytes": len(content), } + + def read(source: Path) -> dict: + check_path(source) + document, content = read_json(source, limit=min(64 * 1024 * 1024, MAX_BUNDLE_BYTES - total)) + track(source, content) return document + def raw(source: Path, limit: int) -> bytes: + check_path(source) + content = read_bytes(source, limit=min(limit, MAX_BUNDLE_BYTES - total)) + track(source, content) + return content + def evaluation(source: Path) -> dict: document = read(source) validate_evaluation(document) return document - recorded = read(path) - schema = recorded.get("schema_version") - regressions = None - if schema == "evalarc.evaluation.v2": - validate_evaluation(recorded) - kind = "evaluation" - valid, resolved = recorded["valid"], recorded["resolved"] - elif schema == "evalarc.repetition.v1": - kind = "repetition" + def repetition(source: Path, document: dict | None = None) -> dict: + recorded = read(source) if document is None else document + if recorded.get("schema_version") != "evalarc.repetition.v1": + raise ValueError("expected repetition schema v1") requested, completed = ( recorded.get("requested_attempts"), recorded.get("completed_attempts"), @@ -85,22 +133,119 @@ def evaluation(source: Path) -> dict: or not 1 <= completed <= requested <= 100 ): raise ValueError("repetition must contain 1–100 completed/requested attempts") - directory = root / "attempts" - if directory.is_symlink(): - raise ValueError("attempts directory must not be a symlink") + directory = source.parent / "attempts" expected = {f"{index:04d}" for index in range(1, completed + 1)} - # Bound enumeration too; extra files or directories cannot hide attempts. - found = set() - for child in directory.iterdir(): - found.add(child.name) - if len(found) > 100: - raise ValueError("too many attempt entries") - if found != expected: - raise ValueError("attempt inventory differs from completed_attempts") + inventory(directory, expected, 100) reports = [evaluation(directory / name / "evaluation.json") for name in sorted(expected)] computed = summarize_attempts(reports, requested) if not same_summary(recorded, computed): raise ValueError("repetition summary differs from its attempt evaluations") + return computed + + def suite(recorded: dict) -> dict: + content = raw(root / "suite.toml", MAX_CONFIG_BYTES) + config = parse_suite_config(content) + manifest_hash = hashlib.sha256(content).hexdigest() + plan = read(root / "plan.json") + planned, rows = plan.get("jobs"), recorded.get("jobs") + if ( + not isinstance(planned, list) + or not isinstance(rows, list) + or len(planned) != len(config["jobs"]) + or len(rows) != len(planned) + or any(not isinstance(row, dict) for row in [*planned, *rows]) + ): + raise ValueError("suite job inventory differs from its configuration") + inventory(root / "jobs", {job["id"] for job in config["jobs"]}, 100) + expected_rows, expected_plan = [], [] + candidates = {} + for spec, saved_plan, row in zip(config["jobs"], planned, rows, strict=True): + identity = spec["id"] + if saved_plan.get("id") != identity or row.get("id") != identity: + raise ValueError("suite job order differs from its configuration") + candidate = saved_plan.get("candidate") + if not isinstance(candidate, str) or not candidate or len(candidate) > 4096: + raise ValueError("invalid recorded candidate path") + # Paths describe the original machine. They are never resolved/read. + # A shared frozen source must still have one recorded byte identity. + summary = repetition(root / "jobs" / identity / "repetition.json") + if ( + candidates.setdefault(candidate, summary["candidate_sha256"]) + != summary["candidate_sha256"] + ): + raise ValueError("shared suite candidate has inconsistent fingerprints") + if ( + summary["task"]["id"] != spec["task"] + or summary["seeds"] != spec["seeds"] + or summary["requested_attempts"] != spec["attempts"] + ): + raise ValueError( + f"job {identity}: task, seeds or attempts differ from configuration" + ) + options = spec["runtime"] + runtime_fields = { + "backend": options["backend"], + "image": options["image"] if options["backend"] == "docker" else None, + "response_timeout_seconds": options["timeout"], + "case_timeout_seconds": options["case_timeout"], + "session_output_limit_bytes": options["output_limit"], + } + if not same_summary( + runtime_fields, {key: summary["runtime"].get(key) for key in runtime_fields} + ): + raise ValueError(f"job {identity}: runtime differs from configuration") + dimensions = {name for case in summary["cases"] for name in case["checks"]} + if not set(spec["gate"]["required_dimensions"]) <= dimensions: + raise ValueError(f"job {identity}: gate names an unknown dimension") + expected_rows.append( + summarize_job(identity, summary, Gate(**spec["gate"]), duration(row)) + ) + expected_plan.append( + { + **spec, + "candidate": candidate, + "cases_per_attempt": len(summary["cases"]), + } + ) + case_count = sum(job["attempts"] * job["cases_per_attempt"] for job in expected_plan) + if case_count > 100_000: + raise ValueError("suite exceeds 100000 planned case executions") + if not same_summary( + plan, + { + "schema_version": "evalarc.suite-plan.v1", + "name": config["name"], + "manifest_sha256": manifest_hash, + "jobs": expected_plan, + "planned_attempts": sum(job["attempts"] for job in expected_plan), + "planned_case_executions": case_count, + }, + ): + raise ValueError("suite plan differs from configuration or attempt evidence") + computed = summarize_suite(config["name"], manifest_hash, expected_rows, duration(recorded)) + if not same_summary(recorded, computed): + raise ValueError("suite summary or gate decision differs from its inputs") + xml = raw(root / "junit.xml", 4 * 1024 * 1024).decode("utf-8") + if " dict: raise ValueError("comparison summary differs from its input evaluations") valid, resolved = current["valid"], current["resolved"] regressions = computed["has_regressions"] + elif schema == "evalarc.suite.v1": + kind = "suite" + computed = suite(recorded) + valid = computed["valid"] + resolved = computed["fully_resolved_jobs"] == computed["total_jobs"] else: raise ValueError( - "unsupported evidence schema; expected evaluation, repetition or comparison" + "unsupported evidence schema; expected evaluation, repetition, comparison or suite" ) result = { "schema_version": "evalarc.verification.v1", @@ -125,4 +275,17 @@ def evaluation(source: Path) -> dict: } if regressions is not None: result["has_regressions"] = regressions + if kind == "suite": + result.update( + { + "accepted": computed["accepted"], + "total_jobs": computed["total_jobs"], + "accepted_jobs": computed["accepted_jobs"], + "fully_resolved_jobs": computed["fully_resolved_jobs"], + } + ) + result["scope"] += ( + " Suite configuration, plan, gates and JUnit are checked. " + "Original candidate paths and timings remain reported metadata." + ) return result diff --git a/tests/test_site.py b/tests/test_site.py index 4c33e47..a12004e 100644 --- a/tests/test_site.py +++ b/tests/test_site.py @@ -4,10 +4,13 @@ import json import shutil import tomllib +import zipfile from pathlib import Path import pytest +from evalarc.verify import verify as verify_evidence + spec = importlib.util.spec_from_file_location( "build_site", Path(__file__).resolve().parents[1] / "scripts" / "build_site.py" ) @@ -15,6 +18,24 @@ spec.loader.exec_module(builder) +def test_downloaded_suite_preserves_bytes_and_verifies_offline(tmp_path): + first, second = tmp_path / "first.zip", tmp_path / "second.zip" + builder.write_suite_bundle(first) + builder.write_suite_bundle(second) + assert first.read_bytes() == second.read_bytes() + with zipfile.ZipFile(first) as archive: + assert len(archive.namelist()) == 12 + archive.extractall(tmp_path / "received") + received = tmp_path / "received" / "suite-evidence" + result = verify_evidence(received) + assert result["verified"] and not result["accepted"] + assert result["accepted_jobs"] == 2 + assert result["fully_resolved_jobs"] == 1 + original = builder.ROOT / "examples" / "suite" + for name in result["files"]: + assert (received / name).read_bytes() == (original / name).read_bytes() + + def test_bundle_rejects_changed_evidence_and_extra_files(tmp_path): folder = tmp_path / "site" builder.build(folder) diff --git a/tests/test_suite.py b/tests/test_suite.py index 04c678c..f0176b7 100644 --- a/tests/test_suite.py +++ b/tests/test_suite.py @@ -12,6 +12,7 @@ from evalarc.junit import render_junit from evalarc.repetition import summarize_attempts from evalarc.suite import Gate, assess_gate, load_suite, run_suite +from evalarc.verify import verify def job(identity, candidate, task="support-routing", *, extra="", backend="local", attempts=1): @@ -291,6 +292,7 @@ def test_mixed_domain_suite_writes_complete_evidence_and_matching_progress( "evalarc.durable-kv", "evalarc.support-routing", } + assert verify(output)["accepted"] def test_invalid_job_does_not_hide_later_jobs_and_junit_errors_are_distinct(tmp_path, reference): @@ -317,6 +319,9 @@ def test_invalid_job_does_not_hide_later_jobs_and_junit_errors_are_distinct(tmp_ assert xml.find(".//testcase[@name='invalid']/error") is not None assert xml.find(".//testcase[@name='failed']/failure") is not None assert xml.find(".//testcase[@name='passing']/failure") is None + checked = verify(output) + assert checked["verified"] and not checked["records_valid"] + assert main(["verify", str(output), "--require-accepted"]) == 2 def test_permissive_acceptance_still_exposes_unresolved_outcomes(tmp_path): @@ -332,6 +337,10 @@ def test_permissive_acceptance_still_exposes_unresolved_outcomes(tmp_path): assert data["accepted"] and data["fully_resolved_jobs"] == 0 assert "Gate accepted with unresolved task outcomes" in (output / "index.html").read_text() assert ET.parse(output / "junit.xml").getroot().attrib["failures"] == "0" + checked = verify(output) + assert checked["accepted"] and not checked["fully_resolved"] + assert main(["verify", str(output), "--require-accepted"]) == 0 + assert main(["verify", str(output), "--require-resolved"]) == 1 path.write_text(path.read_text() + 'required_dimensions = ["notes"]\n') assert ( main( diff --git a/tests/test_verify.py b/tests/test_verify.py index e022173..dfffdc4 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -20,6 +20,7 @@ ("repetition", "repetition", 4, True), ("repetition-faulty", "repetition", 4, False), ("comparison", "comparison", 3, False), + ("suite", "suite", 12, False), ], ) def test_archived_evidence_verifies_without_runtime_or_writes( @@ -110,12 +111,147 @@ def test_pipe_is_rejected_without_waiting_for_writer(tmp_path): verify(path) -def test_suite_and_ambiguous_folder_are_explicitly_unsupported(tmp_path): - with pytest.raises(ValueError, match="for a suite"): - verify(EXAMPLES / "suite") +def test_ambiguous_folder_requires_an_explicit_report(tmp_path): shutil.copytree(EXAMPLES / "repetition", tmp_path / "bundle") shutil.copyfile( EXAMPLES / "evaluation" / "evaluation.json", tmp_path / "bundle/evaluation.json" ) with pytest.raises(ValueError, match="choose one"): verify(tmp_path / "bundle") + + +def change_json(path, update): + data = json.loads(path.read_text()) + update(data) + path.write_text(json.dumps(data)) + + +@pytest.mark.parametrize( + "change", + [ + "accepted", + "job-decision", + "boolean-count", + "job-order", + "runtime", + "manifest-gate", + "repetition", + "junit-failure", + "junit-output", + "missing-job", + "extra-job", + "missing-junit", + "missing-plan", + ], +) +def test_suite_rejects_inconsistent_configuration_decisions_and_junit(tmp_path, capsys, change): + folder = tmp_path / "suite" + shutil.copytree(EXAMPLES / "suite", folder) + if change == "accepted": + change_json(folder / "suite.json", lambda d: d.update(accepted=True)) + elif change == "job-decision": + change_json(folder / "suite.json", lambda d: d["jobs"][2]["decision"].update(accepted=True)) + elif change == "boolean-count": + change_json(folder / "suite.json", lambda d: d.update(fully_resolved_jobs=True)) + elif change == "job-order": + change_json(folder / "plan.json", lambda d: d["jobs"].reverse()) + elif change == "runtime": + change_json(folder / "plan.json", lambda d: d["jobs"][0]["runtime"].update(timeout=500)) + elif change == "manifest-gate": + config = folder / "suite.toml" + config.write_text( + config.read_text().replace( + 'required_dimensions = ["notes"]', "required_dimensions = []" + ) + ) + digest = hashlib.sha256(config.read_bytes()).hexdigest() + for name in ("suite.json", "plan.json"): + change_json(folder / name, lambda d: d.update(manifest_sha256=digest)) + change_json( + folder / "plan.json", lambda d: d["jobs"][2]["gate"].update(required_dimensions=[]) + ) + elif change == "repetition": + change_json( + folder / "jobs/support-partial/repetition.json", lambda d: d.update(mean_score=1) + ) + elif change.startswith("junit-"): + path = folder / "junit.xml" + old, new = ( + ("failure", "error") + if change == "junit-failure" + else ('"fully_resolved": false', '"fully_resolved": true') + ) + path.write_text(path.read_text().replace(old, new)) + elif change == "missing-job": + shutil.rmtree(folder / "jobs/support-protected") + elif change == "extra-job": + (folder / "jobs/unreported").mkdir() + else: + (folder / ("junit.xml" if change == "missing-junit" else "plan.json")).unlink() + assert main(["verify", str(folder), "--json"]) == 2 + result = json.loads(capsys.readouterr().out) + assert not result["verified"] and result["error"] + + +@pytest.mark.parametrize("filename", ["suite.toml", "junit.xml", "plan.json"]) +def test_suite_rejects_symlinked_inputs(tmp_path, filename): + folder = tmp_path / "suite" + shutil.copytree(EXAMPLES / "suite", folder) + (folder / filename).unlink() + (folder / filename).symlink_to(EXAMPLES / "suite" / filename) + with pytest.raises(ValueError, match="symlink"): + verify(folder) + + +def test_suite_verification_never_resolves_original_paths_or_prepares_runtime(monkeypatch): + from evalarc.runner import Runtime + + def forbidden(*args, **kwargs): + raise AssertionError("offline evidence checking cannot prepare an execution") + + with monkeypatch.context() as patch: + patch.setattr(Runtime, "prepare", forbidden) + patch.setattr(Path, "resolve", forbidden) + patch.setattr(subprocess, "Popen", forbidden) + result = verify(EXAMPLES / "suite") + assert result["accepted_jobs"] == 2 + assert result["fully_resolved_jobs"] == 1 + assert not result["accepted"] + assert {"suite.toml", "plan.json", "junit.xml"} <= result["files"].keys() + + +def test_suite_gate_requirement_differs_from_record_consistency(capsys): + path = str(EXAMPLES / "suite") + assert main(["verify", path, "--json"]) == 0 + assert json.loads(capsys.readouterr().out)["verified"] + assert main(["verify", path, "--json", "--require-accepted"]) == 1 + assert not json.loads(capsys.readouterr().out)["accepted"] + assert main(["verify", str(EXAMPLES / "repetition"), "--json", "--require-accepted"]) == 2 + assert "requires suite" in json.loads(capsys.readouterr().out)["error"] + + +@pytest.mark.parametrize( + "xml", + [ + ']>&x;', + "" + "" * 20 + "" * 20 + "", + "", + ], +) +def test_junit_rejects_entities_invalid_xml_and_deep_structures(tmp_path, xml): + folder = tmp_path / "suite" + shutil.copytree(EXAMPLES / "suite", folder) + (folder / "junit.xml").write_text(xml) + with pytest.raises(ValueError): + verify(folder) + + +@pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="POSIX named pipes") +@pytest.mark.parametrize("filename", ["suite.toml", "junit.xml"]) +def test_suite_non_json_inputs_must_be_regular_files(tmp_path, filename): + folder = tmp_path / "suite" + shutil.copytree(EXAMPLES / "suite", folder) + (folder / filename).unlink() + os.mkfifo(folder / filename) + with pytest.raises(ValueError, match="regular file"): + verify(folder)