Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Changelog

## 0.8.0 · 2026-09-14 · Research preview

- Verify whole suite handoffs from the original TOML, plan, every repetition and
attempt, recomputed custom gates and JUnit failure/error records.
- Add `--require-accepted` for configured suite gates, distinct from record
consistency and full task resolution. Original candidate paths and timings
remain reported metadata; no candidate or grader is executed.
- Download the featured suite as a deterministic ZIP from the evidence lab.
Preserve all 12 original input files and verify the extracted archive offline.
- Check received suites from the installed wheel outside the checkout with an
empty PATH, including rejection of a modified JUnit record.

Task contracts, scoring rules and historical evidence bytes are unchanged.


## 0.7.1 · 2026-09-14 · Research preview

- Share and restore a specific task pack, control, seed, case and trace step in
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ details to the case list, and retry failed sections independently.

[![EvalArc v0.3: score rises from 90% to 93.75% while a check regresses](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.8: verify the whole handoff.** Download the suite evidence ZIP from the
lab, then run `evalarc verify suite-evidence --json` to recompute its original
TOML, plan, five attempts, custom gates and JUnit. Add `--require-accepted`
for CI acceptance. Consistency, configured acceptance and full resolution are
reported separately. No candidate execution is required.
[Offline verification and limits](docs/verification.md).

**Two working task packs** share an evidence format and
configurable candidate commands:

Expand Down
6 changes: 6 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ EvalArc 关注智能体实际完成的结果,以及支撑评分结论的证据

[![EvalArc v0.3:分数上升,一项检查却退步](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.8:完整验收证据可离线复核。** 从首页下载套件 ZIP,解压后运行
`evalarc verify suite-evidence --json`,复核原始 TOML、执行计划、五次尝试、
验收规则与 JUnit。使用 `--require-accepted` 接入 CI 验收;证据一致、规则接受、
任务完全完成分别报告。整个检查不执行候选程序。
[离线验证流程与边界](docs/verification.md)。

**已实现 coding 和业务工具两个场景。**

| 任务 | 交互方式 | 验证内容 |
Expand Down
18 changes: 9 additions & 9 deletions docs/outreach/hellogithub-submission.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,31 +12,31 @@ https://github.com/noteflowai/evalarc

### 项目描述

EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python/JavaScript 候选、TOML 套件、JUnit 和离线报告复核,保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例;新版可分享到具体任务、用例与调用步骤,并独立重试加载失败的分区。提供可筛选 HF Casebook 和中英文文档,现有演示为脚本对照。
EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python/JavaScript 候选、TOML 套件、JUnit 和离线报告复核,保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例;0.8.0 可下载完整套件证据,并离线复算原始配置、执行计划、验收规则和 JUnit;浏览器可分享到具体调用步骤。提供可筛选 HF Casebook 和中英文文档,现有演示为脚本对照。

### 亮点

- **把讨论定位到同一步**:0.7.1 的 Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点,键盘可从详情返回案例列表;受限剪贴板有手动复制入口。
- **把讨论定位到同一步**: Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点,键盘可从详情返回案例列表;受限剪贴板有手动复制入口。
- **失败可恢复**:suite、重复尝试、前后比较和任务包可以分别重试;一个任务包不可用时仍可检查另一个。页面经过 320/390/1440 像素与键盘路径验证。
- **高分不代表完成**:工单策略得 93.75% 却重复写备注;代码实现得 92.5% 却混淆 JSON true 与 1。可检查全部 15 种声明缺陷,以及三项 suite 作业、六条重复尝试和 167 条审计用例的数据表。
- **可复核交付**:`evalarc verify` 不执行候选代码,重算单次、重复和对照报告;`--require-resolved` 另行要求完整通过。Python/JavaScript 参考实现共享任务约定。
- **可复核交付**:`evalarc verify` 不执行候选代码,重算单次、重复、对照和完整套件报告,包括 TOML、plan、五次尝试与 JUnit;`--require-accepted` 检查配置的验收规则,`--require-resolved` 另行要求任务完整通过。Python/JavaScript 参考实现共享任务约定。

本账号为维护者,项目与 AI 结对开发,采用 MIT 许可,处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照,不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器;suite 级规则和 JUnit 暂不在 verify 的核验范围
本账号为维护者,项目与 AI 结对开发,采用 MIT 许可,处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照,不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器,也不认证报告作者;原始候选路径与计时仍是报告元数据

### 示例代码

安装发布的 Python wheel 后,核对收到的重复评测目录,无需 Docker 或 Node:
安装发布的 Python wheel,从首页下载 suite-evidence.zip 并解压后,无需 Docker 或 Node 即可核验;默认退出 0 表示证据一致,添加 --require-accepted 退出 1 表示严格备注规则拒绝该结果

```sh
evalarc verify received/repetition --json
evalarc verify suite-evidence --json

# Also require valid, fully resolved results:
evalarc verify received/repetition --json --require-resolved
# Require every configured suite gate:
evalarc verify suite-evidence --json --require-accepted
```

### 截图或演示视频

在线体验:https://huggingface.co/spaces/glayguo/evalarc
版本:https://github.com/noteflowai/evalarc/releases/tag/v0.7.1
版本:https://github.com/noteflowai/evalarc/releases/tag/v0.8.0

![分享具体失败证据,检查智能体评分器的盲点](https://github.com/noteflowai/evalarc/raw/main/docs/assets/suite-lab.png)
8 changes: 4 additions & 4 deletions docs/outreach/huggingface-introduction.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Select a case and trace step, then use **Copy evidence link**. The recipient reopens the same task, implementation, seed, case and state change. Clipboard-denied embeds offer a selectable link. Keyboard users can move into case details and return to the list.
**Download the evidence. Check every gate.** EvalArc 0.8.0 verifies a whole suite from its original TOML, plan, every repetition and attempt, custom acceptance rules and JUnit. It runs offline without a candidate process, Docker, Node or the original candidate directory.

Failed suite, repetition, comparison and audit requests can be retried independently. One unavailable task pack no longer blocks the other; retrying can restore the originally linked view. The explorer fits 320px screens and keeps JSON scrolling inside its cards. Saved reports and scoring rules are unchanged.
Download **suite-evidence.zip** from the lab and unzip it. `evalarc verify suite-evidence --json` checks all 12 original input files and exits 0 for consistency. Adding `--require-accepted` exits 1: the strict notes gate rejects the faulty policy. Two of three jobs are accepted; only one is fully resolved. Those are separate outcomes.

[Open the Space](https://huggingface.co/spaces/glayguo/evalarc) · [Release 0.7.1](https://github.com/noteflowai/evalarc/releases/tag/v0.7.1) · [Source and methods](https://github.com/noteflowai/evalarc)
[Open the evidence lab](https://huggingface.co/spaces/glayguo/evalarc) · [Release 0.8.0](https://github.com/noteflowai/evalarc/releases/tag/v0.8.0) · [Verification workflow and limits](https://github.com/noteflowai/evalarc/blob/main/docs/verification.md)

The same public casebook still provides 167 audit cases, six repeated attempts and three suite jobs. These are scripted development controls, not frontier-model scores or unseen-task reliability estimates. EvalArc remains a research preview.
The public Casebook still provides 167 audit cases, six repeated attempts and three suite jobs. Historical source bytes and scoring rules are unchanged. These are scripted development controls; offline record consistency does not authenticate the producer or independently rerun the grader. Original candidate paths and durations remain reported metadata. EvalArc remains a research preview.

Maintainer update to the existing introduction, developed with AI assistance. Independent community project; no upstream or Hugging Face endorsement is implied.
18 changes: 9 additions & 9 deletions docs/outreach/weekly-submission.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
维护者自荐:**evalarc**。EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python/JavaScript 候选、TOML 套件、JUnit 和离线报告复核,保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例;新版可分享到具体任务、用例与调用步骤,并独立重试加载失败的分区。提供可筛选 HF Casebook 和中英文文档,现有演示为脚本对照。
维护者自荐:**evalarc**。EvalArc 是面向 AI 智能体的开源评测与验收工具。支持 Python/JavaScript 候选、TOML 套件、JUnit 和离线报告复核,保留每轮原始记录。交互实验室展示高分仍违反关键业务约束的案例;0.8.0 可下载完整套件证据,并离线复算原始配置、执行计划、验收规则和 JUnit;浏览器可分享到具体调用步骤。提供可筛选 HF Casebook 和中英文文档,现有演示为脚本对照。

在线体验:https://huggingface.co/spaces/glayguo/evalarc

- **把讨论定位到同一步**:0.7.1 的 Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点,键盘可从详情返回案例列表;受限剪贴板有手动复制入口。
- **把讨论定位到同一步**: Copy evidence link 保存任务、对照实现、种子、案例与 trace 步骤。接收者打开同一观察点,键盘可从详情返回案例列表;受限剪贴板有手动复制入口。
- **失败可恢复**:suite、重复尝试、前后比较和任务包可以分别重试;一个任务包不可用时仍可检查另一个。页面经过 320/390/1440 像素与键盘路径验证。
- **高分不代表完成**:工单策略得 93.75% 却重复写备注;代码实现得 92.5% 却混淆 JSON true 与 1。可检查全部 15 种声明缺陷,以及三项 suite 作业、六条重复尝试和 167 条审计用例的数据表。
- **可复核交付**:`evalarc verify` 不执行候选代码,重算单次、重复和对照报告;`--require-resolved` 另行要求完整通过。Python/JavaScript 参考实现共享任务约定。
- **可复核交付**:`evalarc verify` 不执行候选代码,重算单次、重复、对照和完整套件报告,包括 TOML、plan、五次尝试与 JUnit;`--require-accepted` 检查配置的验收规则,`--require-resolved` 另行要求任务完整通过。Python/JavaScript 参考实现共享任务约定。

```sh
evalarc verify received/repetition --json
evalarc verify suite-evidence --json

# Also require valid, fully resolved results:
evalarc verify received/repetition --json --require-resolved
# Require every configured suite gate:
evalarc verify suite-evidence --json --require-accepted
```

安装发布的 Python wheel 后,核对收到的重复评测目录,无需 Docker 或 Node:
安装发布的 Python wheel,从首页下载 suite-evidence.zip 并解压后,无需 Docker 或 Node 即可核验;默认退出 0 表示证据一致,添加 --require-accepted 退出 1 表示严格备注规则拒绝该结果

项目:https://github.com/noteflowai/evalarc
版本:https://github.com/noteflowai/evalarc/releases/tag/v0.7.1
版本:https://github.com/noteflowai/evalarc/releases/tag/v0.8.0

![分享具体失败证据,检查智能体评分器的盲点](https://github.com/noteflowai/evalarc/raw/main/docs/assets/suite-lab.png)

本账号为维护者,项目与 AI 结对开发,采用 MIT 许可,处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照,不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器;suite 级规则和 JUnit 暂不在 verify 的核验范围
本账号为维护者,项目与 AI 结对开发,采用 MIT 许可,处于研究预览阶段。现有展示来自已保存的公开开发任务和脚本 Docker 对照,不提供真实大模型排名或 RL 收益结论。离线一致性校验不等于重新运行评分器,也不认证报告作者;原始候选路径与计时仍是报告元数据
8 changes: 8 additions & 0 deletions docs/suites.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,3 +161,11 @@ cancellation; keep that file outside candidates and the output directory.
The suite stream adds job IDs to the existing `evalarc.event.v1` events.

See the [validation record](validation-v0.5.md) for the checks actually run.

## Verify a received suite without executing it

With EvalArc 0.8+, keep the original TOML, plan, suite JSON, JUnit and all job
attempts together. Run `evalarc verify received/suite --json --require-accepted`
to recompute configured gates offline. Full resolution is a separate condition,
available through `--require-resolved`. Original candidate paths are metadata;
do not rewrite them when moving the evidence. [Handoff guide](verification.md).
7 changes: 7 additions & 0 deletions docs/suites.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,10 @@ JUnit 中一个 testcase 对应一个 **job 验收门槛**:
把外部日志放在候选及输出目录之外。

完整字段与 CI 示例见[英文指南](suites.md),本轮实测范围见[验证记录](validation-v0.5.md)。

## 不执行候选程序,复核收到的套件

EvalArc 0.8 起,保留原始 TOML、计划、套件 JSON、JUnit 和各作业全部尝试,
运行 `evalarc verify received/suite --json --require-accepted`,即可离线复算验收规则。
`--require-resolved` 另行要求任务完全完成。原始候选路径作为元数据保留,
移动证据时不要改写。[完整交付指南](verification.md)。
Loading
Loading