Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,24 @@ jobs:
strategy:
matrix:
task: ["durable-kv", "support-routing"]
language: ["python", "javascript"]
include:
- language: python
image: python:3.12-slim
- language: javascript
image: node:22-slim
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install .
- run: docker pull python:3.12-slim
- run: evalarc audit --task ${{ matrix.task }} --seeds 17 --output runs/docker-audit
- run: docker pull ${{ matrix.image }}
- run: evalarc audit --task ${{ matrix.task }} --language ${{ matrix.language }} --image ${{ matrix.image }} --seeds 17 --output runs/docker-audit
- uses: actions/upload-artifact@v4
if: always()
with:
name: grader-audit-${{ matrix.task }}
name: grader-audit-${{ matrix.task }}-${{ matrix.language }}
path: runs/docker-audit/
site:
runs-on: ubuntu-latest
Expand Down
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# Changelog

## 0.6.0 · 2026-09-14 · Research preview

- `init --language python|javascript` supplies starters and references for both
tasks. JavaScript workspaces include a portable command manifest, the complete
task contract, and runtime guidance; Python remains the default.
- An independent Node.js durable service preserves numeric source text, integer
precision, float/integer distinctions, signed floating zero, nested values,
and unordered object equality. A complete snapshot precedes write acknowledgement.
- `audit --language javascript` exercises the same eight coding and seven
simulated-ticket fault models. Local audits resolve Node from the host PATH;
Docker image selection remains explicit.
- Workspace initialization stages complete files before publishing and removes
partial output after failures. Existing files and directories remain protected.
- Protocol tests cover values beyond the public task cases, SIGKILL recovery,
invalid batches, special keys, failed persistence, and corrupt snapshots.
- Docker CI audits now cover both languages and both tasks. The Python package
includes all JavaScript templates without adding a Python runtime dependency.

Task contracts, grading/runtime source files, and evidence schemas are unchanged.
Comparisons still require matching recorded commands, runtime, grader, and cases;
different-language executions do not become a matched comparison automatically.
No TypeScript SDK, Rust worker, or model-provider adapter is introduced.

## 0.5.0 · 2026-09-14 · Research preview

- `evalarc suite` executes versioned TOML plans across multiple candidates and
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
type: software
authors:
- name: EvalArc contributors
version: 0.5.0
version: 0.6.0
date-released: 2026-09-14
license: MIT
repository-code: "https://github.com/noteflowai/evalarc"
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,21 @@ committed Docker audits without a model API or installation.

[![EvalArc v0.3: score rises from 90% to 93.75% while a check regresses](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.5 includes two working task packs**, a shared evidence format, and
**v0.6 includes two working task packs**, a shared evidence format, and
configurable candidate commands:

| Task | Interaction | Host verification | Declared faults |
| --- | --- | --- | ---: |
| `durable-kv` | Run a coding agent's completed service | Responses, transactions, restart durability | 8 |
| `support-routing` | Drive a policy through simulated ticket tools | Routing, exact notes, closure, unrelated state, protocol | 7 |

v0.6 adds **Python and JavaScript workspace templates for both tasks**.
Use `init --language javascript` for a starter or `--reference` for a scripted
control, and `audit --language javascript` to check the same 15 fault models
with independent Node.js implementations. JavaScript requires Node.js 22+;
Docker runs explicitly select `--image node:22-slim`. See the
[multilanguage guide](docs/languages.md).

v0.5 adds `evalarc suite`: declare tasks, candidates, repeats, budgets, and
acceptance gates in TOML. Preview the plan, execute all jobs, and inspect HTML,
JSON, and JUnit results. Scores remain task-specific. See the
Expand Down
8 changes: 7 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,19 @@ EvalArc 关注智能体实际完成的结果,以及支撑评分结论的证据

[![EvalArc v0.3:分数上升,一项检查却退步](docs/assets/regression-lab.png)](https://huggingface.co/spaces/glayguo/evalarc)

**v0.5 已实现 coding 和业务工具两个场景。**
**v0.6 已实现 coding 和业务工具两个场景。**

| 任务 | 交互方式 | 验证内容 |
| --- | --- | --- |
| `durable-kv` | 执行代码智能体交付的服务 | 读写、事务、CAS、持久化与异常恢复 |
| `support-routing` | 策略通过工具操作模拟工单 | 路由、精确备注、条件关闭、无关数据保护与协议完成 |

v0.6 为两个任务都提供 **Python 和 JavaScript 工作区模板**:
`init --language javascript` 生成起步代码,添加 `--reference` 生成脚本对照;
`audit --language javascript` 使用独立的 Node.js 实现检查相同的 15 类故障。
JavaScript 需要 Node.js 22+,Docker 模式显式指定 `--image node:22-slim`。
详见[多语言接入指南](docs/languages.zh-CN.md)。

v0.5 新增 `evalarc suite`:用 TOML 声明任务、候选、轮次、预算及验收门槛,
先预览执行计划,再批量运行并输出 HTML、JSON 和 JUnit。各任务单独评分。
详见[套件与 CI 指南](docs/suites.zh-CN.md)。
Expand Down
13 changes: 9 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Architecture and domain expansion

EvalArc's direction is open environments and auditable evaluations for
software agents. This document distinguishes the working v0.5 implementation
software agents. This document distinguishes the working v0.6 implementation
from interfaces proposed for subsequent releases.

## Current implementation
Expand Down Expand Up @@ -92,10 +92,15 @@ when an interactive viewer or JavaScript SDK is
implemented. Introduce a Rust worker only after measured execution or
distribution requirements justify a separate component.

The `main.py` restriction now applies only to the default command. A separate
JavaScript support policy validates the protocol with Node. This release has
no TypeScript SDK or tested Rust submission/worker. See
The `main.py` restriction applies only to the default command. Both task packs
now have Python and JavaScript starters, references, and declared fault audits.
Templates live outside grading source files. Node's durable reference uses
JSON snapshots, independently of the Python SQLite reference; it preserves
numeric source text and type-sensitive equality through process restarts.
The runtime still executes the manifest command without a language-specific
worker or bridge. See the [multilanguage guide](languages.md) and
[candidate commands](candidate-commands.md) for image and executable requirements.
This release has no TypeScript SDK or tested Rust submission/worker.

## Metrics

Expand Down
14 changes: 9 additions & 5 deletions docs/candidate-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
The evaluator selects a task with `--task`; the default remains `durable-kv`.
Candidate configuration cannot choose a task, grader, backend, limits, or image.

`evalarc init workspace/js --language javascript --task support-routing`
generates a JavaScript starter and command manifest. Both tasks also support
`--reference`; Python remains the default. See the [multilanguage guide](languages.md).

To replace a task's default Python entrypoint, put `evalarc.toml` in the
candidate workspace:

Expand Down Expand Up @@ -45,11 +49,11 @@ Candidates have no network in Docker. Local subprocesses receive a minimal
environment with the system executable search path; use an absolute executable
path for a runtime outside that path.

The repository validates Python in both backends and the independent JavaScript
support policy with local Node.js. Compiled TypeScript can use the JavaScript
entrypoint, but no TypeScript SDK or compiler integration is provided. Rust
workers, Rust submissions, and Node-in-Docker execution have not been validated
in this release.
The repository validates both tasks with Python and Node.js 22+ in local and
Docker execution. JavaScript images must be selected explicitly, for example
`--image node:22-slim`. Compiled TypeScript can use the JavaScript entrypoint,
but no TypeScript SDK or compiler integration is provided. Rust workers and Rust
submissions have not been validated in this release.

Command templates are included in report runtime metadata. A command change
makes checkpoint runs incomparable even if the task and source files match.
Expand Down
107 changes: 107 additions & 0 deletions docs/languages.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Python and JavaScript candidates

EvalArc's evaluator remains Python 3.11+. Version 0.6 packages Python and
JavaScript starters and scripted references for both task packs. Candidate
language does not select the verifier or change the scoring contract.

| Command | Python | JavaScript |
| --- | --- | --- |
| `init` | Default; creates `main.py` and `TASK.md` | `--language javascript`; also creates `evalarc.toml` and `RUNTIME.md` |
| `init --reference` | SQLite service or ticket policy | JSON-snapshot service or independent ticket policy |
| `audit` | Default; existing controls | `--language javascript`; the same 8 coding / 7 support faults |
| `evaluate`, `repeat`, `suite` | Execute the candidate manifest or default command | Execute the generated manifest; no language flag needed |

## Generate and evaluate

After installing EvalArc, run from the repository root:

```bash
docker pull python:3.12-slim
docker pull node:22-slim
evalarc init workspace/kv-python --reference
evalarc init workspace/kv-javascript --language javascript --reference
evalarc init workspace/support-javascript --task support-routing --language javascript --reference

evalarc doctor --candidate workspace/kv-javascript --image node:22-slim
evalarc evaluate workspace/kv-javascript --image node:22-slim --output runs/kv-javascript
evalarc evaluate workspace/support-javascript --task support-routing \
--image node:22-slim --output runs/support-javascript
```

Omit `--reference` to generate unfinished starter code. The starter produces
valid protocol responses but does not complete the task. Initialization needs
neither Node nor Docker, refuses existing destinations, and publishes a
complete workspace only after all files have been written.

Docker image selection is explicit. The language option does not replace the
default Python image. Node.js 22+ must be in the chosen image; no packages are
installed at evaluation time. The exact immutable image ID is recorded.

## Audit the controls

```bash
evalarc audit --language javascript --image node:22-slim \
--seeds 17 --output runs/js-coding-audit
evalarc audit --task support-routing --language javascript --image node:22-slim \
--seeds 17 --output runs/js-support-audit
```

Audits apply the existing named defects to each language's reference source,
then run every control through the existing host verifier. Success requires
the reference to resolve and each defect to fail in its declared dimension.
The [recorded audits](../examples/javascript-audits/README.md) retain the
commands, candidate fingerprints, image IDs, checks, and traces. These are
scripted controls on public development tasks, not model performance results
or a bound on undiscovered grader defects.

## Run multiple languages together

The generated workspaces above match the paths in the example suite:

```bash
evalarc suite examples/multilanguage/suite.toml --dry-run
evalarc suite examples/multilanguage/suite.toml --output runs/multilanguage
```

The three jobs choose their own task, image, cases, and full-resolution gate.
All candidates and images are preflighted before the first job starts. Results
retain separate scores and a JUnit testcase per job gate.
The [recorded Docker suite](../examples/multilanguage/run/index.html) preserves
all three jobs and their 34 case executions.

`compare` and checkpoint trajectories still require matching recorded runtime
commands and conditions. A Python/Node pair, or a pair using different images,
is not automatically a matched comparison. Shared task checks let you inspect
conformance; they do not establish a language ranking.

## Local execution and other languages

For trusted candidates only, append `--backend local --trust-local` to an
evaluation or audit. Candidate subprocesses use a minimal PATH. If Node lives
in nvm or another version manager, put its absolute executable path in the
workspace's `evalarc.toml`. Local JavaScript audits resolve Node from the host
PATH themselves and record the resulting command. Docker manifests should
keep the container command, usually `"node"`.

TypeScript can be compiled before evaluation and launched through the same
command manifest. EvalArc does not include a TypeScript compiler or SDK.
A prebuilt Rust executable can use the generic command interface, but this
release does not supply or validate a Rust reference/worker.

## Durable reference semantics

The Node reference is independent of the Python SQLite implementation. It
stores a complete JSON snapshot, flushes it, and renames it before acknowledging
each mutation. Only acknowledged mutations surviving process crashes are in
the task contract; this is not a production database or a power-loss claim.

It retains numeric source text using the Node 22 JSON APIs. This preserves
integers larger than JavaScript's exact Number range and distinguishes `1`
from `1.0`. CAS compares nested JSON values independent of object property
order, while retaining Boolean/number, integer/float, and signed-float-zero
distinctions. Equivalent floating spellings compare equally. Map-based storage
supports keys such as `__proto__` without treating them as object metadata.

The [validation record](validation-v0.6.md) describes protocol edge cases,
fault detection, packaging, and runtime checks. Tests do not expand the
published grader's case set or alter its fingerprint.
89 changes: 89 additions & 0 deletions docs/languages.zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Python 与 JavaScript 候选接入

EvalArc 评测核心仍使用 Python 3.11+。v0.6 为两个任务都打包了 Python 和
JavaScript 起步模板、参考实现与故障审计。候选使用的语言不改变任务契约和评分规则。

| 命令 | Python | JavaScript |
| --- | --- | --- |
| `init` | 默认生成 `main.py`、`TASK.md` | 加 `--language javascript`,同时生成命令清单和运行说明 |
| `init --reference` | SQLite 服务或工单策略 | JSON 快照服务或独立工单策略 |
| `audit` | 默认审计 Python 对照 | `--language javascript`,检查相同的 coding 8 类、工单 7 类故障 |
| `evaluate`、`repeat`、`suite` | 使用默认命令或候选清单 | 执行生成的命令清单,无需再指定语言 |

## 创建并运行

安装 EvalArc 后,在仓库根目录执行:

```bash
docker pull python:3.12-slim
docker pull node:22-slim
evalarc init workspace/kv-python --reference
evalarc init workspace/kv-javascript --language javascript --reference
evalarc init workspace/support-javascript --task support-routing --language javascript --reference

evalarc doctor --candidate workspace/kv-javascript --image node:22-slim
evalarc evaluate workspace/kv-javascript --image node:22-slim --output runs/kv-javascript
evalarc evaluate workspace/support-javascript --task support-routing \
--image node:22-slim --output runs/support-javascript
```

不加 `--reference` 就会生成待实现的起步代码,它能响应协议,但不能完成任务。
创建模板不需要 Node 或 Docker;目标路径已存在时拒绝覆盖,写入中途失败会清理
不完整工作区。

JavaScript 需要 Node.js 22+。镜像必须显式选择:`--language javascript`
不会自动替换默认 Python 镜像。评测时不安装依赖,证据保存实际使用的不可变镜像 ID。

## 审计并检查证据

```bash
evalarc audit --language javascript --image node:22-slim \
--seeds 17 --output runs/js-coding-audit
evalarc audit --task support-routing --language javascript --image node:22-slim \
--seeds 17 --output runs/js-support-audit
```

通过条件包含参考实现完全完成任务,以及每个故障在预定维度被发现。
[已保存的审计记录](../examples/javascript-audits/README.md)保留命令、候选指纹、
镜像、逐项检查及交互轨迹。这些是公开开发任务上的脚本对照结果,不能代替模型评测,
也不能保证评分器不存在其他缺陷。

上面的三个工作区可直接用于混合语言套件:

```bash
evalarc suite examples/multilanguage/suite.toml --dry-run
evalarc suite examples/multilanguage/suite.toml --output runs/multilanguage
```

每项作业独立声明任务、镜像和验收门槛。第一项开始前,所有候选快照和镜像都必须
通过预检查。各任务分数保持独立,JUnit 为每个作业门槛生成一项测试。
[已保存的 Docker 套件](../examples/multilanguage/run/index.html)包含三项作业和
34 次场景执行的完整记录。

`compare` 和检查点曲线仍要求记录中的命令及运行条件一致。
Python 与 Node 的两次运行不会自动成为可配对比较的记录;共同的任务检查可用于
核验契约遵守情况,但不能据此给语言排名。

## 本机与其他语言

可信代码可用 `--backend local --trust-local`。候选子进程只有最小 PATH:
Node 如果位于 nvm 等版本管理器目录,需要把工作区 `evalarc.toml` 中的 `"node"`
改为绝对路径。本机 JavaScript 审计会自行从宿主 PATH 找到 Node,并记录实际命令。
Docker 清单使用容器内命令,通常保留 `"node"`。

TypeScript 可以先编译,再通过命令清单启动产物;当前不提供 TypeScript 编译器
或 SDK。预编译的 Rust 程序可使用通用命令接口,本版尚未提供或验证 Rust 参考实现
及 worker。

## 持久化参考实现的边界

Node 参考实现使用完整 JSON 快照,与 Python SQLite 实现相互独立。
每次变更先写盘、刷新并替换快照,再返回确认。契约检查的是已确认变更在进程崩溃后
仍可恢复,不能将结果扩展为生产数据库性能、并发写入或断电保证。

实现保留 JSON 数字原文,避免大整数精度丢失以及 `1` 和 `1.0` 被合并。
CAS 递归比较嵌套结构,忽略对象属性顺序,但区分布尔值和数字、整数和小数、
浮点正零和负零;数值相同的小数写法可以相等。Map 存储支持 `__proto__` 等特殊键。

[验证记录](validation-v0.6.md)列出额外协议用例、故障检测、容器和安装包检查。
这些测试没有修改公开评分场景或评分器指纹。
19 changes: 19 additions & 0 deletions docs/migration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,24 @@
# Migration notes

## EvalArc v0.5 → v0.6

`init` and `audit` accept `--language python|javascript`; Python remains the
default. Default Python workspace bytes and audit source controls are unchanged.
JavaScript templates need Node.js 22+. Choose `--image node:22-slim` explicitly
for Docker; image selection does not follow the language flag.

JavaScript workspaces contain `main.js`, a command manifest, the task contract,
and runtime notes. `evaluate`, `repeat`, and `suite` execute that manifest with
their existing options. Local JavaScript audits resolve Node from the host PATH
and record the absolute executable path. See the [language guide](languages.md).

Initialization now stages all files before publishing the workspace. Existing
destinations are still refused; a write failure removes partial output.

No evidence schema, task contract, task case, or grading/runtime source changes.
Existing matching v0.4/v0.5 records remain comparable. Different-language or
different-image runs still fail the matched-command/runtime requirements.

## EvalArc v0.4 → v0.5

`evalarc suite CONFIG.toml` adds declarative, sequential multi-job execution.
Expand Down
Loading
Loading