diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 95a5184..bb5a124 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -31,18 +31,24 @@ jobs:
strategy:
matrix:
task: ["durable-kv", "support-routing"]
+ language: ["python", "javascript"]
+ include:
+ - language: python
+ image: python:3.12-slim
+ - language: javascript
+ image: node:22-slim
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install .
- - run: docker pull python:3.12-slim
- - run: evalarc audit --task ${{ matrix.task }} --seeds 17 --output runs/docker-audit
+ - run: docker pull ${{ matrix.image }}
+ - run: evalarc audit --task ${{ matrix.task }} --language ${{ matrix.language }} --image ${{ matrix.image }} --seeds 17 --output runs/docker-audit
- uses: actions/upload-artifact@v4
if: always()
with:
- name: grader-audit-${{ matrix.task }}
+ name: grader-audit-${{ matrix.task }}-${{ matrix.language }}
path: runs/docker-audit/
site:
runs-on: ubuntu-latest
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6e8670..711b278 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,28 @@
# Changelog
+## 0.6.0 · 2026-09-14 · Research preview
+
+- `init --language python|javascript` supplies starters and references for both
+ tasks. JavaScript workspaces include a portable command manifest, the complete
+ task contract, and runtime guidance; Python remains the default.
+- An independent Node.js durable service preserves numeric source text, integer
+ precision, float/integer distinctions, signed floating zero, nested values,
+ and unordered object equality. A complete snapshot precedes write acknowledgement.
+- `audit --language javascript` exercises the same eight coding and seven
+ simulated-ticket fault models. Local audits resolve Node from the host PATH;
+ Docker image selection remains explicit.
+- Workspace initialization stages complete files before publishing and removes
+ partial output after failures. Existing files and directories remain protected.
+- Protocol tests cover values beyond the public task cases, SIGKILL recovery,
+ invalid batches, special keys, failed persistence, and corrupt snapshots.
+- Docker CI audits now cover both languages and both tasks. The Python package
+ includes all JavaScript templates without adding a Python runtime dependency.
+
+Task contracts, grading/runtime source files, and evidence schemas are unchanged.
+Comparisons still require matching recorded commands, runtime, grader, and cases;
+different-language executions do not become a matched comparison automatically.
+No TypeScript SDK, Rust worker, or model-provider adapter is introduced.
+
## 0.5.0 · 2026-09-14 · Research preview
- `evalarc suite` executes versioned TOML plans across multiple candidates and
diff --git a/CITATION.cff b/CITATION.cff
index 75afcf6..de2515f 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -4,7 +4,7 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
type: software
authors:
- name: EvalArc contributors
-version: 0.5.0
+version: 0.6.0
date-released: 2026-09-14
license: MIT
repository-code: "https://github.com/noteflowai/evalarc"
diff --git a/README.md b/README.md
index 85105e9..870a694 100644
--- a/README.md
+++ b/README.md
@@ -30,7 +30,7 @@ committed Docker audits without a model API or installation.
[](https://huggingface.co/spaces/glayguo/evalarc)
-**v0.5 includes two working task packs**, a shared evidence format, and
+**v0.6 includes two working task packs**, a shared evidence format, and
configurable candidate commands:
| Task | Interaction | Host verification | Declared faults |
@@ -38,6 +38,13 @@ configurable candidate commands:
| `durable-kv` | Run a coding agent's completed service | Responses, transactions, restart durability | 8 |
| `support-routing` | Drive a policy through simulated ticket tools | Routing, exact notes, closure, unrelated state, protocol | 7 |
+v0.6 adds **Python and JavaScript workspace templates for both tasks**.
+Use `init --language javascript` for a starter or `--reference` for a scripted
+control, and `audit --language javascript` to check the same 15 fault models
+with independent Node.js implementations. JavaScript requires Node.js 22+;
+Docker runs explicitly select `--image node:22-slim`. See the
+[multilanguage guide](docs/languages.md).
+
v0.5 adds `evalarc suite`: declare tasks, candidates, repeats, budgets, and
acceptance gates in TOML. Preview the plan, execute all jobs, and inspect HTML,
JSON, and JUnit results. Scores remain task-specific. See the
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 1921117..0a1d702 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -24,13 +24,19 @@ EvalArc 关注智能体实际完成的结果，以及支撑评分结论的证据
[](https://huggingface.co/spaces/glayguo/evalarc)
-**v0.5 已实现 coding 和业务工具两个场景。**
+**v0.6 已实现 coding 和业务工具两个场景。**
| 任务 | 交互方式 | 验证内容 |
| --- | --- | --- |
| `durable-kv` | 执行代码智能体交付的服务 | 读写、事务、CAS、持久化与异常恢复 |
| `support-routing` | 策略通过工具操作模拟工单 | 路由、精确备注、条件关闭、无关数据保护与协议完成 |
+v0.6 为两个任务都提供 **Python 和 JavaScript 工作区模板**：
+`init --language javascript` 生成起步代码，添加 `--reference` 生成脚本对照；
+`audit --language javascript` 使用独立的 Node.js 实现检查相同的 15 类故障。
+JavaScript 需要 Node.js 22+，Docker 模式显式指定 `--image node:22-slim`。
+详见[多语言接入指南](docs/languages.zh-CN.md)。
+
v0.5 新增 `evalarc suite`：用 TOML 声明任务、候选、轮次、预算及验收门槛，
先预览执行计划，再批量运行并输出 HTML、JSON 和 JUnit。各任务单独评分。
详见[套件与 CI 指南](docs/suites.zh-CN.md)。
diff --git a/docs/architecture.md b/docs/architecture.md
index fdfed65..b68c913 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -1,7 +1,7 @@
# Architecture and domain expansion
EvalArc's direction is open environments and auditable evaluations for
-software agents. This document distinguishes the working v0.5 implementation
+software agents. This document distinguishes the working v0.6 implementation
from interfaces proposed for subsequent releases.
## Current implementation
@@ -92,10 +92,15 @@ when an interactive viewer or JavaScript SDK is
implemented. Introduce a Rust worker only after measured execution or
distribution requirements justify a separate component.
-The `main.py` restriction now applies only to the default command. A separate
-JavaScript support policy validates the protocol with Node. This release has
-no TypeScript SDK or tested Rust submission/worker. See
+The `main.py` restriction applies only to the default command. Both task packs
+now have Python and JavaScript starters, references, and declared fault audits.
+Templates live outside grading source files. Node's durable reference uses
+JSON snapshots, independently of the Python SQLite reference; it preserves
+numeric source text and type-sensitive equality through process restarts.
+The runtime still executes the manifest command without a language-specific
+worker or bridge. See the [multilanguage guide](languages.md) and
[candidate commands](candidate-commands.md) for image and executable requirements.
+This release has no TypeScript SDK or tested Rust submission/worker.
## Metrics
diff --git a/docs/candidate-commands.md b/docs/candidate-commands.md
index b08ca19..22c7b9c 100644
--- a/docs/candidate-commands.md
+++ b/docs/candidate-commands.md
@@ -3,6 +3,10 @@
The evaluator selects a task with `--task`; the default remains `durable-kv`.
Candidate configuration cannot choose a task, grader, backend, limits, or image.
+`evalarc init workspace/js --language javascript --task support-routing`
+generates a JavaScript starter and command manifest. Both tasks also support
+`--reference`; Python remains the default. See the [multilanguage guide](languages.md).
+
To replace a task's default Python entrypoint, put `evalarc.toml` in the
candidate workspace:
@@ -45,11 +49,11 @@ Candidates have no network in Docker. Local subprocesses receive a minimal
environment with the system executable search path; use an absolute executable
path for a runtime outside that path.
-The repository validates Python in both backends and the independent JavaScript
-support policy with local Node.js. Compiled TypeScript can use the JavaScript
-entrypoint, but no TypeScript SDK or compiler integration is provided. Rust
-workers, Rust submissions, and Node-in-Docker execution have not been validated
-in this release.
+The repository validates both tasks with Python and Node.js 22+ in local and
+Docker execution. JavaScript images must be selected explicitly, for example
+`--image node:22-slim`. Compiled TypeScript can use the JavaScript entrypoint,
+but no TypeScript SDK or compiler integration is provided. Rust workers and Rust
+submissions have not been validated in this release.
Command templates are included in report runtime metadata. A command change
makes checkpoint runs incomparable even if the task and source files match.
diff --git a/docs/languages.md b/docs/languages.md
new file mode 100644
index 0000000..3e1fc19
--- /dev/null
+++ b/docs/languages.md
@@ -0,0 +1,107 @@
+# Python and JavaScript candidates
+
+EvalArc's evaluator remains Python 3.11+. Version 0.6 packages Python and
+JavaScript starters and scripted references for both task packs. Candidate
+language does not select the verifier or change the scoring contract.
+
+| Command | Python | JavaScript |
+| --- | --- | --- |
+| `init` | Default; creates `main.py` and `TASK.md` | `--language javascript`; also creates `evalarc.toml` and `RUNTIME.md` |
+| `init --reference` | SQLite service or ticket policy | JSON-snapshot service or independent ticket policy |
+| `audit` | Default; existing controls | `--language javascript`; the same 8 coding / 7 support faults |
+| `evaluate`, `repeat`, `suite` | Execute the candidate manifest or default command | Execute the generated manifest; no language flag needed |
+
+## Generate and evaluate
+
+After installing EvalArc, run from the repository root:
+
+```bash
+docker pull python:3.12-slim
+docker pull node:22-slim
+evalarc init workspace/kv-python --reference
+evalarc init workspace/kv-javascript --language javascript --reference
+evalarc init workspace/support-javascript --task support-routing --language javascript --reference
+
+evalarc doctor --candidate workspace/kv-javascript --image node:22-slim
+evalarc evaluate workspace/kv-javascript --image node:22-slim --output runs/kv-javascript
+evalarc evaluate workspace/support-javascript --task support-routing \
+ --image node:22-slim --output runs/support-javascript
+```
+
+Omit `--reference` to generate unfinished starter code. The starter produces
+valid protocol responses but does not complete the task. Initialization needs
+neither Node nor Docker, refuses existing destinations, and publishes a
+complete workspace only after all files have been written.
+
+Docker image selection is explicit. The language option does not replace the
+default Python image. Node.js 22+ must be in the chosen image; no packages are
+installed at evaluation time. The exact immutable image ID is recorded.
+
+## Audit the controls
+
+```bash
+evalarc audit --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-coding-audit
+evalarc audit --task support-routing --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-support-audit
+```
+
+Audits apply the existing named defects to each language's reference source,
+then run every control through the existing host verifier. Success requires
+the reference to resolve and each defect to fail in its declared dimension.
+The [recorded audits](../examples/javascript-audits/README.md) retain the
+commands, candidate fingerprints, image IDs, checks, and traces. These are
+scripted controls on public development tasks, not model performance results
+or a bound on undiscovered grader defects.
+
+## Run multiple languages together
+
+The generated workspaces above match the paths in the example suite:
+
+```bash
+evalarc suite examples/multilanguage/suite.toml --dry-run
+evalarc suite examples/multilanguage/suite.toml --output runs/multilanguage
+```
+
+The three jobs choose their own task, image, cases, and full-resolution gate.
+All candidates and images are preflighted before the first job starts. Results
+retain separate scores and a JUnit testcase per job gate.
+The [recorded Docker suite](../examples/multilanguage/run/index.html) preserves
+all three jobs and their 34 case executions.
+
+`compare` and checkpoint trajectories still require matching recorded runtime
+commands and conditions. A Python/Node pair, or a pair using different images,
+is not automatically a matched comparison. Shared task checks let you inspect
+conformance; they do not establish a language ranking.
+
+## Local execution and other languages
+
+For trusted candidates only, append `--backend local --trust-local` to an
+evaluation or audit. Candidate subprocesses use a minimal PATH. If Node lives
+in nvm or another version manager, put its absolute executable path in the
+workspace's `evalarc.toml`. Local JavaScript audits resolve Node from the host
+PATH themselves and record the resulting command. Docker manifests should
+keep the container command, usually `"node"`.
+
+TypeScript can be compiled before evaluation and launched through the same
+command manifest. EvalArc does not include a TypeScript compiler or SDK.
+A prebuilt Rust executable can use the generic command interface, but this
+release does not supply or validate a Rust reference/worker.
+
+## Durable reference semantics
+
+The Node reference is independent of the Python SQLite implementation. It
+stores a complete JSON snapshot, flushes it, and renames it before acknowledging
+each mutation. Only acknowledged mutations surviving process crashes are in
+the task contract; this is not a production database or a power-loss claim.
+
+It retains numeric source text using the Node 22 JSON APIs. This preserves
+integers larger than JavaScript's exact Number range and distinguishes `1`
+from `1.0`. CAS compares nested JSON values independent of object property
+order, while retaining Boolean/number, integer/float, and signed-float-zero
+distinctions. Equivalent floating spellings compare equally. Map-based storage
+supports keys such as `__proto__` without treating them as object metadata.
+
+The [validation record](validation-v0.6.md) describes protocol edge cases,
+fault detection, packaging, and runtime checks. Tests do not expand the
+published grader's case set or alter its fingerprint.
diff --git a/docs/languages.zh-CN.md b/docs/languages.zh-CN.md
new file mode 100644
index 0000000..afcc1fd
--- /dev/null
+++ b/docs/languages.zh-CN.md
@@ -0,0 +1,89 @@
+# Python 与 JavaScript 候选接入
+
+EvalArc 评测核心仍使用 Python 3.11+。v0.6 为两个任务都打包了 Python 和
+JavaScript 起步模板、参考实现与故障审计。候选使用的语言不改变任务契约和评分规则。
+
+| 命令 | Python | JavaScript |
+| --- | --- | --- |
+| `init` | 默认生成 `main.py`、`TASK.md` | 加 `--language javascript`，同时生成命令清单和运行说明 |
+| `init --reference` | SQLite 服务或工单策略 | JSON 快照服务或独立工单策略 |
+| `audit` | 默认审计 Python 对照 | `--language javascript`，检查相同的 coding 8 类、工单 7 类故障 |
+| `evaluate`、`repeat`、`suite` | 使用默认命令或候选清单 | 执行生成的命令清单，无需再指定语言 |
+
+## 创建并运行
+
+安装 EvalArc 后，在仓库根目录执行：
+
+```bash
+docker pull python:3.12-slim
+docker pull node:22-slim
+evalarc init workspace/kv-python --reference
+evalarc init workspace/kv-javascript --language javascript --reference
+evalarc init workspace/support-javascript --task support-routing --language javascript --reference
+
+evalarc doctor --candidate workspace/kv-javascript --image node:22-slim
+evalarc evaluate workspace/kv-javascript --image node:22-slim --output runs/kv-javascript
+evalarc evaluate workspace/support-javascript --task support-routing \
+ --image node:22-slim --output runs/support-javascript
+```
+
+不加 `--reference` 就会生成待实现的起步代码，它能响应协议，但不能完成任务。
+创建模板不需要 Node 或 Docker；目标路径已存在时拒绝覆盖，写入中途失败会清理
+不完整工作区。
+
+JavaScript 需要 Node.js 22+。镜像必须显式选择：`--language javascript`
+不会自动替换默认 Python 镜像。评测时不安装依赖，证据保存实际使用的不可变镜像 ID。
+
+## 审计并检查证据
+
+```bash
+evalarc audit --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-coding-audit
+evalarc audit --task support-routing --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-support-audit
+```
+
+通过条件包含参考实现完全完成任务，以及每个故障在预定维度被发现。
+[已保存的审计记录](../examples/javascript-audits/README.md)保留命令、候选指纹、
+镜像、逐项检查及交互轨迹。这些是公开开发任务上的脚本对照结果，不能代替模型评测，
+也不能保证评分器不存在其他缺陷。
+
+上面的三个工作区可直接用于混合语言套件：
+
+```bash
+evalarc suite examples/multilanguage/suite.toml --dry-run
+evalarc suite examples/multilanguage/suite.toml --output runs/multilanguage
+```
+
+每项作业独立声明任务、镜像和验收门槛。第一项开始前，所有候选快照和镜像都必须
+通过预检查。各任务分数保持独立，JUnit 为每个作业门槛生成一项测试。
+[已保存的 Docker 套件](../examples/multilanguage/run/index.html)包含三项作业和
+34 次场景执行的完整记录。
+
+`compare` 和检查点曲线仍要求记录中的命令及运行条件一致。
+Python 与 Node 的两次运行不会自动成为可配对比较的记录；共同的任务检查可用于
+核验契约遵守情况，但不能据此给语言排名。
+
+## 本机与其他语言
+
+可信代码可用 `--backend local --trust-local`。候选子进程只有最小 PATH：
+Node 如果位于 nvm 等版本管理器目录，需要把工作区 `evalarc.toml` 中的 `"node"`
+改为绝对路径。本机 JavaScript 审计会自行从宿主 PATH 找到 Node，并记录实际命令。
+Docker 清单使用容器内命令，通常保留 `"node"`。
+
+TypeScript 可以先编译，再通过命令清单启动产物；当前不提供 TypeScript 编译器
+或 SDK。预编译的 Rust 程序可使用通用命令接口，本版尚未提供或验证 Rust 参考实现
+及 worker。
+
+## 持久化参考实现的边界
+
+Node 参考实现使用完整 JSON 快照，与 Python SQLite 实现相互独立。
+每次变更先写盘、刷新并替换快照，再返回确认。契约检查的是已确认变更在进程崩溃后
+仍可恢复，不能将结果扩展为生产数据库性能、并发写入或断电保证。
+
+实现保留 JSON 数字原文，避免大整数精度丢失以及 `1` 和 `1.0` 被合并。
+CAS 递归比较嵌套结构，忽略对象属性顺序，但区分布尔值和数字、整数和小数、
+浮点正零和负零；数值相同的小数写法可以相等。Map 存储支持 `__proto__` 等特殊键。
+
+[验证记录](validation-v0.6.md)列出额外协议用例、故障检测、容器和安装包检查。
+这些测试没有修改公开评分场景或评分器指纹。
diff --git a/docs/migration.md b/docs/migration.md
index 66b072d..3d81b34 100644
--- a/docs/migration.md
+++ b/docs/migration.md
@@ -1,5 +1,24 @@
# Migration notes
+## EvalArc v0.5 → v0.6
+
+`init` and `audit` accept `--language python|javascript`; Python remains the
+default. Default Python workspace bytes and audit source controls are unchanged.
+JavaScript templates need Node.js 22+. Choose `--image node:22-slim` explicitly
+for Docker; image selection does not follow the language flag.
+
+JavaScript workspaces contain `main.js`, a command manifest, the task contract,
+and runtime notes. `evaluate`, `repeat`, and `suite` execute that manifest with
+their existing options. Local JavaScript audits resolve Node from the host PATH
+and record the absolute executable path. See the [language guide](languages.md).
+
+Initialization now stages all files before publishing the workspace. Existing
+destinations are still refused; a write failure removes partial output.
+
+No evidence schema, task contract, task case, or grading/runtime source changes.
+Existing matching v0.4/v0.5 records remain comparable. Different-language or
+different-image runs still fail the matched-command/runtime requirements.
+
## EvalArc v0.4 → v0.5
`evalarc suite CONFIG.toml` adds declarative, sequential multi-job execution.
diff --git a/docs/outreach/huggingface-introduction.md b/docs/outreach/huggingface-introduction.md
index c31c069..affe478 100644
--- a/docs/outreach/huggingface-introduction.md
+++ b/docs/outreach/huggingface-introduction.md
@@ -1,7 +1,14 @@
I'm publishing EvalArc as its maintainer: an open toolkit for auditing the
graders behind AI-agent evaluations.
-**Updated for v0.5: same score, different acceptance gate.**
+**Updated for v0.6: Python and JavaScript, with explicit acceptance gates.**
+
+Generate starters or independent scripted references in either language, and
+run the same task contracts through the existing host graders. The Node Docker
+audits detect all 15 declared faults; the recorded mixed-language suite preserves
+three resolved jobs and 34 case executions. Runtime images are explicit, and
+this is conformance evidence rather than a language or model ranking.
+[Language guide](https://github.com/noteflowai/evalarc/blob/main/docs/languages.md).
**New data companion:** the
[EvalArc Casebook](https://huggingface.co/datasets/glayguo/evalarc-casebook)
diff --git a/docs/outreach/launch.md b/docs/outreach/launch.md
index 3b35967..08f1907 100644
--- a/docs/outreach/launch.md
+++ b/docs/outreach/launch.md
@@ -12,6 +12,8 @@ EvalArc 是面向 AI 智能体的开源评测与评分器审计工具，帮助
并保留重复评测、版本回归与全部运行证据，适合 PAI 智能体创新场景的评测设计、
验收验证与技术交流。HF Casebook 进一步提供可筛选、可用 Python 读取的公开
证据表，分别保留 167 条用例、6 次重复尝试和 3 项验收作业及其原始记录。
+v0.6 支持生成 Python／JavaScript 候选，并通过独立实现验证同一任务约定，
+可在统一套件中为不同语言明确指定运行镜像。
当前为研究预览版，演示使用脚本对照，尚未给出真实大模型
性能或强化学习收益结论。
@@ -44,6 +46,11 @@ suite jobs filterable and readable from Python as separate development
configurations. Each row retains a pointer and hash for its original source
record. This is a tabular view of existing evidence, not additional model trials.
+Version 0.6 supplies Python and JavaScript starters and independent references
+for both tasks. The Node audits exercise the same declared faults, and a
+mixed-language suite retains three resolved jobs. Shared task contracts do not
+establish a language ranking.
+
## Entry points
- Source: https://github.com/noteflowai/evalarc
diff --git a/docs/roadmap.md b/docs/roadmap.md
index cf20c18..884b159 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -18,6 +18,7 @@ question needs substantially more evidence.
| v0.3, implemented | Readiness checks, individual HTML reports, matched-run comparisons, protected output paths | Read-only inspection, per-check regression detection, input consistency checks, and preserved prior runs |
| v0.4, implemented | Fixed-candidate repeated evaluation, per-check variability, case deadlines, progress events, process diagnostics | Fresh state per attempt, explicit invalid denominators, shared restart budgets, cancellation and cleanup tests |
| v0.5, implemented | Declarative suites, per-job acceptance gates, HTML/JSON and JUnit export | All-candidate preflight, workload preview, protected dimensions, distinct invalid outcomes, no cross-domain score average |
+| v0.6, implemented | Python/JavaScript templates and independent controls for both tasks | The same 15 declared faults, Node Docker execution, lossless numeric handling, process-crash recovery, atomic workspace initialization |
| Next | Broader language coverage for coding | A non-Python Durable KV submission passes the same full contract |
| Next | Independent defect packs and task authors | Reviewer-authored faults and a third independently authored task |
| Next | Real model-provider adapter | End-to-end run with measured usage, task evidence, repeated attempts |
diff --git a/docs/validation-v0.6.md b/docs/validation-v0.6.md
new file mode 100644
index 0000000..9e1af82
--- /dev/null
+++ b/docs/validation-v0.6.md
@@ -0,0 +1,70 @@
+# Validation record · v0.6.0 · 2026-09-14
+
+Checks ran locally on Linux with Python 3.12.3 and local Node.js 22.23.2.
+The release adds candidate templates and independent controls around the
+existing evaluators. No model API was invoked.
+
+| Check | Observed result |
+| --- | --- |
+| Python tests in an isolated Git worktree | 205 passed, including 33 new template/protocol tests |
+| Repository Ruff lint and formatting | Passed |
+| JavaScript durable reference and declared defects, Docker | Reference resolved 15/15 cases; all 8 faults detected in their declared dimensions |
+| JavaScript support reference and declared defects, Docker | Reference resolved 4/4 cases; all 7 faults detected in their declared dimensions |
+| Local JavaScript controls, seed 17 | Both references resolved; all 15 declared faults detected |
+| Generated JavaScript references, seeds 41 and 97 | Both tasks resolved every case |
+| Local protocol edge cases | Large integers, integer/float distinctions, signed zero, exponent spellings, nested objects, property-order independence, special keys, malformed JSON, rejected batches |
+| Persistence edge cases | Values and CAS survive SIGKILL; write failures produce no acknowledgement; duplicate-key corrupt snapshots are not silently reset |
+| Initialization | Python default bytes retained; no runtime needed for scaffolding; existing destinations protected; partial writes cleaned up |
+| Installed wheel outside the development checkout | Generated workspaces, doctor, explicit-image Docker suite, starter failures, repeat, and JavaScript audit worked |
+| Installed-wheel mixed-language Docker suite | 3/3 gates accepted and jobs fully resolved; 34 case executions; JUnit has 3 tests, 0 failures, 0 errors |
+| Installed-wheel JavaScript starters | Both correctly fail to resolve; coding score 0, support score 0.35 from preserved-state checks |
+| Installed-wheel JavaScript support repetition | 2/2 resolved attempts, no observed variable checks |
+| Historical Python comparison | Matching v0.4/v0.6 coding records have identical grading/case/runtime metadata, score delta 0, and no regression |
+| Container cleanup | No EvalArc containers remained after validation |
+
+The [JavaScript audit records](../examples/javascript-audits/README.md) contain
+167 total case executions: 135 coding and 32 support cases across references and
+faulty controls. The [mixed-language suite](../examples/multilanguage/run/index.html)
+preserves another 34 cases from the installed wheel, bringing the Docker
+validation total to 201. It retains the original TOML, JUnit, every attempt,
+per-job reports, and progress events.
+
+The Node image used by both audits and the JavaScript suite jobs is
+`sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5`.
+The Python suite job uses the existing image
+`sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea`.
+These IDs identify the local resolved images; future pulls of mutable tags may
+resolve differently.
+
+The documented suite is parsed by a regression test after generating its
+workspaces. Its runtime settings use `[jobs.runtime]` tables, as required by
+the existing strict configuration schema. All three commands are preflighted
+before the first suite job starts.
+
+Final packaging includes all four JavaScript sources, runtime notes, and the
+new template module. Wheel sources/assets are checked against the installation
+used above. The source distribution retains the new suite configuration and
+audit/suite evidence, including JSONL and JUnit XML. Documentation links and
+standalone report links are checked against the final tree.
+
+Task contracts, grading source files, and evidence schemas did not change.
+Node-specific protocol tests are additional development checks, not new
+published grader cases. Existing promotion pages and historical records retain
+their original provenance. Docker CI is configured for both tasks and both
+languages; this record describes local execution, not a hosted CI run.
+
+These controls establish observed conformance and detection of named defects.
+They do not establish production database durability, resistance to arbitrary
+reward hacking, model reliability, or TypeScript/Rust SDK support.
+
+## Publication integration
+
+The completed local change was merged with the separately verified HF
+Casebook publication. The combined tree passed 210 Python tests, lint and
+formatting. A fresh mixed-language Docker suite again resolved all three jobs;
+the archived JavaScript audit evaluations were independently validated.
+
+The casebook keeps its original 167 audit cases, six repeated attempts and
+three suite jobs from earlier versions. It does not relabel those records as
+v0.6 language comparisons. Site and dataset artifacts share the guarded
+publication workflow; the Docker matrix now audits both languages and tasks.
diff --git a/examples/javascript-audits/README.md b/examples/javascript-audits/README.md
new file mode 100644
index 0000000..9750b79
--- /dev/null
+++ b/examples/javascript-audits/README.md
@@ -0,0 +1,24 @@
+# JavaScript control audits
+
+These records exercise independent Node.js references and deliberately faulty
+variants through the existing task graders. They use seed 17 in Docker.
+
+| Task | Reference | Detected faults | Evidence |
+| --- | --- | --- | --- |
+| Durable KV | 15/15 cases | 8/8 | [Report](durable-kv/index.html), [JSON](durable-kv/audit.json), [events](durable-kv/events.jsonl) |
+| Simulated support | 4/4 cases | 7/7 | [Report](support-routing/index.html), [JSON](support-routing/audit.json), [events](support-routing/events.jsonl) |
+
+Reproduce after installing EvalArc 0.6 and pulling `node:22-slim`:
+
+```bash
+evalarc audit --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-coding-audit
+evalarc audit --task support-routing --language javascript --image node:22-slim \
+ --seeds 17 --output runs/js-support-audit
+```
+
+The [validation record](../../docs/validation-v0.6.md) identifies the image used.
+Each JSON preserves the resolved image ID and the full per-control evaluation.
+The archived output is evidence of these executions, not a model benchmark or
+proof of general grader robustness. The original Python examples retain their
+historical provenance.
diff --git a/examples/javascript-audits/durable-kv/audit.json b/examples/javascript-audits/durable-kv/audit.json
new file mode 100644
index 0000000..de58163
--- /dev/null
+++ b/examples/javascript-audits/durable-kv/audit.json
@@ -0,0 +1,4257 @@
+{
+ "schema_version": "evalarc.audit.v2",
+ "valid": true,
+ "reference_passed": true,
+ "reference": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:53:49.832260+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "fab9d6c65ef16a3c8bfb0fdf83ea73fddda5c9484a304f625cf67bff8ec0d097",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.528938,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.550636,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.578917,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.648721,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.70211,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.769467,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.646747,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.754041,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.900915,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.192842,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 1.28181,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.151454,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.134884,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.855847,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.862338,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 14.576101,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ },
+ "mutants": [
+ {
+ "name": "ack-without-work",
+ "target_dimension": "basic",
+ "killed": true,
+ "valid": true,
+ "score": 0.0,
+ "failing_cases": [
+ "json-types-and-overwrite",
+ "read-write-delete",
+ "seeded-state-machine"
+ ],
+ "candidate_sha256": "4a2182c4aeb53fccd91851d943a3f6fb55cf27d03941012cbd23942df81df51e",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:00.654947+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "4a2182c4aeb53fccd91851d943a3f6fb55cf27d03941012cbd23942df81df51e",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.0,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 0,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 0.0
+ },
+ "validation": {
+ "passed": 0,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 0.0
+ },
+ "transactions": {
+ "passed": 0,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 0.0
+ },
+ "compare_swap": {
+ "passed": 0,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 0.0
+ },
+ "persistence": {
+ "passed": 0,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 0.0
+ },
+ "crash_recovery": {
+ "passed": 0,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 0.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 0,
+ "duration_seconds": 0.549895,
+ "transcript_sha256": "e9d1c1a4ce952f925f5a3d00a84348bf482007670ba13602f73d0129cc422a60",
+ "error": "response mismatch at request 1; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.584373,
+ "transcript_sha256": "ad684a2e4e62254b35236e279cda7ec03bfe6bf6a01e3c0b24a1810e6c8088d2",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": null}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.584876,
+ "transcript_sha256": "932f3ec3f66ee636feddf02978271d1af20d352d113d51580182f4956132fcbc",
+ "error": "response mismatch at request 2; expected {\"error\": \"invalid_request\", \"ok\": false}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.578509,
+ "transcript_sha256": "dd5c4b23bf9661f05363e4be2791dffa554d3bb7a6226d1be64e58adc97e7907",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.547386,
+ "transcript_sha256": "660d89f2f1ccd8d228c1871197f23f08289d1c1fe244918053aa9aca3a9ce576",
+ "error": "response mismatch at request 2; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.53177,
+ "transcript_sha256": "8d05dbc72469fa9ab22e4943d0e9ed13a8733f443dc95a84a306daa02b2031b4",
+ "error": "response mismatch at request 2; expected {\"error\": \"invalid_request\", \"ok\": false}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.549798,
+ "transcript_sha256": "92c53f111ed9d1c600b448aaf9e5a529f32d2f085433ad700cd369bd71c55185",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.559346,
+ "transcript_sha256": "88cbf896bd041b5c3a80f99fc01f19d0b473a1a39cdb4f59d621a8abbcaa35e4",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 1.051951,
+ "transcript_sha256": "d01e5e8ceee977002ba418a5bf134418cd85c9b36813ea2fadfa7908c141b718",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.926142,
+ "transcript_sha256": "5c70c6333e7c60969ecad930a4b839122a1582a89cdf257e323ffa6fbbf3036f",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.903025,
+ "transcript_sha256": "d01e5e8ceee977002ba418a5bf134418cd85c9b36813ea2fadfa7908c141b718",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.980445,
+ "transcript_sha256": "5c70c6333e7c60969ecad930a4b839122a1582a89cdf257e323ffa6fbbf3036f",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.526977,
+ "transcript_sha256": "7ef0ed9705196a8514631dacec2c92eeb5e34cd9811d281f9816e348c50d1a17",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": true}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 24,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 1.481053,
+ "transcript_sha256": "165fee513b6ad33052636d17d8a1f492e3226d1b5f19b246f01f1c20289f5525",
+ "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 0.451502,
+ "transcript_sha256": "ce6e264c70b36102aabe44145d2a7558766913a2725261b8f120efea98fa8659",
+ "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 36,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 10.820054,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "memory-only",
+ "target_dimension": "persistence",
+ "killed": true,
+ "valid": true,
+ "score": 0.78333333,
+ "failing_cases": [
+ "restart-preserves-acknowledged-write",
+ "restart-preserves-cas",
+ "restart-preserves-transaction"
+ ],
+ "candidate_sha256": "65617a32e2dec95ea2ce1457efae7d1dbe6e26094a6adc5c67514f333813bc23",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:10.972898+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "65617a32e2dec95ea2ce1457efae7d1dbe6e26094a6adc5c67514f333813bc23",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.78333333,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 0,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 0.0
+ },
+ "crash_recovery": {
+ "passed": 1,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 0.3333333333333333
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.46604,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.474375,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.475647,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.458584,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.464691,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.455858,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.457636,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.454052,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.928483,
+ "transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.958207,
+ "transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.937998,
+ "transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.975821,
+ "transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": -9,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 0.924555,
+ "transcript_sha256": "497082839111501c6aa4912312fe30686befcab05beb84ebbcb99be0d1e8aef1",
+ "error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": 321}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": -9,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.389628,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": -9,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.483543,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 10.317339,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "partial-batch",
+ "target_dimension": "transactions",
+ "killed": true,
+ "valid": true,
+ "score": 0.9,
+ "failing_cases": [
+ "rollback-batch"
+ ],
+ "candidate_sha256": "25c9faee716ab4c4d81dea864df1e8d8560d01110e153cd84c199ad6ffb0ddb3",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:22.312752+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "25c9faee716ab4c4d81dea864df1e8d8560d01110e153cd84c199ad6ffb0ddb3",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.9,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 1,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 0.5
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.461501,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.530414,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.470365,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.491789,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.479108,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 0.472393,
+ "transcript_sha256": "f6015865dce884224cf6ccccc9fbdd367baad17842dde00fbee476f06b1aab0d",
+ "error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": true, \"ok\": true, \"value\": \"must roll back\"}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.492097,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.497769,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.426448,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.929136,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 0.987817,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.923095,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.96986,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.474352,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.718944,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 11.339695,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "cas-always-wins",
+ "target_dimension": "compare_swap",
+ "killed": true,
+ "valid": true,
+ "score": 0.85,
+ "failing_cases": [
+ "cas-match-and-mismatch",
+ "cas-type-sensitivity"
+ ],
+ "candidate_sha256": "6ab2bd9883cf0eff59df9198d55c7934827b6fb539453f6b7eb17a9db113a8b8",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:33.188001+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "6ab2bd9883cf0eff59df9198d55c7934827b6fb539453f6b7eb17a9db113a8b8",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.85,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 0,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 0.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.570831,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.583597,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.547586,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.522098,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.508744,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.494022,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.490188,
+ "transcript_sha256": "71199e89272547e9c1a1961a85a650b93c30e2f848912b6be28ea262907eda56",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.530264,
+ "transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.323756,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.841123,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 0.84372,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.848225,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.826512,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.292735,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.642089,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 10.874828,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "boolean-equals-one",
+ "target_dimension": "compare_swap",
+ "killed": true,
+ "valid": true,
+ "score": 0.925,
+ "failing_cases": [
+ "cas-type-sensitivity"
+ ],
+ "candidate_sha256": "202d2f3820eb71874cffb461ddc9f695af43ed3d4126aac7784475a97f1b6bb5",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:44.053559+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "202d2f3820eb71874cffb461ddc9f695af43ed3d4126aac7784475a97f1b6bb5",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.925,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 1,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 0.5
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.460583,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.473433,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.433965,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.396075,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.455688,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.432909,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.450149,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.435249,
+ "transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
+ "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.268524,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.865693,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 1.084807,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.982003,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": -9,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.977971,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.427617,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.710429,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 10.86503,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "delete-noop",
+ "target_dimension": "basic",
+ "killed": true,
+ "valid": true,
+ "score": 0.65,
+ "failing_cases": [
+ "read-write-delete",
+ "seeded-state-machine"
+ ],
+ "candidate_sha256": "0500d54aca41a5403568b3950be43efd2ecadbf09c2f003043c7a0a7324f2807",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:54:55.080156+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "0500d54aca41a5403568b3950be43efd2ecadbf09c2f003043c7a0a7324f2807",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.65,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 1,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 0.3333333333333333
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 1,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 0.5
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 2,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 0.6666666666666666
+ },
+ "crash_recovery": {
+ "passed": 2,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 0.6666666666666666
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 4,
+ "duration_seconds": 0.484824,
+ "transcript_sha256": "4122b55bc205e4703a3a9f62a5397de448a833f424e54edc914bbbf5465c70d1",
+ "error": "response mismatch at request 5; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 216,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.543204,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.555981,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.475617,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 0.435511,
+ "transcript_sha256": "da7df3388998e4f6fd4743177baa1ddcd8625d54fded8d5c6cfbdf813d50cdba",
+ "error": "response mismatch at request 2; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.433667,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.449461,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.435147,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 3,
+ "duration_seconds": 1.273556,
+ "transcript_sha256": "0ae3e20082ef3346a8027b09f61e373e1c733234603e3e12d5e61853d6df9c8f",
+ "error": "response mismatch at request 4; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.025015,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 0.934191,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.852946,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.050153,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 1.495149,
+ "transcript_sha256": "679e7ab1f7f833fb12551b29af4fdc1ad8e1051810e594a4da85fdc3420b8c42",
+ "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 8,
+ "duration_seconds": 0.569837,
+ "transcript_sha256": "0132892f9b4bac6264eb6ef2dae9da8b7bac7a6a158b56357f1319c853615de7",
+ "error": "response mismatch at request 9; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 193,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 11.0261,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "accept-nonstring-keys",
+ "target_dimension": "validation",
+ "killed": true,
+ "valid": true,
+ "score": 0.925,
+ "failing_cases": [
+ "reject-and-continue"
+ ],
+ "candidate_sha256": "e17d1eeddba4a649c867c98b683de7ce64c4cbd5da448e8e29723569c8c0de5f",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:55:07.877997+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "e17d1eeddba4a649c867c98b683de7ce64c4cbd5da448e8e29723569c8c0de5f",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.925,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 1,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 0.5
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.505065,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.527997,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 4,
+ "duration_seconds": 0.523412,
+ "transcript_sha256": "dc07c86f871afd6b2f208e9751bdd459e52368d54d008ca0622a8edd61a7008b",
+ "error": "response mismatch at request 5; expected {\"error\": \"invalid_request\", \"ok\": false}, got {\"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 141,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.504849,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.509386,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.51418,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.537505,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.502504,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.441501,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.975848,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 1.016199,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.075213,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.133081,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.903977,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 1.11267,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 12.797362,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ },
+ {
+ "name": "commit-on-exit",
+ "target_dimension": "crash_recovery",
+ "killed": true,
+ "valid": true,
+ "score": 0.9,
+ "failing_cases": [
+ "sigkill-after-acknowledgement",
+ "sigkill-after-atomic-batch",
+ "sigkill-after-delete"
+ ],
+ "candidate_sha256": "884f21ad792983fc73e53f6b55ba9531b003b4cbaf51b362690b1c17431981e7",
+ "evaluation": {
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:55:22.421082+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "884f21ad792983fc73e53f6b55ba9531b003b4cbaf51b362690b1c17431981e7",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "failed",
+ "score": 0.9,
+ "resolved": false,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 0,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 0.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.749596,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.669885,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.682262,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.800129,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.694564,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.645486,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.891929,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.673617,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 2.004075,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.184324,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 1.130183,
+ "transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 1,
+ "duration_seconds": 1.0895,
+ "transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
+ "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}",
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.085192,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": false
+ },
+ "passed": false,
+ "status": "failed",
+ "checks_completed": 2,
+ "duration_seconds": 1.640536,
+ "transcript_sha256": "679e7ab1f7f833fb12551b29af4fdc1ad8e1051810e594a4da85fdc3420b8c42",
+ "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}",
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.581022,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 14.54055,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+ }
+ }
+ ],
+ "mutation_score": 1.0,
+ "killed": 8,
+ "total": 8,
+ "passed": true,
+ "interpretation": "Coverage of 8 declared behavioral fault models only. This is not a bound on reward hacking or a model benchmark."
+}
diff --git a/examples/javascript-audits/durable-kv/events.jsonl b/examples/javascript-audits/durable-kv/events.jsonl
new file mode 100644
index 0000000..cb88290
--- /dev/null
+++ b/examples/javascript-audits/durable-kv/events.jsonl
@@ -0,0 +1,288 @@
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:35.260307+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "fab9d6c65ef16a3c8bfb0fdf83ea73fddda5c9484a304f625cf67bff8ec0d097", "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:35.260437+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:35.789718+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.528938, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:35.789914+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:36.340761+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.550636, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:36.342270+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:36.921345+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.578917, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:36.921521+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:37.570497+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.648721, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:37.570723+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:38.272971+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.70211, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:38.273169+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:39.042801+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.769467, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:39.042991+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:39.689869+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.646747, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:39.690092+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:40.444263+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.754041, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:40.444443+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:42.345630+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.900915, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:42.346431+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:43.539664+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 1.192842, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:43.539909+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:44.821839+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 1.28181, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:44.822013+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:45.973700+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 1.151454, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:45.974017+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:47.109042+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 1.134884, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:47.110040+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:48.966035+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.855847, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:48.966220+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:49.828846+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.862338, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:49.834812+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 1.0, "status": "passed", "valid": true, "control": "reference"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:49.839779+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "4a2182c4aeb53fccd91851d943a3f6fb55cf27d03941012cbd23942df81df51e", "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:49.839895+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:50.389926+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "failed", "duration_seconds": 0.549895, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:50.390126+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:50.974735+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "failed", "duration_seconds": 0.584373, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:50.975019+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:51.560054+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "failed", "duration_seconds": 0.584876, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:51.560266+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:52.138908+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "failed", "duration_seconds": 0.578509, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:52.139075+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:52.687076+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "failed", "duration_seconds": 0.547386, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:52.687292+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:53.219214+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "failed", "duration_seconds": 0.53177, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:53.219435+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:53.769400+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "failed", "duration_seconds": 0.549798, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:53.769626+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:54.329133+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "failed", "duration_seconds": 0.559346, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:54.329314+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:55.381423+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "failed", "duration_seconds": 1.051951, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:55.381628+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:56.308619+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "failed", "duration_seconds": 0.926142, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:56.308805+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:57.211939+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "failed", "duration_seconds": 0.903025, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:57.212121+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:58.192718+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "failed", "duration_seconds": 0.980445, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:58.192940+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:58.720057+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "failed", "duration_seconds": 0.526977, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:53:58.720294+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.201494+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "failed", "duration_seconds": 1.481053, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.201723+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.653492+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "failed", "duration_seconds": 0.451502, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.655291+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.0, "status": "failed", "valid": true, "control": "ack-without-work"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.657084+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "65617a32e2dec95ea2ce1457efae7d1dbe6e26094a6adc5c67514f333813bc23", "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:00.657182+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:01.123353+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.46604, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:01.123587+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:01.598094+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.474375, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:01.598270+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.074082+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.475647, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.074259+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.532979+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.458584, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.533198+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.998012+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.464691, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:02.998205+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:03.454184+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.455858, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:03.454375+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:03.912145+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.457636, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:03.912350+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:04.366534+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.454052, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:04.366834+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:05.295482+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "failed", "duration_seconds": 0.928483, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:05.296486+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:06.254843+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "failed", "duration_seconds": 0.958207, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:06.255084+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:07.193209+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "failed", "duration_seconds": 0.937998, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:07.193387+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:08.169364+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "failed", "duration_seconds": 0.975821, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:08.169572+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:09.094274+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "failed", "duration_seconds": 0.924555, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:09.094464+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.484214+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.389628, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.484400+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.970946+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.483543, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.973088+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.78333333, "status": "failed", "valid": true, "control": "memory-only"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.974980+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "25c9faee716ab4c4d81dea864df1e8d8560d01110e153cd84c199ad6ffb0ddb3", "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:10.976846+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:11.438463+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.461501, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:11.439028+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:11.969627+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.530414, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:11.969901+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:12.440415+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.470365, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:12.440680+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:12.932608+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.491789, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:12.933479+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:13.412759+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.479108, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:13.412999+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:13.885522+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "failed", "duration_seconds": 0.472393, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:13.885755+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:14.378002+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.492097, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:14.378248+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:14.876143+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.497769, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:14.876330+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:16.302926+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.426448, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:16.303128+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:17.232417+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 0.929136, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:17.232639+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:18.220632+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 0.987817, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:18.222742+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:19.146016+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 0.923095, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:19.146252+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:20.116253+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 0.96986, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:20.116435+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:21.590928+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.474352, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:21.591164+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.310389+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.718944, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.313116+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.9, "status": "failed", "valid": true, "control": "partial-batch"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.315079+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "6ab2bd9883cf0eff59df9198d55c7934827b6fb539453f6b7eb17a9db113a8b8", "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.315187+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.886146+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.570831, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:22.886333+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:23.470086+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.583597, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:23.470288+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:24.018021+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.547586, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:24.018265+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:24.540495+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.522098, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:24.540765+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:25.049665+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.508744, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:25.049883+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:25.544049+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.494022, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:25.544272+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:26.034611+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "failed", "duration_seconds": 0.490188, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:26.034860+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:26.565238+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "failed", "duration_seconds": 0.530264, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:26.565424+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:27.889339+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.323756, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:27.889607+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:28.730915+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 0.841123, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:28.731160+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:29.575017+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 0.84372, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:29.575225+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:30.423571+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 0.848225, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:30.423781+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:31.250438+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 0.826512, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:31.250712+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:32.543581+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.292735, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:32.543797+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.186080+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.642089, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.188394+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.85, "status": "failed", "valid": true, "control": "cas-always-wins"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.190452+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "202d2f3820eb71874cffb461ddc9f695af43ed3d4126aac7784475a97f1b6bb5", "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.190571+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.651303+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.460583, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:33.651549+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.125140+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.473433, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.125388+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.559497+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.433965, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.559767+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.955981+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.396075, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:34.956209+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:35.412043+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.455688, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:35.412301+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:35.845343+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.432909, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:35.845541+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:36.295811+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.450149, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:36.296010+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:36.731375+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "failed", "duration_seconds": 0.435249, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:36.731574+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:38.000216+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.268524, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:38.000473+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:38.866291+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 0.865693, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:38.866489+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:39.951430+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 1.084807, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:39.951666+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:40.933852+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 0.982003, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:40.934111+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:41.912219+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 0.977971, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:41.912411+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:43.340175+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.427617, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:43.340426+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.051149+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.710429, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.053945+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.925, "status": "failed", "valid": true, "control": "boolean-equals-one"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.056285+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "0500d54aca41a5403568b3950be43efd2ecadbf09c2f003043c7a0a7324f2807", "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.056406+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.541376+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "failed", "duration_seconds": 0.484824, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:44.541588+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:45.085082+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.543204, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:45.085331+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:45.641463+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.555981, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:45.641763+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.117516+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.475617, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.117740+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.553379+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "failed", "duration_seconds": 0.435511, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.553589+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.987425+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.433667, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:46.987691+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:47.437299+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.449461, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:47.437527+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:47.872806+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.435147, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:47.873076+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:49.146780+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "failed", "duration_seconds": 1.273556, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:49.146978+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:50.172122+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 1.025015, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:50.172322+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:51.106638+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 0.934191, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:51.106902+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:51.959979+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 0.852946, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:51.960169+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:53.010455+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 1.050153, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:53.011264+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:54.506611+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "failed", "duration_seconds": 1.495149, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:54.506874+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.077076+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "failed", "duration_seconds": 0.569837, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.080500+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.65, "status": "failed", "valid": true, "control": "delete-noop"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.082745+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "e17d1eeddba4a649c867c98b683de7ce64c4cbd5da448e8e29723569c8c0de5f", "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.082935+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.588137+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.505065, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:55.588381+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:56.116578+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.527997, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:56.116843+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:56.640411+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "failed", "duration_seconds": 0.523412, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:56.640682+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:57.145738+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.504849, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:57.145982+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:57.655503+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.509386, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:57.655746+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:58.170054+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.51418, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:58.170245+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:58.707909+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.537505, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:58.708172+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:59.210797+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.502504, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:54:59.210999+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:00.652663+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.441501, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:00.652893+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:01.628867+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 0.975848, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:01.629055+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:02.645389+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 1.016199, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:02.645634+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:03.721046+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 1.075213, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:03.721286+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:04.854525+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 1.133081, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:04.854764+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:06.758895+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.903977, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:06.759086+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:07.872015+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 1.11267, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:07.878365+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.925, "status": "failed", "valid": true, "control": "accept-nonstring-keys"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:07.884031+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "884f21ad792983fc73e53f6b55ba9531b003b4cbaf51b362690b1c17431981e7", "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:07.884148+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:08.633872+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.749596, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:08.634054+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:09.304107+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.669885, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:09.304303+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:09.986740+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.682262, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:09.986957+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:10.787223+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.800129, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:10.787396+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:11.482076+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.694564, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:11.482221+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:12.127857+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.645486, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:12.128076+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:13.020153+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.891929, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:13.021119+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:13.695755+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.673617, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:13.695945+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:15.700161+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 2.004075, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:15.700370+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:16.884854+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 1.184324, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:16.886098+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.016707+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "failed", "duration_seconds": 1.130183, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.017350+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:19.106977+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "failed", "duration_seconds": 1.0895, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:19.107153+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.192469+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 1.085192, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.192701+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:21.833393+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "failed", "duration_seconds": 1.640536, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:21.833595+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.414891+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.581022, "control": "commit-on-exit"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.421425+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 0.9, "status": "failed", "valid": true, "control": "commit-on-exit"}
diff --git a/examples/javascript-audits/durable-kv/index.html b/examples/javascript-audits/durable-kv/index.html
new file mode 100644
index 0000000..4c0d6fc
--- /dev/null
+++ b/examples/javascript-audits/durable-kv/index.html
@@ -0,0 +1,37 @@
+
+EvalArc · Grader audit
+
+EVALARC / DEVELOPMENT AUDIT
+Test the grader.
+Behavioral controls for task-specific agent evaluations. Correct and faulty
+submissions are evaluated against the same externally enforced contract.
+8/8
declared faults detected
1.000
reference correctness
Does the grader detect plausible defects? Positive control basic 1.00 3/3
validation 1.00 2/2
transactions 1.00 2/2
compare_swap 1.00 2/2
persistence 1.00 3/3
crash_recovery 1.00 3/3
Reproduction record Task: durable-kv v0.1.0
Complete JSON evidence for all controls
Scope: This is a scripted grader audit, not an AI leaderboard. Public seeds are not held-out evaluation data. Detection rates apply only to the listed fault models. No human time horizon, model quality, or RL improvement is inferred.EvalArc · Grader audit
+
+EVALARC / DEVELOPMENT AUDIT
+Test the grader.
+Behavioral controls for task-specific agent evaluations. Correct and faulty
+submissions are evaluated against the same externally enforced contract.
+7/7
declared faults detected
1.000
reference correctness
Does the grader detect plausible defects? Positive control routing 1.00 4/4
notes 1.00 4/4
closure 1.00 4/4
scope 1.00 4/4
protocol 1.00 4/4
Reference execution evidence route-open-ticket · seed 17 · passed {
+ "seed": 17,
+ "case_id": "route-open-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.57553,
+ "transcript_sha256": "7b5b4973e934a60c55b95bda6eedf79fcff62764fbc1b81723c5126608caf4bd",
+ "initial_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "queue": "queue-0-2"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "body": "Reviewed request T-85a0bcc1.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 3,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 279,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} close-resolved-ticket · seed 17 · passed {
+ "seed": 17,
+ "case_id": "close-resolved-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.600284,
+ "transcript_sha256": "a260768c11b589ce3f58658cac6aae8a01d36e388a47fc47c7995f8d5c36708e",
+ "initial_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "queue": "queue-1-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "body": "Reviewed request T-c41f9dfd.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 342,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} retry-before-commit · seed 17 · passed {
+ "seed": 17,
+ "case_id": "retry-before-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.581408,
+ "transcript_sha256": "b1d1d91dc01fc4bfb3243aa3739e953a40aa3920caf63a4b2ab73c87ecabda66",
+ "initial_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-3fb81d27"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {}
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "body": "Reviewed request T-3fb81d27.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 362,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} retry-after-commit · seed 17 · passed {
+ "seed": 17,
+ "case_id": "retry-after-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.595072,
+ "transcript_sha256": "c97b016921c8d8fdcdeffd1ebf1f438d5a29be06732c31c86f99fd087bc0b58f",
+ "initial_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "queue": "queue-3-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true,
+ "replayed": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 4,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 5,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 5,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 461,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} Reproduction record Task: support-routing v0.1.0
Complete JSON evidence for all controls
Scope: This is a scripted grader audit, not an AI leaderboard. Public seeds are not held-out evaluation data. Detection rates apply only to the listed fault models. No human time horizon, model quality, or RL improvement is inferred.EvalArc · Suite EVALARC / SUITE
Every job, explicit criteria. Python and JavaScript reference conformance · PASSED
3
jobs with every attempt resolved
Each job has its own task, budget, and acceptance rules. Scores stay within their task; there is no cross-domain average.
Job decisions durable-kv · coding · Gate: ACCEPTED
Mean assessed score: 1. Resolved attempts: 1/1. Completed: 1. Invalid: 0.
Acceptance rules and checks {
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ }
+} durable-kv · coding · Gate: ACCEPTED
Mean assessed score: 1. Resolved attempts: 1/1. Completed: 1. Invalid: 0.
Acceptance rules and checks {
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ }
+} support-routing · business-tools · Gate: ACCEPTED
Mean assessed score: 1. Resolved attempts: 1/1. Completed: 1. Invalid: 0.
Acceptance rules and checks {
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ }
+} Run record Manifest SHA-256: ae478f5b926dab4eee42bb1b4eb0c3a87446da0d09f5b27f8ebd273199802478
Suite JSON · Original configuration · Resolved plan · JUnit acceptance gates · Progress events
Acceptance follows each job's declared gate. No score is averaged across jobs or domains. Gate acceptance does not imply full task resolution. Repeated observations use fixed public cases and are descriptive only. JUnit contains one test per job gate, rather than one test per individual task case.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/evaluation.json b/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/evaluation.json
new file mode 100644
index 0000000..2ccb26f
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/evaluation.json
@@ -0,0 +1,463 @@
+{
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:55:42.533193+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.500171,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.563726,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.5038,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.510414,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.482119,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.456643,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.515152,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.488902,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.38184,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.932079,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 0.926056,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.933188,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.937424,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.418463,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.704395,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 11.266005,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+}
diff --git a/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/index.html b/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/index.html
new file mode 100644
index 0000000..6227ae5
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-coding/attempts/0001/index.html
@@ -0,0 +1,387 @@
+EvalArc · Evaluation EVALARC / EVALUATION
durable-kv coding · task v0.1.0 · RESOLVED
Dimension results Case results Case evidence read-write-delete · seed 17 · passed {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.500171,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 171,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} json-types-and-overwrite · seed 17 · passed {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.563726,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4531,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} reject-and-continue · seed 17 · passed {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.5038,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 407,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} empty-and-unicode-keys · seed 17 · passed {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.510414,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 190,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} commit-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.482119,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 86,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} rollback-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.456643,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 321,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} cas-match-and-mismatch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.515152,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 240,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} cas-type-sensitivity · seed 17 · passed {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.488902,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 143,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-acknowledged-write · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.38184,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 95,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-transaction · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.932079,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-acknowledgement · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 0.926056,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 83,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-atomic-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.933188,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 166,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-cas · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 0.937424,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 39,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 37,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-delete · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.418463,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 12,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 26,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} seeded-state-machine · seed 17 · passed {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.704395,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1101,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} Reproduction record Candidate SHA-256: ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7
Complete JSON evidence
Observed task outcomes only. Full resolution requires every check to pass. Host durations include execution overhead and are not agent completion-time baselines. Unknown model cost and token usage remain null.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/javascript-coding/events.jsonl b/examples/multilanguage/run/jobs/javascript-coding/events.jsonl
new file mode 100644
index 0000000..461d621
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-coding/events.jsonl
@@ -0,0 +1,36 @@
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.267282+00:00", "event": "repetition_started", "task": "durable-kv", "attempts": 1, "candidate_sha256": "ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.267454+00:00", "event": "attempt_started", "attempt": 1, "total_attempts": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.269382+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7", "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.269521+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.769837+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.500171, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.770048+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:32.334736+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.563726, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:32.335029+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:32.839059+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.5038, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:32.839333+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:33.349905+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.510414, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:33.350136+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:33.832405+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.482119, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:33.832623+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:34.289459+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.456643, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:34.289756+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:34.805088+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.515152, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:34.805422+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:35.294494+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.488902, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:35.294819+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:36.676848+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.38184, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:36.677110+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:37.609364+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 0.932079, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:37.609664+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:38.535870+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 0.926056, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:38.536063+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:39.469424+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 0.933188, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:39.469684+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:40.407252+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 0.937424, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:40.407430+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:41.826068+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.418463, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:41.826334+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.531042+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.704395, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.533506+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 1.0, "status": "passed", "valid": true, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.535766+00:00", "event": "attempt_completed", "attempt": 1, "total_attempts": 1, "valid": true, "status": "passed", "score": 1.0}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.536378+00:00", "event": "repetition_completed", "status": "passed", "completed_attempts": 1, "requested_attempts": 1}
diff --git a/examples/multilanguage/run/jobs/javascript-coding/index.html b/examples/multilanguage/run/jobs/javascript-coding/index.html
new file mode 100644
index 0000000..3e6e84f
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-coding/index.html
@@ -0,0 +1,24 @@
+EvalArc · Repeatability EVALARC / REPEATABILITY
See every attempt. durable-kv · PASSED · one frozen candidate, fixed cases, fresh state per attempt
1/1
requested attempts resolved
1
mean score of assessed attempts
0
checks with varying outcomes
Requested: 1. Completed: 1. Assessed: 1. Invalid: 0. Assessed score range: 1–1.
Outcomes by case cas-match-and-mismatch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates compare_swap: 1/1 (100.0%) cas-type-sensitivity · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates compare_swap: 1/1 (100.0%) commit-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates transactions: 1/1 (100.0%) empty-and-unicode-keys · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates json-types-and-overwrite · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates read-write-delete · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates reject-and-continue · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates restart-preserves-acknowledged-write · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) restart-preserves-cas · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) restart-preserves-transaction · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) rollback-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates transactions: 1/1 (100.0%) seeded-state-machine · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates sigkill-after-acknowledgement · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) sigkill-after-atomic-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) sigkill-after-delete · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) Every attempt Reproduction record Candidate SHA-256: ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7
Repetition summary JSON · Progress events
Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed. Case and check denominators count their assessed observations, including those in an otherwise invalid attempt. Repeated success on these cases does not establish general agent reliability.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/javascript-coding/repetition.json b/examples/multilanguage/run/jobs/javascript-coding/repetition.json
new file mode 100644
index 0000000..3ac77d4
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-coding/repetition.json
@@ -0,0 +1,330 @@
+{
+ "schema_version": "evalarc.repetition.v1",
+ "evalarc_version": "0.6.0",
+ "created_at": "2026-09-14T04:55:42.536351+00:00",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "complete": true,
+ "valid": true,
+ "status": "passed",
+ "all_attempts_resolved": true,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "assessed_resolution_rate": 1.0,
+ "mean_score": 1.0,
+ "min_score": 1.0,
+ "max_score": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0,
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "compare_swap": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "compare_swap": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "transactions": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "validation": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "validation": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "transactions": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ }
+ ],
+ "attempts": [
+ {
+ "attempt": 1,
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "created_at": "2026-09-14T04:55:42.533193+00:00"
+ }
+ ],
+ "interpretation": "Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed."
+}
diff --git a/examples/multilanguage/run/jobs/javascript-support/attempts/0001/evaluation.json b/examples/multilanguage/run/jobs/javascript-support/attempts/0001/evaluation.json
new file mode 100644
index 0000000..b9afd41
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-support/attempts/0001/evaluation.json
@@ -0,0 +1,911 @@
+{
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:55:44.411869+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "support-routing",
+ "version": "0.1.0",
+ "domain": "business-tools",
+ "split": "public-development"
+ },
+ "candidate_sha256": "2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143",
+ "grader_sha256": "887aea97efec396118521f720f6d8753be9ae964ab92eaa655d6575185326593",
+ "cases_sha256": "b7cd8663c69da9e1715d5c8e1d707514863dcc333fefc57acc3315053ebc85df",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "dimensions": {
+ "routing": {
+ "passed": 4,
+ "total": 4,
+ "assessed": 4,
+ "weight": 0.3,
+ "score": 1.0
+ },
+ "notes": {
+ "passed": 4,
+ "total": 4,
+ "assessed": 4,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "closure": {
+ "passed": 4,
+ "total": 4,
+ "assessed": 4,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "scope": {
+ "passed": 4,
+ "total": 4,
+ "assessed": 4,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "protocol": {
+ "passed": 4,
+ "total": 4,
+ "assessed": 4,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "route-open-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.480493,
+ "transcript_sha256": "7b5b4973e934a60c55b95bda6eedf79fcff62764fbc1b81723c5126608caf4bd",
+ "initial_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "queue": "queue-0-2"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "body": "Reviewed request T-85a0bcc1.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 3,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 279,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "close-resolved-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.450855,
+ "transcript_sha256": "a260768c11b589ce3f58658cac6aae8a01d36e388a47fc47c7995f8d5c36708e",
+ "initial_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "queue": "queue-1-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "body": "Reviewed request T-c41f9dfd.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 342,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "retry-before-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.473071,
+ "transcript_sha256": "b1d1d91dc01fc4bfb3243aa3739e953a40aa3920caf63a4b2ab73c87ecabda66",
+ "initial_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-3fb81d27"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {}
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "body": "Reviewed request T-3fb81d27.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 362,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "retry-after-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.463068,
+ "transcript_sha256": "c97b016921c8d8fdcdeffd1ebf1f438d5a29be06732c31c86f99fd087bc0b58f",
+ "initial_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "queue": "queue-3-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true,
+ "replayed": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 4,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 5,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 5,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 461,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 1.872363,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+}
diff --git a/examples/multilanguage/run/jobs/javascript-support/attempts/0001/index.html b/examples/multilanguage/run/jobs/javascript-support/attempts/0001/index.html
new file mode 100644
index 0000000..4190c8f
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-support/attempts/0001/index.html
@@ -0,0 +1,854 @@
+EvalArc · Evaluation EVALARC / EVALUATION
support-routing business-tools · task v0.1.0 · RESOLVED
Dimension results Case results Case evidence route-open-ticket · seed 17 · passed {
+ "seed": 17,
+ "case_id": "route-open-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.480493,
+ "transcript_sha256": "7b5b4973e934a60c55b95bda6eedf79fcff62764fbc1b81723c5126608caf4bd",
+ "initial_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-85a0bcc1": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ },
+ "U-6a06e9ab": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "queue": "queue-0-2"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-85a0bcc1",
+ "body": "Reviewed request T-85a0bcc1.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-85a0bcc1": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-0-2",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-85a0bcc1."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 3,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 279,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} close-resolved-ticket · seed 17 · passed {
+ "seed": 17,
+ "case_id": "close-resolved-ticket",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.450855,
+ "transcript_sha256": "a260768c11b589ce3f58658cac6aae8a01d36e388a47fc47c7995f8d5c36708e",
+ "initial_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-c41f9dfd": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "U-b48438b5": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "queue": "queue-1-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd",
+ "body": "Reviewed request T-c41f9dfd.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-c41f9dfd"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-c41f9dfd": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-1-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-c41f9dfd."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 342,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} retry-before-commit · seed 17 · passed {
+ "seed": 17,
+ "case_id": "retry-before-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.473071,
+ "transcript_sha256": "b1d1d91dc01fc4bfb3243aa3739e953a40aa3920caf63a4b2ab73c87ecabda66",
+ "initial_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-3fb81d27": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ },
+ "U-62590992": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-3fb81d27"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {}
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "queue": "queue-2-1"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-3fb81d27",
+ "body": "Reviewed request T-3fb81d27.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-3fb81d27": {
+ "before": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "technical",
+ "queue": "queue-2-1",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-3fb81d27."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 4,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 4,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 362,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} retry-after-commit · seed 17 · passed {
+ "seed": 17,
+ "case_id": "retry-after-commit",
+ "checks": {
+ "routing": true,
+ "notes": true,
+ "closure": true,
+ "scope": true,
+ "protocol": true
+ },
+ "passed": true,
+ "status": "passed",
+ "error": null,
+ "duration_seconds": 0.463068,
+ "transcript_sha256": "c97b016921c8d8fdcdeffd1ebf1f438d5a29be06732c31c86f99fd087bc0b58f",
+ "initial_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "final_state": {
+ "T-cd68615c": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "U-51436d1f": {
+ "category": "account",
+ "queue": "inbox",
+ "resolved": false,
+ "status": "open",
+ "notes": [
+ "Unrelated request; preserve this ticket."
+ ]
+ }
+ },
+ "trace": [
+ {
+ "step": 0,
+ "type": "tool_call",
+ "action": {
+ "tool": "get_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true,
+ "ticket": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 1,
+ "type": "tool_call",
+ "action": {
+ "tool": "assign_queue",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "queue": "queue-3-0"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "inbox",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 2,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": false,
+ "error": {
+ "code": "temporarily_unavailable",
+ "retryable": true
+ }
+ },
+ "status": "tool_error",
+ "error_source": "injected",
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 3,
+ "type": "tool_call",
+ "action": {
+ "tool": "add_note",
+ "arguments": {
+ "ticket_id": "T-cd68615c",
+ "body": "Reviewed request T-cd68615c.",
+ "request_id": "review"
+ }
+ },
+ "result": {
+ "ok": true,
+ "replayed": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {}
+ },
+ {
+ "step": 4,
+ "type": "tool_call",
+ "action": {
+ "tool": "close_ticket",
+ "arguments": {
+ "ticket_id": "T-cd68615c"
+ }
+ },
+ "result": {
+ "ok": true
+ },
+ "status": "ok",
+ "error_source": null,
+ "changes": {
+ "T-cd68615c": {
+ "before": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "open",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ },
+ "after": {
+ "category": "billing",
+ "queue": "queue-3-0",
+ "resolved": true,
+ "status": "closed",
+ "notes": [
+ "Customer contacted support.",
+ "Reviewed request T-cd68615c."
+ ]
+ }
+ }
+ }
+ },
+ {
+ "step": 5,
+ "type": "finish",
+ "action": {
+ "finish": true
+ }
+ }
+ ],
+ "tool_calls": 5,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 461,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} Reproduction record Candidate SHA-256: 2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143
Complete JSON evidence
Observed task outcomes only. Full resolution requires every check to pass. Host durations include execution overhead and are not agent completion-time baselines. Unknown model cost and token usage remain null.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/javascript-support/events.jsonl b/examples/multilanguage/run/jobs/javascript-support/events.jsonl
new file mode 100644
index 0000000..7217735
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-support/events.jsonl
@@ -0,0 +1,14 @@
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.539481+00:00", "event": "repetition_started", "task": "support-routing", "attempts": 1, "candidate_sha256": "2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.539539+00:00", "event": "attempt_started", "attempt": 1, "total_attempts": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.541537+00:00", "event": "evaluation_started", "task": "support-routing", "candidate_sha256": "2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143", "total_cases": 4, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:42.541680+00:00", "event": "case_started", "task": "support-routing", "seed": 17, "case_id": "route-open-ticket", "case_number": 1, "total_cases": 4, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.022387+00:00", "event": "case_completed", "task": "support-routing", "seed": 17, "case_id": "route-open-ticket", "case_number": 1, "total_cases": 4, "status": "passed", "duration_seconds": 0.480493, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.022961+00:00", "event": "case_started", "task": "support-routing", "seed": 17, "case_id": "close-resolved-ticket", "case_number": 2, "total_cases": 4, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.474130+00:00", "event": "case_completed", "task": "support-routing", "seed": 17, "case_id": "close-resolved-ticket", "case_number": 2, "total_cases": 4, "status": "passed", "duration_seconds": 0.450855, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.474371+00:00", "event": "case_started", "task": "support-routing", "seed": 17, "case_id": "retry-before-commit", "case_number": 3, "total_cases": 4, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.947625+00:00", "event": "case_completed", "task": "support-routing", "seed": 17, "case_id": "retry-before-commit", "case_number": 3, "total_cases": 4, "status": "passed", "duration_seconds": 0.473071, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:43.947871+00:00", "event": "case_started", "task": "support-routing", "seed": 17, "case_id": "retry-after-commit", "case_number": 4, "total_cases": 4, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:44.411174+00:00", "event": "case_completed", "task": "support-routing", "seed": 17, "case_id": "retry-after-commit", "case_number": 4, "total_cases": 4, "status": "passed", "duration_seconds": 0.463068, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:44.411945+00:00", "event": "evaluation_completed", "task": "support-routing", "score": 1.0, "status": "passed", "valid": true, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:44.415189+00:00", "event": "attempt_completed", "attempt": 1, "total_attempts": 1, "valid": true, "status": "passed", "score": 1.0}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:44.415751+00:00", "event": "repetition_completed", "status": "passed", "completed_attempts": 1, "requested_attempts": 1}
diff --git a/examples/multilanguage/run/jobs/javascript-support/index.html b/examples/multilanguage/run/jobs/javascript-support/index.html
new file mode 100644
index 0000000..a64d1f3
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-support/index.html
@@ -0,0 +1,24 @@
+EvalArc · Repeatability EVALARC / REPEATABILITY
See every attempt. support-routing · PASSED · one frozen candidate, fixed cases, fresh state per attempt
1/1
requested attempts resolved
1
mean score of assessed attempts
0
checks with varying outcomes
Requested: 1. Completed: 1. Assessed: 1. Invalid: 0. Assessed score range: 1–1.
Outcomes by case close-resolved-ticket · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates routing: 1/1 (100.0%) notes: 1/1 (100.0%) closure: 1/1 (100.0%) scope: 1/1 (100.0%) protocol: 1/1 (100.0%) retry-after-commit · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates routing: 1/1 (100.0%) notes: 1/1 (100.0%) closure: 1/1 (100.0%) scope: 1/1 (100.0%) protocol: 1/1 (100.0%) retry-before-commit · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates routing: 1/1 (100.0%) notes: 1/1 (100.0%) closure: 1/1 (100.0%) scope: 1/1 (100.0%) protocol: 1/1 (100.0%) route-open-ticket · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates routing: 1/1 (100.0%) notes: 1/1 (100.0%) closure: 1/1 (100.0%) scope: 1/1 (100.0%) protocol: 1/1 (100.0%) Every attempt Reproduction record Candidate SHA-256: 2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143
Repetition summary JSON · Progress events
Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed. Case and check denominators count their assessed observations, including those in an otherwise invalid attempt. Repeated success on these cases does not establish general agent reliability.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/javascript-support/repetition.json b/examples/multilanguage/run/jobs/javascript-support/repetition.json
new file mode 100644
index 0000000..fb4c0bd
--- /dev/null
+++ b/examples/multilanguage/run/jobs/javascript-support/repetition.json
@@ -0,0 +1,227 @@
+{
+ "schema_version": "evalarc.repetition.v1",
+ "evalarc_version": "0.6.0",
+ "created_at": "2026-09-14T04:55:44.415724+00:00",
+ "task": {
+ "id": "support-routing",
+ "version": "0.1.0",
+ "domain": "business-tools",
+ "split": "public-development"
+ },
+ "candidate_sha256": "2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143",
+ "grader_sha256": "887aea97efec396118521f720f6d8753be9ae964ab92eaa655d6575185326593",
+ "cases_sha256": "b7cd8663c69da9e1715d5c8e1d707514863dcc333fefc57acc3315053ebc85df",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "complete": true,
+ "valid": true,
+ "status": "passed",
+ "all_attempts_resolved": true,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "assessed_resolution_rate": 1.0,
+ "mean_score": 1.0,
+ "min_score": 1.0,
+ "max_score": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0,
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "close-resolved-ticket",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "routing": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "notes": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "closure": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "scope": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "protocol": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "retry-after-commit",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "routing": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "notes": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "closure": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "scope": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "protocol": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "retry-before-commit",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "routing": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "notes": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "closure": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "scope": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "protocol": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "route-open-ticket",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "routing": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "notes": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "closure": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "scope": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ },
+ "protocol": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ }
+ ],
+ "attempts": [
+ {
+ "attempt": 1,
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "created_at": "2026-09-14T04:55:44.411869+00:00"
+ }
+ ],
+ "interpretation": "Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed."
+}
diff --git a/examples/multilanguage/run/jobs/python-coding/attempts/0001/evaluation.json b/examples/multilanguage/run/jobs/python-coding/attempts/0001/evaluation.json
new file mode 100644
index 0000000..7f6101b
--- /dev/null
+++ b/examples/multilanguage/run/jobs/python-coding/attempts/0001/evaluation.json
@@ -0,0 +1,465 @@
+{
+ "schema_version": "evalarc.evaluation.v2",
+ "created_at": "2026-09-14T04:55:31.258719+00:00",
+ "evalarc_version": "0.6.0",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "python:3.12-slim",
+ "image_id": "sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea",
+ "command": [
+ "{python}",
+ "-I",
+ "-B",
+ "main.py",
+ "{state}/store.db"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "dimensions": {
+ "basic": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.25,
+ "score": 1.0
+ },
+ "validation": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "transactions": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.2,
+ "score": 1.0
+ },
+ "compare_swap": {
+ "passed": 2,
+ "total": 2,
+ "assessed": 2,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "persistence": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.15,
+ "score": 1.0
+ },
+ "crash_recovery": {
+ "passed": 3,
+ "total": 3,
+ "assessed": 3,
+ "weight": 0.1,
+ "score": 1.0
+ }
+ },
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.6496,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 200,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.71574,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4594,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.639067,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 452,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.592289,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 232,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.616036,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 96,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.656281,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 374,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.66439,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 278,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.689647,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 160,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.617219,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 116,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 29,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.136055,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 206,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 1.085292,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 103,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.155548,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 206,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.108908,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 43,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 42,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.632093,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 29,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.781928,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1250,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+ }
+ ],
+ "evaluation_seconds": 13.754527,
+ "agent_cost_usd": null,
+ "agent_tokens": null
+}
diff --git a/examples/multilanguage/run/jobs/python-coding/attempts/0001/index.html b/examples/multilanguage/run/jobs/python-coding/attempts/0001/index.html
new file mode 100644
index 0000000..da57cf6
--- /dev/null
+++ b/examples/multilanguage/run/jobs/python-coding/attempts/0001/index.html
@@ -0,0 +1,387 @@
+EvalArc · Evaluation EVALARC / EVALUATION
durable-kv coding · task v0.1.0 · RESOLVED
Dimension results Case results Case evidence read-write-delete · seed 17 · passed {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 6,
+ "duration_seconds": 0.6496,
+ "transcript_sha256": "c809eed26dc56583f9385d7d4adca2d981d06a672dc1ccbc0ffc52aa228b86f7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 200,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} json-types-and-overwrite · seed 17 · passed {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 16,
+ "duration_seconds": 0.71574,
+ "transcript_sha256": "83f6377f4b57ac659e8162136107ca644158ee089c944892aa9817085e74bd60",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 4594,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} reject-and-continue · seed 17 · passed {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 10,
+ "duration_seconds": 0.639067,
+ "transcript_sha256": "d4e0c26837f54e0af94d4909e3cab21d86dedca8c8842d0daae9e7471198c2a0",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 452,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} empty-and-unicode-keys · seed 17 · passed {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "dimension": "validation",
+ "checks": {
+ "validation": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.592289,
+ "transcript_sha256": "e8abe374fc308ce0744e96d2920495b96bda3b026cec9b86fa055023e1e4e620",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 232,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} commit-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 0.616036,
+ "transcript_sha256": "ac21f2f6bcc47e3c807fa4ed72300af0f5ef39809ba17f93423e37a0e177e718",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 96,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} rollback-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "dimension": "transactions",
+ "checks": {
+ "transactions": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.656281,
+ "transcript_sha256": "1786c093ba29dcad562531a44cdbe5fddbed9d3f4632c1c6ddfe736baa240f35",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 374,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} cas-match-and-mismatch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 7,
+ "duration_seconds": 0.66439,
+ "transcript_sha256": "ecd6148f6f387b4ef5351b877afd4cf27ea2a04c4253a34220fd78de5442d89f",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 278,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} cas-type-sensitivity · seed 17 · passed {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "dimension": "compare_swap",
+ "checks": {
+ "compare_swap": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 5,
+ "duration_seconds": 0.689647,
+ "transcript_sha256": "459a5b74588217d8abcf8e138eb559364a1d01d1f23e8c43ca0127ba28240258",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 160,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-acknowledged-write · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 4,
+ "duration_seconds": 1.617219,
+ "transcript_sha256": "c97b82c67f26a1aa2fdc8ac57f50690dbbe13554d094059ffe6690a592df3a55",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 116,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 29,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-transaction · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.136055,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 206,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-acknowledgement · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 2,
+ "duration_seconds": 1.085292,
+ "transcript_sha256": "544c6be916371ed5a0e901394c5e293733ca84e0ad20079a952f9c4b2821a728",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 103,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-atomic-batch · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.155548,
+ "transcript_sha256": "2d1f704df309d49f8e6e89e674581af2cf81aa22fdde633058e2f2d7b8c8a430",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 206,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} restart-preserves-cas · seed 17 · passed {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "dimension": "persistence",
+ "checks": {
+ "persistence": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.108908,
+ "transcript_sha256": "149c04fd1366e77603e631349740ae38bdff40aa69f6eb4c738979eb13bf3f49",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 43,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 42,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} sigkill-after-delete · seed 17 · passed {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "dimension": "crash_recovery",
+ "checks": {
+ "crash_recovery": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 3,
+ "duration_seconds": 1.632093,
+ "transcript_sha256": "b3dba7f28e6629bb22f49b0439e0f74580ca3263dd98e16061ac9a8f680b5df7",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 137,
+ "output_bytes": 13,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ },
+ {
+ "exit_code": 0,
+ "output_bytes": 29,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} seeded-state-machine · seed 17 · passed {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "dimension": "basic",
+ "checks": {
+ "basic": true
+ },
+ "passed": true,
+ "status": "passed",
+ "checks_completed": 52,
+ "duration_seconds": 0.781928,
+ "transcript_sha256": "2a7e5ad4bedce67faf4d53f289c3a1a7304345a0167b287418fdf7dd411030f8",
+ "error": null,
+ "processes": [
+ {
+ "exit_code": 0,
+ "output_bytes": 1250,
+ "stderr_tail": "",
+ "stderr_tail_limit_bytes": 2048
+ }
+ ]
+} Reproduction record Candidate SHA-256: 378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe
Complete JSON evidence
Observed task outcomes only. Full resolution requires every check to pass. Host durations include execution overhead and are not agent completion-time baselines. Unknown model cost and token usage remain null.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/python-coding/events.jsonl b/examples/multilanguage/run/jobs/python-coding/events.jsonl
new file mode 100644
index 0000000..4199511
--- /dev/null
+++ b/examples/multilanguage/run/jobs/python-coding/events.jsonl
@@ -0,0 +1,36 @@
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:17.506881+00:00", "event": "repetition_started", "task": "durable-kv", "attempts": 1, "candidate_sha256": "378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe"}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:17.506941+00:00", "event": "attempt_started", "attempt": 1, "total_attempts": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:17.508658+00:00", "event": "evaluation_started", "task": "durable-kv", "candidate_sha256": "378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe", "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:17.508790+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.158580+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "read-write-delete", "case_number": 1, "total_cases": 15, "status": "passed", "duration_seconds": 0.6496, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.159815+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.875784+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "json-types-and-overwrite", "case_number": 2, "total_cases": 15, "status": "passed", "duration_seconds": 0.71574, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:18.876003+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:19.515751+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "reject-and-continue", "case_number": 3, "total_cases": 15, "status": "passed", "duration_seconds": 0.639067, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:19.515983+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.108430+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "empty-and-unicode-keys", "case_number": 4, "total_cases": 15, "status": "passed", "duration_seconds": 0.592289, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.108669+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.724853+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "commit-batch", "case_number": 5, "total_cases": 15, "status": "passed", "duration_seconds": 0.616036, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:20.725065+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:21.381534+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "rollback-batch", "case_number": 6, "total_cases": 15, "status": "passed", "duration_seconds": 0.656281, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:21.381750+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.046276+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-match-and-mismatch", "case_number": 7, "total_cases": 15, "status": "passed", "duration_seconds": 0.66439, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.046491+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.736257+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "cas-type-sensitivity", "case_number": 8, "total_cases": 15, "status": "passed", "duration_seconds": 0.689647, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:22.736435+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:24.353808+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-acknowledged-write", "case_number": 9, "total_cases": 15, "status": "passed", "duration_seconds": 1.617219, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:24.354012+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:25.490242+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-transaction", "case_number": 10, "total_cases": 15, "status": "passed", "duration_seconds": 1.136055, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:25.490474+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:26.575916+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-acknowledgement", "case_number": 11, "total_cases": 15, "status": "passed", "duration_seconds": 1.085292, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:26.576165+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:27.731879+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-atomic-batch", "case_number": 12, "total_cases": 15, "status": "passed", "duration_seconds": 1.155548, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:27.732098+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:28.841169+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "restart-preserves-cas", "case_number": 13, "total_cases": 15, "status": "passed", "duration_seconds": 1.108908, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:28.841420+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:30.473708+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "sigkill-after-delete", "case_number": 14, "total_cases": 15, "status": "passed", "duration_seconds": 1.632093, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:30.473944+00:00", "event": "case_started", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.256322+00:00", "event": "case_completed", "task": "durable-kv", "seed": 17, "case_id": "seeded-state-machine", "case_number": 15, "total_cases": 15, "status": "passed", "duration_seconds": 0.781928, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.261516+00:00", "event": "evaluation_completed", "task": "durable-kv", "score": 1.0, "status": "passed", "valid": true, "attempt": 1}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.264111+00:00", "event": "attempt_completed", "attempt": 1, "total_attempts": 1, "valid": true, "status": "passed", "score": 1.0}
+{"schema_version": "evalarc.event.v1", "created_at": "2026-09-14T04:55:31.264690+00:00", "event": "repetition_completed", "status": "passed", "completed_attempts": 1, "requested_attempts": 1}
diff --git a/examples/multilanguage/run/jobs/python-coding/index.html b/examples/multilanguage/run/jobs/python-coding/index.html
new file mode 100644
index 0000000..34ad30c
--- /dev/null
+++ b/examples/multilanguage/run/jobs/python-coding/index.html
@@ -0,0 +1,24 @@
+EvalArc · Repeatability EVALARC / REPEATABILITY
See every attempt. durable-kv · PASSED · one frozen candidate, fixed cases, fresh state per attempt
1/1
requested attempts resolved
1
mean score of assessed attempts
0
checks with varying outcomes
Requested: 1. Completed: 1. Assessed: 1. Invalid: 0. Assessed score range: 1–1.
Outcomes by case cas-match-and-mismatch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates compare_swap: 1/1 (100.0%) cas-type-sensitivity · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates compare_swap: 1/1 (100.0%) commit-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates transactions: 1/1 (100.0%) empty-and-unicode-keys · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates json-types-and-overwrite · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates read-write-delete · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates reject-and-continue · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates restart-preserves-acknowledged-write · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) restart-preserves-cas · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) restart-preserves-transaction · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates persistence: 1/1 (100.0%) rollback-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates transactions: 1/1 (100.0%) seeded-state-machine · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates sigkill-after-acknowledgement · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) sigkill-after-atomic-batch · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) sigkill-after-delete · seed 17 1/1 assessed attempts passed · 0 unassessed
Check pass rates crash_recovery: 1/1 (100.0%) Every attempt Reproduction record Candidate SHA-256: 378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe
Repetition summary JSON · Progress events
Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed. Case and check denominators count their assessed observations, including those in an otherwise invalid attempt. Repeated success on these cases does not establish general agent reliability.
\ No newline at end of file
diff --git a/examples/multilanguage/run/jobs/python-coding/repetition.json b/examples/multilanguage/run/jobs/python-coding/repetition.json
new file mode 100644
index 0000000..cf3404b
--- /dev/null
+++ b/examples/multilanguage/run/jobs/python-coding/repetition.json
@@ -0,0 +1,332 @@
+{
+ "schema_version": "evalarc.repetition.v1",
+ "evalarc_version": "0.6.0",
+ "created_at": "2026-09-14T04:55:31.264634+00:00",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "python:3.12-slim",
+ "image_id": "sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea",
+ "command": [
+ "{python}",
+ "-I",
+ "-B",
+ "main.py",
+ "{state}/store.db"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "complete": true,
+ "valid": true,
+ "status": "passed",
+ "all_attempts_resolved": true,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "assessed_resolution_rate": 1.0,
+ "mean_score": 1.0,
+ "min_score": 1.0,
+ "max_score": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0,
+ "cases": [
+ {
+ "seed": 17,
+ "case_id": "cas-match-and-mismatch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "compare_swap": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "cas-type-sensitivity",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "compare_swap": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "commit-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "transactions": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "empty-and-unicode-keys",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "validation": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "json-types-and-overwrite",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "read-write-delete",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "reject-and-continue",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "validation": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-acknowledged-write",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-cas",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "restart-preserves-transaction",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "persistence": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "rollback-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "transactions": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "seeded-state-machine",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "basic": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-acknowledgement",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-atomic-batch",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ },
+ {
+ "seed": 17,
+ "case_id": "sigkill-after-delete",
+ "passed": 1,
+ "assessed": 1,
+ "failed": 0,
+ "unassessed": 0,
+ "pass_rate": 1.0,
+ "variable": false,
+ "checks": {
+ "crash_recovery": {
+ "passed": 1,
+ "assessed": 1,
+ "pass_rate": 1.0,
+ "variable": false
+ }
+ }
+ }
+ ],
+ "attempts": [
+ {
+ "attempt": 1,
+ "valid": true,
+ "status": "passed",
+ "score": 1.0,
+ "resolved": true,
+ "created_at": "2026-09-14T04:55:31.258719+00:00"
+ }
+ ],
+ "interpretation": "Descriptive outcomes on the same public cases. Invalid attempts are explicit; mean scores and resolution rates use assessed attempts only. No independence assumption, confidence interval, or population success rate is claimed."
+}
diff --git a/examples/multilanguage/run/junit.xml b/examples/multilanguage/run/junit.xml
new file mode 100644
index 0000000..30efc89
--- /dev/null
+++ b/examples/multilanguage/run/junit.xml
@@ -0,0 +1,14 @@
+
+
+
+
+ {"fully_resolved": true, "observed": {"requested_attempts": 1, "completed_attempts": 1, "assessed_attempts": 1, "invalid_attempts": 0, "resolved_attempts": 1, "mean_score": 1.0, "assessed_resolution_rate": 1.0, "variable_cases": 0, "variable_checks": 0}, "gate": {"min_mean_score": 1.0, "min_resolution_rate": 1.0, "required_dimensions": []}, "evidence": "jobs/python-coding/index.html"}
+
+
+ {"fully_resolved": true, "observed": {"requested_attempts": 1, "completed_attempts": 1, "assessed_attempts": 1, "invalid_attempts": 0, "resolved_attempts": 1, "mean_score": 1.0, "assessed_resolution_rate": 1.0, "variable_cases": 0, "variable_checks": 0}, "gate": {"min_mean_score": 1.0, "min_resolution_rate": 1.0, "required_dimensions": []}, "evidence": "jobs/javascript-coding/index.html"}
+
+
+ {"fully_resolved": true, "observed": {"requested_attempts": 1, "completed_attempts": 1, "assessed_attempts": 1, "invalid_attempts": 0, "resolved_attempts": 1, "mean_score": 1.0, "assessed_resolution_rate": 1.0, "variable_cases": 0, "variable_checks": 0}, "gate": {"min_mean_score": 1.0, "min_resolution_rate": 1.0, "required_dimensions": []}, "evidence": "jobs/javascript-support/index.html"}
+
+
+
\ No newline at end of file
diff --git a/examples/multilanguage/run/plan.json b/examples/multilanguage/run/plan.json
new file mode 100644
index 0000000..ba0cc5b
--- /dev/null
+++ b/examples/multilanguage/run/plan.json
@@ -0,0 +1,75 @@
+{
+ "schema_version": "evalarc.suite-plan.v1",
+ "name": "Python and JavaScript reference conformance",
+ "manifest_sha256": "ae478f5b926dab4eee42bb1b4eb0c3a87446da0d09f5b27f8ebd273199802478",
+ "jobs": [
+ {
+ "id": "python-coding",
+ "task": "durable-kv",
+ "candidate": "/tmp/evalarc-v06-package-dyys0bqm/installed-demo-2/workspace/kv-python",
+ "seeds": [
+ 17
+ ],
+ "attempts": 1,
+ "cases_per_attempt": 15,
+ "runtime": {
+ "backend": "docker",
+ "image": "python:3.12-slim",
+ "timeout": 10.0,
+ "case_timeout": 60.0,
+ "output_limit": 1048576
+ },
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ }
+ },
+ {
+ "id": "javascript-coding",
+ "task": "durable-kv",
+ "candidate": "/tmp/evalarc-v06-package-dyys0bqm/installed-demo-2/workspace/kv-javascript",
+ "seeds": [
+ 17
+ ],
+ "attempts": 1,
+ "cases_per_attempt": 15,
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "timeout": 10.0,
+ "case_timeout": 60.0,
+ "output_limit": 1048576
+ },
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ }
+ },
+ {
+ "id": "javascript-support",
+ "task": "support-routing",
+ "candidate": "/tmp/evalarc-v06-package-dyys0bqm/installed-demo-2/workspace/support-javascript",
+ "seeds": [
+ 17
+ ],
+ "attempts": 1,
+ "cases_per_attempt": 4,
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "timeout": 10.0,
+ "case_timeout": 60.0,
+ "output_limit": 1048576
+ },
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ }
+ }
+ ],
+ "planned_attempts": 3,
+ "planned_case_executions": 34
+}
diff --git a/examples/multilanguage/run/suite.json b/examples/multilanguage/run/suite.json
new file mode 100644
index 0000000..dc21077
--- /dev/null
+++ b/examples/multilanguage/run/suite.json
@@ -0,0 +1,223 @@
+{
+ "schema_version": "evalarc.suite.v1",
+ "evalarc_version": "0.6.0",
+ "created_at": "2026-09-14T04:55:44.416536+00:00",
+ "name": "Python and JavaScript reference conformance",
+ "manifest_sha256": "ae478f5b926dab4eee42bb1b4eb0c3a87446da0d09f5b27f8ebd273199802478",
+ "valid": true,
+ "accepted": true,
+ "status": "passed",
+ "total_jobs": 3,
+ "accepted_jobs": 3,
+ "invalid_jobs": 0,
+ "fully_resolved_jobs": 3,
+ "duration_seconds": 27.208387,
+ "jobs": [
+ {
+ "id": "python-coding",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "378209f3f88e8e610eb07203e42dea1890d14e96408ed2e0593468e8f88ec0fe",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "python:3.12-slim",
+ "image_id": "sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea",
+ "command": [
+ "{python}",
+ "-I",
+ "-B",
+ "main.py",
+ "{state}/store.db"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ },
+ "status": "passed",
+ "fully_resolved": true,
+ "observed": {
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "mean_score": 1.0,
+ "assessed_resolution_rate": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0
+ },
+ "duration_seconds": 13.759954
+ },
+ {
+ "id": "javascript-coding",
+ "task": {
+ "id": "durable-kv",
+ "version": "0.1.0",
+ "domain": "coding",
+ "split": "public-development"
+ },
+ "candidate_sha256": "ba49b33370353228d229b1d0738500bdf12f818366051622570e896175bca8b7",
+ "grader_sha256": "2283f5c82c24ac536c4467f5833333fc7098adb46839238fdc56eca017ef2039",
+ "cases_sha256": "842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js",
+ "{state}/store.json"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ },
+ "status": "passed",
+ "fully_resolved": true,
+ "observed": {
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "mean_score": 1.0,
+ "assessed_resolution_rate": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0
+ },
+ "duration_seconds": 11.272241
+ },
+ {
+ "id": "javascript-support",
+ "task": {
+ "id": "support-routing",
+ "version": "0.1.0",
+ "domain": "business-tools",
+ "split": "public-development"
+ },
+ "candidate_sha256": "2ffefac03336d7126393ce0a8311a1990eed4c69d4956d66351f663a0188b143",
+ "grader_sha256": "887aea97efec396118521f720f6d8753be9ae964ab92eaa655d6575185326593",
+ "cases_sha256": "b7cd8663c69da9e1715d5c8e1d707514863dcc333fefc57acc3315053ebc85df",
+ "runtime": {
+ "backend": "docker",
+ "image": "node:22-slim",
+ "image_id": "sha256:83f487e0a63425e5b4d146fb5e5be574bcbe1b7b843d3ebafdd95eaf7767a7e5",
+ "command": [
+ "node",
+ "main.js"
+ ],
+ "response_timeout_seconds": 10.0,
+ "case_timeout_seconds": 60.0,
+ "session_output_limit_bytes": 1048576,
+ "python": "3.12.3",
+ "platform": "Linux-6.17.0-1020-aws-x86_64-with-glibc2.39"
+ },
+ "seeds": [
+ 17
+ ],
+ "gate": {
+ "min_mean_score": 1.0,
+ "min_resolution_rate": 1.0,
+ "required_dimensions": []
+ },
+ "decision": {
+ "accepted": true,
+ "valid": true,
+ "checks": [
+ {
+ "name": "mean_score",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ },
+ {
+ "name": "resolution_rate",
+ "minimum": 1.0,
+ "observed": 1.0,
+ "passed": true
+ }
+ ],
+ "reasons": []
+ },
+ "status": "passed",
+ "fully_resolved": true,
+ "observed": {
+ "requested_attempts": 1,
+ "completed_attempts": 1,
+ "assessed_attempts": 1,
+ "invalid_attempts": 0,
+ "resolved_attempts": 1,
+ "mean_score": 1.0,
+ "assessed_resolution_rate": 1.0,
+ "variable_cases": 0,
+ "variable_checks": 0
+ },
+ "duration_seconds": 1.878601
+ }
+ ],
+ "interpretation": "Acceptance follows each job's declared gate. No score is averaged across jobs or domains. Gate acceptance does not imply full task resolution. Repeated observations use fixed public cases and are descriptive only."
+}
diff --git a/examples/multilanguage/run/suite.toml b/examples/multilanguage/run/suite.toml
new file mode 100644
index 0000000..bbe75e8
--- /dev/null
+++ b/examples/multilanguage/run/suite.toml
@@ -0,0 +1,32 @@
+schema_version = "evalarc.suite-config.v1"
+name = "Python and JavaScript reference conformance"
+
+[[jobs]]
+id = "python-coding"
+task = "durable-kv"
+candidate = "../../workspace/kv-python"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "python:3.12-slim"
+
+[[jobs]]
+id = "javascript-coding"
+task = "durable-kv"
+candidate = "../../workspace/kv-javascript"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "node:22-slim"
+
+[[jobs]]
+id = "javascript-support"
+task = "support-routing"
+candidate = "../../workspace/support-javascript"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "node:22-slim"
diff --git a/examples/multilanguage/suite.toml b/examples/multilanguage/suite.toml
new file mode 100644
index 0000000..bbe75e8
--- /dev/null
+++ b/examples/multilanguage/suite.toml
@@ -0,0 +1,32 @@
+schema_version = "evalarc.suite-config.v1"
+name = "Python and JavaScript reference conformance"
+
+[[jobs]]
+id = "python-coding"
+task = "durable-kv"
+candidate = "../../workspace/kv-python"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "python:3.12-slim"
+
+[[jobs]]
+id = "javascript-coding"
+task = "durable-kv"
+candidate = "../../workspace/kv-javascript"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "node:22-slim"
+
+[[jobs]]
+id = "javascript-support"
+task = "support-routing"
+candidate = "../../workspace/support-javascript"
+seeds = [17]
+attempts = 1
+
+[jobs.runtime]
+image = "node:22-slim"
diff --git a/huggingface/README.md b/huggingface/README.md
index 844da08..4ddcb0d 100644
--- a/huggingface/README.md
+++ b/huggingface/README.md
@@ -18,6 +18,12 @@ tags:
# EvalArc — Look past the score.
+**New in v0.6: Python and JavaScript candidates.** Generate starters or independent
+references for both task packs, audit the declared faults, and combine runtimes
+in one suite. The task contracts and graders are unchanged. See the
+[language guide](https://github.com/noteflowai/evalarc/blob/main/docs/languages.md)
+and recorded mixed-language Docker suite; this does not establish a language ranking.
+
**The score rose from 90% to 93.75%. A previously passing check now fails.**
**New in v0.5: same score, different gate.** Two support jobs use the same
@@ -70,7 +76,7 @@ dependencies; the bundled trusted controls can run on a CPU.
## Scope
-Research preview 0.5.0. These are scripted controls and public development
+Research preview 0.6.0. These are scripted controls and public development
tasks, not held-out frontier-model results. Detection applies only to the
declared faults. No arbitrary reward-hack resistance, human time horizon,
hardware-agent validation or RL improvement is established. Repeated fixed
diff --git a/package-lock.json b/package-lock.json
index 9266f0f..100e1e5 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "evalarc-evidence-site",
- "version": "0.5.0",
+ "version": "0.6.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "evalarc-evidence-site",
- "version": "0.5.0",
+ "version": "0.6.0",
"devDependencies": {
"playwright": "1.63.0"
}
diff --git a/package.json b/package.json
index 0e365f6..c92bf69 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "evalarc-evidence-site",
- "version": "0.5.0",
+ "version": "0.6.0",
"private": true,
"description": "Browser checks for the static EvalArc evidence explorer",
"scripts": {
diff --git a/pyproject.toml b/pyproject.toml
index 7bffc0a..abad1e5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "evalarc"
-version = "0.5.0"
+version = "0.6.0"
description = "Auditable task environments and evaluations for coding and tool-using agents."
readme = "README.md"
requires-python = ">=3.11"
@@ -35,7 +35,7 @@ dev = ["pytest>=8,<9", "ruff>=0.11,<1", "build>=1,<2"]
where = ["src"]
[tool.setuptools.package-data]
-evalarc = ["assets/*.py", "assets/*.md"]
+evalarc = ["assets/*.py", "assets/*.md", "assets/*.js"]
[tool.pytest.ini_options]
testpaths = ["tests"]
diff --git a/site/index.html b/site/index.html
index 053044f..a1e24f5 100644
--- a/site/index.html
+++ b/site/index.html
@@ -19,7 +19,7 @@
-
+
Look pastscore.
A tool returned an error. The write already happened.
EvalArc tests whether an agent's grader catches the difference between plausible progress and a completed task.
@@ -36,6 +36,7 @@
Look pastscore.
Scripted negative control · public seed 17
+
New in v0.6: generate Python or JavaScript candidates and audit independent controls against the same task contracts. Try both runtimes ↗ . The recorded showcases below retain their original versions and fingerprints.
2 task packs
15 declared faults caught
diff --git a/src/evalarc/__init__.py b/src/evalarc/__init__.py
index 113317c..c1b417c 100644
--- a/src/evalarc/__init__.py
+++ b/src/evalarc/__init__.py
@@ -1,3 +1,3 @@
"""Auditable evaluations for AI agents."""
-__version__ = "0.5.0"
+__version__ = "0.6.0"
diff --git a/src/evalarc/assets/JAVASCRIPT.md b/src/evalarc/assets/JAVASCRIPT.md
new file mode 100644
index 0000000..d32bfbc
--- /dev/null
+++ b/src/evalarc/assets/JAVASCRIPT.md
@@ -0,0 +1,35 @@
+# JavaScript candidate runtime
+
+Use Node.js 22 or newer on Linux. The candidate has no npm dependencies.
+`evalarc.toml` declares the exact command; EvalArc does not build or install
+dependencies, choose an image from the language, or run an implicit shell.
+
+For Docker, pull `node:22-slim` before evaluation and pass
+`--image node:22-slim` to `doctor`, `evaluate`, `repeat`, or `audit`.
+In a suite, set `image = "node:22-slim"` under each JavaScript job's
+`[jobs.runtime]` table.
+The resolved immutable image ID is saved in the evaluation evidence.
+
+For trusted local execution, use `--backend local --trust-local`. Candidates
+receive a minimal PATH. If Node is installed through nvm or another version
+manager, replace `"node"` in `evalarc.toml` with its absolute executable path.
+Keep `"node"` for Docker; host paths need not exist inside the image.
+Local JavaScript audits resolve the installed Node executable from the host
+PATH and record that absolute path in their candidate command.
+
+Read TASK.md for the protocol. Use stdout only for JSONL responses; send
+diagnostics to stderr. The evaluator starts a new process for each case.
+The durable service receives a writable state-file path as its first argument;
+acknowledged changes must survive a process restart, including SIGKILL.
+
+The durable reference preserves JSON numeric source text using
+`JSON.parse` reviver context and `JSON.rawJSON`. A normal JavaScript
+parse/stringify round trip can lose large integers and merge `1` with `1.0`,
+which changes the task's type-sensitive CAS behavior. Values and object keys
+also require recursive comparison independent of object property order.
+
+References are transparent scripted controls, not AI agents or model results.
+The durable reference writes a complete snapshot before acknowledging each
+mutation. It targets small, single-process task workloads, not production
+database performance, concurrent writers, or a power-loss guarantee.
+The support reference acts only in the simulated ticket environment.
diff --git a/src/evalarc/assets/reference.js b/src/evalarc/assets/reference.js
new file mode 100644
index 0000000..07d05c7
--- /dev/null
+++ b/src/evalarc/assets/reference.js
@@ -0,0 +1,192 @@
+// Independent durable-kv control: JSON snapshots, no Python or grader imports.
+// Single process, sequential requests; fault toggles are for grader audits only.
+"use strict";
+const fs = require("node:fs");
+const path = require("node:path");
+const readline = require("node:readline");
+
+const DURABLE = true;
+const ATOMIC_BATCH = true;
+const ENFORCE_CAS = true;
+const TYPE_SENSITIVE = true;
+const DELETE_ENABLED = true;
+const VALIDATE_KEYS = true;
+const ACK_ONLY = false;
+const COMMIT_BEFORE_ACK = true;
+
+if (
+ typeof JSON.rawJSON !== "function" ||
+ JSON.parse("1", (_key, _value, context) => context?.source) !== "1"
+) {
+ throw new Error("This reference requires Node.js 22+ with JSON source text access.");
+}
+if (!process.argv[2]) throw new Error("Usage: node main.js STATE_FILE");
+const stateFile = path.resolve(process.argv[2]);
+
+// Ordinary JSON.parse loses large integers and the distinction between 1 and
+// 1.0. Keep numeric source text through requests, snapshots, and responses.
+class JsonNumber {
+ constructor(source) {
+ this.source = source;
+ this.decimal = /[.eE]/.test(source);
+ this.value = this.decimal ? Number(source) : BigInt(source);
+ if (this.decimal && !Number.isFinite(this.value)) {
+ throw new SyntaxError("Only finite JSON numbers are supported.");
+ }
+ }
+ toJSON() {
+ return JSON.rawJSON(this.source);
+ }
+}
+
+function parse(text) {
+ return JSON.parse(text, (_key, value, context) =>
+ typeof value === "number" ? new JsonNumber(context.source) : value
+ );
+}
+
+function object(value) {
+ return value !== null && typeof value === "object" &&
+ !Array.isArray(value) && !(value instanceof JsonNumber);
+}
+
+function equal(left, right) {
+ if (!TYPE_SENSITIVE) {
+ const scalar = (value) => value instanceof JsonNumber ? Number(value.value) : value;
+ if (
+ (left instanceof JsonNumber || typeof left === "boolean") &&
+ (right instanceof JsonNumber || typeof right === "boolean")
+ ) return Number(scalar(left)) === Number(scalar(right));
+ }
+ if (left instanceof JsonNumber || right instanceof JsonNumber) {
+ return left instanceof JsonNumber && right instanceof JsonNumber &&
+ left.decimal === right.decimal && Object.is(left.value, right.value);
+ }
+ if (Array.isArray(left) || Array.isArray(right)) {
+ return Array.isArray(left) && Array.isArray(right) && left.length === right.length &&
+ left.every((value, index) => equal(value, right[index]));
+ }
+ if (object(left) || object(right)) {
+ return object(left) && object(right) &&
+ Object.keys(left).length === Object.keys(right).length &&
+ Object.keys(left).every((key) => Object.hasOwn(right, key) && equal(left[key], right[key]));
+ }
+ return left === right;
+}
+
+function load() {
+ if (!DURABLE) return new Map();
+ let text;
+ try {
+ text = fs.readFileSync(stateFile, "utf8");
+ } catch (error) {
+ if (error.code === "ENOENT") return new Map();
+ throw error;
+ }
+ const entries = parse(text);
+ if (
+ !Array.isArray(entries) ||
+ !entries.every((entry) => Array.isArray(entry) && entry.length === 2 &&
+ typeof entry[0] === "string") ||
+ new Set(entries.map(([key]) => key)).size !== entries.length
+ ) throw new Error("Invalid state snapshot.");
+ return new Map(entries);
+}
+
+let state = load();
+
+function persist(next) {
+ if (!DURABLE) return;
+ const temporary = stateFile + ".pending";
+ const fd = fs.openSync(temporary, "w", 0o600);
+ try {
+ fs.writeFileSync(fd, JSON.stringify([...next]), "utf8");
+ fs.fsyncSync(fd);
+ } finally {
+ fs.closeSync(fd);
+ }
+ fs.renameSync(temporary, stateFile);
+ const directory = fs.openSync(path.dirname(stateFile), "r");
+ try {
+ fs.fsyncSync(directory);
+ } finally {
+ fs.closeSync(directory);
+ }
+}
+
+function commit(next) {
+ if (COMMIT_BEFORE_ACK) persist(next);
+ state = next;
+}
+
+function valid(item, batch = false) {
+ return object(item) &&
+ (batch ? ["put", "delete"] : ["put", "get", "delete", "cas"]).includes(item.op) &&
+ Object.hasOwn(item, "key") && (!VALIDATE_KEYS || typeof item.key === "string") &&
+ (!["put", "cas"].includes(item.op) || Object.hasOwn(item, "value")) &&
+ (item.op !== "cas" || Object.hasOwn(item, "expected"));
+}
+
+function single(item, next) {
+ const { op, key } = item;
+ if (op === "get") {
+ return next.has(key)
+ ? { ok: true, found: true, value: next.get(key) }
+ : { ok: true, found: false };
+ }
+ if (op === "delete") {
+ if (DELETE_ENABLED) next.delete(key);
+ return { ok: true };
+ }
+ if (op === "cas" && ENFORCE_CAS &&
+ !(next.has(key) && equal(next.get(key), item.expected))) {
+ return { ok: true, swapped: false };
+ }
+ next.set(key, item.value);
+ return op === "cas" ? { ok: true, swapped: true } : { ok: true };
+}
+
+const invalid = () => ({ ok: false, error: "invalid_request" });
+
+function handle(item) {
+ if (ACK_ONLY) return { ok: true };
+ if (object(item) && item.op === "batch") {
+ const operations = item.operations;
+ if (!Array.isArray(operations) ||
+ (ATOMIC_BATCH && !operations.every((child) => valid(child, true)))) return invalid();
+ const next = new Map(state);
+ for (const child of operations) {
+ if (!valid(child, true)) {
+ // Deliberately faulty partial commit when ATOMIC_BATCH is disabled.
+ commit(next);
+ return invalid();
+ }
+ single(child, next);
+ }
+ commit(next);
+ return { ok: true };
+ }
+ if (!valid(item)) return invalid();
+ if (item.op === "get") return single(item, state);
+ const next = new Map(state);
+ const response = single(item, next);
+ commit(next);
+ return response;
+}
+
+const input = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
+input.on("line", (line) => {
+ let item;
+ try {
+ item = parse(line);
+ } catch (error) {
+ if (!(error instanceof SyntaxError)) throw error;
+ process.stdout.write(JSON.stringify(invalid()) + "\n");
+ return;
+ }
+ // Storage errors terminate the process. Never acknowledge a failed write.
+ process.stdout.write(JSON.stringify(handle(item)) + "\n");
+});
+input.on("close", () => {
+ if (!COMMIT_BEFORE_ACK) persist(state);
+});
diff --git a/src/evalarc/assets/starter.js b/src/evalarc/assets/starter.js
new file mode 100644
index 0000000..16cd6df
--- /dev/null
+++ b/src/evalarc/assets/starter.js
@@ -0,0 +1,8 @@
+// Implement the durable-kv contract from TASK.md. Node.js 22+.
+"use strict";
+const readline = require("node:readline");
+
+const input = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
+input.on("line", () => {
+ process.stdout.write('{"ok":false,"error":"not_implemented"}\n');
+});
diff --git a/src/evalarc/assets/support_reference.js b/src/evalarc/assets/support_reference.js
new file mode 100644
index 0000000..bfd7f39
--- /dev/null
+++ b/src/evalarc/assets/support_reference.js
@@ -0,0 +1,63 @@
+// Scripted policy, independent of the host verifier. Audit toggles default off.
+"use strict";
+const readline = require("node:readline");
+
+const ACK_ONLY = false;
+const WRONG_TICKET = false;
+const WRONG_QUEUE = false;
+const DUPLICATE_NOTE = false;
+const CLOSE_OPEN = false;
+const SKIP_RETRY = false;
+const NEW_RETRY_KEY = false;
+
+let task;
+let ticketId;
+let lastAction;
+let pending = [];
+const call = (tool, arguments_) => ({ tool, arguments: arguments_ });
+
+function decide(observation) {
+ if (ACK_ONLY) return { finish: true, message: "Completed all requested work." };
+ if (observation.type === "start") {
+ task = observation.task;
+ ticketId = WRONG_TICKET
+ ? task.ticket_ids.find((id) => id !== task.ticket_id)
+ : task.ticket_id;
+ return call("get_ticket", { ticket_id: ticketId });
+ }
+ const result = observation.result;
+ if (!result.ok) {
+ if (result.error.retryable && !SKIP_RETRY) {
+ const retry = structuredClone(lastAction);
+ if (NEW_RETRY_KEY && retry.tool === "add_note") {
+ retry.arguments.request_id += "-retry";
+ }
+ return retry;
+ }
+ return { finish: true, message: "Could not complete the operation." };
+ }
+ if (lastAction.tool === "get_ticket") {
+ const ticket = result.ticket;
+ let queue = task.routing_rules[ticket.category];
+ if (WRONG_QUEUE) queue = Object.values(task.routing_rules).find((value) => value !== queue);
+ pending = [
+ call("assign_queue", { ticket_id: ticketId, queue }),
+ call("add_note", { ticket_id: ticketId, body: task.note, request_id: "review" }),
+ ];
+ if (DUPLICATE_NOTE) {
+ pending.push(call("add_note", {
+ ticket_id: ticketId, body: task.note, request_id: "duplicate",
+ }));
+ }
+ if (ticket.resolved || CLOSE_OPEN) {
+ pending.push(call("close_ticket", { ticket_id: ticketId }));
+ }
+ }
+ return pending.length ? pending.shift() : { finish: true };
+}
+
+const input = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
+input.on("line", (line) => {
+ lastAction = decide(JSON.parse(line));
+ process.stdout.write(JSON.stringify(lastAction) + "\n");
+});
diff --git a/src/evalarc/assets/support_starter.js b/src/evalarc/assets/support_starter.js
new file mode 100644
index 0000000..d9f856c
--- /dev/null
+++ b/src/evalarc/assets/support_starter.js
@@ -0,0 +1,8 @@
+// Replace this policy with an agent that executes the task's tool operations.
+"use strict";
+const readline = require("node:readline");
+
+const input = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
+input.on("line", () => {
+ process.stdout.write('{"finish":true,"message":"Not implemented."}\n');
+});
diff --git a/src/evalarc/audit.py b/src/evalarc/audit.py
index 75c0f45..2a714d0 100644
--- a/src/evalarc/audit.py
+++ b/src/evalarc/audit.py
@@ -2,14 +2,16 @@
from __future__ import annotations
+import shutil
import tempfile
-from importlib.resources import files
+from dataclasses import replace
from pathlib import Path
from evalarc.evaluate import evaluate
from evalarc.events import EventCallback
from evalarc.runner import Runtime
-from evalarc.tasks import get_task
+from evalarc.templates import asset as asset
+from evalarc.templates import candidate_template
MUTANTS = {
"ack-without-work": ("ACK_ONLY = False", "ACK_ONLY = True", "basic"),
@@ -35,10 +37,6 @@
CONTROL_PACKS = {"durable-kv": MUTANTS, "support-routing": SUPPORT_MUTANTS}
-def asset(name: str) -> str:
- return files("evalarc").joinpath("assets", name).read_text()
-
-
def write_candidate(path: Path, source: str) -> Path:
path.mkdir(parents=True)
(path / "main.py").write_text(source)
@@ -57,11 +55,24 @@ def audit(
task_id: str = "durable-kv",
*,
on_event: EventCallback | None = None,
+ language: str = "python",
) -> dict:
- task = get_task(task_id)
- source = asset(task.reference_asset)
+ template = candidate_template(task_id, language, reference=True)
+ if language == "javascript" and runtime.backend == "local":
+ # Local candidates receive a minimal PATH. Resolve our trusted control's
+ # runtime here so installations managed by setup-node/nvm also work.
+ node = shutil.which("node")
+ if node is None:
+ raise ValueError("JavaScript audit requires Node.js 22 or newer on PATH")
+ template = replace(template, command=(str(Path(node).resolve()), *template.command[1:]))
+ source = template.source
controls = CONTROL_PACKS[task_id]
+ def candidate(path: Path, content: str) -> Path:
+ path.mkdir(parents=True)
+ template.write(path, source=content)
+ return path
+
def observer(control: str) -> EventCallback | None:
if on_event is None:
return None
@@ -71,15 +82,18 @@ def observer(control: str) -> EventCallback | None:
with tempfile.TemporaryDirectory(prefix="evalarc-audit-") as directory:
root = Path(directory)
reference = evaluate(
- write_candidate(root / "reference", source),
+ candidate(root / "reference", source),
runtime,
seeds,
task_id,
on_event=observer("reference"),
)
for name, (old, new, target) in controls.items():
- candidate = write_candidate(root / name, mutate(source, old, new))
- result = evaluate(candidate, runtime, seeds, task_id, on_event=observer(name))
+ if language == "javascript":
+ old = old.replace("True", "true").replace("False", "false")
+ new = new.replace("True", "true").replace("False", "false")
+ workspace = candidate(root / name, mutate(source, old, new))
+ result = evaluate(workspace, runtime, seeds, task_id, on_event=observer(name))
failures = [c["case_id"] for c in result["cases"] if c["checks"].get(target) is False]
rows.append(
{
diff --git a/src/evalarc/cli.py b/src/evalarc/cli.py
index c8c4028..176d313 100644
--- a/src/evalarc/cli.py
+++ b/src/evalarc/cli.py
@@ -10,7 +10,7 @@
from evalarc import __version__
from evalarc.artifacts import check_output_location, new_json, new_run
-from evalarc.audit import asset, audit
+from evalarc.audit import audit
from evalarc.compare import compare
from evalarc.doctor import diagnose
from evalarc.evaluate import evaluate, write_json
@@ -20,7 +20,8 @@
from evalarc.report import render_audit, render_comparison, render_evaluation, render_repetition
from evalarc.runner import EnvironmentFailure, Runtime
from evalarc.suite import load_suite, run_suite
-from evalarc.tasks import TASKS, get_task
+from evalarc.tasks import TASKS
+from evalarc.templates import LANGUAGES, initialize
from evalarc.trajectory import summarize
@@ -36,6 +37,7 @@ def parser() -> argparse.ArgumentParser:
init.add_argument("destination", type=Path)
init.add_argument("--reference", action="store_true", help="use the known-good control")
init.add_argument("--task", choices=TASKS, default="durable-kv")
+ init.add_argument("--language", choices=LANGUAGES, default="python")
for name, help_text in (
("evaluate", "grade a candidate directory"),
("audit", "evaluate a task's reference and behavioral negative controls"),
@@ -46,6 +48,13 @@ def parser() -> argparse.ArgumentParser:
command.add_argument("candidate", type=Path)
if name == "repeat":
command.add_argument("--attempts", type=int, default=3)
+ if name == "audit":
+ command.add_argument(
+ "--language",
+ choices=LANGUAGES,
+ default="python",
+ help="control language; JavaScript on Docker needs --image node:22-slim",
+ )
command.add_argument("--task", choices=TASKS, default="durable-kv")
command.add_argument("--backend", choices=["docker", "local"], default="docker")
command.add_argument("--trust-local", action="store_true")
@@ -167,15 +176,10 @@ def main(argv: list[str] | None = None) -> int:
)
return 1 if result["has_regressions"] else 0
if args.command == "init":
- task = get_task(args.task)
- if args.destination.exists():
- raise ValueError("destination already exists; choose a new directory")
- args.destination.mkdir(parents=True)
- (args.destination / "main.py").write_text(
- asset(task.reference_asset if args.reference else task.starter_asset)
- )
- (args.destination / "TASK.md").write_text(asset(task.contract_asset))
+ initialize(args.destination, args.task, args.language, reference=args.reference)
print(f"Created {args.destination}")
+ if args.language == "javascript":
+ print("Requires Node.js 22+. For Docker, select --image node:22-slim.")
return 0
if args.command == "trajectory":
checkpoints = json.loads(args.checkpoints.read_text())
@@ -200,7 +204,9 @@ def main(argv: list[str] | None = None) -> int:
EventLog(output / "events.jsonl", stream=args.progress) as events,
):
runtime.prepare()
- result = audit(runtime, args.seeds, args.task, on_event=events)
+ result = audit(
+ runtime, args.seeds, args.task, on_event=events, language=args.language
+ )
write_json(output / "audit.json", result)
render_audit(result, output / "index.html")
reference_status = (
diff --git a/src/evalarc/templates.py b/src/evalarc/templates.py
new file mode 100644
index 0000000..0377b14
--- /dev/null
+++ b/src/evalarc/templates.py
@@ -0,0 +1,61 @@
+"""Packaged candidate templates; independent of task grading and evidence."""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from importlib.resources import files
+from pathlib import Path
+
+from evalarc.artifacts import new_run
+from evalarc.tasks import get_task
+
+LANGUAGES = ("python", "javascript")
+
+
+def asset(name: str) -> str:
+ return files("evalarc").joinpath("assets", name).read_text(encoding="utf-8")
+
+
+@dataclass(frozen=True)
+class CandidateTemplate:
+ filename: str
+ source: str
+ command: tuple[str, ...] | None = None
+
+ def write(self, path: Path, *, source: str | None = None) -> None:
+ """Write into a caller-owned directory; the caller controls publication."""
+ (path / self.filename).write_text(
+ self.source if source is None else source, encoding="utf-8"
+ )
+ if self.command is not None:
+ (path / "evalarc.toml").write_text(
+ f"command = {json.dumps(self.command)}\n", encoding="utf-8"
+ )
+
+
+def candidate_template(
+ task_id: str, language: str = "python", *, reference: bool = False
+) -> CandidateTemplate:
+ task = get_task(task_id)
+ if language not in LANGUAGES:
+ raise ValueError(f"unknown language {language!r}; choose from {', '.join(LANGUAGES)}")
+ name = task.reference_asset if reference else task.starter_asset
+ if language == "python":
+ return CandidateTemplate("main.py", asset(name))
+ command = ("node", "main.js")
+ if task_id == "durable-kv":
+ command += ("{state}/store.json",)
+ return CandidateTemplate("main.js", asset(name.removesuffix(".py") + ".js"), command)
+
+
+def initialize(
+ destination: Path, task_id: str, language: str = "python", *, reference: bool = False
+) -> None:
+ template = candidate_template(task_id, language, reference=reference)
+ contract = asset(get_task(task_id).contract_asset)
+ with new_run(destination) as staged:
+ template.write(staged)
+ (staged / "TASK.md").write_text(contract, encoding="utf-8")
+ if language == "javascript":
+ (staged / "RUNTIME.md").write_text(asset("JAVASCRIPT.md"), encoding="utf-8")
diff --git a/tests/test_templates.py b/tests/test_templates.py
new file mode 100644
index 0000000..a3d8f2c
--- /dev/null
+++ b/tests/test_templates.py
@@ -0,0 +1,321 @@
+import json
+import shutil
+import subprocess
+import tomllib
+from pathlib import Path
+
+import pytest
+
+from evalarc.audit import asset, audit
+from evalarc.cli import main
+from evalarc.evaluate import evaluate
+from evalarc.runner import CandidateError, Runtime
+from evalarc.suite import load_suite
+from evalarc.tasks import TASKS, get_task
+from evalarc.templates import CandidateTemplate, candidate_template, initialize
+
+
+@pytest.fixture(scope="module")
+def node():
+ executable = shutil.which("node")
+ if not executable:
+ pytest.skip("Node.js 22+ required for JavaScript integration tests")
+ version = subprocess.run(
+ [executable, "-p", "process.versions.node"], capture_output=True, text=True, check=True
+ ).stdout.strip()
+ if int(version.split(".")[0]) < 22:
+ pytest.skip("Node.js 22+ required for JSON numeric source text access")
+ return str(Path(executable).resolve())
+
+
+def javascript_workspace(path, node, task_id="durable-kv", *, reference=True):
+ initialize(path, task_id, "javascript", reference=reference)
+ manifest = path / "evalarc.toml"
+ command = tomllib.loads(manifest.read_text())["command"]
+ command[0] = node
+ manifest.write_text(f"command = {json.dumps(command)}\n")
+ return path
+
+
+@pytest.mark.parametrize("task_id", TASKS)
+@pytest.mark.parametrize("reference", [False, True])
+def test_default_python_template_keeps_original_bytes(tmp_path, task_id, reference):
+ destination = tmp_path / "python"
+ initialize(destination, task_id, reference=reference)
+ task = get_task(task_id)
+ assert sorted(p.name for p in destination.iterdir()) == ["TASK.md", "main.py"]
+ assert (destination / "main.py").read_text() == asset(
+ task.reference_asset if reference else task.starter_asset
+ )
+ assert (destination / "TASK.md").read_text() == asset(task.contract_asset)
+
+
+@pytest.mark.parametrize("task_id", TASKS)
+def test_javascript_init_is_portable_and_needs_no_installed_runtime(tmp_path, monkeypatch, task_id):
+ monkeypatch.setenv("PATH", "")
+ destination = tmp_path / "js"
+ assert main(["init", str(destination), "--task", task_id, "--language", "javascript"]) == 0
+ assert (destination / "main.js").is_file()
+ assert (destination / "TASK.md").read_text() == asset(get_task(task_id).contract_asset)
+ assert (destination / "RUNTIME.md").is_file()
+ command = tomllib.loads((destination / "evalarc.toml").read_text())["command"]
+ assert command[:2] == ["node", "main.js"]
+ assert command[2:] == (["{state}/store.json"] if task_id == "durable-kv" else [])
+
+
+def test_failed_init_removes_partial_workspace(tmp_path, monkeypatch):
+ def fail(self, path, **kwargs):
+ (path / self.filename).write_text("partial")
+ raise OSError("simulated disk failure")
+
+ monkeypatch.setattr(CandidateTemplate, "write", fail)
+ destination = tmp_path / "candidate"
+ with pytest.raises(OSError, match="simulated"):
+ initialize(destination, "durable-kv", "javascript")
+ assert list(tmp_path.iterdir()) == []
+
+
+def test_init_refuses_existing_directory_and_dangling_symlink(tmp_path):
+ existing = tmp_path / "existing"
+ existing.mkdir()
+ (existing / "keep.txt").write_text("user work")
+ with pytest.raises(ValueError, match="already exists"):
+ initialize(existing, "durable-kv", "javascript")
+ assert list(existing.iterdir()) == [existing / "keep.txt"]
+ dangling = tmp_path / "dangling"
+ dangling.symlink_to(tmp_path / "absent")
+ with pytest.raises(ValueError, match="already exists"):
+ initialize(dangling, "durable-kv")
+ assert dangling.is_symlink()
+ assert not (tmp_path / "absent").exists()
+
+
+def test_invalid_template_selection_does_not_create_workspace(tmp_path):
+ for task, language in [("unknown", "python"), ("durable-kv", "rust")]:
+ with pytest.raises(ValueError, match="unknown"):
+ initialize(tmp_path / "candidate", task, language)
+ assert list(tmp_path.iterdir()) == []
+
+
+@pytest.mark.parametrize("task_id", TASKS)
+def test_javascript_references_and_all_declared_faults(node, task_id):
+ events = []
+ result = audit(
+ Runtime(backend="local", timeout=3),
+ [17],
+ task_id,
+ language="javascript",
+ on_event=events.append,
+ )
+ assert result["valid"] and result["passed"]
+ assert result["killed"] == result["total"] == (8 if task_id == "durable-kv" else 7)
+ assert all(row["failing_cases"] for row in result["mutants"])
+ assert result["reference"]["runtime"]["command"][0] == node
+ assert {event["control"] for event in events} == {
+ "reference",
+ *(row["name"] for row in result["mutants"]),
+ }
+
+
+@pytest.mark.parametrize("task_id", TASKS)
+@pytest.mark.parametrize("reference", [False, True])
+def test_generated_javascript_candidates_use_unchanged_grader(tmp_path, node, task_id, reference):
+ candidate = javascript_workspace(tmp_path / "candidate", node, task_id, reference=reference)
+ result = evaluate(candidate, Runtime(backend="local", timeout=3), [41, 97], task_id)
+ assert result["valid"]
+ assert result["resolved"] is reference
+ assert all(case["passed"] is reference for case in result["cases"])
+
+
+def test_missing_local_node_fails_before_any_audit_execution(monkeypatch):
+ import evalarc.audit as module
+
+ def unexpected(*args, **kwargs):
+ pytest.fail("Missing Node must fail before control execution")
+
+ monkeypatch.setattr(module.shutil, "which", lambda _: None)
+ monkeypatch.setattr(module, "evaluate", unexpected)
+ with pytest.raises(ValueError, match="Node.js 22"):
+ audit(Runtime(backend="local"), [17], language="javascript")
+
+
+@pytest.mark.parametrize(
+ "stored,expected,matches",
+ [
+ (9007199254740993, 9007199254740992, False),
+ (10**100 + 1, 10**100 + 1, True),
+ (1, 1.0, False),
+ (True, 1, False),
+ (-0.0, 0.0, False),
+ (-0.0, -0.0, True),
+ (1e-300, 1e-300, True),
+ ([{"b": [1, False], "a": 1.0}], [{"a": 1.0, "b": [1, False]}], True),
+ ({"__proto__": 1, "constructor": [False]}, {"constructor": [0], "__proto__": 1}, False),
+ (
+ {"2": "two", "1": "one", "\U00010000": 1, "\ue000": 2},
+ {"\ue000": 2, "\U00010000": 1, "1": "one", "2": "two"},
+ True,
+ ),
+ (
+ {"value": 42, "source": "9", "decimal": False},
+ {"value": 42, "source": "9", "decimal": False},
+ True,
+ ),
+ ],
+)
+def test_javascript_value_semantics_survive_sigkill(tmp_path, node, stored, expected, matches):
+ candidate = javascript_workspace(tmp_path / "candidate", node)
+ state = tmp_path / "state"
+ state.mkdir()
+ runtime = Runtime(backend="local", timeout=3).for_candidate(
+ candidate, get_task("durable-kv").default_command
+ )
+ with runtime.start(candidate, state) as process:
+ assert process.request({"op": "put", "key": "__proto__", "value": stored}) == {"ok": True}
+ process.finish("kill")
+ with runtime.start(candidate, state) as process:
+ response = process.request({"op": "get", "key": "__proto__"})
+ # Canonical Python serialization preserves both numeric category and sign.
+ assert json.dumps(response["value"], sort_keys=True) == json.dumps(stored, sort_keys=True)
+ assert process.request(
+ {"op": "cas", "key": "__proto__", "expected": expected, "value": "replaced"}
+ ) == {"ok": True, "swapped": matches}
+ process.finish("kill")
+ with runtime.start(candidate, state) as process:
+ response = process.request({"op": "get", "key": "__proto__"})
+ assert json.dumps(response["value"], sort_keys=True) == json.dumps(
+ "replaced" if matches else stored, sort_keys=True
+ )
+ process.finish("eof")
+
+
+def test_javascript_invalid_batch_and_special_keys_preserve_state(tmp_path, node):
+ candidate = javascript_workspace(tmp_path / "candidate", node)
+ state = tmp_path / "state"
+ state.mkdir()
+ runtime = Runtime(backend="local").for_candidate(
+ candidate, get_task("durable-kv").default_command
+ )
+ keys = ["", "__proto__", "constructor", "toString", "hasOwnProperty", "键🔑", "\ud800"]
+ with runtime.start(candidate, state) as process:
+ assert process.request(
+ {"op": "batch", "operations": [{"op": "put", "key": key, "value": key} for key in keys]}
+ ) == {"ok": True}
+ for invalid in [
+ None,
+ [],
+ True,
+ 1,
+ {"op": "put", "key": "__proto__"},
+ {"op": "get", "key": False},
+ {"op": "get", "key": []},
+ {
+ "op": "batch",
+ "operations": [
+ {"op": "delete", "key": "__proto__"},
+ {"op": "batch", "operations": []},
+ ],
+ },
+ ]:
+ assert process.request(invalid) == {"ok": False, "error": "invalid_request"}
+ process.finish("kill")
+ with runtime.start(candidate, state) as process:
+ for key in keys:
+ assert process.request({"op": "get", "key": key}) == {
+ "ok": True,
+ "found": True,
+ "value": key,
+ }
+ process.finish("eof")
+
+
+def test_javascript_cannot_acknowledge_failed_persistence(tmp_path, node):
+ candidate = javascript_workspace(tmp_path / "candidate", node)
+ state = tmp_path / "state"
+ state.mkdir()
+ # A directory where the pending file must be written forces a deterministic
+ # storage error even if the tests are running with elevated permissions.
+ (state / "store.json.pending").mkdir()
+ runtime = Runtime(backend="local").for_candidate(
+ candidate, get_task("durable-kv").default_command
+ )
+ with runtime.start(candidate, state) as process:
+ with pytest.raises(CandidateError, match="exited"):
+ process.request({"op": "put", "key": "x", "value": "must not be acknowledged"})
+ assert not (state / "store.json").exists()
+
+
+def test_javascript_corrupt_snapshot_is_not_reset(tmp_path, node):
+ candidate = javascript_workspace(tmp_path / "candidate", node)
+ state = tmp_path / "state"
+ state.mkdir()
+ (state / "store.json").write_text('[["duplicate",1],["duplicate",2]]')
+ runtime = Runtime(backend="local").for_candidate(
+ candidate, get_task("durable-kv").default_command
+ )
+ with runtime.start(candidate, state) as process:
+ with pytest.raises(CandidateError, match="exited"):
+ process.request({"op": "get", "key": "duplicate"})
+ assert (state / "store.json").read_text() == '[["duplicate",1],["duplicate",2]]'
+
+
+def test_javascript_raw_numeric_lexemes_and_malformed_lines(tmp_path, node):
+ candidate = javascript_workspace(tmp_path / "candidate", node)
+ lines = [
+ '{"op":"put","key":"x","value":1e0}',
+ '{"op":"cas","key":"x","expected":1.00,"value":-0}',
+ '{"op":"cas","key":"x","expected":0,"value":-1e-999}',
+ '{"op":"cas","key":"x","expected":0.0,"value":"incorrect"}',
+ '{"op":"cas","key":"x","expected":-0e99,"value":"correct"}',
+ '{"op":"put","key":"x","value":1e999}',
+ "not JSON",
+ '{"op":"get","key":"x"}',
+ ]
+ result = subprocess.run(
+ [node, str(candidate / "main.js"), str(tmp_path / "store.json")],
+ input="\n".join(lines) + "\n",
+ text=True,
+ capture_output=True,
+ timeout=10,
+ check=True,
+ )
+ assert [json.loads(line) for line in result.stdout.splitlines()] == [
+ {"ok": True},
+ {"ok": True, "swapped": True}, # equivalent floating spellings
+ {"ok": True, "swapped": True}, # integer negative zero equals zero
+ {"ok": True, "swapped": False}, # negative underflow retains its sign
+ {"ok": True, "swapped": True}, # equivalent negative floating zeros
+ {"ok": False, "error": "invalid_request"},
+ {"ok": False, "error": "invalid_request"},
+ {"ok": True, "found": True, "value": "correct"},
+ ]
+
+
+def test_audit_image_is_explicit_and_python_default_is_unchanged():
+ from evalarc.cli import parser
+
+ args = parser().parse_args(["audit", "--language", "javascript"])
+ assert args.image == "python:3.12-slim"
+ assert candidate_template("durable-kv").command is None
+
+
+def test_documented_multilanguage_suite_uses_each_jobs_runtime(tmp_path):
+ source = Path(__file__).resolve().parents[1] / "examples/multilanguage/suite.toml"
+ config = tmp_path / "examples/multilanguage/suite.toml"
+ config.parent.mkdir(parents=True)
+ shutil.copyfile(source, config)
+ for name, task, language in [
+ ("kv-python", "durable-kv", "python"),
+ ("kv-javascript", "durable-kv", "javascript"),
+ ("support-javascript", "support-routing", "javascript"),
+ ]:
+ initialize(tmp_path / "workspace" / name, task, language, reference=True)
+ plan = load_suite(config)
+ assert [job.runtime.image for job in plan.jobs] == [
+ "python:3.12-slim",
+ "node:22-slim",
+ "node:22-slim",
+ ]
+ assert all(job.runtime.backend == "docker" for job in plan.jobs)
+ assert plan.describe()["planned_case_executions"] == 34
+ assert all(job.gate.min_resolution_rate == 1 for job in plan.jobs)