Make agent evaluation evidence filterable in a verified HF casebook

.github/workflows/ci.yml

@@ -56,6 +56,7 @@ jobs:
           node-version: "22"
           cache: npm
       - run: python scripts/build_site.py --output dist/site
+      - run: python scripts/build_dataset.py --output dist/casebook
       - run: npm ci
       - run: npx playwright install --with-deps chromium
       - run: npm run test:site
@@ -64,6 +65,10 @@ jobs:
           name: evidence-site
           path: dist/site/
           include-hidden-files: true
+      - uses: actions/upload-artifact@v4
+        with:
+          name: evidence-casebook
+          path: dist/casebook/
   pages:
     if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' && github.repository == 'noteflowai/evalarc'
     needs: [test, docker-audit, site]
@@ -109,6 +114,10 @@ jobs:
         with:
           name: evidence-site
           path: dist/site
+      - uses: actions/download-artifact@v4
+        with:
+          name: evidence-casebook
+          path: dist/casebook
       - run: python -m pip install huggingface_hub==1.31.0
       - name: Publish and verify public files
         env:
@@ -118,3 +127,4 @@ jobs:
         run: |
           test "$(gh api repos/$GITHUB_REPOSITORY/git/ref/heads/main --jq .object.sha)" = "$GITHUB_SHA"
           python scripts/publish_space.py --bundle dist/site
+          python scripts/publish_dataset.py --bundle dist/casebook

README.md

@@ -8,6 +8,7 @@
 <p align="center">
   <a href="https://huggingface.co/spaces/glayguo/evalarc">Interactive evidence lab</a> ·
   <a href="https://noteflowai.github.io/evalarc/">Web demo</a> ·
+  <a href="https://huggingface.co/datasets/glayguo/evalarc-casebook">Filterable casebook</a> ·
   <a href="https://github.com/noteflowai/evalarc/releases">Releases</a> ·
   <a href="README.zh-CN.md">简体中文</a> ·
   <a href="docs/research.zh-CN.md">Research & papers</a> ·
@@ -49,6 +50,13 @@ remain inspectable. Configured acceptance is separate from task resolution.
 
 [![EvalArc v0.5: the same score meets one gate and fails another](docs/assets/suite-lab.png)](https://glayguo-evalarc.static.hf.space/#suite)
 
+Prefer tables or Python? The [Hugging Face casebook](https://huggingface.co/datasets/glayguo/evalarc-casebook)
+separates 167 audit cases, six repeated attempts and three suite jobs into
+filterable configurations, with unchanged source JSON and provenance.
+Start with `suite_jobs` to compare `gate_accepted` and `fully_resolved`.
+These are scripted public-development records, not a held-out model benchmark.
+[Data guide and reproduction](docs/casebook.md).
+
 `evalarc repeat` freezes one candidate, runs fresh attempts on fixed
 cases, and reports every outcome with per-check pass rates. Runs now record
 JSONL progress, enforce a total case budget, and save bounded process diagnostics.

README.zh-CN.md

@@ -8,6 +8,7 @@ Python 3.11+，Linux 主机，零运行时第三方依赖，MIT 许可证。
 
 [在线交互演示](https://huggingface.co/spaces/glayguo/evalarc) ·
 [网页镜像](https://noteflowai.github.io/evalarc/) ·
+[可筛选证据数据集](https://huggingface.co/datasets/glayguo/evalarc-casebook) ·
 [版本下载](https://github.com/noteflowai/evalarc/releases) ·
 [English](README.md) · [中文调研与论文分析](docs/research.zh-CN.md) ·
 [架构设计](docs/architecture.md) · [方法说明](docs/methodology.md) · [开发路线](docs/roadmap.md)
@@ -41,6 +42,12 @@ v0.5 新增 `evalarc suite`：用 TOML 声明任务、候选、轮次、预算
 
 [![EvalArc v0.5：相同分数，不同验收结果](docs/assets/suite-lab.png)](https://glayguo-evalarc.static.hf.space/#suite)
 
+[Hugging Face Casebook](https://huggingface.co/datasets/glayguo/evalarc-casebook)
+把 167 条审计用例、6 次重复尝试和 3 项验收作业分别整理为可筛选的表，
+保留未经改写的原始 JSON 和版本指纹。可先选择 `suite_jobs`，
+对照 `gate_accepted` 与 `fully_resolved`，或用 Python 读取。
+这些是公开开发任务中的脚本对照，不是隐藏模型测试集。详见[数据说明](docs/casebook.md)。
+
 `evalarc repeat` 固定一份候选快照，在相同场景上重新启动多轮评测，
 保存每轮证据并显示逐项通过率与结果波动。同时补齐场景总时间预算、JSONL 进度
 和受限进程诊断。详见[重复评测指南](docs/reliability.zh-CN.md)。

docs/casebook.md

@@ -0,0 +1,61 @@
+# Casebook: filter and reuse the recorded evidence
+
+The [Hugging Face dataset](https://huggingface.co/datasets/glayguo/evalarc-casebook)
+provides a tabular companion to the interactive lab. The default `suite_jobs`
+view makes an important distinction inspectable: two jobs share a 93.75% mean
+score and zero resolved attempts, but their configured gates disagree.
+
+| Configuration | Rows | One row represents |
+| --- | ---: | --- |
+| `suite_jobs` | 3 | A configured job and its gate decision |
+| `audit_cases` | 167 | A case execution within one of 17 scripted controls |
+| `repetition_attempts` | 6 | A recorded attempt of a frozen support policy |
+
+The row units differ. Do not add them together as independent trials or average
+the parent `evaluation_score` repeated across case rows. Each configuration uses
+a `development` split. No training or held-out model test split is claimed.
+
+```python
+from datasets import load_dataset
+
+jobs = load_dataset("glayguo/evalarc-casebook", "suite_jobs", split="development")
+for job in jobs:
+    if not job["fully_resolved"]:
+        print(job["id"], job["mean_score"], job["gate_accepted"])
+```
+
+Install `datasets` only if using this optional reader. The three JSONL files
+are also readable with Python's standard `json` module. EvalArc itself still
+has no third-party runtime dependency.
+
+Every row carries the original file path, JSON Pointer, file hash, source commit,
+candidate/grader/cases fingerprints and recorded runtime. The bundle includes
+unchanged source JSON, all repeated attempts, suite configuration and JUnit.
+`case_json`, `checks_json`, `gate_json`, `decision_json` and `runtime_json` are
+JSON strings so unrelated task schemas remain readable in a common table.
+
+## Reproduce and publish
+
+From a clean checkout of the commit named in the dataset card:
+
+```bash
+python3 scripts/build_dataset.py --output dist/casebook
+```
+
+The builder reuses the site's evidence checks, validates each audit evaluation,
+recomputes repetitions and suite decisions, and verifies JUnit. It checks row
+counts and unique identities, preserves original evidence bytes and writes a
+file manifest. An existing output directory is never overwritten.
+
+CI builds the dataset and site from the same source, retaining both artifacts.
+The existing publication job requires all tests, Docker audits and browser
+checks to pass. It publishes only the current main commit, uses an expected
+Hub parent revision, and verifies every uploaded file without credentials.
+A retry can initialize an empty dataset owned by the authenticated user; an
+existing unrelated dataset is rejected.
+
+The source records come from EvalArc 0.2, 0.4 and 0.5; their original versions
+and grader hashes are retained. These are authored development fixtures,
+not model-generated trajectories or a matched longitudinal benchmark.
+See the full [dataset card](../huggingface/DATASET.md) for authorship,
+license, provenance and interpretation limits.

docs/outreach/huggingface-introduction.md

@@ -3,6 +3,14 @@ graders behind AI-agent evaluations.
 
 **Updated for v0.5: same score, different acceptance gate.**
 
+**New data companion:** the
+[EvalArc Casebook](https://huggingface.co/datasets/glayguo/evalarc-casebook)
+makes the recorded evidence filterable in Hugging Face: 167 audit cases, six
+repeated attempts and three suite jobs in separate development configurations.
+Start with `suite_jobs` to compare acceptance with full resolution, or load the
+JSONL in Python. Every row links to unchanged source JSON and a fixed code
+revision. This is a small scripted casebook, not a new model benchmark.
+
 The new suite showcase puts the same frozen defective policy through two
 explicit rules. Both support jobs score 93.75% and resolve 0/2 attempts.
 A deliberately permissive gate accepts the partial result; requiring every

docs/outreach/launch.md

@@ -10,7 +10,9 @@ EvalArc 是面向 AI 智能体的开源评测与评分器审计工具，帮助
 和 JUnit 导出：同样是 93.75%、没有完全通过的尝试，宽松规则可允许部分进展，
 要求备注检查全部通过的规则则拒绝。页面分别展示任务完成情况和规则决策，
 并保留重复评测、版本回归与全部运行证据，适合 PAI 智能体创新场景的评测设计、
-验收验证与技术交流。当前为研究预览版，演示使用脚本对照，尚未给出真实大模型
+验收验证与技术交流。HF Casebook 进一步提供可筛选、可用 Python 读取的公开
+证据表，分别保留 167 条用例、6 次重复尝试和 3 项验收作业及其原始记录。
+当前为研究预览版，演示使用脚本对照，尚未给出真实大模型
 性能或强化学习收益结论。
 
 ## English
@@ -37,12 +39,18 @@ permissive threshold but fails a gate requiring every notes check. Three jobs,
 five attempts and the complete configuration remain inspectable. No hosted
 CI importer or model-provider integration is claimed.
 
+The Hugging Face Casebook makes 167 audit cases, six repeated attempts and three
+suite jobs filterable and readable from Python as separate development
+configurations. Each row retains a pointer and hash for its original source
+record. This is a tabular view of existing evidence, not additional model trials.
+
 ## Entry points
 
 - Source: https://github.com/noteflowai/evalarc
 - Demo: https://huggingface.co/spaces/glayguo/evalarc
 - Web mirror: https://noteflowai.github.io/evalarc/
 - Releases: https://github.com/noteflowai/evalarc/releases
+- Data: https://huggingface.co/datasets/glayguo/evalarc-casebook
 
 For each community, write a description appropriate to its rules and audience.
 Disclose maintainer affiliation. Do not present pending editorial submissions

huggingface/DATASET.md

@@ -0,0 +1,139 @@
+---
+pretty_name: EvalArc Casebook — inspect scores, failures and acceptance gates
+license: mit
+language:
+  - en
+size_categories:
+  - n<1K
+task_categories:
+  - other
+tags:
+  - tabular
+  - agent-evaluation
+  - software-testing
+  - reproducibility
+  - synthetic
+configs:
+  - config_name: suite_jobs
+    default: true
+    data_files:
+      - split: development
+        path: data/suite_jobs.jsonl
+  - config_name: audit_cases
+    data_files:
+      - split: development
+        path: data/audit_cases.jsonl
+  - config_name: repetition_attempts
+    data_files:
+      - split: development
+        path: data/repetition_attempts.jsonl
+---
+
+# EvalArc Casebook
+
+**The same 93.75% score can pass one acceptance gate and fail another.**
+Inspect the rules, actual failed checks and original Docker records in a
+filterable table. This is the data companion to the
+[interactive evidence lab](https://huggingface.co/spaces/glayguo/evalarc).
+
+In the default `suite_jobs` view, compare `support-partial` and
+`support-protected`. Both use the same frozen defective policy, score 93.75%
+and fully resolve 0/2 attempts. The deliberately permissive rule accepts partial
+progress; the rule requiring every notes check rejects it. `gate_accepted`
+and `fully_resolved` are separate columns.
+
+## What is included
+
+| Configuration | Rows | Unit and purpose |
+| --- | ---: | --- |
+| `suite_jobs` | 3 | One configured gate per job, backed by five Docker attempts; compare acceptance and full resolution. |
+| `audit_cases` | 167 | One case execution per scripted control: 135 coding and 32 support cases across two references and 15 declared faults. |
+| `repetition_attempts` | 6 | One recorded attempt of a frozen support policy; three reference and three faulty attempts. |
+
+Each configuration has a single **`development`** split. These are different
+units, so their row counts must not be summed into a number of independent
+trials or benchmark examples. `evaluation_score` in the case table is the
+parent evaluation's score, repeated for navigation; averaging that column
+across case rows would reweight evaluations incorrectly.
+
+These are saved **scripted controls on public development tasks**, not runs of
+a trained language model. Support tickets, identifiers and messages are
+synthetic fixtures. No customer data or private logs were collected.
+
+## Use without the web viewer
+
+```python
+from datasets import load_dataset
+
+jobs = load_dataset("glayguo/evalarc-casebook", "suite_jobs", split="development")
+for job in jobs:
+    if not job["fully_resolved"]:
+        print(job["id"], job["mean_score"], job["gate_accepted"])
+```
+
+No dataset loading script, candidate execution or model credential is required.
+You can also download the three JSONL files and read them with Python's standard
+`json` module. The `datasets` package is only needed for the example above.
+
+## Inspect a row's evidence
+
+`source_file` points to an unchanged JSON file in this dataset repository.
+`source_pointer` is a JSON Pointer locating the exact original object; an empty
+pointer means the whole evaluation. `source_sha256` authenticates that file.
+`source_url` links the corresponding file at a fixed GitHub commit.
+
+All configurations preserve candidate, grader and case fingerprints and the
+recorded runtime. The case table additionally includes original check outcomes
+and `case_json`; this retains available support states and tool traces.
+The suite configuration, plan, all five attempts and JUnit are included under
+`evidence/examples/suite/`. Repetition summaries and all six attempts are included
+under their original example directories.
+
+The original records come from EvalArc 0.2, 0.4 and 0.5. Their
+`recorded_evalarc_version` and grader fingerprints are preserved. Do not treat
+the three configurations as matched version comparisons; use the lab's separate
+matched comparison for that question.
+
+Build source:
+[`@SOURCE_COMMIT@`](https://github.com/noteflowai/evalarc/tree/@SOURCE_COMMIT@).
+[`manifest.json`](manifest.json) lists every exported file hash.
+
+```bash
+git clone https://github.com/noteflowai/evalarc.git
+cd evalarc
+git checkout @SOURCE_COMMIT@
+python3 scripts/build_dataset.py --output dist/casebook
+```
+
+The build validates evaluations, recomputes repetitions and suite decisions,
+and checks JUnit before exporting. It does not rerun candidates. See the
+[methodology](https://github.com/noteflowai/evalarc/blob/@SOURCE_COMMIT@/docs/methodology.md)
+for the task contracts and limitations.
+
+## Intended uses and limits
+
+Use this small casebook to learn grader auditing, inspect retries and
+idempotency, test report readers, and discuss acceptance criteria. Reference
+implementations and deliberate faults were authored for these tasks; this is
+not an exhaustive collection of possible defects.
+
+The repeated controls show no observed check variation. They do not establish
+independence, a population reliability estimate or performance on unseen tasks.
+The dataset does not support model rankings, arbitrary reward-hack resistance
+or RL improvement claims. Public development cases should not be presented
+as a held-out benchmark. No hosted CI importer was exercised.
+
+## Authorship and license
+
+Published by the EvalArc maintainer, with AI-assisted development and writing.
+Code, task fixtures and these derived records are MIT-licensed; see
+[`LICENSE`](LICENSE). No third-party model weights are included.
+
+Related maintainer projects:
+[Robot Reel](https://huggingface.co/spaces/glayguo/robot-reel) inspects recorded
+Physical AI experiments;
+[Skills Anywhere](https://huggingface.co/spaces/glayguo/dsh-skills-anywhere)
+provides reusable skill discovery and file checks.
+The [project collection](https://huggingface.co/collections/glayguo/noteflow-ai-open-source-playgrounds-6aa693c382b0184786eb8856)
+groups these independent tools; it does not imply a shared model evaluation
+or upstream endorsement.

huggingface/README.md

@@ -28,6 +28,12 @@ the original TOML, and JUnit output distinguishing a failed gate from an
 environment error. Gate acceptance remains separate from full task resolution.
 A hosted CI importer was not exercised.
 
+**Explore the data as tables:** the
+[EvalArc Casebook](https://huggingface.co/datasets/glayguo/evalarc-casebook)
+offers three separate configurations for 167 audit cases, six repeated attempts
+and three suite jobs. Filter the results or load the JSONL in Python; original
+source records and fingerprints accompany every row.
+
 **Every v0.4 attempt remains visible.** Switch between three
 recorded Docker attempts of the reference and three of the duplicate-write
 control. The reference resolves 3/3 attempts; the faulty control resolves 0/3