Verify received evaluation evidence without executing candidates

.github/workflows/ci.yml

@@ -26,6 +26,13 @@ jobs:
       - run: ruff format --check .
       - run: pytest -q
       - run: python -m build
+      - name: Verify received evidence from the installed wheel without runtime tools
+        run: |
+          python -m venv "$RUNNER_TEMP/installed"
+          "$RUNNER_TEMP/installed/bin/pip" install --no-deps dist/*.whl
+          cd "$RUNNER_TEMP"
+          PYTHONPATH="" "$RUNNER_TEMP/installed/bin/python" \
+            "$GITHUB_WORKSPACE/scripts/check_installed.py" --source "$GITHUB_WORKSPACE"
   docker-audit:
     runs-on: ubuntu-latest
     strategy:

CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## 0.7.0 · 2026-09-14 · Research preview
+
+- Add `evalarc verify` for received evaluation, repetition and comparison
+  records. Recompute summaries and source identities without candidate
+  execution, a source checkout, Docker or the original interpreter.
+- Return machine-readable results and hashes of every checked JSON input.
+  Separate consistency success from `--require-resolved` acceptance.
+- Bound input sizes and attempt inventory; reject duplicate keys, non-finite
+  numbers, symlinks and special files. Keep historical evidence readable.
+- Document unsupported aggregate formats and the distinction between record
+  consistency, independent grader execution and producer authentication.
+
+Task contracts, runtime enforcement and scoring rules are unchanged. This
+release verifies existing evidence; it does not add new model trials.
+
+
 ## 0.6.0 · 2026-09-14 · Research preview
 
 - `init --language python|javascript` supplies starters and references for both

CITATION.cff

@@ -4,7 +4,7 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
 type: software
 authors:
   - name: EvalArc contributors
-version: 0.6.0
+version: 0.7.0
 date-released: 2026-09-14
 license: MIT
 repository-code: "https://github.com/noteflowai/evalarc"

README.md

@@ -85,6 +85,11 @@ ambiguous write outcomes. Python and JavaScript scripted policies use the same
 host verifier. Browser environments, LLM-provider adapters, and RL training
 integrations remain planned. No frontier-model benchmark result is claimed.
 
+**Received a report? Verify it without running the candidate.**
+`evalarc verify path/to/report --json` checks evaluation, repetition or comparison
+evidence and fingerprints every input. Use `--require-resolved` when your handoff
+also requires all checks to pass. [Verification and limits](docs/verification.md).
+
 ## Run an audit
 
 Clone the source, then install in an isolated Python environment:

README.zh-CN.md

@@ -75,6 +75,11 @@ Docker 评测：参考策略 3/3 轮完全通过，重复写入策略虽然平
 
 当前版本尚未经过前沿模型、真实人类工时或强化学习收益标定。
 
+**收到报告后，先独立复核。** `evalarc verify 报告路径 --json` 无需执行候选程序，
+即可重算单次评测、重复运行和前后对照的汇总，并记录每份输入的指纹。
+需要全部任务通过时加 `--require-resolved`。
+[使用说明与校验范围](docs/verification.md)。
+
 ## 直接运行
 
 克隆仓库后，在独立 Python 环境中安装：

docs/verification.md

@@ -0,0 +1,58 @@
+# Verify received evaluation evidence
+
+EvalArc 0.7 adds a read-only handoff check. A customer or CI job can recompute a
+report's claims without installing Node, contacting Docker, locating the
+original candidate or executing its command.
+
+```bash
+evalarc verify received/evaluation.json --json
+evalarc verify received/repetition --json
+evalarc verify received/comparison --json
+```
+
+An evaluation can have any filename. For a repetition, keep `repetition.json`
+with `attempts/0001/evaluation.json`, `0002/evaluation.json`, and so on. For a
+comparison, keep `comparison.json`, `baseline.json` and `current.json` together.
+Pass either the summary file or its directory. A directory must contain exactly
+one supported report type.
+
+The result includes a SHA-256 and byte length for every JSON file actually
+checked. These hashes identify the handoff bytes, including historical package
+versions. It does not rewrite reports or execute commands embedded in metadata.
+
+| Evidence | Recomputed checks |
+| --- | --- |
+| Evaluation v2 | Case/check counts, dimensions, weighted score, validity and resolution |
+| Repetition v1 | Exact attempt inventory, input identities, all aggregate counts, scores and variance |
+| Comparison v1 | Matching task/runtime/grader/cases, score delta, individual regressions and improvements |
+
+Malformed or contradictory evidence exits **2**, with `verified: false` in JSON
+mode. Default exit **0** means the records are internally consistent, even if
+the recorded candidate failed or had an environment error.
+
+To require full resolution as well:
+
+```bash
+evalarc verify received/repetition --json --require-resolved
+```
+
+This exits **0** for valid, fully resolved records, **1** for valid but unresolved
+records, and **2** for invalid records or a verification error. For a comparison,
+full resolution refers to its current evaluation; `has_regressions` is also
+reported separately. This flag is not a replacement for a suite's custom gates.
+
+Suite manifests, custom gate decisions, JUnit, audit summaries and trajectories
+are not currently standalone supported inputs. For a suite, verify each
+`jobs/<id>` repetition directory; this checks its attempts but does not verify
+the suite-level gate decision. The published site additionally checks the
+featured suite against its original configuration and JUnit.
+
+Input files must be regular files without symlinks in the evidence path.
+Reads are limited to 64 MiB per JSON and 256 MiB per handoff. Repetitions contain
+at most 100 attempts. Duplicate JSON keys and non-finite values are rejected.
+Original HTML, logs and source programs are outside the verified inventory.
+
+This is record consistency, not an independent rerun of the business-state
+grader or authentication of the report's producer. A coordinated fabrication
+can be internally consistent. Re-run trusted grading on the candidate when
+execution evidence is required.

huggingface/README.md

@@ -85,3 +85,10 @@ cases do not establish reliability on unseen tasks or a model success rate.
 The source and evidence are MIT licensed. `manifest.json` identifies the source
 commit and SHA-256 of each published file. Publication is performed by the
 maintainer and does not imply endorsement by Hugging Face.
+
+### Verify a handoff offline
+
+With EvalArc 0.7+, run `evalarc verify path/to/report --json` on an evaluation,
+repetition or comparison report. It checks the recorded inputs without executing
+a candidate; `--require-resolved` separately requires fully resolved results.
+[Workflow and limits](https://github.com/noteflowai/evalarc/blob/main/docs/verification.md).

package.json

@@ -1,6 +1,6 @@
 {
   "name": "evalarc-evidence-site",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "private": true,
   "description": "Browser checks for the static EvalArc evidence explorer",
   "scripts": {

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "evalarc"
-version = "0.6.0"
+version = "0.7.0"
 description = "Auditable task environments and evaluations for coding and tool-using agents."
 readme = "README.md"
 requires-python = ">=3.11"

scripts/check_installed.py

@@ -0,0 +1,74 @@
+"""Exercise the installed verifier outside the checkout, with no tools on PATH."""
+
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import tomllib
+from importlib.metadata import version
+from pathlib import Path
+
+
+def check(source: Path) -> dict:
+    import evalarc
+
+    source = source.resolve()
+    module = Path(evalarc.__file__).resolve()
+    if module.is_relative_to(source) or Path.cwd().is_relative_to(source):
+        raise ValueError("use a non-editable wheel installation outside the checkout")
+    expected = tomllib.loads((source / "pyproject.toml").read_text())["project"]["version"]
+    if version("evalarc") != expected:
+        raise ValueError("installed version differs from the source")
+    results = {}
+    with tempfile.TemporaryDirectory(prefix="evalarc-handoff-") as temporary:
+        folder = Path(temporary)
+        environment = {**os.environ, "PATH": "", "PYTHONPATH": "", "PYTHONNOUSERSITE": "1"}
+        entry = str(Path(sys.executable).with_name("evalarc"))
+
+        def run(path: Path, code: int, *extra: str) -> dict:
+            process = subprocess.run(
+                [entry, "verify", str(path), "--json", *extra],
+                cwd=folder,
+                env=environment,
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+            if process.returncode != code:
+                raise ValueError(f"unexpected verifier exit: {process.stdout} {process.stderr}")
+            return json.loads(process.stdout)
+
+        def hashes() -> dict:
+            return {
+                p.relative_to(folder).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
+                for p in folder.rglob("*")
+                if p.is_file()
+            }
+
+        for name in ("evaluation", "repetition", "repetition-faulty", "comparison"):
+            target = folder / name
+            shutil.copytree(source / "examples" / name, target)
+            before = hashes()
+            result = run(target, 0)
+            if not result["verified"] or hashes() != before:
+                raise ValueError("verification failed or changed evidence")
+            results[name] = result
+        run(folder / "repetition", 0, "--require-resolved")
+        run(folder / "repetition-faulty", 1, "--require-resolved")
+        changed = folder / "comparison" / "comparison.json"
+        data = json.loads(changed.read_text())
+        data["score_delta"] = 0.123
+        changed.write_text(json.dumps(data))
+        if run(changed, 2)["verified"]:
+            raise ValueError("changed summary was accepted")
+    return {"version": expected, "module": str(module), "path_empty": True, "verified": results}
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--source", type=Path, required=True)
+    print(json.dumps(check(parser.parse_args().source), indent=2))

site/index.html

@@ -149,6 +149,14 @@ <h1>Look past<br>the <em>score.</em></h1>
       </div>
       <noscript><p>This explorer requires JavaScript. Read the standalone <a href="support/index.html">support report</a> or <a href="coding/index.html">coding report</a> without it.</p></noscript>
     </section>
+    <section class="reproduce" aria-label="Verify a received report">
+      <div><p class="eyebrow">HAND OFF EVIDENCE</p><h2>Received a report?<br>Check its claims.</h2><p>Recompute recorded scores, repeated outcomes and regressions without executing the candidate.</p><a href="https://github.com/noteflowai/evalarc/blob/main/docs/verification.md">Offline verification guide &#8599;</a></div>
+      <div class="terminal"><div>EVALARC 0.7+ / READ-ONLY</div><pre><code>evalarc verify received/repetition --json
+
+# Also require valid, fully resolved results:
+evalarc verify received/repetition \
+  --json --require-resolved</code></pre><p>Consistency is distinct from task success. No grader rerun or producer authentication is implied.</p></div>
+    </section>
     <section class="reproduce">
       <div><p class="eyebrow">FROM THE BROWSER TO YOUR TERMINAL</p><h2>Make the grader<br>earn your trust.</h2><p>Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.</p><a href="https://github.com/noteflowai/evalarc#run-an-audit">Installation &amp; execution guide &#8599;</a></div>
       <div class="terminal"><div>TRUSTED BUNDLED CONTROLS / LINUX / PYTHON 3.11+</div><pre><code>git clone https://github.com/noteflowai/evalarc.git

src/evalarc/__init__.py

@@ -1,3 +1,3 @@
 """Auditable evaluations for AI agents."""
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"

src/evalarc/cli.py

@@ -23,6 +23,7 @@
 from evalarc.tasks import TASKS
 from evalarc.templates import LANGUAGES, initialize
 from evalarc.trajectory import summarize
+from evalarc.verify import SCOPE, verify
 
 
 def parser() -> argparse.ArgumentParser:
@@ -100,11 +101,46 @@ def parser() -> argparse.ArgumentParser:
     suite.add_argument("--docker-command", default=os.getenv("EVALARC_DOCKER", "docker"))
     suite.add_argument("--progress", action="store_true")
     suite.add_argument("--output", type=Path, default=Path("runs/suite"))
+    verification = commands.add_parser(
+        "verify", help="check saved evaluation, repetition or comparison evidence without execution"
+    )
+    verification.add_argument("evidence", type=Path, help="report JSON or its containing directory")
+    verification.add_argument("--json", action="store_true")
+    verification.add_argument(
+        "--require-resolved",
+        action="store_true",
+        help="also require valid, fully resolved results (current result for comparisons)",
+    )
     return root
 
 
 def main(argv: list[str] | None = None) -> int:
     args = parser().parse_args(argv)
+    if args.command == "verify":
+        try:
+            result = verify(args.evidence)
+            code = 0
+            if args.require_resolved:
+                code = 2 if not result["records_valid"] else (0 if result["fully_resolved"] else 1)
+        except (OSError, ValueError, KeyError, TypeError, IndexError, OverflowError) as error:
+            result = {
+                "schema_version": "evalarc.verification.v1",
+                "verified": False,
+                "error": str(error),
+                "scope": SCOPE,
+            }
+            code = 2
+        if args.json:
+            print(json.dumps(result, indent=2))
+        elif result["verified"]:
+            print(
+                f"Verified {result['kind']}: {len(result['files'])} JSON files | "
+                f"Valid records: {result['records_valid']} | "
+                f"Fully resolved: {result['fully_resolved']}\n{SCOPE}"
+            )
+        else:
+            print(f"Verification failed: {result['error']}", file=sys.stderr)
+        return code
     try:
         if args.command == "suite":
             plan = load_suite(args.config)

src/evalarc/records.py

@@ -4,6 +4,8 @@
 
 import json
 import math
+import os
+import stat
 from pathlib import Path
 
 MAX_REPORT_BYTES = 64 * 1024 * 1024
@@ -30,16 +32,30 @@ def _object(pairs: list[tuple[str, object]]) -> dict:
     return result
 
 
-def read_evaluation(path: Path) -> dict:
-    with path.open("rb") as source:
-        content = source.read(MAX_REPORT_BYTES + 1)
-    if len(content) > MAX_REPORT_BYTES:
+def read_json(path: Path, *, limit: int = MAX_REPORT_BYTES) -> tuple[dict, bytes]:
+    """Read finite, unambiguous JSON from a bounded regular file."""
+    fd = os.open(path, os.O_RDONLY | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_NOFOLLOW", 0))
+    with os.fdopen(fd, "rb") as source:
+        info = os.fstat(source.fileno())
+        if not stat.S_ISREG(info.st_mode):
+            raise ValueError("report must be a regular file")
+        if info.st_size > limit:
+            raise ValueError("evaluation report exceeds the 64 MiB read limit")
+        content = source.read(limit + 1)
+    if len(content) > limit:
         raise ValueError("evaluation report exceeds the 64 MiB read limit")
     try:
         data = json.loads(content, object_pairs_hook=_object)
         json.dumps(data, allow_nan=False)
+        if not isinstance(data, dict):
+            raise ValueError("report must be an object")
     except (ValueError, UnicodeDecodeError, RecursionError) as error:
         raise ValueError(f"cannot read finite, unambiguous report JSON: {error}") from error
+    return data, content
+
+
+def read_evaluation(path: Path) -> dict:
+    data, _ = read_json(path)
     validate_evaluation(data)
     return data