feat(diff-scope): measure and capture diffs with Git instead of copying a baseline

Cargo.toml

@@ -44,14 +44,12 @@ jsonschema = { version = "0.46.5", default-features = false }
 regex = { version = "1.12.3", default-features = false, features = ["std", "perf"] }
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = { version = "1.0.150", features = ["preserve_order"] }
-similar = { version = "3.1.1", default-features = false }
 tempfile = "3.27.0"
 thiserror = "2.0.18"
 # Harness descriptor files (harnesses/*.toml). `display` serializes the
 # resolved (layer-merged) descriptor back to authorable TOML for
 # `harness show`.
 toml = { version = "0.9", default-features = false, features = ["parse", "serde", "display"] }
-walkdir = "2.5.0"
 
 [dev-dependencies]
 assert_cmd = "2.2.2"

docs/guides/codebase.md

@@ -68,6 +68,34 @@ Each dispatch gets its own private environment holding:
 An eval that declares no `codebase` still gets a Git repository, initialized on `work`, exactly as
 it always has.
 
+## The baseline ref is what the run is measured against
+
+Nothing writes into an environment after that ref is written, so it names exactly what the agent
+started from — and everything the agent did is the difference from it.
+
+During `ingest`, Git measures that difference. Each run gets:
+
+- `diff-scope.json` — `files_touched`, `lines_added`, `lines_removed`, and `hunks`, plus the list of
+  changed files with a status of `added`, `modified`, or `deleted`
+- `diff.patch` — the diff itself, which is the evidence a judge reads to answer whether the work was
+  any good. It always exists; for a run that changed nothing it is empty. A diff past the capture
+  cap is cut at a line boundary and carries a marker saying so, and `patch.truncated` in
+  `diff-scope.json` records it.
+
+What counts is what Git counts, under the same rules the baseline commit was built under:
+
+- The codebase's own `.gitignore` holds, so a run that compiles does not report its build output as
+  thousands of touched files.
+- Fixtures and staged skills count even when the codebase ignores their paths — they are committed
+  into the baseline regardless, so a change to one is always visible.
+- Framework artifacts under `.eval-magic-outputs/` never count.
+- A nested repository's internals never count: Git tracks no path with a `.git` component.
+- A rename counts as two touched files, one created and one deleted.
+- A binary file counts as one touched file, contributing no lines.
+
+A `diff_scope` assertion gates `max_files_touched`, `max_lines_changed` (added plus removed), or
+both, against exactly these numbers.
+
 ## One checkout per iteration
 
 Every environment a run provisions — each `(eval, condition, run)` cell — is built from one cached
@@ -103,7 +131,8 @@ paths. Seeding a task-specific file into a real project is the common case:
 A fixture overwrites a codebase file of the same path.
 
 The baseline the runner commits respects the codebase's `.gitignore`, so ignored build output stays
-out of it. Fixtures and staged skills are committed regardless of what the codebase ignores.
+out of it. Fixtures and staged skills are committed regardless of what the codebase ignores — which
+is also what keeps them inside every later measurement.
 
 ## A `path` source is not reproducible elsewhere
 
@@ -133,6 +162,16 @@ git status --porcelain
 `git remote -v` and `git status --porcelain` are both empty, and the two revisions match: the
 baseline ref names exactly what the agent started from.
 
+After a dispatch and `ingest`, read what the run produced:
+
+```sh
+jq '{files_touched, lines_added, lines_removed, hunks, files, patch}' diff-scope.json
+head -50 diff.patch
+```
+
+The same difference, spelled by Git itself, is `git diff refs/eval-magic/baseline` inside the
+environment.
+
 The resolved commit appears in `conditions.json`, each `run.json`, `benchmark.json`, and the
 `BASELINE.md` written by `promote-baseline` — alongside the skill the run measured, which
 is recorded the same way:

docs/progressive-enhancements.md

@@ -37,8 +37,9 @@ A harness qualifies at baseline with no harness-specific code beyond naming itse
 
 That baseline already yields a working eval: `llm_judge` assertions grade soft behavior,
 runner-owned `command_check` assertions can inject held-out files and execute deterministically,
-runner-owned final-environment metrics land in `diff-scope.json`, `diff_scope` assertions gate
-files/lines deterministically, and the `detect-stray-writes` post-pass (folded into `ingest`) audits
+runner-owned final-environment metrics land in `diff-scope.json` with the diff itself in
+`diff.patch`, `diff_scope` assertions gate files/lines deterministically, and the
+`detect-stray-writes` post-pass (folded into `ingest`) audits
 writes that leave the private task environment. Run records without transcript ingest are assembled
 from `outputs/final-message.md` or by hand per `schema/run-record.schema.json`.
 
@@ -91,19 +92,34 @@ generic fresh-session fallback can preserve the meaning of a canned reply.
 ## Runner-owned environment checks are baseline
 
 Every canonical `(eval, condition, run)` gets a distinct `eval_root`. After fixtures, staging, and
-guard installation, `run` recreates a runner-owned Git repository at that root, commits the task
-state on branch `work`, runs shadow preflight at the resulting repository boundary, and snapshots
-the task environment. Git is therefore a runtime prerequisite; each task starts clean and has no
-remotes. During `ingest`, before any held-out setup is injected, the runner compares that baseline
-with the final environment and writes raw `files_touched`, `lines_added`, `lines_removed`, and
-zero-context Myers `hunks` to `diff-scope.json`. Framework artifacts under the task root's
-`.eval-magic-outputs/` and runner-owned `.git/` are excluded; nested repository metadata and all
-other new files count. `benchmark.json` preserves these metrics per run even without a `diff_scope`
-assertion. An assertion may gate `max_files_touched`, `max_lines_changed` (added plus removed), or
-both.
+guard installation, `run` establishes a runner-owned Git repository at that root, commits the task
+state, marks it with `refs/eval-magic/baseline`, and runs shadow preflight at the resulting
+repository boundary. Git is therefore a runtime prerequisite; each task starts clean and has no
+remotes. Nothing writes into an environment after the ref is written, so it names exactly what the
+agent started from.
+
+During `ingest`, before any held-out setup is injected, Git measures the final environment against
+that ref. The runner seeds a scratch index from the baseline, brings it up to the working tree with
+one `git add`, and diffs the two trees — so creations, modifications, and deletions all fall out of
+one pass, and an untracked creation is not missed. Raw `files_touched`, `lines_added`,
+`lines_removed`, and zero-context `hunks` go to `diff-scope.json`, alongside the changed-file list;
+the diff itself goes to `diff.patch` beside it, capped and marked when a diff exceeds the cap.
+`benchmark.json` preserves the metrics per run even without a `diff_scope` assertion. An assertion
+may gate `max_files_touched`, `max_lines_changed` (added plus removed), or both.
+
+**What counts is what Git counts.** The measurement runs under the same rules the baseline commit
+was built under: the codebase's own `.gitignore` holds, so a run that compiles does not report its
+build output as thousands of touched files, and the `.git/info/exclude` entry keeps framework
+artifacts under `.eval-magic-outputs/` out. Paths the runner force-added despite those rules — the
+harness config directories and the declared fixture overlay — are tracked in the baseline and stay
+measured. Git indexes no path with a `.git` component, so a nested repository's internals are
+invisible, not just the runner-owned root `.git`. Renames are switched off deliberately: a rename is
+two touched files, one created and one deleted, which is what the metric has always meant. A binary
+file counts as one touched file with no countable lines.
 
 This is deliberately a secondary signal: a smaller diff can be focused, but it can also be
-incomplete. Pair a scope gate with a correctness assertion.
+incomplete. Pair a scope gate with a correctness assertion. The patch is the evidence that closes
+that gap — it is what a judge reads to answer whether the work was any good.
 
 `command_check` is intentionally not a harness enhancement. `run` detects the assertion before
 dispatch so it can validate held-out sources before building. After diff-scope capture, `ingest`

schema/diff-scope.schema.json

@@ -2,14 +2,40 @@
   "$schema": "http://json-schema.org/draft-07/schema#",
   "$id": "https://slow-powers.dev/schemas/diff-scope.schema.json",
   "title": "Diff Scope Metrics",
-  "description": "Runner-owned final-environment diff metrics for one eval run. Compares the complete task environment with its post-staging, post-guard baseline while excluding .eval-magic-outputs framework artifacts. Lives beside run.json as diff-scope.json.",
+  "description": "Runner-owned final-environment diff evidence for one eval run. Git measures the complete task environment against the refs/eval-magic/baseline ref its environment was marked with, honoring the codebase's own .gitignore and the .eval-magic-outputs framework exclusion. Lives beside run.json as diff-scope.json, with the diff itself in diff.patch.",
   "type": "object",
   "required": ["files_touched", "lines_added", "lines_removed", "hunks"],
   "additionalProperties": false,
   "properties": {
     "files_touched": { "type": "integer", "minimum": 0 },
-    "lines_added": { "type": "integer", "minimum": 0, "description": "Byte-lines inserted by a Myers diff." },
-    "lines_removed": { "type": "integer", "minimum": 0, "description": "Byte-lines deleted by a Myers diff." },
-    "hunks": { "type": "integer", "minimum": 0, "description": "Contiguous non-equal operation groups, with zero context." }
+    "lines_added": { "type": "integer", "minimum": 0, "description": "Lines inserted, as git diff --numstat counts them. A binary file contributes none." },
+    "lines_removed": { "type": "integer", "minimum": 0, "description": "Lines deleted, as git diff --numstat counts them. A binary file contributes none." },
+    "hunks": { "type": "integer", "minimum": 0, "description": "Contiguous non-equal operation groups, counted at zero context." },
+    "files": {
+      "type": "array",
+      "description": "Every changed file, ordered by path as Git reports them. Omitted for iterations created before the changed-file list.",
+      "items": {
+        "type": "object",
+        "required": ["path", "status", "lines_added", "lines_removed"],
+        "additionalProperties": false,
+        "properties": {
+          "path": { "type": "string", "description": "Environment-relative path, spelled with forward slashes as Git spells it." },
+          "status": { "type": "string", "enum": ["added", "modified", "deleted"] },
+          "lines_added": { "type": "integer", "minimum": 0 },
+          "lines_removed": { "type": "integer", "minimum": 0 }
+        }
+      }
+    },
+    "patch": {
+      "type": "object",
+      "description": "The captured diff beside this record. Omitted for iterations created before patch capture.",
+      "required": ["path", "bytes", "truncated"],
+      "additionalProperties": false,
+      "properties": {
+        "path": { "type": "string", "description": "Run-relative name of the patch file." },
+        "bytes": { "type": "integer", "minimum": 0, "description": "Size of the written patch, including any truncation marker." },
+        "truncated": { "type": "boolean", "description": "True when the diff exceeded the capture cap and the file carries a marker in place of the rest." }
+      }
+    }
   }
 }

schema/evals.schema.json

@@ -278,12 +278,12 @@
         "max_files_touched": {
           "type": "integer",
           "minimum": 0,
-          "description": "Maximum number of changed, deleted, or newly-created files allowed in the final task environment. Framework files under the task root's .eval-magic-outputs and runner-owned .git are excluded; nested .git metadata remains measurable."
+          "description": "Maximum number of changed, deleted, or newly-created files allowed in the final task environment, as Git reports them against the refs/eval-magic/baseline ref. The codebase's own .gitignore applies, so ignored build output does not count; framework files under .eval-magic-outputs and anything under a .git directory never count; a rename counts as two files."
         },
         "max_lines_changed": {
           "type": "integer",
           "minimum": 0,
-          "description": "Maximum total byte-lines added plus byte-lines removed allowed. Diffing uses Myers operations with zero-context hunks."
+          "description": "Maximum total lines added plus lines removed allowed, as git diff --numstat counts them. A binary file contributes no lines. Hunks are counted at zero context."
         }
       }
     }

src/cli/args.rs

@@ -624,8 +624,10 @@ pub(crate) enum Commands {
     /// grade. Assembles each task's `run.json` + `timing.json`, scans for stray
     /// writes, and maps raw per-env guard logs through `dispatch.json` into
     /// `guard-denials.json` (including tasks without `run.json`). Malformed raw
-    /// records fail with their source path and line number. It captures always-on
-    /// final-environment files/lines/hunks in `diff-scope.json`, grades
+    /// records fail with their source path and line number. It measures the
+    /// finished environment against the `eval-magic/baseline` ref it was marked
+    /// with, writing always-on files/lines/hunks and the changed-file list to
+    /// `diff-scope.json` and the diff itself to `diff.patch`, grades
     /// `transcript_check` assertions, prepares
     /// `diff_scope` grading for finalize, injects held-out
     /// `command_check.setup_files`, and executes each
@@ -645,7 +647,9 @@ pub(crate) enum Commands {
     /// runner-owned `command_check` results, and deterministic `diff_scope`
     /// files/lines thresholds into normal `grading.json` files, then writes
     /// `benchmark.json` with a per-assertion `passed`/`n` rollup from observed
-    /// assertion results and raw per-run metrics from `diff-scope.json`. If a live
+    /// assertion results and raw per-run metrics from `diff-scope.json`. The
+    /// per-run changed-file list and `diff.patch` stay beside each run rather
+    /// than being rolled up. If a live
     /// guard remains armed — the cwd guard, or any per-task Cli env guard — prints
     /// a `teardown` reminder before source edits. Requires `--iteration`.
     Finalize(CommonArgs),
@@ -698,12 +702,16 @@ pub(crate) enum Commands {
     DetectStrayWrites(CommonArgs),
     /// Grade run records (runner checks + LLM-judge task emission).
     ///
-    /// Captures always-on final-environment files/lines/hunks in `diff-scope.json`
+    /// Captures always-on final-environment files/lines/hunks plus the
+    /// changed-file list in `diff-scope.json`, writes the diff itself to
+    /// `diff.patch` beside it (truncated with a marker past its size cap),
     /// and evaluates `transcript_check` assertions directly: regex against
     /// tool invocations or, for scripted evals, assistant messages across rounds.
     /// Checks can require a match before the final completion claim or before the
     /// first write/patch tool call. A `diff_scope` assertion gates the captured file count
-    /// and/or added-plus-removed line count. Grade captures scope before it injects
+    /// and/or added-plus-removed line count. Git supplies both, so the codebase's
+    /// own `.gitignore` decides what counts and ignored build output stays out.
+    /// Grade captures scope before it injects
     /// held-out `command_check.setup_files` and executes each runner-owned command
     /// in its task environment, applying fixed environment overrides and running
     /// every environment matrix cell; completed command and diff-scope results
@@ -729,7 +737,8 @@ pub(crate) enum Commands {
     /// grouped findings in schema-v2 `plugin-shadow.json` (legacy unversioned
     /// reports remain readable) unless it records the resolved descriptor's
     /// `isolates_live_sources = true` assertion), and raw per-run files/lines/hunks
-    /// from `diff-scope.json`. Shadow findings retain their intrinsic warning or
+    /// from `diff-scope.json`. Each run's changed-file list and its `diff.patch`
+    /// stay in the run directory. Shadow findings retain their intrinsic warning or
     /// comparison-invalid severity, per-cell appearances, resolution, and
     /// remediation. A timing metric with `n: 0` is unavailable, not a measured
     /// zero. The top-level `diff_scope` field is omitted for compatible older

src/cli/run/orchestrate/build.rs

@@ -416,10 +416,12 @@ pub(super) fn post_build(
     // exist, but before project-local skill discovery inspects ancestor state.
     // Recreating `.git` also resets explicit iteration rebuilds to one clean,
     // runner-owned baseline with no inherited history or remotes.
+    //
+    // This is also where the diff baseline is captured: the `eval-magic/baseline`
+    // ref written here marks the state every later measurement is the difference
+    // from. Nothing below writes into an environment, so the ref stays exact.
     super::git::initialize_task_repositories(ctx, r)?;
 
     super::shadow_preflight::run(ctx, opts, r, staged, &targets)?;
-    crate::pipeline::capture_iteration_baselines(&r.iteration_dir)
-        .map_err(|error| RunError::msg(error.to_string()))?;
     Ok(())
 }