fix(grade): skip incomplete command checks

docs/guides/judging.md

@@ -66,10 +66,19 @@ assertion set, so a benchmark can be read against the instrument that produced i
 cannot be read leaves the run-time copy in place with a warning; one that fails validation stops
 grading rather than measuring with assertions you have already replaced.
 
-Cached results are keyed by assertion id. Editing an assertion in place — rewording an `llm_judge`
-rubric, changing a `command_check` command — leaves the verdict or result the previous definition
-produced, and `grade` reports every one it reused. Re-run `eval-magic grade --overwrite` to
-re-execute command checks and `eval-magic dispatch --judges --overwrite` to re-judge.
+Cached judge responses are keyed by assertion id. Rewording an `llm_judge` rubric under the same id
+leaves the previous verdict in place; use `eval-magic dispatch --judges --overwrite` to re-judge it.
+
+Command-check results use both the authored definition digest and the exact `run.json` digest as
+their cache key. `grade` reuses only an exact match. A missing or mismatched digest executes the
+check again and replaces the result, so a legacy result, an edited check, or a changed run record
+refreshes without `eval-magic grade --overwrite`; that option forces even an exact match to execute
+again.
+
+A command check is eligible only after its task has a runner-owned `run.json`. Partial ingest leaves
+an incomplete task environment untouched: no held-out setup, command execution, or result write. If
+an incomplete task already carries a cached command-check result, `grade` warns that an older grader
+may have contaminated the environment. That task must not be resumed; build a fresh iteration.
 
 ## What the bundle contains
 

docs/progressive-enhancements.md

@@ -121,8 +121,9 @@ that gap — it is what a judge reads to answer whether the work was any good.
 
 `command_check` is intentionally not a harness enhancement. `run` detects the assertion before
 dispatch so it can validate held-out sources before building. After diff-scope capture, `ingest`
-copies the assertion's held-out `setup_files` from the skill's `evals/` directory into that root and
-executes the trusted command through the platform shell. Root `.git` paths are reserved for both
+grades only tasks with a runner-owned `run.json`, copies the assertion's held-out `setup_files` from
+the skill's `evals/` directory into that root, and executes the trusted command through the platform
+shell. Root `.git` paths are reserved for both
 visible overlays and held-out setup files, while nested repositories remain valid. The runner clears
 inherited Git routing variables before optional `env` values override the environment; optional
 `matrix` values execute every Cartesian-product cell and persist per-cell results. The files are

schema/command-check.schema.json

@@ -39,7 +39,11 @@
     },
     "definition_digest": {
       "type": "string",
-      "description": "Digest of the authored command_check this result came from. Reuse is keyed by assertion id, so this is what lets a later grade report that the check under that id has been edited since. Absent in results written before it was recorded, which make no claim either way."
+      "description": "Digest of the authored command_check this result came from. A result is reusable only when this and run_record_digest match the current check and run record. Absent in legacy results, which are rerun."
+    },
+    "run_record_digest": {
+      "type": "string",
+      "description": "Digest of the exact runner-owned run.json bytes this result graded. A result is reusable only when this and definition_digest match the current run record and check. Absent in legacy results, which are rerun."
     },
     "cells": {
       "type": "array",

src/cli/args.rs

@@ -679,8 +679,12 @@ pub(crate) enum Commands {
     /// `diff_scope` grading for finalize, injects held-out
     /// `command_check.setup_files`, and executes each
     /// runner-owned command check in its task environment, applying its
-    /// environment overrides and running every environment matrix cell. Diff
-    /// scope is captured before held-out files are injected. Then stops at the
+    /// environment overrides and running every environment matrix cell. A task
+    /// is eligible only after its runner-owned `run.json` exists, so a partial
+    /// ingest leaves incomplete task environments untouched. A cached result is
+    /// reused only when its authored definition and run-record digests match;
+    /// legacy or stale results are executed again. Diff scope is captured before
+    /// held-out files are injected. Then stops at the
     /// judge hand-off, writing one bounded `judge-evidence.md` per recorded run
     /// and listing the effective sample count of judge tasks per `llm_judge`
     /// assertion. The exact evidence bundle is shared by that run's tasks and
@@ -761,8 +765,11 @@ pub(crate) enum Commands {
     /// Grade captures scope before it injects
     /// held-out `command_check.setup_files` and executes each runner-owned command
     /// in its task environment, applying fixed environment overrides and running
-    /// every environment matrix cell; completed command and diff-scope results
-    /// are reused. Before emitting tasks, writes one `judge-evidence.md` beside
+    /// every environment matrix cell. Tasks without `run.json` are skipped before
+    /// setup injection or command execution. A command result is reused only when
+    /// its authored definition and run-record digests match; legacy or stale
+    /// results execute again. Completed diff-scope results are reused. Before
+    /// emitting tasks, writes one `judge-evidence.md` beside
     /// every recorded run. This 98,304-byte bounded bundle combines task context,
     /// completion state, diff evidence, conversation, tool summary, and source
     /// paths; its exact bytes are inlined into each run's LLM-judge prompts. The
@@ -793,10 +800,11 @@ pub(crate) enum Commands {
     /// assertions from the run's own evidence, after the dispatch they grade. Every
     /// invocation prints the file it read them from, and each `grading.json` records it
     /// under `assertion_source`. An unreadable live file leaves the run-time copy in
-    /// place with a warning; an invalid one stops grading. Cached judge verdicts and
-    /// command-check results are keyed by assertion id, so an assertion edited in place
-    /// is reported rather than silently reused: `--overwrite` re-executes command
-    /// checks; `dispatch --judges --overwrite` re-judges. See `eval-magic docs judging`.
+    /// place with a warning; an invalid one stops grading. Cached judge verdicts are
+    /// keyed by assertion id, so `dispatch --judges --overwrite` re-judges an edited
+    /// assertion. Command checks invalidate automatically when their definition or
+    /// run record changes; `grade --overwrite` also re-executes an exact cache match.
+    /// See `eval-magic docs judging`.
     Grade(GradeArgs),
     /// Aggregate before/after benchmark deltas.
     ///

src/cli/commands/pipeline.rs

@@ -342,10 +342,10 @@ pub(crate) fn run_grade(args: GradeArgs) -> anyhow::Result<()> {
             diffs.measured, diffs.reused, diffs.missing_baseline, diffs.shared_environment
         );
         let commands = pipeline::grade_command_checks(&dir, &instrument, common.overwrite)?;
-        if commands.executed + commands.reused > 0 {
+        if commands.executed + commands.reused + commands.skipped_incomplete > 0 {
             println!(
-                "Command checks: {} executed, {} reused, {} failed",
-                commands.executed, commands.reused, commands.failed
+                "Command checks: {} executed, {} reused, {} failed, {} skipped (missing run.json)",
+                commands.executed, commands.reused, commands.failed, commands.skipped_incomplete
             );
         }
         for w in &commands.warnings {

src/pipeline/grade/command_check.rs

@@ -14,8 +14,12 @@ use crate::pipeline::error::PipelineError;
 use crate::pipeline::grade::instrument::GradingInstrument;
 use crate::validation::{SchemaName, validate_against_schema};
 
+use cache::{definition_digest, has_cached_results, run_record_digest};
+
 const DIAGNOSTIC_LIMIT: usize = 2 * 1024;
 
+mod cache;
+
 /// The schema-gated intermediate result persisted before finalize converts it
 /// into a normal [`crate::core::AssertionResult`].
 #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -29,12 +33,14 @@ pub struct CommandCheckResult {
     pub stderr: String,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub cells: Option<Vec<CommandCheckCellResult>>,
-    /// Digest of the `command_check` this result came from. Reuse is keyed by
-    /// assertion id, so this is what tells a later grade that the check under
-    /// that id has been edited since. Absent in results that predate the record,
-    /// which make no claim either way.
+    /// Digest of the `command_check` this result came from. Reuse requires it to
+    /// match the current definition; an absent legacy value is never reusable.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub definition_digest: Option<String>,
+    /// Digest of the exact runner-owned `run.json` bytes this result graded.
+    /// Absent in legacy results, which are never reusable as a cache entry.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub run_record_digest: Option<String>,
 }
 
 /// The result of one environment-matrix cell.
@@ -53,8 +59,10 @@ pub struct CommandCheckSummary {
     pub executed: usize,
     pub reused: usize,
     pub failed: usize,
-    /// Reused results whose check has since been edited. Returned rather than
-    /// printed: the CLI handler owns how a warning reads.
+    /// Tasks skipped because their runner-owned `run.json` is absent.
+    pub skipped_incomplete: usize,
+    /// Safety warnings returned rather than printed: the CLI handler owns how
+    /// user-facing warnings read.
     pub warnings: Vec<String>,
 }
 
@@ -68,12 +76,14 @@ struct DispatchFile {
 struct DispatchTask {
     eval_id: String,
     condition: String,
+    #[serde(default)]
+    run_index: Option<u32>,
     eval_root: Option<String>,
     run_record_path: String,
 }
 
-/// Inject held-out setup files and execute all command checks in declaration
-/// order for every matching dispatch task.
+/// For each matching task with a completed run record, inject held-out setup
+/// files and execute all command checks in declaration order.
 pub fn grade_command_checks(
     iteration_dir: &Path,
     instrument: &GradingInstrument,
@@ -127,6 +137,26 @@ pub fn grade_command_checks(
             continue;
         }
 
+        let run_record_path = Path::new(&task.run_record_path);
+        let run_dir = run_record_path.parent().ok_or_else(|| {
+            PipelineError::Message(format!(
+                "command_check task '{}'/{} has no run directory in run_record_path",
+                task.eval_id, task.condition
+            ))
+        })?;
+        let results_dir = run_dir.join("command-checks");
+        if !run_record_path.exists() {
+            summary.skipped_incomplete += 1;
+            if has_cached_results(&results_dir)? {
+                summary.warnings.push(format!(
+                    "command_check results already exist for {}, but run.json is missing. An older eval-magic version may have copied held-out setup files or executed commands in this task environment; it may already be contaminated and must not be resumed. Build a fresh iteration.",
+                    task_label(task)
+                ));
+            }
+            continue;
+        }
+        let run_record_digest = run_record_digest(run_record_path)?;
+
         let eval_root = task
             .eval_root
             .as_deref()
@@ -138,13 +168,6 @@ pub fn grade_command_checks(
             ));
         }
         let eval_root = Path::new(eval_root);
-        let run_dir = Path::new(&task.run_record_path).parent().ok_or_else(|| {
-            PipelineError::Message(format!(
-                "command_check task '{}'/{} has no run directory in run_record_path",
-                task.eval_id, task.condition
-            ))
-        })?;
-        let results_dir = run_dir.join("command-checks");
 
         for check in checks {
             validate_assertion_id(&check.id)?;
@@ -157,21 +180,17 @@ pub fn grade_command_checks(
                     &value,
                     &result_path.to_string_lossy(),
                 )?;
-                if reused
-                    .definition_digest
-                    .is_some_and(|recorded| recorded != digest)
+                if reused.definition_digest.as_deref() == Some(digest.as_str())
+                    && reused.run_record_digest.as_deref() == Some(run_record_digest.as_str())
                 {
-                    summary.warnings.push(format!(
-                        "command_check '{}' for {}/{} changed since its cached result was produced; that result is reused as-is. Re-run grade with --overwrite to execute the edited check.",
-                        check.id, task.eval_id, task.condition
-                    ));
+                    summary.reused += 1;
+                    continue;
                 }
-                summary.reused += 1;
-                continue;
             }
 
             inject_setup_files(check, instrument.setup_root_for(&task.eval_id), eval_root)?;
-            let result = execute_command_check(check, eval_root)?;
+            let mut result = execute_command_check(check, eval_root)?;
+            result.run_record_digest = Some(run_record_digest.clone());
             if !result.passed {
                 summary.failed += 1;
             }
@@ -189,14 +208,12 @@ pub fn grade_command_checks(
     Ok(summary)
 }
 
-/// Digest of a check's authored definition, so reuse can tell an edited check
-/// from the one that produced the cached result.
-fn definition_digest(check: &AssertionCommandCheck) -> String {
-    crate::core::fs::fnv1a_hex(
-        serde_json::to_string(check)
-            .expect("an authored command_check serializes")
-            .as_bytes(),
-    )
+fn task_label(task: &DispatchTask) -> String {
+    let run = task
+        .run_index
+        .map(|index| format!("/run-{index}"))
+        .unwrap_or_default();
+    format!("{}/{}{run}", task.eval_id, task.condition)
 }
 
 fn isolation_error(task: &DispatchTask, detail: &str) -> PipelineError {
@@ -274,6 +291,7 @@ pub(super) fn execute_command_check(
             stderr: cell.stderr,
             cells: None,
             definition_digest: Some(definition_digest(assertion)),
+            run_record_digest: None,
         });
     };
 
@@ -312,6 +330,7 @@ pub(super) fn execute_command_check(
         stderr: String::new(),
         cells: Some(cells),
         definition_digest: Some(definition_digest(assertion)),
+        run_record_digest: None,
     })
 }
 

src/pipeline/grade/command_check/cache.rs

@@ -0,0 +1,40 @@
+//! Command-check cache identity and legacy-result detection.
+
+use std::fs;
+use std::path::Path;
+
+use crate::core::AssertionCommandCheck;
+use crate::pipeline::error::PipelineError;
+
+/// Digest of a check's authored definition, so reuse never crosses an edit.
+pub(super) fn definition_digest(check: &AssertionCommandCheck) -> String {
+    crate::core::fs::fnv1a_hex(
+        serde_json::to_string(check)
+            .expect("an authored command_check serializes")
+            .as_bytes(),
+    )
+}
+
+/// Digest the exact runner-owned record bytes that a result grades.
+pub(super) fn run_record_digest(path: &Path) -> Result<String, PipelineError> {
+    Ok(crate::core::fs::fnv1a_hex(&fs::read(path)?))
+}
+
+/// True when an incomplete task already carries a result left by an older
+/// grader. Presence alone is enough for the safety warning: a malformed legacy
+/// result still means held-out setup or its command may have touched the env.
+pub(super) fn has_cached_results(results_dir: &Path) -> Result<bool, PipelineError> {
+    if !results_dir.exists() {
+        return Ok(false);
+    }
+    for entry in fs::read_dir(results_dir)? {
+        if entry?
+            .path()
+            .extension()
+            .is_some_and(|extension| extension == "json")
+        {
+            return Ok(true);
+        }
+    }
+    Ok(false)
+}