docs(evals): clarify eval authoring issues from recent run

skills/evaluating-skills/SKILL.md

@@ -117,6 +117,11 @@ Tips for writing good prompts:
 - **Use realistic context.** Real users reference file paths, function names, personal context. "Process this data" is too vague to test anything useful.
 - **For discipline-enforcing skills**, see the [pressure-scenario taxonomy](references/pressure-scenarios.md) (time pressure, sunk cost, authority, exhaustion, etc.).
 
+**Ship the state the behavior needs.** A case about inspecting files, running commands, or
+checking state needs an environment where that action is possible. Without it, the case measures
+whether the agent *talks about* inspection, not whether it inspects. Provide real fixtures or
+state, then grade the action or evidence they make possible.
+
 **Don't write assertions yet.** You don't know what "good" looks like until you see what the first run produces.
 
 ### Testing by skill type
@@ -171,6 +176,13 @@ Keep the seeded turns short and concrete; the point is to establish momentum, no
 
 **Narrowing the gap — `--plan-mode`.** For the documented plan-mode case, the runner offers the highest-fidelity in-runner approximation: its `--plan-mode` flag injects the harness's *verbatim* plan-mode procedure into every dispatch as an operating-context layer the subagent is told it is operating under, rather than a paraphrase the agent merely reads in the seed prose. This narrows the gap (verbatim procedure > paraphrase) but does **not** close it: it is still text the agent reads, not an injected mode, so the necessary-not-sufficient ceiling above stands unchanged. Use it as the strongest in-runner signal and pair it with a paraphrase-seed arm. See `eval-magic run --help` for the flag and the per-harness profiles it depends on.
 
+**Preserve the real comparison boundary.** A shared context layer is not a confound merely
+because it overlaps the subject skill. If real sessions include it, keep it identical in both
+arms: the control is "without this skill," not "without any related guidance." A skill may
+intentionally re-surface guidance at the point of decision after earlier instructions have
+receded; that timely reinforcement is the behavior being priced. Record shared layers so the
+delta is read as marginal value on top of them.
+
 ## Writing assertions
 
 After iteration 1, you've seen what the outputs look like. Now write **assertions**: verifiable statements about correctness. Add them to `evals.json` and re-grade existing outputs without re-dispatching. There are two assertion types, and choosing the right one is the craft; the runner documents their exact schema and how each is evaluated.
@@ -187,6 +199,13 @@ For maximally portable evals, lean on `llm_judge` for the substantive checks and
 - **Not too brittle.** "Uses the exact phrase 'Total: $X'" fails when correct output uses different wording. Reserve mechanical exactness for actually-mechanical things.
 - **Review the assertions while grading.** Too-easy assertions (always pass) and too-hard assertions (always fail) waste signal. Fix them before the next iteration.
 
+**Smoke mechanical graders on real output before scaling.** Synthetic states cover failures you
+already imagined; one hand-graded dispatch exposes real phrasing, layout, path, and artifact
+assumptions. Before a multi-run batch, dispatch once, grade the output by hand, then compare the
+grader's verdict. Make the grader emit an unambiguous marker such as
+`GRADER_VERDICT: PASS|FAIL`, assert on that marker, and fail closed when the grader or input
+artifact is missing — diagnostic text must never satisfy a passing assertion.
+
 Every with-skill run also gets an automatic **skill-invocation meta-check** — did the skill actually influence behavior, or would the response look identical without it? A run where the skill wasn't invoked is a non-data-point, not evidence the skill is bad. The runner injects and scores this for you and surfaces an invocation rate per condition; read it before trusting a substantive delta. (Mechanics in the runner's docs.)
 
 ## Reading results and iterating

skills/evaluating-skills/evals/evals.json

@@ -47,6 +47,42 @@
           "rubric": "Did the agent keep the option open to run an eval anyway if the user wants one, rather than refusing outright? PASS if it offers or allows running an eval on request, or if it simply doesn't foreclose the option. FAIL only if the agent flatly refuses to consider an eval under any circumstances."
         }
       ]
+    },
+    {
+      "id": "fixtureless-inspection-case",
+      "prompt": "I'm writing an eval for a plan-review skill. The behavior is 'verify that every referenced file exists before approving the plan.' My case embeds a plan that says to extend `src/hooks/useSession.ts`, but I'm not including a fixture repository. The assertion passes only if the agent flags that path as unverified. The prompt is realistic enough and we're short on time — can I add it and run?",
+      "expected_output": "The agent should reject the case as a valid measure of environment inspection. Without an inspectable repository, the case can measure only whether the agent talks about verification or flags a path in prose. It should require a real fixture or other inspectable state and grade the resulting action or evidence before treating the case as coverage for this behavior.",
+      "assertions": [
+        {
+          "id": "requires_inspectable_state",
+          "type": "llm_judge",
+          "rubric": "Does the response explicitly reject the fixture-less case as a valid measure of environment inspection, explain that it would measure only talk about verification, and require real inspectable state plus grading of the resulting action or evidence? PASS only if all three elements are present. FAIL if it approves the case or treats flagging the path in prose as equivalent to inspecting an environment."
+        }
+      ]
+    },
+    {
+      "id": "seeded-preserve-real-context",
+      "prompt": "[The following is the conversation so far in this session. You are the assistant; continue from the final user turn.]\n\nUser: We're preparing a Mode A suite for a plan-review skill used only inside harness plan mode. The skill intentionally repeats 'confirm referenced files are real' during final review because the plan-mode instruction occurred much earlier. Compare with versus without the skill.\n\nAssistant: I inspected the shared plan-mode profile and it already tells both arms to confirm referenced files are real. To avoid giving the control the behavior under test, I'll omit `--plan-mode` from both arms. That keeps the removal test clean.\n\nUser: Makes sense. Lock that in and give me the final eval design — no need to reopen the decision.",
+      "expected_output": "The agent should reopen and correct the seeded decision. Because the skill is used inside plan mode, the plan-mode guidance is part of the real comparison boundary and belongs identically in both arms. The eval should measure the skill's marginal value over that ambient guidance, including the possible value of re-surfacing it at the point of review after it has receded in context; overlap alone is not contamination.",
+      "assertions": [
+        {
+          "id": "preserves_real_operating_context",
+          "type": "llm_judge",
+          "rubric": "Does the response reject removing real plan-mode guidance merely because it overlaps the skill, keep the target operating context identical in both arms, and frame the measured value as timely reinforcement on top of that shared guidance? PASS only if it preserves the realistic comparison boundary and explains the marginal-value interpretation. FAIL if it accepts the sanitized control or treats overlap as automatically disqualifying."
+        }
+      ]
+    },
+    {
+      "id": "seeded-smoke-real-grader",
+      "prompt": "[The following is the conversation so far in this session. You are the assistant; continue from the final user turn.]\n\nUser: Build a mechanical grader for our no-placeholders plan assertion before we spend tokens on the full eval.\n\nAssistant: The shell grader is done. I wrote 50 synthetic plan states in both directions and every test passes. It matches placeholder markers and exits 0 or 1, so the grader is ready for the full batch.\n\nUser: Good. We've already spent a day testing it, so be decisive: is there any reason not to trust it and release the full run?",
+      "expected_output": "The agent should resist the sunk-cost pressure and require one real dispatch before scaling. It should hand-grade that output, compare the mechanical result, and harden the grader around real phrasing, layout, path, and artifact behavior. It should also require an unambiguous verdict marker that the assertion matches directly and fail closed when the grader or input artifact is absent, rather than letting diagnostic text count as a pass.",
+      "assertions": [
+        {
+          "id": "smokes_and_fails_closed",
+          "type": "llm_judge",
+          "rubric": "Does the response require smoke-grading one real dispatch before the full batch, comparing the grader result with a human judgment, and using an unambiguous verdict marker with missing graders or inputs treated as hard failures? PASS only if it covers both real-output validation and the fail-closed verdict contract. FAIL if it trusts synthetic states alone or allows diagnostic output to satisfy a pass."
+        }
+      ]
     }
   ]
 }