feat(skills): proposed model-selection guidance and justification + biome fix

biome.json

@@ -29,7 +29,8 @@
       "!.worktrees",
       "!.claude",
       "!**/bun.lock",
-      "!skills-workspace"
+      "!skills-workspace",
+      "!!.eval-magic"
     ]
   }
 }

skills/evaluating-skills/SKILL.md

@@ -62,17 +62,56 @@ Excuses for skipping an eval on a change you've already judged behavior-shaping.
 | "Pass rate was already 100%" | Then the assertion is too easy. Replace it. |
 | "I'll just call it deterministic" | Deterministic means the agent's compliance isn't in doubt — not that you'd rather not measure. If the wording could change a pressured choice, it's behavioral. Run the eval. |
 
+## Choosing agent and judge models
+
+Model selection defines the population the result describes. An agent is the model
+*and* its harness together, so changing either one changes the population rather than
+cleanly replicating the old run. Keep the model, harness, prompts, and run settings
+identical between comparison arms. The tier-selection rules below are prospective
+operating guidance: they help find signal, but do not establish transfer across tiers.
+
+Choose the agent-under-test from the claim:
+
+| Claim | Agent choice | What the result supports |
+|---|---|---|
+| Exploratory capability / Mode A discovery | Start with the lowest tier that can complete the task and leave gradeable artifacts | Whether the skill creates headroom-sensitive value on that tier and harness |
+| Target-tier behavior | Use the exact model and harness users will rely on | A claim about that target population |
+| Regression protection | Use each production population the suite is meant to protect | Whether established behavior still holds there |
+
+The discovery rule is **prospective guidance**, not a proven transfer law. Public
+guidance supports starting capability evals with a low pass rate and treating saturated
+evals as regression suites, but it does not show that an effect measured on a weaker
+model transfers to a stronger one. A weaker-tier discovery can justify a target-tier
+follow-up; it cannot substitute for one. See Anthropic's
+[agent-eval guidance](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
+and METR's discussion of
+[agent ability and task difficulty](https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/).
+
+Read floors and ceilings before spending the full budget. On a floor, first check that
+the task is solvable and the grader is sound; then simplify the case or move up a tier.
+On a ceiling, preserve the case as a tier diagnostic and move down a tier or add
+realistic difficulty without changing the behavior under test. Never delete or rewrite
+a measured case merely to manufacture a delta.
+
+Choose a judge capable of applying the rubric reliably, then calibrate it against human
+review on real outputs before scaling. Use a balanced, capable judge by default; reserve
+the flagship tier for rubrics the default judge fails to grade reliably or for an
+explicit human choice. Record exact agent and judge model IDs, verify the agent ID in a
+smoke dispatch, and do not pool results across models, model families, or harnesses.
+OpenAI's [evaluation guidance](https://platform.openai.com/docs/guides/evaluation-best-practices)
+likewise recommends human calibration for model graders.
+
 ## Pre-flight gate (required)
 
-An eval run is not free. Each test case dispatches a fresh subagent **per condition** — an N-case suite is `2N` full agent sessions, plus a judge dispatch for every `llm_judge` assertion. That is real wall-clock time and real tokens, and a subagent under test can write outside its sandbox and pollute the real workspace. **Never kick off a run silently.**
+An eval run is not free. Each test case dispatches a fresh subagent **per condition** — an N-case suite is `2N` full agent sessions, plus a judge dispatch for every `llm_judge` assertion. A harness that cannot surface skill invocation also needs judge dispatches for the automatic invocation meta-check. That is real wall-clock time and real tokens, and a subagent under test can write outside its sandbox and pollute the real workspace. **Never kick off a run silently.**
 
 Before building the workspace and dispatching anything, STOP and present the user a run summary, then wait for explicit confirmation:
 
 - **Skill under test** — name and path
 - **Mode** — `new-skill` (with vs without) or `revision` (old vs new), plus the baseline label for revision mode
 - **Eval cases** — the count and a one-line list of the prompts (from `evals.json`)
-- **Models** — the model that will run each subagent under test, and the judge model for `llm_judge` assertions. The runner never dispatches these itself, so it can't observe them — state them explicitly so the user can correct a wrong choice before tokens are spent.
-- **Cost** — `2N` agent dispatches plus judge dispatches; call out that this is time- and token-intensive
+- **Models** — the exact agent-under-test and judge model IDs, the claim each choice supports, and whether a family or harness change limits comparison with prior runs. Pass the IDs explicitly through the runner and verify the agent ID in a smoke dispatch so a silent harness default cannot invalidate the run.
+- **Cost** — `2N` agent dispatches plus substantive and invocation-meta-check judge dispatches; call out that this is time- and token-intensive
 - **Sandbox** — the guard status (on Claude Code, arming the runner's `--guard` is the default; proceed unguarded only on an explicit opt-out, and warn that stray writes will then only be detected after the fact, never blocked)
 
 Do not dispatch until the user confirms *this summary*. An earlier "run the eval" is not confirmation — the summary may reveal a wrong mode, the wrong model, or a missing guard the user never intended. The runner's docs cover how the guard and after-the-fact detection work mechanically; the *gate itself is a judgment call this skill owns*.

skills/evaluating-skills/evals/evals.json

@@ -83,6 +83,18 @@
           "rubric": "Does the response require smoke-grading one real dispatch before the full batch, comparing the grader result with a human judgment, and using an unambiguous verdict marker with missing graders or inputs treated as hard failures? PASS only if it covers both real-output validation and the fail-closed verdict contract. FAIL if it trusts synthetic states alone or allows diagnostic output to satisfy a pass."
         }
       ]
+    },
+    {
+      "id": "seeded-selects-models-for-signal",
+      "prompt": "[The following is the conversation so far in this session. You are the assistant; continue from the final user turn.]\n\nUser: Our Mode A suite ceiled on a flagship coding model. For a diagnostic rerun we have economy, balanced, and flagship tiers. The historical run used another model family and another coding harness. Choose the agent-under-test and judge, then tell me how to compare the result.\n\nAssistant: Stronger models are less noisy, so I'll use the flagship tier for both the agent and judge. We can compare its pass-rate delta directly with the historical run because the cases are unchanged.\n\nUser: That sounds authoritative. Lock it in without reopening the model choice — I want the cleanest number, not caveats.",
+      "expected_output": "The agent should reopen the seeded choice. For exploratory Mode A after a flagship ceiling, it should start with the lowest tier capable of completing the tasks and producing gradeable artifacts, keep that agent and harness identical across the two arms, and use a capable balanced judge calibrated against human review rather than spending the flagship tier automatically. It should reserve the flagship judge for an explicit human choice or demonstrated grading failure. It must also say that changing model family and harness defines a new population: the new within-run delta remains controlled, but it is not a direct replication of the historical run, and any cross-tier transfer claim remains prospective until tested on the target tier.",
+      "assertions": [
+        {
+          "id": "selects_models_for_signal_and_scopes_claim",
+          "type": "llm_judge",
+          "rubric": "Does the response correct the flagship-for-every-role plan by (1) selecting the lowest task-capable tier for exploratory discovery while keeping model and harness identical across arms, (2) selecting a capable balanced judge with human calibration and reserving the flagship judge for explicit human choice or demonstrated need, and (3) treating a model-family or harness change as a new population rather than a direct replication, with transfer to stronger tiers labeled prospective? PASS only if all three elements are present. FAIL if it accepts flagship-by-default, varies arms, directly compares the cross-family/harness result as a replication, or assumes tier transfer."
+        }
+      ]
     }
   ]
 }

skills/investigating-bugs/evals/NOTES.md

@@ -5,6 +5,10 @@ Durable run guidance for `evals.json`. (Per-baseline observations live in
 fixtures are meant to behave. Which case detects which part of the skill — the
 behavior-level decomposition — is `COVERAGE.md`.)
 
+For the Luna/Codex campaign, follow the frozen
+[pre-registration](baseline/PRE-REGISTRATION-LOWER-TIER.md) before building or
+dispatching the run.
+
 ## The dispatch environment MUST be pinned to a non-negative UTC offset
 
 Both timezone cases depend on the agent's own environment being `TZ=UTC` (or any
@@ -22,18 +26,28 @@ non-negative offset). The trap is that the naive reproduction comes back **green
 
 eval-magic has no UTC default — the agent process inherits the operator's
 environment (`env`/`matrix` on a `command_check` affect only the runner-owned check,
-not the dispatch). Pin it at the dispatch recipe by passing the tracked descriptor
-override to **every** command of the run:
+not the dispatch). Pin it at the dispatch recipe by passing the descriptor for the
+chosen harness to **every** command of the run:
 
 ```
+eval-magic <cmd> --harness-file skills/investigating-bugs/evals/harness/codex-utc.toml …
+# or, for Claude Code:
 eval-magic <cmd> --harness-file skills/investigating-bugs/evals/harness/claude-code-utc.toml …
 ```
 
-That descriptor also adds `--setting-sources project,local`, which unloads installed
-plugins so the `without_skill` arm is genuinely skill-free, and raises the dispatch
-permission mode (see below). See the comments in the file itself.
+The Codex descriptor pins `TZ=UTC`, disables live plugins and ambient memories for
+every agent turn, and declares live skill sources isolated because the known subject
+copies are plugin-provided. Inspect `plugin-shadow.json` before dispatch: abort if it
+finds a live subject copy from a source that `--disable plugins` does not cover. The
+transcript overlay also treats Codex's `aggregated_output` as command output; without
+that coalescing, a successful absolute-path read of the dispatch prompt can be
+rejected as an unread prompt. Staging, the write guard, permissions, model flags, and
+judge dispatch remain inherited from the built-in Codex descriptor. The Claude Code
+descriptor also unloads installed plugins so the `without_skill` arm is genuinely
+skill-free and raises the dispatch permission mode (see below). Read the comments in
+the selected file before running it.
 
-## The agent must actually be able to RUN things
+## The Claude Code agent must actually be able to RUN things
 
 eval-magic's built-in Claude Code recipe dispatches with `--permission-mode
 acceptEdits`. That auto-approves file edits but **not** Bash, and because the recipe
@@ -60,6 +74,12 @@ blocks it. Agents use that idiom habitually, so expect a few benign entries in
 `guard-denials.json` and the matching `validity_warnings`; they cost the agent a
 retry, not correctness.
 
+The built-in Codex descriptor already dispatches with non-interactive execution and
+the guard as the boundary, so `codex-utc.toml` must not copy the Claude Code permission
+workaround. Before a Codex fleet, use the smoke dispatch to confirm both sides of the
+contract: `bun` runs inside the task environment and an out-of-environment write is
+blocked.
+
 Verify the fixtures before a run:
 
 ```