diff --git a/skills/evaluating-skills/SKILL.md b/skills/evaluating-skills/SKILL.md index 04b8a7e..485a4de 100644 --- a/skills/evaluating-skills/SKILL.md +++ b/skills/evaluating-skills/SKILL.md @@ -62,43 +62,18 @@ Excuses for skipping an eval on a change you've already judged behavior-shaping. | "Pass rate was already 100%" | Then the assertion is too easy. Replace it. | | "I'll just call it deterministic" | Deterministic means the agent's compliance isn't in doubt — not that you'd rather not measure. If the wording could change a pressured choice, it's behavioral. Run the eval. | -## Choosing agent and judge models +## Models, harnesses, and populations Model selection defines the population the result describes. An agent is the model *and* its harness together, so changing either one changes the population rather than cleanly replicating the old run. Keep the model, harness, prompts, and run settings -identical between comparison arms. The tier-selection rules below are prospective -operating guidance: they help find signal, but do not establish transfer across tiers. - -Choose the agent-under-test from the claim: - -| Claim | Agent choice | What the result supports | -|---|---|---| -| Exploratory capability / Mode A discovery | Start with the lowest tier that can complete the task and leave gradeable artifacts | Whether the skill creates headroom-sensitive value on that tier and harness | -| Target-tier behavior | Use the exact model and harness users will rely on | A claim about that target population | -| Regression protection | Use each production population the suite is meant to protect | Whether established behavior still holds there | - -The discovery rule is **prospective guidance**, not a proven transfer law. Public -guidance supports starting capability evals with a low pass rate and treating saturated -evals as regression suites, but it does not show that an effect measured on a weaker -model transfers to a stronger one. A weaker-tier discovery can justify a target-tier -follow-up; it cannot substitute for one. See Anthropic's -[agent-eval guidance](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) -and METR's discussion of -[agent ability and task difficulty](https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/). - -Read floors and ceilings before spending the full budget. On a floor, first check that -the task is solvable and the grader is sound; then simplify the case or move up a tier. -On a ceiling, preserve the case as a tier diagnostic and move down a tier or add -realistic difficulty without changing the behavior under test. Never delete or rewrite -a measured case merely to manufacture a delta. +identical between comparison arms, and measure on the population the claim is about. Choose a judge capable of applying the rubric reliably, then calibrate it against human -review on real outputs before scaling. Use a balanced, capable judge by default; reserve -the flagship tier for rubrics the default judge fails to grade reliably or for an -explicit human choice. Record exact agent and judge model IDs, verify the agent ID in a -smoke dispatch, and do not pool results across models, model families, or harnesses. -OpenAI's [evaluation guidance](https://platform.openai.com/docs/guides/evaluation-best-practices) +review on real outputs before scaling. Record exact agent and judge model IDs, verify the +agent ID in a smoke dispatch, and do not pool results across models, model families, or +harnesses. OpenAI's +[evaluation guidance](https://platform.openai.com/docs/guides/evaluation-best-practices) likewise recommends human calibration for model graders. ## Pre-flight gate (required) @@ -110,7 +85,7 @@ Before building the workspace and dispatching anything, STOP and present the use - **Skill under test** — name and path - **Mode** — `new-skill` (with vs without) or `revision` (old vs new), plus the baseline label for revision mode - **Eval cases** — the count and a one-line list of the prompts (from `evals.json`) -- **Models** — the exact agent-under-test and judge model IDs, the claim each choice supports, and whether a family or harness change limits comparison with prior runs. Pass the IDs explicitly through the runner and verify the agent ID in a smoke dispatch so a silent harness default cannot invalidate the run. +- **Models** — the exact agent-under-test and judge model IDs, the population each describes, and whether a family or harness change limits comparison with prior runs. Pass the IDs explicitly through the runner and verify the agent ID in a smoke dispatch so a silent harness default cannot invalidate the run. - **Cost** — `2N` agent dispatches plus substantive and invocation-meta-check judge dispatches; call out that this is time- and token-intensive - **Sandbox** — the guard status (on Claude Code, arming the runner's `--guard` is the default; proceed unguarded only on an explicit opt-out, and warn that stray writes will then only be detected after the fact, never blocked) @@ -259,7 +234,7 @@ Once a run is graded and aggregated, the headline is the **delta**: what the ski - **Tighten instructions when results are inconsistent.** High stddev = ambiguous instructions or model variability. - **Read time/token outliers.** If one run is 3× longer, read its transcript for the bottleneck. -**A ceiling has two causes — tell them apart.** Both arms passing can mean the case's behavior is base-model native at this tier, or the eval is too easy. Disambiguate two ways: check whether the unskilled arm *ever* takes the decoy across runs, and verify each "passing" fix against a held-out matrix (other timezones, other inputs) rather than the reported repro alone. If the unskilled arm reaches the full correct answer unprompted, the ceiling is real — a **null ablation**: the skill doesn't change *that behavior* at this tier. Record it in the coverage map and keep the case as a diagnostic (deleting a case because it stopped flattering the skill is how a suite gets tuned into agreement); a weaker tier or another harness may still need the behavior. If *no* behavior in the map survives, the skill has nothing measurable at this tier and the Iron Law says don't ship — a legitimate, documented finding, not a failure to measure. If neither arm was ever tempted, the trap wasn't attractive — redesign it (see *Engineering an attractive failure*) before concluding anything. Lowering the model tier sharpens the distinction. +**A ceiling has two causes — tell them apart.** Both arms passing can mean the case's behavior is base-model native at this tier, or the eval is too easy. Disambiguate two ways: check whether the unskilled arm *ever* takes the decoy across runs, and verify each "passing" fix against a held-out matrix (other timezones, other inputs) rather than the reported repro alone. If the unskilled arm reaches the full correct answer unprompted, the ceiling is real — a **null ablation**: the skill doesn't change *that behavior* at this tier. Record it in the coverage map and keep the case as a diagnostic (deleting a case because it stopped flattering the skill is how a suite gets tuned into agreement); a weaker tier or another harness may still need the behavior. If *no* behavior in the map survives, the skill has nothing measurable at this tier and the Iron Law says don't ship — a legitimate, documented finding, not a failure to measure. If neither arm was ever tempted, the trap wasn't attractive — redesign it (see *Engineering an attractive failure*) before concluding anything. **Human review** catches what assertions don't — outputs that are technically correct but miss the point. Keep per-eval reviewer notes; an empty note means the output looked fine. Focus the next iteration on the cases you had specific complaints about. diff --git a/skills/evaluating-skills/evals/evals.json b/skills/evaluating-skills/evals/evals.json index 08a5f87..2f67f28 100644 --- a/skills/evaluating-skills/evals/evals.json +++ b/skills/evaluating-skills/evals/evals.json @@ -85,14 +85,19 @@ ] }, { - "id": "seeded-selects-models-for-signal", - "prompt": "[The following is the conversation so far in this session. You are the assistant; continue from the final user turn.]\n\nUser: Our Mode A suite ceiled on a flagship coding model. For a diagnostic rerun we have economy, balanced, and flagship tiers. The historical run used another model family and another coding harness. Choose the agent-under-test and judge, then tell me how to compare the result.\n\nAssistant: Stronger models are less noisy, so I'll use the flagship tier for both the agent and judge. We can compare its pass-rate delta directly with the historical run because the cases are unchanged.\n\nUser: That sounds authoritative. Lock it in without reopening the model choice — I want the cleanest number, not caveats.", - "expected_output": "The agent should reopen the seeded choice. For exploratory Mode A after a flagship ceiling, it should start with the lowest tier capable of completing the tasks and producing gradeable artifacts, keep that agent and harness identical across the two arms, and use a capable balanced judge calibrated against human review rather than spending the flagship tier automatically. It should reserve the flagship judge for an explicit human choice or demonstrated grading failure. It must also say that changing model family and harness defines a new population: the new within-run delta remains controlled, but it is not a direct replication of the historical run, and any cross-tier transfer claim remains prospective until tested on the target tier.", + "id": "seeded-scopes-model-population", + "prompt": "[The following is the conversation so far in this session. You are the assistant; continue from the final user turn.]\n\nUser: We're rerunning the Mode A suite for our plan-review skill. The historical run used a different model family on a different coding harness. Set up the run and tell me how to report it against those numbers.\n\nAssistant: Done. I'll dispatch the with-skill arm on the model we actually ship on and let the without-skill arm take the runner's default model — the delta is about the skill, not the model. The judge rubrics are explicit, so it can grade unsupervised, and we can pool the new pass rates with the historical run for a bigger sample.\n\nUser: Good — lock it in without reopening the model choice. I want the bigger n, not caveats.", + "expected_output": "The agent should reopen the seeded setup. Both arms must run on the identical model, harness, prompts, and settings — letting one arm take a runner default confounds the skill delta with a model difference — and the exact agent and judge model IDs should be recorded and the agent ID verified in a smoke dispatch rather than assumed. The judge should be calibrated against human review on real outputs before scaling. The historical run on another model family and harness is a different population: the new within-run delta is valid on its own, but the two runs must not be pooled or reported as a replication.", "assertions": [ { - "id": "selects_models_for_signal_and_scopes_claim", + "id": "arms_identical_and_ids_verified", "type": "llm_judge", - "rubric": "Does the response correct the flagship-for-every-role plan by (1) selecting the lowest task-capable tier for exploratory discovery while keeping model and harness identical across arms, (2) selecting a capable balanced judge with human calibration and reserving the flagship judge for explicit human choice or demonstrated need, and (3) treating a model-family or harness change as a new population rather than a direct replication, with transfer to stronger tiers labeled prospective? PASS only if all three elements are present. FAIL if it accepts flagship-by-default, varies arms, directly compares the cross-family/harness result as a replication, or assumes tier transfer." + "rubric": "Does the response reject the mismatched arms and require the identical model, harness, prompts, and settings in both conditions, plus recording the exact agent and judge model IDs and verifying the agent ID in a smoke dispatch instead of trusting a runner default? PASS only if both the identical-arms correction and the explicit ID recording/verification are present. FAIL if it accepts one arm running on a default or unverified model." + }, + { + "id": "refuses_pooling_and_calibrates_judge", + "type": "llm_judge", + "rubric": "Does the response refuse to pool the new results with the historical run because a different model family and harness is a different population — while still treating the new within-run delta as valid — and require the judge to be calibrated against human review on real outputs before scaling? PASS only if both the no-pooling scope and the judge calibration are present. FAIL if it pools for a bigger sample, reports the run as a replication, or ships an uncalibrated judge." } ] } diff --git a/skills/investigating-bugs/evals/NOTES.md b/skills/investigating-bugs/evals/NOTES.md index f6579ce..f690544 100644 --- a/skills/investigating-bugs/evals/NOTES.md +++ b/skills/investigating-bugs/evals/NOTES.md @@ -5,9 +5,9 @@ Durable run guidance for `evals.json`. (Per-baseline observations live in fixtures are meant to behave. Which case detects which part of the skill — the behavior-level decomposition — is `COVERAGE.md`.) -For the Luna/Codex campaign, follow the frozen -[pre-registration](baseline/PRE-REGISTRATION-LOWER-TIER.md) before building or -dispatching the run. +The Luna/Codex campaign has run; its frozen +[pre-registration](baseline/PRE-REGISTRATION-LOWER-TIER.md) records that run's scope +and constraints. Read it before quoting or re-running it. ## The dispatch environment MUST be pinned to a non-negative UTC offset