Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,10 @@ eval-magic run --mode revision

The command help and generated runbook describe baseline selection and the rest of the workflow.

An eval can treat coordinated skills as one treatment by setting `skill_name` to an ordered list.
Pass one listed member with `--skill`; it remains the eval owner and supplies fixtures. See
`eval-magic docs isolation` for the complete configuration, Mode A/B behavior, and provenance.

## How it works

Each eval case runs once per condition and repetition in its own clean Git repository. The two arms
Expand Down
61 changes: 42 additions & 19 deletions docs/guides/isolation.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,40 +144,63 @@ then use `isolates_live_sources` to record the operator assertion.
dispatch's setting-source selection. A plugin can appear there and remain absent from the dispatch,
or the reverse. Use the dispatch's init event.

## The skill under test is a copy

Every skill an eval stages is copied into the eval home before any dispatch runs, and
each condition stages from that copy. Nothing the agent can reach is read from your own
skill directory, so editing a skill mid-campaign cannot change what a prepared iteration
measures.
## Treatment skills are copies

`skill_name` in the `evals/evals.json` file accepts either one skill name or an ordered,
non-empty list:

```json
{
"skill_name": ["review-workflow", "review-verification"],
"evals": [
{
"id": "review-change",
"prompt": "Review this change.",
"expected_output": "A prioritized review."
}
]
}
```

The copy is the working tree as it sits on disk, not a checkout of a commit —
evaluating an uncommitted revision is the ordinary case, and in a `--mode revision` run
the edit under test is uncommitted by definition. What the run measured is recorded rather than inferred, in
`conditions.json`, each `run.json`, `benchmark.json`, and the `BASELINE.md` written by
With a list, `--skill` selects the eval owner: the member whose `evals/` directory supplies the
definitions and fixtures, and whose name owns the workspace and promotion destination. The owner
must appear in the list. `--stage-name` is unavailable because one override cannot name several
staged skills.

Every treatment member is copied into the eval home before any dispatch runs, and each condition
stages from those copies. Mode A stages all treatment members in `with_skill` and none in
`without_skill`. Other siblings from `--skill-dir` remain ambient in both arms. Mode B snapshots
and stages the complete set in both revisions. A scalar `skill_name` retains the existing
single-skill paths and artifacts.

Each copy is the working tree as it sits on disk, not a checkout of a commit. Evaluating an
uncommitted revision is the ordinary case, and in a `--mode revision` run the edit under test is
uncommitted by definition. What the run measured is recorded rather than inferred in
`conditions.json`, each `run.json`, `benchmark.json`, and the `BASELINE.md` file written by
`promote-baseline`:

```sh
jq '.skill_source' conditions.json
```

`dirty: true` means the recorded revision alone does not identify what ran. Commit the
skill before a run whose result you intend to publish.
`dirty: true` means the recorded revision alone does not identify what ran. Commit the treatment
skills before a run whose result you intend to publish.

Sibling skills staged by `--skill-dir` are copied the same way, and the roster is
captured once when the run resolves. The `siblings` field names exactly what every
environment received.
Ambient skills staged by `--skill-dir` are copied the same way, and the roster is captured once
when the run resolves. For a multi-skill treatment, `skill_source.eval_owner` names the owner and
`skill_source.skills` records every treatment member's resolved source and revision. The
`siblings` field, when present, names ambient skills staged in both arms.

The eval home sits outside the skill's own repository: under `$XDG_DATA_HOME/eval-magic`
(or `~/.local/share/eval-magic`), in a directory named for the skill directory it serves.
`run` prints the path it chose, and every command it suggests carries `--workspace-dir`,
so there is nothing to remember. `EVAL_MAGIC_WORKSPACE_DIR` moves the default;
`--workspace-dir` overrides both.

Copying does not remove the live directory from the machine, so a dispatch can still read
it by absolute path. `detect-stray-writes` reports that as a live-source read, and
`aggregate` carries it into `validity_warnings` for the same reason a discoverable
plugin copy is carried there: the arm may not be comparing what it claims to.
Copying does not remove the live directories from the machine, so a dispatch can still read one by
absolute path. `detect-stray-writes` checks every treatment source and reports that as a live-source
read. `aggregate` carries it into `validity_warnings` for the same reason a discoverable plugin
copy is carried there: the arm may not be comparing what it claims to.

## The task repository is a separate boundary

Expand Down
10 changes: 9 additions & 1 deletion docs/guides/judging.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,15 @@ An authored `llm_judge` assertion can request several independent verdicts for t

Use `run --judge-samples N` to set a campaign-wide default. An assertion's `samples` field takes
precedence over that default. The effective count must be at least one. The framework-injected
`__skill_invoked` meta-check is not substantive grading and remains single-shot.
`__skill_invoked` checks are not substantive grading and remain single-shot per treatment member.

For a multi-skill treatment, deterministic transcript grading checks every staged slug separately.
The response files use `__skill_invoked__skill-N.json`, and each meta result names its
`skill_name`, so partial and complete invocation are distinguishable. Harness descriptors supply
the tool and argument signature; a harness without deterministic invocation events receives one
LLM fallback task per member. The suite-level `meta_summary.skill_invoked` value is true when any
treatment member was invoked. The `benchmark.json` file retains the suite rate and adds per-skill
counts and rates. A scalar treatment keeps the `__skill_invoked.json` filename and artifact shape.

Each sample is a separate judge task and response, but every sample for a run receives the exact
same bounded `judge-evidence.md`. The agent is not rerun, and eval-magic does not rebuild or expand
Expand Down
42 changes: 41 additions & 1 deletion schema/benchmark.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -166,9 +166,36 @@
"type": "array",
"items": { "type": "string" },
"description": "Sibling skills staged alongside the skill under test, as the roster was captured at resolution."
},
"eval_owner": {
"type": "string",
"description": "CLI-selected skill that owns the eval definitions and workspace namespace. Present for multi-skill treatments."
},
"skills": {
"type": "array",
"minItems": 1,
"items": { "$ref": "#/definitions/skillSourceEntry" },
"description": "Ordered resolved source and revision for every multi-skill treatment member."
}
}
},
"skillSourceEntry": {
"type": "object",
"required": ["name", "kind", "source", "branch"],
"additionalProperties": false,
"properties": {
"name": { "type": "string" },
"kind": { "type": "string", "enum": ["git", "path"] },
"source": { "type": "string" },
"resolved_path": { "type": "string" },
"ref": { "type": "string" },
"revision": { "type": "string" },
"origin_url": { "type": "string" },
"branch": { "type": "string" },
"host_local": { "type": "boolean" },
"dirty": { "type": "boolean" }
}
},
"assertionCount": {
"type": "object",
"required": ["passed", "n"],
Expand Down Expand Up @@ -234,7 +261,20 @@
"duration_ms": { "$ref": "#/definitions/stats" },
"total_tokens": { "$ref": "#/definitions/stats" },
"skill_invocation_n": { "type": "integer" },
"skill_invocation_rate": { "type": ["number", "null"] }
"skill_invocation_rate": { "type": ["number", "null"] },
"skill_invocations": {
"type": "object",
"description": "Per-treatment-member invocation counts and rates for multi-skill evals.",
"additionalProperties": {
"type": "object",
"required": ["n", "rate"],
"additionalProperties": false,
"properties": {
"n": { "type": "integer", "minimum": 0 },
"rate": { "type": "number", "minimum": 0, "maximum": 1 }
}
}
}
}
},
"diffScopeRun": {
Expand Down
12 changes: 10 additions & 2 deletions schema/evals.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,16 @@
"additionalProperties": false,
"properties": {
"skill_name": {
"type": "string",
"description": "Name of the skill being evaluated. Should match the skill directory name."
"description": "Name of the skill being evaluated, or an ordered non-empty set of coordinated skills. The CLI-selected eval owner must be a member of a set.",
"oneOf": [
{ "type": "string", "minLength": 1 },
{
"type": "array",
"minItems": 1,
"uniqueItems": true,
"items": { "type": "string", "minLength": 1 }
}
]
},
"codebase": {
"$ref": "#/definitions/codebase",
Expand Down
20 changes: 18 additions & 2 deletions schema/grading.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"meta_results": {
"type": "array",
"description": "Framework-injected meta-assertions (e.g. skill-invocation check). Reserved id prefix: __ (double underscore). Tracked separately from substantive assertion_results so they do not pollute the skill effectiveness pass_rate.",
"items": { "$ref": "#/definitions/binaryAssertionResult" }
"items": { "$ref": "#/definitions/metaResult" }
},
"meta_summary": {
"type": "object",
Expand All @@ -27,13 +27,29 @@
"failed": { "type": "integer", "minimum": 0 },
"total": { "type": "integer", "minimum": 0 },
"skill_invoked": {
"description": "True when the skill-invocation meta-check passed; false when the judge found no evidence the skill influenced behavior; null when no skill was loaded for this run.",
"description": "True when at least one treatment member's invocation meta-check passed; false when none did; null when no treatment skill was loaded for this run.",
"type": ["boolean", "null"]
}
}
}
},
"definitions": {
"metaResult": {
"type": "object",
"required": ["id", "passed", "evidence"],
"additionalProperties": false,
"properties": {
"id": { "type": "string" },
"skill_name": { "type": "string", "description": "Treatment member checked; absent for scalar legacy artifacts." },
"passed": { "type": "boolean" },
"evidence": { "type": "string" },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 },
"grader": {
"type": "string",
"enum": ["transcript_check", "llm_judge", "command_check", "diff_scope"]
}
}
},
"assertionResult": {
"oneOf": [
{ "$ref": "#/definitions/binaryAssertionResult" },
Expand Down
4 changes: 4 additions & 0 deletions schema/judge-tasks.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
"description": "1-based run index within a multi-run (eval, condition) cell; absent for single-run cells."
},
"assertion_id": { "type": "string" },
"skill_name": {
"type": "string",
"description": "Treatment member checked by a multi-skill invocation meta task."
},
"sample_index": {
"type": "integer",
"minimum": 1,
Expand Down
42 changes: 42 additions & 0 deletions schema/run-record.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@
"type": ["string", "null"],
"description": "Absolute path to the SKILL.md the subagent could load, or null if no skill was provided (without_skill condition)."
},
"skills": {
"type": "array",
"description": "Ordered treatment roster for list-authored evals. Empty in the control arm and absent in scalar legacy records.",
"items": { "$ref": "#/definitions/conditionSkill" }
},
"prompt": {
"type": "string",
"description": "The user prompt as dispatched to the subagent."
Expand Down Expand Up @@ -95,6 +100,16 @@
}
},
"definitions": {
"conditionSkill": {
"type": "object",
"required": ["name", "skill_path", "staged_skill_slug"],
"additionalProperties": false,
"properties": {
"name": { "type": "string" },
"skill_path": { "type": "string" },
"staged_skill_slug": { "type": ["string", "null"] }
}
},
"responderOutcome": {
"type": "object",
"required": ["ending"],
Expand Down Expand Up @@ -321,9 +336,36 @@
"type": "array",
"items": { "type": "string" },
"description": "Sibling skills staged alongside the skill under test, as the roster was captured at resolution."
},
"eval_owner": {
"type": "string",
"description": "CLI-selected skill that owns the eval definitions and workspace namespace. Present for multi-skill treatments."
},
"skills": {
"type": "array",
"minItems": 1,
"items": { "$ref": "#/definitions/skillSourceEntry" },
"description": "Ordered resolved source and revision for every multi-skill treatment member."
}
}
},
"skillSourceEntry": {
"type": "object",
"required": ["name", "kind", "source", "branch"],
"additionalProperties": false,
"properties": {
"name": { "type": "string" },
"kind": { "type": "string", "enum": ["git", "path"] },
"source": { "type": "string" },
"resolved_path": { "type": "string" },
"ref": { "type": "string" },
"revision": { "type": "string" },
"origin_url": { "type": "string" },
"branch": { "type": "string" },
"host_local": { "type": "boolean" },
"dirty": { "type": "boolean" }
}
},
"conversationTool": {
"type": "object",
"required": ["type", "ordinal", "round", "name"],
Expand Down
22 changes: 19 additions & 3 deletions src/adapters/skill_shadow/grouping.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,10 @@ impl PluginShadowReport {
subject_skill_name: &str,
expected_cells: &[(String, String)],
) -> Self {
Self::from_observed_sources_with_class(
Self::from_observed_sources_for_subjects_with_class(
config_dir,
sources,
subject_skill_name,
&[subject_skill_name],
expected_cells,
ShadowFindingClass::OperatorEnvironment,
)
Expand All @@ -63,6 +63,22 @@ impl PluginShadowReport {
subject_skill_name: &str,
expected_cells: &[(String, String)],
class: ShadowFindingClass,
) -> Self {
Self::from_observed_sources_for_subjects_with_class(
config_dir,
sources,
&[subject_skill_name],
expected_cells,
class,
)
}

pub(crate) fn from_observed_sources_for_subjects_with_class(
config_dir: impl Into<String>,
sources: Vec<ShadowSource>,
subject_skill_names: &[&str],
expected_cells: &[(String, String)],
class: ShadowFindingClass,
) -> Self {
let mut merged = Vec::<ShadowSource>::new();
for mut source in sources {
Expand All @@ -88,7 +104,7 @@ impl PluginShadowReport {
}
for finding in &mut report.findings {
finding.class = class;
finding.role = if finding.skill_name == subject_skill_name {
finding.role = if subject_skill_names.contains(&finding.skill_name.as_str()) {
ShadowSkillRole::Subject
} else {
ShadowSkillRole::Sibling
Expand Down
Loading
Loading