progressive-enhancements.md

Harness progressive enhancements

Audience: developers working on the eval-magic codebase. CLI help and the shipped guides are the user-facing docs; this file explains how harness support is structured in code and what wiring more of it buys. Per-harness implementation notes live in claude-notes.md, codex-notes.md, and opencode-notes.md. Pointing eval-magic at a harness it doesn't know — user-supplied descriptor files, layering, harness list/show/lint — is the user-facing BYOH guide.

Harness compatibility is not a parity checklist to audit — it is a runner-ready baseline every harness satisfies, plus optional enhancements a harness's adapter opts into. Dispatch and transcript recovery are mandatory because the runner owns execution and run-record assembly. Native conversation resume remains optional; an eval that declares scripted turns or a responder is rejected when the harness cannot preserve one session.

One dispatch mechanism

eval-magic dispatch runs every task — one-shot, scripted, and judge alike — through the harness CLI, one subprocess per round, --jobs tasks at a time. A scripted eval resumes one harness-native session across its rounds in one eval_root. The generated artifacts are the runtime source of truth for how to dispatch: run writes RUNBOOK.md and dispatch-manifest.md naming the exact command for the selected harness — hand-maintained docs never carry command recipes. Eval-agent environment defaults are harness-independent: [dispatch.env] layers merge by key and repeatable run --agent-env KEY=VALUE entries override them. The resolved map is recorded in conditions.json / dispatch.json and applied by the runner when it spawns each task; judges and runner-owned command_check subprocesses remain separate.

The baseline contract

A harness qualifies at baseline when its descriptor provides:

1. A headless exec command — some way to invoke the harness with a prompt from a chosen cwd and let it run to completion.
2. A parseable transcript — the command captures the harness's event stream under the task's round output directory, and a named or declarative reader normalizes a non-empty final response.
3. --no-stage when native staging isn't wired — each SKILL.md is inlined into its dispatch prompt instead of staged for native discovery.

That baseline already yields a working eval: llm_judge assertions grade soft behavior, runner-owned command_check assertions can inject held-out files and execute deterministically, runner-owned final-environment metrics land in diff-scope.json with the diff itself in diff.patch, diff_scope assertions gate files/lines deterministically, and the detect-stray-writes post-pass (folded into ingest) audits writes that leave the private task environment. dispatch records monotonic eval-agent subprocess duration for every runner-ready harness, and record-runs assembles every run.json and timing.json from the runner-owned dispatch metadata, completion artifact, and per-round transcripts.

In descriptor terms the baseline is label, [dispatch].exec_template, [transcript] with one primary reader, and [tools] beside that reader. run rejects a harness that cannot dispatch or recover transcripts. A harness without skills_dir forces --no-stage; without a declared guard the run continues unguarded behind the detect-stray-writes audit; requested models without a model flag are recorded as provenance only. Supported enhancements are provided automatically — the write guard auto-arms wherever a harness declares one and staging is active (--no-guard opts out). A selected eval with turns or a responder also requires [conversation]; no generic fresh-session fallback can preserve the meaning of a follow-up reply.

Where this lives in code

• harnesses/<label>.toml — one embedded descriptor file per built-in harness, carrying every declarative value (labels, dirs, capability booleans, phrases, command templates, banner prose). Descriptors are schema-gated (schema/harness-descriptor.schema.json) and invariant-checked at load time.
• src/adapters/registry.rs — the label-keyed registry: init_registry loads every layer (embedded → user-global → project-local → --harness-file) with field-level merge per label, and adapter_for() resolves each Harness handle to its descriptor-backed adapter — still the single dispatch point. Broken discovered files skip with a warning; the layer model itself is documented in the BYOH guide.
• src/adapters/descriptor/layers.rs — layer discovery (config-root resolution, per-directory scans, the --harness-file top layer) and the user-layer restrictions (no [guard]).
• src/adapters/harness.rs — the HarnessAdapter trait, tiered into baseline and enhancement sections.
• src/adapters/descriptor.rs — the descriptor model, the per-file schema gate (parse_descriptor_value), field-level merge (merge_descriptor_value), and finalize_descriptor (the load-time invariants, with merged-file provenance in every message); src/adapters/descriptor_adapter.rs — the one generic DescriptorAdapter implementing the trait from a descriptor.
• src/adapters/capabilities.rs — the named capabilities: closed enums a descriptor references by kebab-case name for everything that is real code (transcript summary/denial readers, slug generation, shadow preflight). The write guard needs no named capability: it is pure [guard] data rendered by the one generic engine in src/adapters/guard.rs.
• src/adapters/<harness>/ — only the code behind those capabilities: transcript summary/denial readers, harness-native shadow scans, the OpenCode slug sanitizer.
• run_capabilities() (descriptor table [run]) + harness_run_preflight() (src/cli/run/util.rs) — the run preflight: it resolves the guard tri-state (auto-arm when the harness declares a guard and staging is active; --guard/--no-guard make it explicit), rejects descriptors without a dispatch command or transcript reader, and adjusts optional capabilities (guard forced off, missing skills_dir forces --no-stage); only contradictory flag combinations (--bootstrap/--stage-name where the descriptor declares them incompatible with --no-stage) and an explicit --guard on a user-descriptor-only harness reject.

Runner-owned environment checks are baseline

Every canonical (eval, condition, run) gets a distinct eval_root provisioned from its effective codebase. After overlays, staging, and guard installation, run commits the task state, marks it with refs/eval-magic/baseline, and runs shadow preflight at the resulting repository boundary. Git is therefore a runtime prerequisite; each task starts clean and has no remotes. Nothing writes into an environment after the ref is written, so it names exactly what the agent started from.

During ingest, before any held-out setup is injected, Git measures the final environment against that ref. The runner seeds a scratch index from the baseline, brings it up to the working tree with one git add, and diffs the two trees — so creations, modifications, and deletions all fall out of one pass, and an untracked creation is not missed. Raw files_touched, lines_added, lines_removed, and zero-context hunks go to diff-scope.json, alongside the changed-file list; the diff itself goes to diff.patch beside it, capped and marked when a diff exceeds the cap. benchmark.json preserves the metrics per run even without a diff_scope assertion. An assertion may gate max_files_touched, max_lines_changed (added plus removed), or both.

What counts is what Git counts. The measurement runs under the same rules the baseline commit was built under: the codebase's own .gitignore holds, so a run that compiles does not report its build output as thousands of touched files, and the .git/info/exclude entry keeps framework artifacts under .eval-magic-outputs/ out. Paths the runner force-added despite those rules — the harness config directories and the declared file overlay — are tracked in the baseline and stay measured. Git indexes no path with a .git component, so a nested repository's internals are invisible, not just the runner-owned root .git. Renames are switched off deliberately: a rename is two touched files, one created and one deleted, which is what the metric has always meant. A binary file counts as one touched file with no countable lines.

This is deliberately a secondary signal: a smaller diff can be focused, but it can also be incomplete. Pair a scope gate with a correctness assertion. The patch is the evidence that closes that gap — it is what a judge reads to answer whether the work was any good.

command_check is intentionally not a harness enhancement. run detects the assertion before dispatch so it can validate held-out sources before building. After diff-scope capture, ingest grades only tasks with a runner-owned run.json, copies the assertion's held-out setup_files from the skill's evals/ directory into that root, and executes the trusted command through the platform shell. Root .git paths are reserved for both visible overlays and held-out setup files, while nested repositories remain valid. The runner clears inherited Git routing variables before optional env values override the environment; optional matrix values execute every Cartesian-product cell and persist per-cell results. The files are never staged or mentioned to the agent, and therefore never inflate scope metrics.

Beyond the runner-ready dispatch and transcript baseline, this path needs no additional parser signals, tool vocabulary, or model flag, so it behaves the same for built-ins and descriptor-only harnesses. It also does not use harness tools: an armed agent write guard can remain installed while the runner executes the command. finalize converts the schema-gated intermediate result into an ordinary grading result, leaving aggregation harness-agnostic.

The enhancements

Each enhancement is a group of descriptor fields — plus a named capability where code is involved — with defaults. Wire them together; load-time descriptor validation (validate_descriptor in src/adapters/descriptor.rs) rejects the combinations that must move in lockstep, with an actionable message per violation.

Transcript ingest

Why harness-specific: every harness persists a different event stream (Claude Code's -p stream-json, Codex's item.completed JSONL) — but not all of them need per-harness code. Ingest is two-tier: a flat stream (each tool event self-contained) is a declarative mapping the generic extract engine (src/adapters/extract.rs) executes from descriptor data alone, while a stream needing cross-event state — keyed tool_use/tool_result joins, shape-dependent content coercion — is real per-harness code behind a named capability. If a stream needs more than the extract primitives, it's a code capability, not a bigger DSL.

What it unlocks: transcript_check assertions, token capture, historical duration fallback, automatic run.json/timing.json assembly by ingest, and — where the transcript exposes a skill-tool event — a deterministic __skill_invoked meta-check. Paired with [tools], tool patterns are portable: grading identifies a native tool name's role from the run's own descriptor and retries the pattern against every spelling all_tool_vocabulary() declares for that role, so one authored assertion measures the same behavior on every harness (#308).

Sub-capability: permission-denied tool results. A refused tool call can be reported in the event stream or a paired harness capture while the overall dispatch still exits 0. On its own that is easy to miss: the run grades normally and may degrade to static reasoning. A named denial reader that can distinguish a refusal from an ordinary tool error makes ingest write permission-denials.json and aggregate warn once per affected task. Why harness-specific: refusals are not a shared shape — Claude Code reports them in the terminal result event's structured permission_denials, Codex reports pre-execution router rejections and PreToolUse hook blocks in the stderr capture beside its JSONL, and OpenCode records a refusal as an ordinary tool_use event with part.state.status:"error" whose state.error is the string OpenCode itself authors at its permission layer (a fixed PermissionDeniedError/PermissionRejectedError prefix, or the shared eval guard: reason for a guard block). Each denial reader recognizes only its harness's own refusal strings; ordinary failed tool calls — including ambiguous DNS and OS-process failures, and OpenCode tool-body errors like oldString not found — remain unclassified rather than creating false positives. Fallback: no report and no warning — a run that degraded to static reasoning is only visible in the raw captures, which run preflight states once naming that fallback. Capability: transcript.permission_denials_parser names the reader independently of the primary summary tier. It may accompany a named parser or declarative [transcript.extract] summary; claude-stream-json, cline-json, codex-items, and opencode-events surface denials. For compatibility, a descriptor with only transcript.parser still uses that parser's bundled denial support. The eval write guard denies through the same permission mechanism, so its blocks land in the report as well; they are attributed by the eval guard: reason prefix and excluded from the warning so one denial is not reported twice.

There is no no-transcript fallback for a runner-ready harness. A descriptor without a primary transcript reader is rejected before the workspace is built. A valid reader may still omit optional token, duration, denial, or skill-invocation evidence; those individual signals remain unavailable or use their documented grading fallback.

Descriptor fields: the [transcript] table — events_filename (gate: an absent table means the ingest pipeline never reads a transcript), one primary summary reader, and surfaces_skill_invocation. A deterministic native skill event uses skill_tool / skill_arg; a successful exact-path shell read uses the mutually exclusive skill_access table with its tool, command argument, exit-code argument, and declared read-command basenames. The primary reader is either parser or the summary outputs under extract; validation rejects both, neither, and a surface-only extract. The extract sub-table is the declarative tier: equality where filters, final and ordered assistant-text picks, a session-id pick, flat tool-item mapping, token sum/subtract reduction, duration rule, and the auxiliary session-surface mapping represented in the descriptor schema and visible in resolved descriptors. Capability: transcript.parser names the code that stitches a non-flat stream (claude-stream-json, cline-json, codex-items, opencode-events) — a new harness emitting compatible captures reuses one with zero code. The built-in Codex descriptor uses declarative summary extraction plus permission_denials_parser = "codex-items"; the named parser remains available as a compatibility/reference implementation and is covered by a differential summary test. The tool names the transcript yields must be declared in [tools] (see the write-guard enhancement) or detect-stray-writes audits nothing for the harness — validation rejects the combination.

Native conversation resume

Why harness-specific: each CLI spells same-session continuation differently and exposes its session identifier in a different transcript event.

What it unlocks: an eval's ordered turns array and its responder policy. dispatch starts the normal one-shot command, extracts the native session id, asks the eval's turn source what follows each round, and resumes the same session for each delivered follow-up. It writes raw round transcripts under outputs/turn-N/ and atomically commits conversation.json only after a complete or normal guardrail-stopped scenario. ingest skips an interrupted task with no completion artifact.

A scripted turn is gated by agent_asks (?) plus the optional response regex. A responder instead derives each turn by consulting a small model, once after every round, and records that origin on the turn itself. The responder needs no descriptor field and no named capability of its own: it reads the round's last assistant message out of final_text, which every transcript parser already normalizes, and it dispatches its own consultations through the same [dispatch].exec_template a judge uses. Every harness that resolves a resume template gets it for free, and none can be "missing" it.

That portability is not a happy accident, it is forced. A dispatch runs headless with stdin detached, so a harness-native question tool has no channel to be answered on; the runner can only send free text as the next user turn. Text is therefore the only mechanism that fits, and it is the one every transcript parser already normalizes into final_text.

A consultation binds the exec template's placeholders the way a judge dispatch does — guard arguments off, its own capture directory, its own prompt — with one addition: <eval-root> is the run's responder/turn-N/ directory rather than the task env. A consultation must not be able to write into the codebase under measurement, nor inherit that codebase's CLAUDE.md as instructions to itself.

Fallback: none. run rejects selected multi-turn evals — scripted or responder-driven — when the harness omits this capability; silently starting a fresh session would make the answer meaningless.

Descriptor fields: [conversation].resume_exec_template, with required <eval-root>, <outputs_dir>, {session_arg}, and {prompt_arg} placeholders, plus optional token_usage_aggregation (sum by default, last for cumulative session reports). It requires dispatch.exec_template and transcript parsing. Declarative transcript extraction must include assistant_messages and session_id; named built-in parsers provide the same normalized fields.

Native skill staging + skills block

Why harness-specific: each harness has its own project-local discovery dir and its own way of surfacing discoverable skills to a session (Claude Code's Skill-tool list, Codex's ## Skills markdown, OpenCode's <available_skills> XML), and some constrain skill naming.

What it unlocks: environment parity — the staged skill is discovered the way a real install would discover it, instead of being pasted into the prompt.

Fallback: --no-stage inlines each SKILL.md into its dispatch prompt.

Descriptor fields: skills_dir, config_dirs, and the [staging] table — slug_template (or slug_capability), stage_name_pattern/stage_name_max_len/stage_name_invalid_message (declarative naming rules), rewrites_frontmatter_name, advertises_staged_slug_name, surface_phrase, unresolved_phrase — plus the [skills_block] table (header/item/footer format strings with {name}/{description}/{path} placeholders, rendered by the shared skills_block renderer). Capability: staging.slug_capability for naming rules that need sanitization/truncation beyond a format string (opencode); simpler rules stay declarative.

Model flag

Why harness-specific: the CLI flag (and its position in the command) differs per harness.

What it unlocks: --agent-model / --judge-model actually select models in the generated recipes; judge tasks resolve a per-task model.

Fallback: the models are recorded as provenance in conditions.json only; dispatches run on the harness's default model.

Descriptor fields: the [model] table — flag (consumed as {model_arg} by the dispatch templates).

Write guard

Why harness-specific: the guard arms a native pre-tool hook — hook surface, blocking mechanism, trust model, and deny-verdict shape are all harness-native (Claude Code's settings.local.json + hookSpecificOutput, Codex's hooks.json + {"decision": "block"}, OpenCode's auto-loaded project plugin that blocks by throwing the reason).

What it unlocks: out-of-bounds writes are blocked before they happen instead of detected afterwards. The guard is provided automatically: every staged run of a guard-declaring built-in arms it unless --no-guard opts out (--guard makes the request explicit, turning silent can't-arm cases into a warning or error). Every block is also recorded as privacy-safe metadata under the task env and collected into guard-denials.json; aggregate emits one validity warning per affected task because even an intentional boundary block changed agent behavior.

Fallback: detect-stray-writes audits after the fact. (It also flags live-source reads — an arm whose subagent read the live skill source instead of its staged copy, which contaminates the arm; fatal in revision mode, where the old_skill arm then sees new-skill content.)

Descriptor fields: the [guard] table — verdict_template and armed_message for every engine, engine (the install-mechanism discriminator, default json-hooks), and the per-engine fields: json-hooks (Claude Code, Codex) declares hooks_file, matcher, command_template, and hook_entry; opencode-plugin and cline-plugin declare plugin_file instead — plus [tools] (the write/patch/shell/read vocabulary) and run.supports_guard (validated to stay in lockstep with the [guard] table). There is no guard code capability: one engine module (src/adapters/guard.rs) holds three install arms selected by engine and a single shared verdict path. json-hooks merges the rendered hook entry into hooks_file; opencode-plugin stages an embedded JS project plugin (harnesses/opencode-guard-plugin.js, {exe}/{marker} substituted as JSON string literals) whose tool.execute.before hook forwards every tool call to the generic entry point and throws the verdict's reason to block — OpenCode auto-loads project plugins by directory convention, so no dispatch flag is needed. cline-plugin stages an embedded JS project plugin directory (harnesses/cline-guard-plugin.js as .cline/plugins/.../index.js) whose beforeTool hook forwards every tool call the same way — joining run_commands' commands array into one command string first — and returns {skip: true, reason} to block; Cline auto-loads project plugin dirs in headless dispatches too. The templates' authored JSON key order is serialized verbatim — the verdict bytes are the harness's on-disk contract. Validation proves the per-engine shape (the schema's conditional requiredness, plus load-time checks barring the other engine's fields), that every hooked matcher tool is declared in [tools] (json-hooks), that the templates parse as JSON, and that their {command}/{matcher}/{reason} placeholders sit in string values. Patch payload extraction accepts structured files plus raw command/patch/input/content/patchText bodies and validates every source and destination. The quote-aware Bash scanner resolves literal redirect and tee targets from the invocation cwd; dynamic, malformed, or outside targets remain denied. It allows local Git inspection, staging, commits, and branching within the task boundary, but denies outside/dynamic repository routing, git worktree add, remote-capable subcommands, mutating git remote, and remote/url config writes. Read-only git remote, git remote -v, git remote get-url, and config reads remain available. --no-guard opts out of these blocks, though the task repository still begins without remotes. The guard arbiter and detect-stray-writes classify tool names against the cross-harness vocabulary union (all_tool_vocabulary), so wiring a guard or transcript ingest without declaring the harness's tool names is rejected at descriptor load. The hidden guard / guard-codex subcommands are frozen hook entry-point aliases (a stable on-disk contract); a new guard-capable built-in uses the generic guard-hook --harness <label> entry point (OpenCode's plugin spawns exactly that) — a descriptor [guard] block is all it takes, no bespoke install/verdict code beyond the engine arms. Shared marker/manifest/teardown machinery lives in src/sandbox/. The adapters ↔ sandbox module dependency is intentional: adapters own the guard descriptor and native hook surface, while sandbox owns harness-neutral boundary enforcement and lifecycle. New code follows that same rule rather than choosing a side by call direction. User-supplied descriptors may not declare [guard] — the guard fails open, so a mistyped user guard block would silently disarm it. On such a harness auto-arm quietly stays off (the preflight warns naming the detect-stray-writes fallback); only an explicit --guard is rejected in preflight, since a run the user asked to guard must not continue silently unguarded.

Shadow preflight

Why harness-specific: what "discoverable from the live environment" means is harness-native. Claude Code loads enabled plugins and its global skills dir. Codex loads repository-ancestor, user, and admin skill directories plus enabled installed plugins. OpenCode loads project and global .opencode, .claude, and .agents skill dirs — including skills installed for other harnesses. A logical eval skill present in any such source can contaminate the with/without comparison when dispatches load that source, even when the staged copy uses a unique slug.

What it unlocks: a build-time contamination warning (shared banner + schema-v3 plugin-shadow.json in the iteration dir), which aggregate folds into benchmark.json validity warnings. The runner scans every matrix environment and the shared policy groups scanner facts by logical skill and source class, records live/staged sources and affected cells, and assigns role-aware severity. operator-environment findings come from inherited global/plugin sources; codebase-sourced findings come from project roots the harness descriptor declares. Subject and asymmetric sibling collisions invalidate the comparison; symmetric sibling collisions warn. Because the scan runs before dispatch it reports risk, so the banner states the consequence conditionally; the verdict is settled afterwards by the session-surface sub-capability below. When the resolved descriptor declares isolates_live_sources = true, operator-source scan facts, intrinsic severity, and artifact are retained, but the banner becomes informational and aggregate omits those findings. Codebase findings use the eval's separate exclude_skill_sources policy and remain warnings when preserved. Schema-v2 and historical unversioned artifacts remain readable.

Session surface (sub-capability of transcript ingest)

Why harness-specific: only some CLIs announce what a session could discover. Claude Code's stream-json opens with a system/init event carrying skills and plugins; Codex announces a bare thread_id on thread.started and OpenCode's envelope carries no roster at all.

What it unlocks: verified shadow findings. record-runs collects each dispatch's reported roster — per round, so resumed turns are covered — into session-surface.json, then resolves every finding to resolved_severity in plugin-shadow.json. A finding refuted in every expected cell becomes isolated and produces no validity warning; a confirmed one keeps its severity and names the cells; an unsettled one says why. Refuting requires every expected cell to have reported and none to have seen the source, so a missing transcript yields unverified, never a refutation. The intrinsic severity is never rewritten. Where the assertion and the evidence disagree, the evidence wins and the contradiction is reported.

Fallback: no session-surface.json is written, every finding stays unverified, and isolates_live_sources remains the only way to record applied isolation. Absence of the file always means "cannot report", never "nothing loaded".

Descriptor fields: [transcript.extract.session_surface] selects records with an optional string-equality where filter, then reads skills_field and/or plugins_field. Plugin objects map through required plugin_name_field and optional plugin_id_field / plugin_version_field dotted paths; bare plugin strings remain valid names. The last matching record that supplies at least one configured array wins. Explicitly empty arrays are positive evidence of an empty roster, while no usable matching record means the harness could not report. The mapping is auxiliary and therefore must accompany either transcript.parser or declarative summary outputs. When explicitly declared it is authoritative, even if a named parser also has legacy surface support.

Capability: session-surface extraction is generic descriptor data. Parser-only descriptors retain named-parser fallback for compatibility; the built-in Claude descriptor uses the declarative mapping. This is additive descriptor schema surface, so it does not change the version or shape of session-surface.json.

Fallback: no preflight — the run proceeds with no shadow report. This does not prove the live environment is clean; the operator must check any harness-native global discovery sources.

Descriptor fields: the [shadow] table — preflight, plus optional isolates_live_sources (false by default). The latter records that every reported source is excluded from every initial and resumed eval-agent dispatch. It is checked against transcript evidence where the harness reports a session surface, and taken on trust where it does not. A built-in overlay may set it without repeating the inherited preflight; a new harness must still resolve to a preflight. It must not be used for partial isolation, and eval-magic never infers it by parsing shell templates — the operator-facing recipes and verification procedure are eval-magic docs isolation.

Capability: shadow.preflight names the scan (claude-plugins, cline-skills, codex-skills, or opencode-skills). It returns the harness-neutral PluginShadowReport from src/adapters/skill_shadow.rs. Harness modules emit discovery/root/remediation facts; grouping, severity, artifact serialization, the banner, and aggregate warnings are shared. The capability also selects a shared resolution policy (precedence or coexistence), with OpenCode's selected path obtained from a best-effort opencode debug skill probe only for duplicate runtime IDs. Codex's scan does not enumerate bundled system skills (no stable listing exists), and OpenCode's does not scan config-declared skills.paths/skills.urls sources.

Native plan mode

Why harness-specific: the read-only planning mode is the harness's own — a permission mode for Claude Code, a built-in agent for OpenCode — and so is the way the agent presents its plan.

What it unlocks: evals that declare plan_mode: true. The driver dispatches the opening round with the planning arguments, lets the agent present a plan, approves it with one fixed message, and resumes the same session with the act arguments; the eval's turns or responder then proceed as usual. The approved plan is saved as outputs/plan.md and rendered in the judge evidence bundle. plan_file is the deterministic signal that the plan was presented; without one the eval's responder decides, which is why run requires a responder on a harness without a plan file.

Fallback: none. run rejects a plan-mode eval for a harness without [plan_mode], before any environment is built, the way it rejects multi-turn evals for a harness without [conversation].

Descriptor fields: the [plan_mode] table — plan_args and act_args fill the {mode_args} slot that both dispatch.exec_template and conversation.resume_exec_template must carry (the act arguments also render for judge, responder, and probe dispatches); the optional [plan_mode.plan_file] table names the file the harness writes its plan to (root, ~-expanded, and the write tool's content_field). Writes under that root are allowed by the write guard and the stray-write audit. Requires [conversation]. Validation rejects a table without the slot in both templates, and a slot with no table to fill it.

Capability: plan-mode in harness list. Claude Code and OpenCode declare it; Codex exec has no plan-mode flag and Cline cannot resume a session, so neither does.

Dispatch commands

Why harness-specific: the command line the runner spawns is the harness's CLI.

What it unlocks: eval-magic dispatch itself. Without an exec_template there is nothing for the runner to run, and dispatch fails for that harness.

Fallback: none. The run preflight rejects a descriptor without exec_template, so the gap surfaces before a workspace is built.

Descriptor fields: the [dispatch] table — env, exec_template, next_steps_template, manifest_template, guard_args, model_note. Templates carry {model_arg}/{guard_args} slots the renderer fills for eval-agent dispatches, plus {mode_args} when a [plan_mode] table backs it; a judge dispatch reuses exec_template with guard_args deliberately empty, because judges run from the iteration directory outside every guarded task env. env contains non-secret eval-agent defaults, applied per task when the runner spawns the command; unset keys inherit the host environment and no timezone default is imposed. Validation rejects a template whose placeholder has no backing field.

Current support

The resolved descriptor registry is the source of truth for which harness has which enhancement. eval-magic harness list summarizes the registered capabilities; eval-magic harness show <label> prints the resolved descriptor behind that summary. Both surfaces update from descriptor data rather than a hand-maintained support table.

Adding a new harness

1. Start as a user descriptor — scaffold it with eval-magic harness init <label> (a commented template plus a notes skeleton, lint-clean as written), fill in verified values per the BYOH guide, and iterate with harness lint/show and real runs. No Rust, no rebuild; this is also where the descriptor's field set gets proven.
2. Promote to a built-in once it earns bundling: move the file to harnesses/<label>.toml and add it to EMBEDDED_DESCRIPTORS (src/adapters/descriptor.rs). The registry keys on the descriptor's label. This is the data-only descriptor-contribution PR described in byoh.md's "Upstreaming your descriptor" — open it with .github/PULL_REQUEST_TEMPLATE/harness-descriptor.md.
3. Create docs/<harness>-notes.md with the implementation notes discovered along the way — promoted from the scaffolded .eval-magic/harnesses/<label>-notes.md. The PR template requires it: the notes file is where the don't-guess guardrail's verification evidence lives.
4. Confirm eval-magic harness list and harness show <label> report the built-in and only the capabilities its descriptor actually declares.
5. Wire enhancements in leverage order — the dispatch command and transcript ingest first (they carry the most fidelity and are prerequisites for conversation resume), then conversation resume, staging, model flag, guard (guard requires built-in status — user descriptors may not declare one). Most enhancements are descriptor fields; add a named capability in src/adapters/capabilities.rs (plus its src/adapters/<harness>/ module) only when the harness's stream or hooks are incompatible with every existing capability.

Guardrails

• Cross-harness compatibility is enforced. A change for one harness must not regress another; load-time descriptor validation (src/adapters/descriptor.rs), the golden-artifact fixtures under tests/golden/ (re-bless deliberate output changes with GOLDEN_BLESS=1 cargo test golden_), the per-value pins in src/adapters/harness.rs, and the per-harness integration tests under tests/run/ are the floor.
• One enhancement per PR. Wiring a harness happens one capability at a time.
• Don't guess harness details. CLI flags, hook shapes, and event vocabularies come from the harness's own documentation or observed output — record what you verified in the harness's notes file. The harness init scaffold embeds these prompts inline at every field, and the harness-descriptor PR template requires the per-field source attestation.