feat(eval): support selectable evaluation agent CLIs

[benym]

🔎 Existing issue check

• I have searched the existing issues.

🧭 Problem

comet eval currently launches the Local or LangSmith harness through a Claude Code-specific execution path. Users cannot select Codex CLI or Qoder CLI as the platform that runs the evaluated Skill, the auto_user simulator, and the optional LLM Judge, so cross-platform evaluation is not reproducible through the existing CLI or manifest.

✨ Proposed solution

Extend the existing comet eval command with one option; do not add a new command:

comet eval <target> --agent claude-code|codex|qoder

Allow the same choice to be persisted in comet/eval.yaml:

execution:
  agent: codex

Resolve the platform with this precedence:

--agent > execution.agent > claude-code

claude-code remains the default so existing invocations and manifests retain their current behavior.

Agent adapter architecture

Introduce one shared Agent Adapter contract with three first-party implementations:

• ClaudeCodeAdapter
• CodexAdapter
• QoderAdapter

Resolve one adapter per eval run and use it for every LLM role:

• subject: executes the evaluated Skill;
• simulator: acts as the user in auto_user mode;
• judge: performs the optional post-run LLM evaluation.

The three roles use the same selected platform but separate sessions and context. Existing role-specific model, credential, prompt, timeout, and Judge configuration semantics should remain intact; the adapter maps those settings to the selected CLI/backend.

Each adapter owns platform-specific CLI discovery, version reporting, argument construction, single-turn and resumable multi-turn execution, and normalization of output, token/cost telemetry, tool events, and observed Skill invocation evidence. Task orchestration, treatments, Docker/workspace isolation, validators, Rubric, Pass@k/Pass^k, quality gates, and report generation remain platform-independent.

Execution and evidence rules

• Validate the agent identifier before execution and check the selected CLI, credentials, model configuration, and required session-continuation capability before starting workloads.
• interaction.mode = none runs one subject session.
• interaction.mode = auto_user alternates isolated subject and simulator sessions through the selected adapter.
• The optional Judge runs afterward in a third isolated session through the same adapter.
• Persist the selected platform, CLI version, role models, session/turn metadata, normalized telemetry, and raw-log references in experiment evidence.
• Record unavailable token or cost telemetry as N/A, never 0.
• Preserve raw logs when a platform cannot expose structured tool events.
• When require_skill_invocation = true, lack of observable invocation evidence must fail explicitly; artifacts must not be used to infer that a Skill was invoked.
• Subject or simulator execution failures are harness/agent failures. Optional Judge failures preserve deterministic Rubric results and record the Judge failure without inventing a score.
• --collect validates the identifier and manifest but does not require the selected CLI or credentials to be installed.
• --suite and --agent remain orthogonal so Local, LangSmith, and future suites reuse the same adapters.

Acceptance coverage

Add focused coverage for:

• CLI/manifest/default precedence and invalid agent values;
• unchanged Claude Code default behavior;
• all three roles using one selected platform with isolated sessions;
• single-turn and auto_user execution;
• adapter event and telemetry normalization;
• N/A handling and required Skill invocation evidence;
• missing CLI, credentials, model, and continuation capability errors;
• --collect avoiding runtime probes;
• Local/LangSmith routing with each supported agent.

🎯 Primary area

Eval (domains/eval)

🪐 Workflow phase

Not phase-specific

🔀 Alternatives considered

1. Add platform branches directly to the existing shell scripts. This is initially smaller but entangles three roles with three CLIs and makes future suites and agents harder to maintain.
2. Allow arbitrary executable/argument templates. This is flexible but cannot reliably validate session behavior or normalize Skill invocation, token, and cost evidence.

The first-party Adapter boundary keeps the public CLI small while isolating platform-specific behavior.

🧰 Compatibility notes

This is additive: one --agent option and one optional execution.agent manifest field. Existing commands, manifests, task definitions, Claude Code behavior, Local/LangSmith suites, deterministic validators, Rubric calculations, and report semantics must remain compatible.

🧩 Additional context

Comet already recognizes Codex and Qoder as installation platforms. This issue concerns the Agent backend that executes the complete eval interaction and Judge flow, not Skill installation alone.