docs: clarify eval and experiment boundaries (#1517)

Entire-Checkpoint: 335841c2d678

Author: christso

docs/adr/2026-06-17-harbor-runner-boundary.md

@@ -37,42 +37,110 @@ Harbor should own:
 - Harbor `task.toml` files and Harbor YAML config;
 - Opik trace upload through Harbor when enabled.
 
+## Alignment with experiment separation
+
+The 2026-06-23 experiment/eval separation decision makes runtime binding an
+experiment concern. Harbor execution should follow the same split:
+
+- AgentV eval YAML remains the authoring or selection surface for what benchmark
+  suite is being evaluated.
+- AgentV experiment YAML selects or pins the Harbor runner, candidate
+  agent/model, run policy, and other runtime binding.
+- Harbor-authored YAML remains Harbor's own config surface when the standard
+  suite needs Harbor-specific task packaging or verifier settings.
+
+This means the examples below describe the desired logical fields, but new
+runtime fields should be placed on an experiment unless they are genuinely part
+of the benchmark suite identity. Do not put candidate agent/model binding in the
+eval file for new AgentV-native examples.
+
 ## Minimal future config surface
 
-The AgentV eval file should select Harbor with a nested runner config:
+An AgentV eval suite can select the benchmark source without copying Harbor's
+task schema or claiming to be the runtime runner:
 
 ```yaml
-name: swebench-verified-codex
+name: swebench-verified
 
-execution:
-  runner: harbor
-  harbor:
-    dataset: swebench-verified
-    agent: codex
-    model: openai/gpt-5-mini
+source:
+  type: harbor
+  dataset: swebench-verified
+```
+
+The corresponding experiment selects how that suite runs:
+
+```yaml
+name: swebench-verified-codex
+target: codex-gpt5-mini
+evals: evals/swebench-verified.eval.yaml
+runner:
+  type: harbor
+  options:
     opik:
       enabled: true
 ```
 
 For a Harbor-authored YAML file, use `config` instead of `dataset`:
 
 ```yaml
-execution:
-  runner: harbor
-  harbor:
-    config: ./harbor/swebench-verified.yaml
+source:
+  type: harbor
+  config: ./harbor/swebench-verified.yaml
 ```
 
 The first implementation should accept exactly one Harbor source selector:
 `dataset` for a known Harbor dataset id, or `config` for an existing Harbor YAML
 file. There should be no precedence rule between them. If both are set, fail
 validation and ask the user to choose one.
 
-Keep Harbor-specific options nested under `execution.harbor`. Do not add
-top-level AgentV fields for Harbor task packaging, verifier images, task patches,
-or Docker/Compose adapter settings. If a Harbor option becomes too specific to
-standardize, users should put it in the referenced Harbor YAML file instead of
-AgentV adding a pass-through field.
+Do not combine Harbor suite selection with candidate binding in the eval file:
+
+```yaml
+# Avoid in eval.yaml
+execution:
+  runner: harbor
+  harbor:
+    dataset: swebench-verified
+    agent: codex
+    model: openai/gpt-5-mini
+```
+
+Split that shape across the suite and experiment instead:
+
+```yaml
+# evals/swebench-verified.eval.yaml
+name: swebench-verified
+source:
+  type: harbor
+  dataset: swebench-verified
+```
+
+```yaml
+# experiments/swebench-verified-codex.yaml
+name: swebench-verified-codex
+target: codex
+model: openai/gpt-5-mini
+evals: evals/swebench-verified.eval.yaml
+runner:
+  type: harbor
+```
+
+Keep Harbor suite source selection under `source` in the eval suite. Keep
+experiment-side runner selection under `runner.type`, with runner knobs under
+`runner.options`. The eval suite answers "where do these cases come from?"; the
+experiment answers "how is this run executed?" Do not use `execution.runner` in
+new eval-suite examples because that name collides with the experiment runner.
+Do not repeat the runner discriminator as `runner.harbor.options`; `type:
+harbor` already provides that namespace.
+
+Do not add top-level AgentV fields for Harbor task packaging, verifier images,
+task patches, or Docker/Compose adapter settings. If a Harbor option becomes too
+specific to standardize, users should put it in the referenced Harbor YAML file
+instead of AgentV adding a pass-through field.
+
+If the Harbor integration later changes the eval source or experiment runner
+schema, this ADR should be updated with the final shape. The boundary decision
+is stable: Harbor runtime binding is not an eval-case schema extension.
 
 ## CLI invocation strategy
 
@@ -82,8 +150,9 @@ Native evals continue to run with the existing command:
 agentv eval evals/native.eval.yaml --target codex
 ```
 
-Harbor-backed evals should use the same top-level entrypoint and dispatch based
-on `execution.runner`:
+Harbor-backed evals should use the same top-level entrypoint. If no explicit
+experiment runner is configured, AgentV may infer Harbor execution from
+`source.type: harbor`:
 
 ```bash
 agentv eval evals/swebench-harbor.eval.yaml
@@ -105,9 +174,9 @@ agentv results import harbor --job <harbor-job-id>
 ```
 
 Do not overload native `--target` semantics in the first Harbor runner slice.
-Harbor `agent`, `model`, and matrix behavior should come from
-`execution.harbor` or the referenced Harbor YAML until repeated usage proves a
-shared AgentV flag is needed.
+Harbor `agent`, `model`, and matrix behavior should come from the experiment or
+the referenced Harbor YAML until repeated usage proves a shared AgentV flag is
+needed.
 
 ## Unsupported fields and non-goals
 
@@ -128,8 +197,9 @@ standard-suite path.
 ## Implementation sequencing
 
 1. Document the native-vs-Harbor boundary and commit alias rules.
-2. Add schema validation for optional `execution.runner` and
-   `execution.harbor`, with no changes to native workspace acquisition.
+2. Add schema validation for eval-suite `source.type: harbor` and exactly one of
+   `source.dataset` or `source.config`, plus experiment `runner.type` and
+   `runner.options`, with no changes to native workspace acquisition.
 3. Add a Harbor launch adapter that records job identity and status.
 4. Add a Harbor result importer that maps rewards, exceptions, timings,
    artifacts, and Opik trace URLs into AgentV run bundles.

docs/adr/2026-06-23-experiments-vs-eval-separation.md

@@ -37,10 +37,22 @@ This decision must also preserve AgentV's existing product boundary:
 
 ## Vocabulary
 
-An eval is a frozen task definition. It includes the prompt or dataset, expected
-behavior, task-owned workspace fixtures, and assertions. AgentV's LLM-judge,
-code-grader, deterministic assertions, and hidden or explicit evaluation
-criteria belong here.
+An eval suite is a frozen task-definition boundary. It includes suite metadata,
+shared prompt/context, case references, shared assertions or graders, and
+task-owned workspace fixtures. AgentV's LLM-judge, code-grader, deterministic
+assertions, and hidden or explicit evaluation criteria belong here.
+
+An eval case is one atomic task inside a suite. It includes the case id, prompt
+or input, criteria, expected output or reference behavior, case metadata, and
+case-specific workspace overrides. A suite can inline cases, point to
+`cases.yaml`/JSONL, or use a directory convention where each case owns files such
+as `TASK.txt`, `PROMPT.md`, `answer/`, or `grader.test.ts`.
+
+In that directory-convention form, `EVAL.yaml` may be thin or inferred by a
+loader, but the suite layer is still present conceptually: the directory
+convention plus runner adapter is the suite contract. This distinction matters
+because AgentV is a reusable framework, not a single benchmark harness whose
+suite semantics can live only in code.
 
 An experiment is a committed or generated run definition. It declares which
 agent, target, provider, model, harness options, setup steps, run count, timeout,
@@ -62,6 +74,13 @@ task inputs, datasets, assertions, and task fixtures. They should not be the
 canonical place for which agent, model, harness, setup injection, sandbox, or run
 matrix executes the task.
 
+For simple projects, an eval-only run remains valid. AgentV treats the implicit
+experiment label as `default` unless a committed experiment is configured. For
+specialized harnesses that already have a strong directory contract, AgentV may
+support loaders that infer the suite from the directory instead of requiring a
+separate YAML file, but those loaders must still lower into the same suite/case
+concepts.
+
 Experiment files will live under `experiments/` by convention. AgentV will
 support YAML as the canonical authoring path for the abstraction story and TypeScript
 as the power-user escape hatch:
@@ -85,6 +104,48 @@ setup:
   - script: cp skills/copilot/AGENTS.md AGENTS.md
 ```
 
+## Workspace boundary
+
+Workspace config belongs with the eval suite or case when it defines the task
+scenario being replayed. Examples:
+
+- clone `org/repo` at a specific `commit` or `base_commit`;
+- copy starter files, failing tests, fixtures, or issue prompts;
+- run task-owned setup hooks that prepare the repo state required by the case;
+- declare per-case repo pins or fixture overrides.
+
+Experiment setup belongs with the experiment when it changes the runtime
+condition being compared. Examples:
+
+- choose `codex` versus `claude` targets;
+- inject an `AGENTS.md`, skill, guideline file, or tool config for an A/B run;
+- choose repeat/run policy, timeout, workers, budget, or sandbox mode;
+- select a subset of suites or cases for a run campaign.
+
+Rule of thumb: if changing it changes the task being evaluated, put it in the
+suite or case workspace. If changing it changes the candidate or run condition
+measured against the same task, put it in the experiment.
+
+## Directory-style evals
+
+Convex-style harnesses are a useful counterexample to requiring YAML for every
+case. A product-specific benchmark can encode each case as a directory with a
+task prompt, reference solution, and executable grader. In AgentV terms, that is
+not `experiment -> eval case` with no suite; it is an implicit suite contract
+provided by the loader:
+
+```text
+evals/<category>/<case>/
+  TASK.txt          # case input
+  answer/           # reference fixture
+  grader.test.ts    # code-grader assertion
+```
+
+AgentV should support this as an import/loader shape when useful, but the core
+contract remains `experiment -> eval suite -> eval case`. The experiment applies
+runtime bindings to the selected suites/cases; it does not own the prompt,
+expected behavior, or grading contract.
+
 `config.yaml` will gain a default experiment pointer so existing `agentv eval`
 usage keeps working: