docs: clarify eval and experiment boundaries

Entire-Checkpoint: 335841c2d678

Author: christso

docs/adr/2026-06-17-harbor-runner-boundary.md

@@ -37,19 +37,46 @@ Harbor should own:
 - Harbor `task.toml` files and Harbor YAML config;
 - Opik trace upload through Harbor when enabled.
 
+## Alignment with experiment separation
+
+The 2026-06-23 experiment/eval separation decision makes runtime binding an
+experiment concern. Harbor execution should follow the same split:
+
+- AgentV eval YAML remains the authoring or selection surface for what benchmark
+  suite is being evaluated.
+- AgentV experiment YAML selects the Harbor runner, candidate agent/model, run
+  policy, and other runtime binding.
+- Harbor-authored YAML remains Harbor's own config surface when the standard
+  suite needs Harbor-specific task packaging or verifier settings.
+
+This means the examples below describe the desired logical fields, but new
+runtime fields should be placed on an experiment unless they are genuinely part
+of the benchmark suite identity. Do not put candidate agent/model binding in the
+eval file for new AgentV-native examples.
+
 ## Minimal future config surface
 
-The AgentV eval file should select Harbor with a nested runner config:
+An AgentV eval suite can select the benchmark identity without copying Harbor's
+task schema:
 
 ```yaml
-name: swebench-verified-codex
+name: swebench-verified
 
 execution:
   runner: harbor
   harbor:
     dataset: swebench-verified
-    agent: codex
-    model: openai/gpt-5-mini
+```
+
+The corresponding experiment selects how that suite runs:
+
+```yaml
+name: swebench-verified-codex
+target: codex-gpt5-mini
+evals: evals/swebench-verified.eval.yaml
+runner:
+  type: harbor
+  harbor:
     opik:
       enabled: true
 ```
@@ -74,6 +101,11 @@ or Docker/Compose adapter settings. If a Harbor option becomes too specific to
 standardize, users should put it in the referenced Harbor YAML file instead of
 AgentV adding a pass-through field.
 
+If the Harbor integration later chooses to move `runner.type: harbor` entirely
+into experiment files, this ADR should be updated with the final schema. The
+boundary decision is stable: Harbor runtime binding is not an eval-case schema
+extension.
+
 ## CLI invocation strategy
 
 Native evals continue to run with the existing command:
@@ -105,9 +137,9 @@ agentv results import harbor --job <harbor-job-id>
 ```
 
 Do not overload native `--target` semantics in the first Harbor runner slice.
-Harbor `agent`, `model`, and matrix behavior should come from
-`execution.harbor` or the referenced Harbor YAML until repeated usage proves a
-shared AgentV flag is needed.
+Harbor `agent`, `model`, and matrix behavior should come from the experiment or
+the referenced Harbor YAML until repeated usage proves a shared AgentV flag is
+needed.
 
 ## Unsupported fields and non-goals
 

docs/adr/2026-06-23-experiments-vs-eval-separation.md

@@ -37,10 +37,22 @@ This decision must also preserve AgentV's existing product boundary:
 
 ## Vocabulary
 
-An eval is a frozen task definition. It includes the prompt or dataset, expected
-behavior, task-owned workspace fixtures, and assertions. AgentV's LLM-judge,
-code-grader, deterministic assertions, and hidden or explicit evaluation
-criteria belong here.
+An eval suite is a frozen task-definition boundary. It includes suite metadata,
+shared prompt/context, case references, shared assertions or graders, and
+task-owned workspace fixtures. AgentV's LLM-judge, code-grader, deterministic
+assertions, and hidden or explicit evaluation criteria belong here.
+
+An eval case is one atomic task inside a suite. It includes the case id, prompt
+or input, criteria, expected output or reference behavior, case metadata, and
+case-specific workspace overrides. A suite can inline cases, point to
+`cases.yaml`/JSONL, or use a directory convention where each case owns files such
+as `TASK.txt`, `PROMPT.md`, `answer/`, or `grader.test.ts`.
+
+In that directory-convention form, `EVAL.yaml` may be thin or inferred by a
+loader, but the suite layer is still present conceptually: the directory
+convention plus runner adapter is the suite contract. This distinction matters
+because AgentV is a reusable framework, not a single benchmark harness whose
+suite semantics can live only in code.
 
 An experiment is a committed or generated run definition. It declares which
 agent, target, provider, model, harness options, setup steps, run count, timeout,
@@ -62,6 +74,13 @@ task inputs, datasets, assertions, and task fixtures. They should not be the
 canonical place for which agent, model, harness, setup injection, sandbox, or run
 matrix executes the task.
 
+For simple projects, an eval-only run remains valid. AgentV treats the implicit
+experiment label as `default` unless a committed experiment is configured. For
+specialized harnesses that already have a strong directory contract, AgentV may
+support loaders that infer the suite from the directory instead of requiring a
+separate YAML file, but those loaders must still lower into the same suite/case
+concepts.
+
 Experiment files will live under `experiments/` by convention. AgentV will
 support YAML as the canonical authoring path for the abstraction story and TypeScript
 as the power-user escape hatch:
@@ -85,6 +104,48 @@ setup:
   - script: cp skills/copilot/AGENTS.md AGENTS.md
 ```
 
+## Workspace boundary
+
+Workspace config belongs with the eval suite or case when it defines the task
+scenario being replayed. Examples:
+
+- clone `org/repo` at a specific `commit` or `base_commit`;
+- copy starter files, failing tests, fixtures, or issue prompts;
+- run task-owned setup hooks that prepare the repo state required by the case;
+- declare per-case repo pins or fixture overrides.
+
+Experiment setup belongs with the experiment when it changes the runtime
+condition being compared. Examples:
+
+- choose `codex` versus `claude` targets;
+- inject an `AGENTS.md`, skill, guideline file, or tool config for an A/B run;
+- choose repeat/run policy, timeout, workers, budget, or sandbox mode;
+- select a subset of suites or cases for a run campaign.
+
+Rule of thumb: if changing it changes the task being evaluated, put it in the
+suite or case workspace. If changing it changes the candidate or run condition
+measured against the same task, put it in the experiment.
+
+## Directory-style evals
+
+Convex-style harnesses are a useful counterexample to requiring YAML for every
+case. A product-specific benchmark can encode each case as a directory with a
+task prompt, reference solution, and executable grader. In AgentV terms, that is
+not `experiment -> eval case` with no suite; it is an implicit suite contract
+provided by the loader:
+
+```text
+evals/<category>/<case>/
+  TASK.txt          # case input
+  answer/           # reference fixture
+  grader.test.ts    # code-grader assertion
+```
+
+AgentV should support this as an import/loader shape when useful, but the core
+contract remains `experiment -> eval suite -> eval case`. The experiment applies
+runtime bindings to the selected suites/cases; it does not own the prompt,
+expected behavior, or grading contract.
+
 `config.yaml` will gain a default experiment pointer so existing `agentv eval`
 usage keeps working: