feat(core): support flatter eval imports

Author: christso

apps/web/src/content/docs/docs/evaluation/eval-files.mdx

@@ -17,15 +17,15 @@ other suites. Raw case files are reusable data inputs, not a second runnable
 experiment format.
 
 - A **task suite** is eval YAML that owns task context: `workspace`, shared
-  `input`, shared `assertions`, and test cases. It can run directly or be
-  imported with `type: suite`.
+  `input`, shared `assertions`, fixtures, graders, and test cases. It can run
+  directly or be imported through `imports.suites`.
 - A **raw case file** is a YAML/JSONL array, directory, or glob of cases. Import
-  it with `tests: ./cases.yaml`, string shorthand, or `type: tests`; parent
+  it with `imports.tests`, `tests: ./cases.yaml`, or string shorthand; parent
   suite context applies because raw cases do not carry their own suite context.
 - A **wrapper eval** is eval YAML that imports one or more suites with
-  `type: suite` and binds runtime policy in its inline `experiment:` block.
+  `imports.suites` and binds runtime policy in its inline `experiment:` block.
   Wrapper evals can live anywhere in the repo. A wrapper that imports suites
-  with `type: suite` must not define parent `workspace`; imported suites own
+  with `imports.suites` must not define parent `workspace`; imported suites own
   task environment. Machine-local existing workspace paths belong in CLI flags
   or `config.local.yaml`, not eval YAML.
 
@@ -61,25 +61,29 @@ A wrapper eval stays ordinary eval YAML while choosing runtime policy:
 ```yaml
 # experiments/refunds-codex.eval.yaml
 experiment:
-  name: refunds-codex
   target: codex-gpt5
   repeat:
     count: 2
     strategy: pass_all
 
+imports:
+  suites:
+    - path: ../evals/suites/refunds.eval.yaml
+  tests:
+    - path: ../evals/cases/refund-smoke.cases.yaml
+
 tests:
-  - include: ../evals/suites/refunds.eval.yaml
-    type: suite
-  - include: ../evals/cases/refund-smoke.cases.yaml
-    type: tests
+  - id: local-edge-case
+    input: Can a final-sale item be refunded after damage in transit?
+    expected_output: Explain the final-sale exception for damaged transit.
 ```
 
 The `experiments/` directory in that example is optional and user-owned. AgentV
 does not infer behavior from the path; the wrapper runs because it is eval YAML
 with an inline `experiment:` block. The wrapper owns runtime policy only. Put
 workspace setup in imported child suites. Parent workspace-affecting fields,
 including top-level `workspace`, are for parent-owned raw cases, including
-cases imported with `type: tests`. Runtime workspace path overrides belong in
+cases imported with `imports.tests`. Runtime workspace path overrides belong in
 CLI flags or `.agentv/config.local.yaml`; repos, hooks, templates, Docker
 config, env checks, and isolation belong in top-level or case-level
 `workspace`.
@@ -112,7 +116,8 @@ tests:
 | `category` | Optional slash-delimited analytics taxonomy path. Overrides the category derived from the eval file path. |
 | `experiment` | Runtime policy (`target`, `targets`, `workers`, `repeat`, `threshold`, `timeout_seconds`, `budget_usd`, etc.) |
 | `workspace` | Suite-level task environment — inline object or string path to an [external workspace file](/docs/guides/workspace-pool/#external-workspace-config). Repo entries declare identity and checkout pins; acquisition is covered in [Workspace Architecture](/docs/guides/workspace-architecture/#repo-provenance-vs-acquisition). |
-| `tests` | Array of individual tests, include entries, or a string path to an external file or directory. Tests and include entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. |
+| `imports` | Optional import groups. `imports.suites` imports full child eval suites with their task context. `imports.tests` imports raw test rows into this file's context. Import entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. |
+| `tests` | Inline raw tests or a string path to an external raw-case file or directory. Legacy `tests[].include` entries still load with a migration warning; prefer `imports.suites` or `imports.tests`. |
 | `assertions` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test |
 | `input` | Suite-level input messages prepended to each test's input unless `execution.skip_defaults: true` is set on the test |
 
@@ -332,13 +337,26 @@ use direct paths, directories, or globs:
 ```yaml
 tests:
   - ./cases/*.cases.yaml
-  - include: ./suites/*.eval.yaml
-    type: suite
 ```
 
-String shorthand is raw-case-only. Import reusable task suites with object
-entries using `include:` and `type: suite`; use `type: tests` when you want to
-drop suite context and import only raw cases.
+String shorthand is raw-case-only. Import reusable task suites through
+`imports.suites`; use `imports.tests` when you want to drop suite context and
+import only raw cases into the parent context:
+
+```yaml
+imports:
+  suites:
+    - path: ./suites/*.eval.yaml
+  tests:
+    - path: ./cases/regression.jsonl
+
+tests:
+  - id: local-edge-case
+    input: ...
+```
+
+Legacy `tests[].include` entries still load with a migration warning for older
+eval files, but new evals should use `imports.suites` or `imports.tests`.
 
 ### Raw Cases as Directory Paths
 

apps/web/src/content/docs/docs/evaluation/experiments.mdx

@@ -64,75 +64,81 @@ experiment:
   workers: 2
 
 tests:
-  - include: ../evals/suites/refunds.eval.yaml
-    type: suite
-  - include: ../evals/cases/refund-smoke.cases.yaml
-    type: tests
+  - id: local-edge-case
+    input: Check a damaged final-sale refund.
+
+imports:
+  suites:
+    - path: ../evals/suites/refunds.eval.yaml
+  tests:
+    - path: ../evals/cases/refund-smoke.cases.yaml
 ```
 
 The `experiments/` folder is optional and user-owned. AgentV does not scan it
 for special files or infer runtime behavior from the path; the same wrapper eval
 could live under `evals/wrappers/`, `benchmarks/`, or beside the suite it runs.
 
-## Tests Imports
+## Suite And Test Imports
 
-Use `tests[]` for composition, imports, and selection.
+Use `imports.suites` for full child suites and `imports.tests` for raw test
+rows. Inline `tests` remain raw cases owned by the current file.
 
 ```yaml
+imports:
+  suites:
+    - path: evals/support/*.eval.yaml
+      select:
+        test_ids:
+          - refund-*
+          - missing-order-date
+        tags: regression
+        metadata:
+          priority: high
+      run:
+        threshold: 1.0
+        repeat:
+          count: 2
+          strategy: pass_all
+  tests:
+    - path: cases/*.cases.yaml
+    - path: cases/regression.jsonl
+
 tests:
-  - include: evals/support/*.eval.yaml
-    type: suite
-    select:
-      test_ids:
-        - refund-*
-        - missing-order-date
-      tags: regression
-      metadata:
-        priority: high
-    run:
-      threshold: 1.0
-      repeat:
-        count: 2
-        strategy: pass_all
-  - include: cases/*.cases.yaml
-    type: tests
-  - include: cases/regression.jsonl
-    type: tests
   - cases/smoke/*.cases.yaml
 ```
 
-`type: suite` preserves the imported suite's task contract: metadata,
+`imports.suites` preserves the imported suite's task contract: metadata,
 `workspace`, shared `input`, shared `assertions`, and tests. The parent eval
 still owns the single run bundle and runtime policy. Child suite
 `experiment:` blocks are ignored when imported; use parent `experiment:` for
-run policy and `tests[].run` for scoped threshold, repeat, timeout, or budget
+run policy and import `run:` for scoped threshold, repeat, timeout, or budget
 overrides.
 
-A parent eval that imports any `type: suite` entry must not define top-level
+A parent eval that imports any `imports.suites` entry must not define top-level
 `workspace`. Imported suites own task environment. If the parent should provide
-workspace context, import raw cases with `type: tests` or shorthand paths
+workspace context, import raw cases with `imports.tests` or shorthand paths
 instead of importing an eval suite.
 
-`type: tests` imports only raw test entries. It intentionally drops shared
+`imports.tests` imports only raw test entries. It intentionally drops shared
 context from an imported eval suite, so parent suite fields apply to those raw
 cases.
 
-`tests[].select.test_ids` filters imported test IDs with glob patterns.
-`tests[].select.tags` filters each imported case's effective `metadata.tags`.
+Import `select.test_ids` filters imported test IDs with glob patterns.
+Import `select.tags` filters each imported case's effective `metadata.tags`.
 Effective case tags are suite-first and deduped:
 `suite.tags + suite.metadata.tags + test.metadata.tags`. Top-level suite `tags`
 still remain suite identity metadata for discovery and reporting; selection reads
-the merged case metadata view. `tests[].select.metadata` filters case metadata by
+the merged case metadata view. Import `select.metadata` filters case metadata by
 key/value, where selector values may be scalars or lists. Globbed include paths
 are resolved in deterministic path order, then test order.
 
 String-valued `tests` and string entries inside `tests[]` are raw-case import
-shorthand. They are equivalent to `include` with `type: tests` and may point at
+shorthand. They are equivalent to `imports.tests` and may point at
 raw case files, directories, or globs. Importing another eval suite must use
-object form with `include:` and `type: suite`.
+`imports.suites`.
 
-Suite imports are resolved as a deterministic include graph. Circular `type:
-suite` imports fail validation with the import chain; raw-case shorthand does
+Suite imports are resolved as a deterministic include graph. Circular
+`imports.suites` imports fail validation with the import chain; raw-case shorthand does
 not recursively load suite runtime blocks.
 
 Imported suite rows keep their source suite metadata in `run_manifest.jsonl`. Use each
@@ -145,7 +151,7 @@ Use scoped `run:` blocks for result interpretation and scheduling policies that
 vary by include group or test case. Precedence is:
 
 ```text
-test.run > tests[].run > parent experiment
+test.run > import run > parent experiment
 ```
 
 ```yaml
@@ -156,26 +162,26 @@ experiment:
     count: 3
     strategy: pass_at_k
 
-tests:
-  - include: ./evals/flaky-agentic/**/*.eval.yaml
-    type: suite
-    select:
-      tags: [agentic]
-    run:
-      repeat:
-        count: 3
-        strategy: pass_at_k
-
-  - include: ./evals/regression/**/*.eval.yaml
-    type: suite
-    select:
-      tags: [must-pass]
-    run:
-      threshold: 1.0
-      repeat:
-        count: 2
-        strategy: pass_all
+imports:
+  suites:
+    - path: ./evals/flaky-agentic/**/*.eval.yaml
+      select:
+        tags: [agentic]
+      run:
+        repeat:
+          count: 3
+          strategy: pass_at_k
+
+    - path: ./evals/regression/**/*.eval.yaml
+      select:
+        tags: [must-pass]
+      run:
+        threshold: 1.0
+        repeat:
+          count: 2
+          strategy: pass_all
 
+tests:
   - id: critical-case
     input: "..."
     criteria: Must pass exactly

docs/adr/0006-separate-experiments-from-eval-definitions.md

@@ -159,17 +159,22 @@ multi-file command such as `agentv eval a.eval.yaml b.eval.yaml` remains a batch
 of independent eval-suite runs, rather than one implicit wrapper experiment with
 shared mutable setup.
 
-## Tests Import Surface
+## Suite And Test Import Surface
 
-`tests:` is the only composition, import, and selection surface.
+`imports.suites`, `imports.tests`, and inline `tests` are the authored
+composition surfaces.
 
-`include:` accepts direct paths and glob patterns. Include entries are
-structurally identified by the `include` field, so their `type:` field can use
-the ordinary AgentV wire-format discriminator without ambiguity:
+`imports.suites` imports full child eval suites. `imports.tests` imports raw
+case rows into the parent file's task context. Inline `tests` are raw cases
+owned by the current file. Legacy `tests[].include` entries remain a migration
+path for existing eval files, but new evals should use the flatter import
+groups.
 
-- `include: **/*.eval.yaml` imports eval suites.
-- `include: **/*.cases.yaml` imports raw cases.
-- `include: **/*.jsonl` imports raw cases.
+`path:` accepts direct paths and glob patterns:
+
+- `imports.suites[].path: **/*.eval.yaml` imports eval suites.
+- `imports.tests[].path: **/*.cases.yaml` imports raw cases.
+- `imports.tests[].path: **/*.jsonl` imports raw cases.
 
 `select:` filters imported cases with the same general selection shape used by
 old experiment suite selection. `select.test_ids` filters by test id and maps
@@ -216,29 +221,29 @@ In that example, `case-1` matches `select.tags: cargowise`.
 Imported tests run in deterministic order: resolved path first, then the test
 order inside each resolved source.
 
-`type: suite` preserves the imported suite task contract. That includes suite
+`imports.suites` preserves the imported suite task contract. That includes suite
 metadata, `workspace`, shared `input`, shared `assertions`, and tests. The
 parent eval still owns one run bundle and one runtime policy. Child suite
-`experiment:` blocks are ignored when imported with `type: suite`; they do not
+`experiment:` blocks are ignored when imported through `imports.suites`; they do not
 fall back into the parent run. Scoped runtime overrides that the parent wants
-to apply to imported tests live in `tests[].run`.
+to apply to imported tests live on the import entry's `run:` block.
 
-A parent eval that imports any child eval suite with `type: suite` must not
+A parent eval that imports any child eval suite with `imports.suites` must not
 define parent `workspace`. The wrapper owns runtime policy, not task
 environment. Imported child suites keep their own `workspace`, including
 `workspace.repos[]`, templates, hooks, and isolation. Existing local workspace
 paths are machine-local bindings supplied through CLI flags or
 `config.local.yaml`. If the parent should own workspace context, import raw cases
-with `type: tests` or shorthand paths instead of importing an eval suite.
+with `imports.tests` or shorthand paths instead of importing an eval suite.
 
-`type: tests` imports only raw test entries. It intentionally drops shared
+`imports.tests` imports only raw test entries. It intentionally drops shared
 suite context such as workspace, shared input, and shared assertions. Use this
 mode only when the imported file is a case corpus or when dropping suite context
 is the desired behavior.
 
 Suite files still support raw-case shorthand imports. A string-valued `tests`
-field or a string entry inside the `tests` list is equivalent to an include
-entry with `type: tests`:
+field or a string entry inside the `tests` list is equivalent to an
+`imports.tests` entry:
 
 ```yaml
 tests: ./cases.yaml
@@ -247,16 +252,13 @@ tests: ./cases.yaml
 ```yaml
 tests:
   - ./cases/*.cases.yaml
-  - include: ./suites/*.eval.yaml
-    type: suite
 ```
 
 The shorthand is only for raw case files or directories. Importing another eval
-suite must use object form with `include:` and `type: suite`, so authors cannot
-accidentally drop or preserve suite context without saying which behavior they
-want.
+suite must use `imports.suites`, so authors cannot accidentally drop or
+preserve suite context without saying which behavior they want.
 
-Do not use `import:` or `kind:` for `tests:` include entries.
+Do not use `import:` or `kind:` for import entries.
 
 Parent suite-level task fields should not silently override imported suite task
 fields. Explicit override syntax can be considered later if a concrete use case