docs(readme): clarify experiment layout and rubric schema

Author: christso

README.md

@@ -2,35 +2,9 @@
 
 **Evaluate AI targets against real repos from the terminal. No server. No signup.**
 
-```bash
-npm install -g agentv
-agentv init
-agentv eval evals/example.yaml
-```
-
-That's it. Results in seconds, not minutes.
-
-## What it does
-
-AgentV runs evaluation cases against configured targets and scores them with deterministic code graders + customizable LLM graders. Everything lives in Git — YAML eval files, markdown judge prompts, JSONL results.
-
-```yaml
-# evals/math.yaml
-description: Math problem solving
-tests:
-  - id: addition
-    input: What is 15 + 27?
-    expected_output: "42"
-    assertions:
-      - type: contains
-        value: "42"
-```
+AgentV is a repo-native eval runner for comparing AI targets on real workspace tasks with portable results for local development, CI, Dashboard, compare, and trend.
 
-```bash
-agentv eval evals/math.yaml
-```
-
-## Why AgentV?
+## Why?
 
 - **Local-first** — runs on your machine, no cloud accounts or API keys for eval infrastructure
 - **Repo-backed workspaces** — reuse real repos, setup scripts, and existing harnesses instead of rebuilding synthetic tasks
@@ -44,15 +18,15 @@ agentv eval evals/math.yaml
 
 - **Suite / imports / tests** are the task corpus: the prompts, cases, datasets, and imported benchmarks you want to evaluate.
 - **Workspace / fixtures / graders** are task-owned context: repos, setup scripts, files, fixtures, deterministic checks, and LLM grading prompts.
-- **Target** is the system under test: an agent, model/provider, gateway, replay target, CLI wrapper, transcript provider, or future app/service wrapper.
-- **Experiment** names comparison intent: target/model, variant, repeats, gates, timeout/runtime policy, and result grouping.
+- **Target** is the system under test: an agent, model/provider, gateway, replay target, CLI wrapper, transcript provider, or future app/service wrapper. Use target ids such as `copilot--claude-opus-4.8` to name the concrete provider/model/variant being compared.
+- **Experiment** names comparison intent: target id, repeats, gates, timeout/runtime policy, and result grouping.
 - **Run** is one concrete execution that writes portable artifacts for readers such as Dashboard, compare, and trend.
 
 ```mermaid
 flowchart LR
   corpus["Suite / imports / tests<br/>task corpus"]
   context["Workspace / fixtures / graders<br/>task-owned context"]
-  experiment["Experiment<br/>target + variant + runtime policy"]
+  experiment["Experiment<br/>target id + runtime policy"]
   run["Run<br/>concrete execution"]
   artifacts["Run artifacts<br/>summary.json + index.jsonl + sidecars"]
   readers["Dashboard / compare / trend<br/>derived readers"]
@@ -76,11 +50,20 @@ agentv init
 
 **3. Create an eval** in `evals/`:
 ```yaml
+name: backend-with-skills
 description: Code generation quality
 
 experiment:
-  target: copilot
+  name: backend-with-skills
+  target: copilot--claude-opus-4.8
+  repeat:
+    count: 3
+    strategy: pass_at_k
+  early_exit: true
+  timeout_seconds: 600
   threshold: 0.8
+  budget_usd: 5
+  sandbox: auto
 
 tests:
   - id: fizzbuzz
@@ -104,30 +87,30 @@ agentv eval evals/my-eval.yaml
 
 **5. Compare two runs** (pass two `index.jsonl` manifests — e.g. before and after a change):
 ```bash
-agentv compare .agentv/results/<experiment>/<before-timestamp>/default/index.jsonl .agentv/results/<experiment>/<after-timestamp>/default/index.jsonl
+agentv compare .agentv/results/backend-without-skills/<timestamp>/copilot--claude-opus-4.8/index.jsonl .agentv/results/backend-with-skills/<timestamp>/copilot--claude-opus-4.8/index.jsonl
 ```
 
 ## Results
 
-Each run writes a timestamped bundle under `.agentv/results/<experiment>/<timestamp>/<run-id>/`. The flat `index.jsonl` manifest is the portable surface used by scripts, CI, and `agentv compare`:
+Each run writes a timestamped invocation directory under `.agentv/results/<experiment>/<timestamp>/`. In this example, the experiment is `backend-with-skills` and the resolved target id is `copilot--claude-opus-4.8`; model or variant choices are encoded in that target id, not in an extra results folder. The flat `index.jsonl` manifest is the portable surface used by scripts, CI, and `agentv compare`:
 
 ```bash
-agentv eval evals/my-eval.yaml --output ./run   # writes ./run/default/index.jsonl
-cat ./run/default/index.jsonl                    # JSONL results for scripts/CI
+agentv eval evals/my-eval.yaml
+cat .agentv/results/backend-with-skills/<timestamp>/copilot--claude-opus-4.8/index.jsonl
 ```
 
 Run bundle layout:
 
 ```
 .agentv/results/
-└── my-eval/                          # <experiment> — comparison/run grouping
+└── backend-with-skills/              # <experiment> — comparison/run grouping
     └── 2026-06-30T08-30-00-000Z/     # <timestamp> — one run
-        └── default/                  # <run-id>
+        └── copilot--claude-opus-4.8/ # <target> — resolved system under test
             ├── index.jsonl           # flat per-test results (scripts/CI, `agentv compare`)
             ├── summary.json          # run rollup: pass rate, counts, cost
-            └── fizzbuzz--a1b2c3d4/   # <case-allocation>
+            └── fizzbuzz--a1b2c3d4/   # <result_dir> for one test case
                 ├── summary.json      # per-test rollup across runs
-                ├── task/             # frozen inputs, for reproducibility
+                ├── test/             # generated test bundle: frozen inputs for reproducibility
                 │   ├── EVAL.yaml     #   resolved eval spec
                 │   ├── targets.yaml  #   resolved target config
                 │   └── graders/      #   grader files used
@@ -149,7 +132,7 @@ Use `evaluate()` when your application owns the run:
 import { evaluate } from '@agentv/sdk';
 
 const { results, summary } = await evaluate({
-  target: { name: 'copilot', provider: 'copilot' },
+  target: { name: 'copilot--claude-opus-4.8', provider: 'copilot' },
   threshold: 0.8,
   tests: [
     {
@@ -179,8 +162,15 @@ import { defineEval } from '@agentv/sdk';
 export default defineEval({
   description: 'Code generation quality',
   experiment: {
-    target: 'copilot',
+    name: 'backend-with-skills',
+    target: 'copilot--claude-opus-4.8',
+    repeat: {
+      count: 3,
+      strategy: 'pass_at_k',
+    },
     threshold: 0.8,
+    budgetUsd: 5,
+    sandbox: 'auto',
   },
   tests: [
     {

packages/core/src/evaluation/validation/eval-file.schema.ts

@@ -87,6 +87,8 @@ const RubricItemSchema = z.object({
   score_ranges: z.array(ScoreRangeSchema).optional(),
 });
 
+const RubricCriterionSchema = z.union([z.string().min(1), RubricItemSchema]);
+
 // --- Type-specific evaluator schemas ---
 
 const CodeGraderSchema = EvaluatorCommonSchema.extend({
@@ -237,7 +239,7 @@ const EqualsSchema = EvaluatorCommonSchema.extend({
 
 const RubricsSchema = EvaluatorCommonSchema.extend({
   type: z.literal('rubrics'),
-  criteria: z.array(RubricItemSchema).min(1),
+  criteria: z.array(RubricCriterionSchema).min(1),
 });
 
 /** Union of all grader types */

packages/core/test/evaluation/validation/eval-file-schema.test.ts

@@ -116,6 +116,24 @@ describe('EvalFileSchema input shorthand', () => {
     expect(result.success).toBe(true);
   });
 
+  it('accepts explicit rubrics criteria string shorthand', () => {
+    const result = EvalFileSchema.safeParse({
+      tests: [
+        {
+          ...baseTest,
+          assertions: [
+            {
+              type: 'rubrics',
+              criteria: ['Must be polite', 'Must be accurate'],
+            },
+          ],
+        },
+      ],
+    });
+
+    expect(result.success).toBe(true);
+  });
+
   it('accepts flatter imports with optional inline tests', () => {
     const result = EvalFileSchema.safeParse({
       name: 'wrapper',