refactor(evals): replace stale self skill fixtures

apps/web/src/content/docs/docs/getting-started/installation.mdx

@@ -17,10 +17,11 @@ Install the AgentV CLI:
 npm install -g agentv
 ```
 
-Then load the onboarding skill and follow the instructions:
+Then load a bundled skill and follow its instructions. For eval authoring,
+`agentv-eval-writer` is the best starting point:
 
 ```bash
-agentv skills get agentv-onboarding
+agentv skills get agentv-eval-writer
 ```
 
 Paste the output to your AI agent and ask it to set up AgentV in your repository.

evals/README.md

@@ -1,36 +1,50 @@
-# AgentV Self-Evaluation
+# AgentV Plugin-Aligned Self Evals
 
-This folder contains evaluations to ensure AgentV's `AGENTS.md` design principles don't regress.
+This directory holds the repo-owned self-evaluation suites, split to match the
+current plugin boundary:
+
+- `agentv-self` covers AgentV's own repo guidance and self-eval workspace
+  behavior.
+- `agentv-dev` covers the bundled `agentv skills` CLI surface and skill content
+  shipped with the developer plugin. It reads live repo files from
+  `plugins/agentv-dev/`, `skills-data/`, and current docs instead of
+  checked-in transcript fixtures.
 
 ## Structure
 
-```
+```text
 evals/
-├── targets.yaml              # Target definitions
-├── design-principles.yaml    # Tests
-└── design-principles-grader.md # LLM grader prompt
+├── agentv-self/
+│   ├── agentv-self.eval.yaml
+│   ├── azure-smoke.eval.yaml
+│   └── scripts/setup.mjs
+├── agentv-dev/
+│   └── skills/
+│       ├── *.eval.yaml
+│       └── README.md
+└── agentic-engineering/
 ```
 
-## Known Issue: pi-coding-agent on Windows
-
-The pi-coding-agent provider has issues passing multi-line prompts on Windows. When running pi directly from CLI, it works correctly, but through the provider the prompt content is not received by pi.
-
-**Workaround**: Use mock provider for testing eval structure, or run on macOS/Linux.
+## Running
 
-**To fix**: Investigate `packages/core/src/evaluation/providers/pi-coding-agent.ts` - specifically how `spawn()` handles arguments with newlines on Windows.
-
-## Running the Evals
+Use the local CLI source from the repo root:
 
 ```bash
-# Dry run to verify structure
-bun agentv eval evals/design-principles.yaml --targets evals/targets.yaml --dry-run
-
-# Full run (requires working pi-coding-agent or alternative target)
-bun agentv eval evals/design-principles.yaml --targets evals/targets.yaml
+# Validate the renamed suites
+bun apps/cli/src/cli.ts validate evals/agentv-self/agentv-self.eval.yaml
+bun apps/cli/src/cli.ts validate evals/agentv-self/azure-smoke.eval.yaml
+bun apps/cli/src/cli.ts validate evals/agentv-dev/skills/*.eval.yaml
+
+# Prepare one agentv-self case and inspect the materialized workspace
+bun apps/cli/src/cli.ts prepare \
+  evals/agentv-self/agentv-self.eval.yaml \
+  --test-id guidance-split-paths \
+  --target codex
+
+# Run the agentv-dev skills suite against a target
+bun apps/cli/src/cli.ts eval run evals/agentv-dev/skills/*.eval.yaml --target <target>
 ```
 
-## Test Cases
-
-1. **violates-lightweight-core** - Proposes adding complex built-in (should fail)
-2. **violates-ai-first-design** - Proposes rigid CLI instructions (should fail)
-3. **follows-principles** - Proposes plugin/example approach (should pass)
+`agentv-self/agentv-self.eval.yaml` uses a `before_all` hook to copy the current repo
+checkout into the eval workspace. That keeps `/AGENTS.md` and `/.agents/`
+current without declaring extra repos in workspace config.

evals/agentv-dev/skills/README.md

@@ -0,0 +1,90 @@
+# `agentv-dev` skill evals
+
+Three-category eval suite covering the bundled `agentv skills` CLI surface
+owned by the `agentv-dev` plugin (`list`, `get`, `path`). Each category
+isolates one aspect of the skill UX so a regression in any single
+dimension is easy to read off the run report.
+
+## Coverage Model
+
+These evals intentionally avoid checked-in transcript fixtures. They read the
+live repo content directly:
+
+- `plugins/agentv-dev/skills/agentv-dev/SKILL.md` for the plugin-aligned skill
+  catalog and command surface
+- `skills-data/*/SKILL.md` for the actual bundled skill bodies shipped by the
+  CLI
+- `apps/web/src/content/docs/docs/getting-started/installation.mdx` for the
+  live CLI usage examples
+
+That keeps the suite aligned with the current repo instead of stale snapshots.
+
+## Categories
+
+| File | What it tests | Tests |
+|------|---------------|-------|
+| `skill-invocation.eval.yaml` | Does the agent invoke the right CLI command + flag for a given task? | 8 |
+| `skill-selection.eval.yaml` | Does the agent pick the right skill for a natural-language task? | 8 |
+| `output-correctness.eval.yaml` | Does the agent produce structurally and factually correct output from live skill/docs content? | 7 |
+
+## Running
+
+From the worktree root:
+
+```bash
+# Validate the suite
+bun apps/cli/src/cli.ts validate evals/agentv-dev/skills
+
+# All three categories against one target
+bun apps/cli/src/cli.ts eval run evals/agentv-dev/skills/*.eval.yaml --target <target>
+
+# A single category
+bun apps/cli/src/cli.ts eval run \
+  evals/agentv-dev/skills/skill-selection.eval.yaml --target azure
+
+# A single test
+bun apps/cli/src/cli.ts eval run \
+  evals/agentv-dev/skills/skill-invocation.eval.yaml \
+  --test-id invoke-get-full-flag --target azure
+```
+
+`<target>` is any name resolvable from `.agentv/targets.yaml` in the worktree
+(for example `azure`, `claude`, or `codex`).
+
+## Adding Test Cases
+
+Tests are plain entries under `tests:`. Each test must have:
+
+- `id` (kebab-case, unique within the file)
+- `criteria` — one-line human description
+- `input` — either a bare string or a `[{role, content: [...]}]` block
+- `assertions` — at least one entry; prefer deterministic types
+  (`contains`, `regex`, `icontains-any`) over `rubrics` so the eval
+  stays cheap and stable. Use `rubrics` only for genuinely qualitative
+  checks.
+
+Pattern for repo-driven file tests:
+
+```yaml
+- id: my-new-test
+  criteria: One-line description
+  input:
+    - role: user
+      content:
+        - type: file
+          value: /skills-data/agentv-eval-writer/SKILL.md
+        - type: text
+          value: |
+            <prompt referring to the live repo file>
+  assertions:
+    - type: contains
+      value: <expected substring>
+    - type: rubrics
+      criteria:
+        - <criterion 1>
+        - <criterion 2>
+```
+
+When multiple tests share the same live file, declare it once at the top level
+under `input:` (suite-level input, prepended to every test) instead of
+repeating it per-test — see `skill-selection.eval.yaml`.

evals/self/skills/output-correctness.eval.yaml → evals/agentv-dev/skills/output-correctness.eval.yaml

@@ -1,7 +1,9 @@
+name: agentv-dev-output-correctness
 description: >-
   Tests whether agent-produced output is structurally and factually correct
-  given a bundled-skill fixture as ground truth — covers eval YAML structure,
-  CLI command syntax, fixture-content accuracy, and grader configuration.
+  given live AgentV skill and docs content as ground truth — covers eval YAML
+  structure, CLI command syntax, skill-content accuracy, and grader
+  configuration.
 
 tags: [agent, skills]
 
@@ -12,10 +14,10 @@ tests:
       - role: user
         content:
           - type: file
-            value: fixtures/agentv-eval-writer.txt
+            value: /skills-data/agentv-eval-writer/SKILL.md
           - type: text
             value: |
-              Using the agentv-eval-writer skill above as your reference,
+              Using the live agentv-eval-writer skill above as your reference,
               produce a minimal but valid `.eval.yaml` snippet (one `description`
               and one test in `tests:`) for an eval that grades whether an LLM
               correctly summarises a paragraph.
@@ -41,11 +43,11 @@ tests:
       - role: user
         content:
           - type: file
-            value: fixtures/agentv-eval-writer.txt
+            value: /skills-data/agentv-eval-writer/SKILL.md
           - type: text
             value: |
-              Using the agentv-eval-writer skill as reference, write a single
-              test entry under `tests:` that uses **three** assertions:
+              Using the live agentv-eval-writer skill as reference, write a
+              single test entry under `tests:` that uses **three** assertions:
               one `contains`, one `regex`, and one `rubrics` (with at least
               two criteria). Output only the YAML for the test entry.
     assertions:
@@ -58,22 +60,23 @@ tests:
       - type: rubrics
         criteria:
           - "Each assertion has both a `type:` and the field that type expects (`value:` for contains/regex, `criteria:` list for rubrics)"
-          - Does not invent assertion types that are not in the agentv-eval-writer skill (e.g., `like`, `match`, `expect`)
-          - Uses kebab-case (e.g., `llm-grader`, `is-json`) not snake_case (e.g., `llm_grader`) for any multi-word grader types referenced
+          - Does not invent assertion types that are not in the agentv-eval-writer skill (for example `like`, `match`, or `expect`)
+          - Uses kebab-case (for example `llm-grader`, `is-json`) not snake_case (for example `llm_grader`) for any multi-word grader types referenced
 
   - id: cli-command-correct-syntax
     criteria: Agent produces correct `agentv skills` CLI commands with valid subcommands and flags.
     input:
       - role: user
         content:
           - type: file
-            value: fixtures/agentv-bench.txt
+            value: /apps/web/src/content/docs/docs/getting-started/installation.mdx
           - type: text
             value: |
-              Give me three shell commands, one per line, that exercise the
+              Using the live installation doc above as your reference, give me
+              three shell commands, one per line, that exercise the
               `agentv skills` subcommand:
               1. List all bundled skills as JSON.
-              2. Print the agentv-bench skill including its references and templates.
+              2. Print the agentv-bench skill including its bundled support files.
               3. Print the on-disk path of the agentv-bench skill.
     assertions:
       - type: regex
@@ -83,16 +86,16 @@ tests:
       - type: regex
         value: "agentv\\s+skills\\s+path\\s+agentv-bench"
 
-  - id: fixture-content-accurate
-    criteria: Agent's description of a skill matches the skill's actual SKILL.md content.
+  - id: skill-content-accurate
+    criteria: Agent's description of a skill matches the skill's live SKILL.md content.
     input:
       - role: user
         content:
           - type: file
-            value: fixtures/agentv-trace-analyst.txt
+            value: /skills-data/agentv-trace-analyst/SKILL.md
           - type: text
             value: |
-              Read the agentv-trace-analyst skill above. In one paragraph,
+              Read the live agentv-trace-analyst skill above. In one paragraph,
               describe what this skill does and when it should be invoked.
               Quote at most one short phrase from the skill text; do not
               invent capabilities the skill does not mention.
@@ -101,20 +104,21 @@ tests:
         criteria:
           - Description accurately reflects what agentv-trace-analyst does (analysing eval traces and failure modes)
           - Does not claim the skill writes or runs evals (those are agentv-eval-writer and agentv-bench respectively)
-          - Any quoted phrase appears verbatim in the fixture
-          - Does not hallucinate sub-features not present in the SKILL.md
+          - Any quoted phrase appears verbatim in the provided SKILL.md
+          - Does not hallucinate sub-features not present in the skill
 
-  - id: skill-list-complete
-    criteria: Agent lists all six bundled skills when given the JSON fixture.
+  - id: plugin-skill-list-complete
+    criteria: Agent lists all bundled skills from the live `agentv-dev` plugin stub.
     input:
       - role: user
         content:
           - type: file
-            value: fixtures/skills-list-all.txt
+            value: /plugins/agentv-dev/skills/agentv-dev/SKILL.md
           - type: text
             value: |
-              The JSON above is the output of `agentv skills list --json`.
-              Reproduce the list of skill names as a markdown bulleted list.
+              The file above is the live `agentv-dev` plugin entrypoint.
+              Reproduce the list of available bundled skill names as a markdown
+              bulleted list.
     assertions:
       - type: contains
         value: agentv-bench
@@ -124,53 +128,49 @@ tests:
         value: agentv-eval-writer
       - type: contains
         value: agentv-governance
-      - type: contains
-        value: agentv-onboarding
       - type: contains
         value: agentv-trace-analyst
       - type: rubrics
         criteria:
-          - Lists exactly the six skill names from the JSON, no more and no fewer
-          - Does not invent additional skill names not present in the fixture
+          - Lists exactly the five skill names from the live plugin file, no more and no fewer
+          - Does not invent additional skill names not present in the plugin file
 
-  - id: error-message-accurate
-    criteria: Agent correctly interprets the not-found error and proposes a recoverable next step.
+  - id: governance-ref-command-accurate
+    criteria: Agent gives the correct command for loading a specific governance reference file.
     input:
       - role: user
         content:
           - type: file
-            value: fixtures/skills-get-nonexistent.txt
+            value: /skills-data/agentv-governance/SKILL.md
           - type: text
             value: |
-              The output above came from running `agentv skills get does-not-exist`.
-              In two sentences, explain (a) what error the user hit and
-              (b) the single most useful command to run next.
+              Using the live agentv-governance skill above as reference, what
+              single command should I run to load the `lint-rules` reference
+              file without pulling in the whole skill?
+              Reply with the command on its own line.
     assertions:
-      - type: rubrics
-        criteria:
-          - Identifies that `does-not-exist` is not a recognised skill name
-          - "Proposes either `agentv skills list` or selecting one of the names from the error's \"Available skills:\" line"
-          - Does not blame missing installation or recommend re-running `npm install -g agentv`
+      - type: regex
+        value: "agentv\\s+skills\\s+get\\s+agentv-governance\\s+--ref\\s+lint-rules"
 
   - id: grader-config-valid
     criteria: Agent configures an llm-grader with the fields the AgentV registry expects.
     input:
       - role: user
         content:
           - type: file
-            value: fixtures/agentv-eval-writer.txt
+            value: /skills-data/agentv-eval-writer/SKILL.md
           - type: text
             value: |
-              Using the agentv-eval-writer skill as reference, produce a YAML
-              snippet for **one** assertion that uses `type: llm-grader` to
-              grade an answer's correctness. Include whatever fields the
-              skill says an `llm-grader` requires (e.g., target/model, rubric
-              criteria). Output only the YAML for the single assertion.
+              Using the live agentv-eval-writer skill as reference, produce a
+              YAML snippet for **one** assertion that uses `type: llm-grader`
+              to grade an answer's correctness. Include whatever fields the
+              skill says an `llm-grader` requires (for example target/model or
+              rubric criteria). Output only the YAML for the single assertion.
     assertions:
       - type: regex
         value: "type:\\s+llm-grader"
       - type: rubrics
         criteria:
           - "Uses `type: llm-grader` (kebab-case), not `type: llm_grader` or `type: llmGrader`"
-          - Includes the fields the skill documents for llm-grader (typically a rubric/criteria field; may also include a target/model)
-          - Does not invent fields not mentioned in the skill (e.g., a fabricated `model_provider` field)
+          - Includes the fields the skill documents for llm-grader (typically a rubric or criteria field; may also include a target or model)
+          - Does not invent fields not mentioned in the skill (for example a fabricated `model_provider` field)