diff --git a/.aipm/generated-root.json b/.aipm/generated-root.json new file mode 100644 index 0000000..66ac555 --- /dev/null +++ b/.aipm/generated-root.json @@ -0,0 +1,17 @@ +{ + "version": 1, + "paths": [ + ".kiro/agents/experimenter.json", + ".kiro/agents/test-subject.json", + "GEMINI.md", + "POWER.md", + "agents/experimenter.md", + "agents/test-subject.md", + "commands/evaluate.toml", + "gemini-extension.json", + "hooks/hooks.json", + "mcp.json", + "skills/evaluate-skill/SKILL.md", + "steering/evaluation-workflow.md" + ] +} diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 83b3c34..922eafe 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -4,7 +4,6 @@ "name": "AI Plugin Marketplace Template" }, "metadata": { - "version": "0.0.1", "description": "Universal AI Plugin Marketplace — author once, distribute to all platforms" }, "plugins": [ @@ -12,7 +11,12 @@ "name": "skill-evaluator", "source": "./plugins/skill-evaluator", "description": "Evaluate AI skills across model tiers with blind testing and refinement recommendations", - "tags": ["evaluation", "testing", "skills", "model-tiers"] + "tags": [ + "evaluation", + "testing", + "skills", + "model-tiers" + ] } ] } diff --git a/.cursor-plugin/marketplace.json b/.cursor-plugin/marketplace.json index 83b3c34..922eafe 100644 --- a/.cursor-plugin/marketplace.json +++ b/.cursor-plugin/marketplace.json @@ -4,7 +4,6 @@ "name": "AI Plugin Marketplace Template" }, "metadata": { - "version": "0.0.1", "description": "Universal AI Plugin Marketplace — author once, distribute to all platforms" }, "plugins": [ @@ -12,7 +11,12 @@ "name": "skill-evaluator", "source": "./plugins/skill-evaluator", "description": "Evaluate AI skills across model tiers with blind testing and refinement recommendations", - "tags": ["evaluation", "testing", "skills", "model-tiers"] + "tags": [ + "evaluation", + "testing", + "skills", + "model-tiers" + ] } ] } diff --git a/.kiro/agents/experimenter.json b/.kiro/agents/experimenter.json new file mode 100644 index 0000000..798ddfc --- /dev/null +++ b/.kiro/agents/experimenter.json @@ -0,0 +1,21 @@ +{ + "name": "experimenter", + "description": "Orchestrates blind skill evaluation across model tiers", + "prompt": "# Experimenter Agent\n\nYou are the experimenter in a blind skill evaluation. Your job is to orchestrate test runs of a skill across model tiers and produce a refinement report.\n\n## Principles\n\n- **Blind testing**: Never reveal expected outcomes to test subjects\n- **Structured protocol**: Define pass/fail criteria BEFORE running tests\n- **Systematic comparison**: Evaluate each tier independently before comparing across tiers\n- **Actionable output**: Every identified failure must include a specific recommendation\n\n## Workflow\n\n1. Receive the skill content and test cases from the evaluate-skill skill\n2. For each model tier (opus, sonnet, haiku):\n a. For each test case, spawn a test-subject agent at the appropriate tier\n b. Provide only the skill content and the input — never the expected outcome\n c. Collect and store the output\n3. Compare outputs against expected outcomes\n4. Generate a structured refinement report\n\n## Report Format\n\n```\n# Skill Evaluation Report\n\n## Summary\n- Skill: [name]\n- Clarity Floor: [lowest passing tier]\n- Overall Pass Rate: [X/Y]\n\n## Per-Tier Results\n### Opus\n| Test Case | Pass/Fail | Notes |\n|-----------|-----------|-------|\n| ... | ... | ... |\n\n### Sonnet\n...\n\n### Haiku\n...\n\n## Failure Analysis\n### [Test Case N at Tier X]\n- **Symptom**: [what went wrong]\n- **Root Cause**: [why the lower-tier agent failed]\n- **Recommendation**: [specific improvement to the skill]\n\n## Recommendations\n1. [Ordered list of improvements, highest impact first]\n```", + "mcpServers": {}, + "tools": [ + "delegate", + "read", + "write", + "glob", + "grep", + "shell" + ], + "toolAliases": {}, + "allowedTools": [], + "resources": [], + "hooks": {}, + "toolsSettings": {}, + "includeMcpJson": true, + "model": null +} diff --git a/.kiro/agents/test-subject.json b/.kiro/agents/test-subject.json new file mode 100644 index 0000000..1c1e065 --- /dev/null +++ b/.kiro/agents/test-subject.json @@ -0,0 +1,20 @@ +{ + "name": "test-subject", + "description": "Blind agent that executes a skill and produces output without knowledge of expected outcomes", + "prompt": "# Test Subject Agent\n\n\n\nYou are a test subject in a blind skill evaluation. You will receive a skill and an input. Execute the skill to the best of your ability and produce your output.\n\n## Rules\n\n- You do NOT know what the expected outcome is — just do your best\n- Follow the skill instructions exactly as written\n- If the skill instructions are ambiguous, make your best interpretation and note the ambiguity\n- Do not ask clarifying questions — work with what you have\n- Produce your output in a clear, structured format\n\n## Process\n\n1. Read the skill content provided to you\n2. Read the input provided to you\n3. Execute the skill's instructions against the input\n4. Produce your output\n\n## Output Format\n\nProduce your result in this format:\n\n```\n## Output\n\n[Your skill execution output here]\n\n## Notes\n\n- [Any ambiguities encountered]\n- [Assumptions made]\n- [Difficulties faced]\n```", + "mcpServers": {}, + "tools": [ + "read", + "write", + "shell", + "glob", + "grep" + ], + "toolAliases": {}, + "allowedTools": [], + "resources": [], + "hooks": {}, + "toolsSettings": {}, + "includeMcpJson": true, + "model": null +} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 70e49a4..0ec557e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,17 +9,19 @@ cloning, run: pnpm install ``` -This requires `@ai-plugin-marketplace/cli` and `@ai-plugin-marketplace/core` to -be **published to npm**. A fresh `pnpm install` regenerates `pnpm-lock.yaml` -against the published versions. +`@ai-plugin-marketplace/cli` and `@ai-plugin-marketplace/core` are published to +npm, so a fresh `pnpm install` resolves them and writes `pnpm-lock.yaml` against +the published versions. ## Workflow 1. Add or edit a plugin under `plugins//`. Use `aipm scaffold ` to create a new one or `aipm add-target ` to expand an existing plugin's support envelope. -2. Run `aipm build` to regenerate toolkit-owned artifacts (hook JSON and the - `dist/` bundles). Commit both your authored sources and the regenerated +2. Run `aipm build` to regenerate toolkit-owned artifacts: hook JSON, the + `dist/` bundles, the marketplace registries (generated from + `aipm.workspace.ts`), and — for a single-plugin marketplace — the repo-root + Gemini/Kiro artifacts. Commit both your authored sources and the regenerated output — CI enforces that the tree is clean after a build (freshness). 3. Run `aipm validate` and resolve any hard findings. diff --git a/GEMINI.md b/GEMINI.md new file mode 100644 index 0000000..71f535d --- /dev/null +++ b/GEMINI.md @@ -0,0 +1,26 @@ +# Skill Evaluator + +Evaluate AI skills across model tiers with blind testing and refinement recommendations. + +## Overview + +This extension tests AI skills across model tiers (opus → sonnet → haiku) using blind sub-agent testing. It identifies where skill clarity degrades and provides actionable refinement recommendations. + +## Usage + +Use the `/evaluate` command to start an evaluation: + +``` +/evaluate +``` + +- `skill-path`: Path to the SKILL.md file to evaluate +- `test-cases-path`: Path to a JSON file with test cases (array of `{ input, expectedOutcome }`) + +## How It Works + +1. The experimenter agent reads the skill and test cases +2. Separate blind test-subject agents are spawned per test case at the highest tier (opus) +3. The process repeats at lower tiers (sonnet, then haiku) +4. Results are compared against expected outcomes per tier +5. A refinement report identifies where clarity breaks down and recommends improvements diff --git a/POWER.md b/POWER.md new file mode 100644 index 0000000..fc9f2ae --- /dev/null +++ b/POWER.md @@ -0,0 +1,28 @@ +--- +schemaVersion: 0.1.0 +name: skill-evaluator +description: Evaluate AI skills across model tiers with blind testing and refinement recommendations +version: 0.0.1 +--- + +# Skill Evaluator + +This power evaluates AI skills across model tiers (opus → sonnet → haiku) using blind sub-agent testing. + +## Capabilities + +- **Blind evaluation**: Test subjects execute skills without knowledge of expected outcomes +- **Multi-tier testing**: Run skills at progressively lower model tiers to find clarity breakpoints +- **Refinement recommendations**: Generate specific suggestions to improve skill clarity + +## Workflow + +1. Provide a skill path and test cases (input/expected-outcome pairs) +2. The experimenter agent orchestrates blind test runs across model tiers +3. Results are compared against expected outcomes +4. A refinement report is generated with specific recommendations + +## Related Files + +- `steering/evaluation-workflow.md` — Evaluation workflow steering +- `mcp.json` — MCP server configuration (empty — this power is skill/agent-based) diff --git a/README.md b/README.md index a2f2af9..966c041 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,8 @@ then: pnpm install ``` -> **Note:** `pnpm install` requires the `@ai-plugin-marketplace/*` packages to be -> published to npm. See [CONTRIBUTING.md](CONTRIBUTING.md). +The `@ai-plugin-marketplace/*` packages are published to npm, so a fresh +`pnpm install` resolves them directly. See [CONTRIBUTING.md](CONTRIBUTING.md). ## Authoring plugins @@ -43,11 +43,20 @@ repo-root marketplace registries. aipm build ``` -`aipm build` regenerates every toolkit-owned artifact — per-plugin hook JSON -(`hooks/claude.json`, `hooks/hooks.json`) and the standalone bundles under -`dist/gemini/` and `dist/kiro/`. Both authored sources and generated outputs are -committed, so plugins stay browsable on GitHub and pull-request diffs stay -honest. +`aipm build` regenerates every toolkit-owned artifact: + +- per-plugin hook JSON (`hooks/claude.json`, `hooks/hooks.json`); +- the standalone bundles under `dist/gemini/` and `dist/kiro/`; +- the marketplace registries (`.claude-plugin/marketplace.json`, + `.cursor-plugin/marketplace.json`) — generated from `aipm.workspace.ts` plus + each plugin's `aipm.config.ts` (see [Marketplace metadata](#marketplace-metadata-and-generated-registries)); +- for a single-plugin marketplace, the **repo-root Gemini extension and Kiro + power** (`gemini-extension.json`, `GEMINI.md`, `POWER.md`, `commands/`, + `skills/`, `steering/`, `.kiro/`, …) so the repo installs natively into hosts + that have no marketplace concept. + +Both authored sources and generated outputs are committed, so plugins stay +browsable on GitHub and pull-request diffs stay honest. ### Validate @@ -57,8 +66,8 @@ aipm validate Validation checks the support envelope, every target manifest's schema, cross-target name consistency, MCP server-key sync, marketplace registration, -and freshness (that the committed generated artifacts match what `aipm build` -would produce). +and freshness — that the committed generated artifacts (registries, bundles, +and the repo-root Gemini/Kiro emission) match what `aipm build` would produce. `package.json` exposes these as `pnpm build`, `pnpm check`, and `pnpm scaffold` if you prefer the npm-script entry points. @@ -81,24 +90,76 @@ plugin that carries files for a target outside its envelope. To expand a plugin's envelope, run `aipm add-target ` to scaffold the skeleton files for a new target, then fill in the manifest fields. +## Marketplace metadata and generated registries + +Repo-level marketplace metadata lives in `aipm.workspace.ts` at the repo root: + +```ts +import { defineWorkspace } from '@ai-plugin-marketplace/core'; + +export default defineWorkspace({ + marketplace: { + name: 'ai-plugin-marketplace', + owner: { name: 'AI Plugin Marketplace Template' }, + description: 'Universal AI Plugin Marketplace — author once, distribute to all platforms', + }, +}); +``` + +The **presence of this file opts the repo into generated registries.** Instead +of hand-maintaining `.claude-plugin/marketplace.json` and +`.cursor-plugin/marketplace.json`, `aipm build` generates them from the +workspace metadata plus each plugin's `aipm.config.ts` (its `description` and +`keywords` become the registry entry's `description` and `tags`). The generated +registries are committed and freshness-checked like every other artifact. + +### Gemini and Kiro: the single-plugin marketplace + +Gemini CLI and Kiro have **no marketplace concept** — a repo is installed as one +extension (Gemini) or one power (Kiro) from its root. So when a marketplace +exposes **exactly one plugin**, `aipm build` additionally emits that plugin's +Gemini/Kiro artifacts at the **repo root** (`gemini-extension.json`, `GEMINI.md`, +`POWER.md`, `commands/`, `skills/`, `steering/`, `.kiro/`), letting the repo +install natively into those hosts. A sidecar at `.aipm/generated-root.json` +records exactly which root paths the toolkit owns. + +If you add a **second plugin**, the repo can no longer be a single Gemini/Kiro +artifact: `aipm validate` reports a `single-artifact-host` finding, and you keep +full Claude/Cursor marketplace support (which can host many plugins) while +choosing one plugin to expose to Gemini/Kiro. Because the repo root is the +single Gemini/Kiro artifact, the plugin does **not** carry its own `LICENSE` or +`README.md` — the repo-root `LICENSE`/`README.md` are canonical and are never +overwritten by generation. + ## Repository structure ``` . +├── aipm.workspace.ts # Marketplace metadata (opts into generated registries) ├── .claude-plugin/ -│ └── marketplace.json # Claude Code marketplace registry +│ └── marketplace.json # Claude Code registry (GENERATED) ├── .cursor-plugin/ -│ └── marketplace.json # Cursor marketplace registry +│ └── marketplace.json # Cursor registry (GENERATED) ├── plugins/ │ └── / # One directory per plugin (authored sources) -│ ├── aipm.config.ts # Support envelope + plugin version +│ ├── aipm.config.ts # Support envelope + version + description/keywords │ └── ... # Manifests, agents, skills, commands, hooks, rules ├── dist/ # Generated standalone bundles (committed) │ ├── gemini/ │ └── kiro/ +│ +│ # Repo-root Gemini/Kiro artifacts — emitted for a single-plugin marketplace +│ # (GENERATED; tracked in .aipm/generated-root.json): +├── gemini-extension.json +├── GEMINI.md +├── POWER.md +├── commands/ skills/ steering/ agents/ hooks/ mcp.json .kiro/ +├── .aipm/ +│ └── generated-root.json # Records which repo-root paths the toolkit owns +│ ├── package.json # Depends on @ai-plugin-marketplace/cli + core -├── LICENSE -└── README.md +├── LICENSE # Canonical — also serves the Gemini/Kiro artifact +└── README.md # Canonical — also serves the Gemini/Kiro artifact ``` ## Upgrading the toolkit @@ -119,9 +180,18 @@ when. ## Example plugin: skill-evaluator The included `skill-evaluator` plugin demonstrates the full multi-platform -pattern end-to-end — it evaluates AI skills across model tiers (opus → sonnet → -haiku) using blind sub-agent testing. See -[plugins/skill-evaluator/README.md](plugins/skill-evaluator/README.md). +pattern end-to-end. Given a skill (`SKILL.md`) and a set of test cases, it: + +1. runs the skill with blind test-subject agents at different model tiers + (opus → sonnet → haiku); +2. compares outputs against expected outcomes; +3. identifies where skill clarity degrades at lower tiers; +4. generates actionable refinement recommendations. + +Invoke it in any host with `/evaluate path/to/SKILL.md path/to/test-cases.json`, +where each test case is `{ "input": "...", "expectedOutcome": "..." }`. The skill +definition lives at +[plugins/skill-evaluator/skills/evaluate-skill/SKILL.md](plugins/skill-evaluator/skills/evaluate-skill/SKILL.md). ## License diff --git a/agents/experimenter.md b/agents/experimenter.md new file mode 100644 index 0000000..f687abe --- /dev/null +++ b/agents/experimenter.md @@ -0,0 +1,65 @@ +--- +name: experimenter +description: Orchestrates blind skill evaluation across model tiers +tools: + - activate_skill + - read_file + - write_file + - glob + - search_file_content + - run_shell_command +model: opus +--- + +# Experimenter Agent + +You are the experimenter in a blind skill evaluation. Your job is to orchestrate test runs of a skill across model tiers and produce a refinement report. + +## Principles + +- **Blind testing**: Never reveal expected outcomes to test subjects +- **Structured protocol**: Define pass/fail criteria BEFORE running tests +- **Systematic comparison**: Evaluate each tier independently before comparing across tiers +- **Actionable output**: Every identified failure must include a specific recommendation + +## Workflow + +1. Receive the skill content and test cases from the evaluate-skill skill +2. For each model tier (opus, sonnet, haiku): + a. For each test case, spawn a test-subject agent at the appropriate tier + b. Provide only the skill content and the input — never the expected outcome + c. Collect and store the output +3. Compare outputs against expected outcomes +4. Generate a structured refinement report + +## Report Format + +``` +# Skill Evaluation Report + +## Summary +- Skill: [name] +- Clarity Floor: [lowest passing tier] +- Overall Pass Rate: [X/Y] + +## Per-Tier Results +### Opus +| Test Case | Pass/Fail | Notes | +|-----------|-----------|-------| +| ... | ... | ... | + +### Sonnet +... + +### Haiku +... + +## Failure Analysis +### [Test Case N at Tier X] +- **Symptom**: [what went wrong] +- **Root Cause**: [why the lower-tier agent failed] +- **Recommendation**: [specific improvement to the skill] + +## Recommendations +1. [Ordered list of improvements, highest impact first] +``` diff --git a/agents/test-subject.md b/agents/test-subject.md new file mode 100644 index 0000000..e0583d5 --- /dev/null +++ b/agents/test-subject.md @@ -0,0 +1,49 @@ +--- +name: test-subject +description: Blind agent that executes a skill and produces output without knowledge of expected outcomes +model: sonnet +tools: + - read_file + - write_file + - run_shell_command + - glob + - search_file_content + - replace +--- + +# Test Subject Agent + + + +You are a test subject in a blind skill evaluation. You will receive a skill and an input. Execute the skill to the best of your ability and produce your output. + +## Rules + +- You do NOT know what the expected outcome is — just do your best +- Follow the skill instructions exactly as written +- If the skill instructions are ambiguous, make your best interpretation and note the ambiguity +- Do not ask clarifying questions — work with what you have +- Produce your output in a clear, structured format + +## Process + +1. Read the skill content provided to you +2. Read the input provided to you +3. Execute the skill's instructions against the input +4. Produce your output + +## Output Format + +Produce your result in this format: + +``` +## Output + +[Your skill execution output here] + +## Notes + +- [Any ambiguities encountered] +- [Assumptions made] +- [Difficulties faced] +``` diff --git a/aipm.workspace.ts b/aipm.workspace.ts new file mode 100644 index 0000000..80d040e --- /dev/null +++ b/aipm.workspace.ts @@ -0,0 +1,15 @@ +import { defineWorkspace } from '@ai-plugin-marketplace/core'; + +/** + * Marketplace metadata for this repo. Its presence opts the repo into generated marketplace + * registries (`.claude-plugin/`, `.cursor-plugin/`, and — when Codex is targeted — + * `.agents/plugins/`) instead of hand-authored JSON. Because this marketplace exposes a single + * plugin, the toolkit also emits the repo-root Gemini extension and Kiro power for it. + */ +export default defineWorkspace({ + marketplace: { + name: 'ai-plugin-marketplace', + owner: { name: 'AI Plugin Marketplace Template' }, + description: 'Universal AI Plugin Marketplace — author once, distribute to all platforms', + }, +}); diff --git a/commands/evaluate.toml b/commands/evaluate.toml new file mode 100644 index 0000000..7aefd1d --- /dev/null +++ b/commands/evaluate.toml @@ -0,0 +1,10 @@ +description = "Evaluate a skill across model tiers using blind testing" +prompt = """ +Evaluate the skill at the path provided as the first argument, using test cases from the path provided as the second argument. + +Arguments: {{args}} + +Run the skill with blind test agents at opus, sonnet, and haiku tiers. +Compare outputs against expected outcomes. +Generate a refinement report with specific recommendations. +""" diff --git a/dist/gemini/skill-evaluator/LICENSE b/dist/gemini/skill-evaluator/LICENSE deleted file mode 100644 index cf29a8e..0000000 --- a/dist/gemini/skill-evaluator/LICENSE +++ /dev/null @@ -1,15 +0,0 @@ -ISC License - -Copyright (c) 2025 - -Permission to use, copy, modify, and/or distribute this software for any -purpose with or without fee is hereby granted, provided that the above -copyright notice and this permission notice appear in all copies. - -THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH -REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY -AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, -INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM -LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR -OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR -PERFORMANCE OF THIS SOFTWARE. diff --git a/dist/gemini/skill-evaluator/README.md b/dist/gemini/skill-evaluator/README.md deleted file mode 100644 index 2a2b201..0000000 --- a/dist/gemini/skill-evaluator/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# Skill Evaluator - -Evaluate AI skills across model tiers with blind testing and refinement recommendations. - -## What It Does - -Given a skill (SKILL.md) and a set of test cases (input/expected-outcome pairs), this plugin: - -1. Runs the skill with blind test-subject agents at different model tiers (opus → sonnet → haiku) -2. Compares outputs against expected outcomes -3. Identifies where skill clarity degrades at lower tiers -4. Generates actionable refinement recommendations - -## Usage - -### Claude Code / Cursor - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -### Gemini CLI - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -## Test Cases Format - -Create a JSON file with an array of test cases: - -```json -[ - { - "input": "Description of the input scenario", - "expectedOutcome": "Description of what the skill should produce" - }, - { - "input": "Another scenario", - "expectedOutcome": "Expected result for this scenario" - } -] -``` - -## Platform Support - -| Platform | Install Method | -|----------|---------------| -| Claude Code | `claude plugin add --path plugins/skill-evaluator` | -| Cursor | `cursor plugin add --path plugins/skill-evaluator` | -| Gemini CLI | `gemini extensions install` (use standalone build in `dist/gemini/skill-evaluator`) | -| Kiro | Install as power (use standalone build in `dist/kiro/skill-evaluator`) | -| Skills CLI | `npx skills add ` (auto-discovers SKILL.md files) | - -## Architecture - -- **evaluate-skill** (skill): Main entry point — orchestrates the evaluation -- **experimenter** (agent): Sets up blind tests, compares outcomes, generates report -- **test-subject** (agent): Executes skills blindly at various model tiers -- **evaluation-protocol** (rule): Ensures structured protocol adherence - -## License - -ISC diff --git a/dist/gemini/skill-evaluator/hooks/hooks.json b/dist/gemini/skill-evaluator/hooks/hooks.json index 948272b..9cdff81 100644 --- a/dist/gemini/skill-evaluator/hooks/hooks.json +++ b/dist/gemini/skill-evaluator/hooks/hooks.json @@ -3,11 +3,11 @@ "PostToolUse": [ { "matcher": "write_file", - "description": "Log evaluation report writes to a structured log file", + "description": "Append one JSON object per evaluation-report write to a JSON Lines log", "hooks": [ { "type": "command", - "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Evaluation report written\" >> .evaluation-log.jsonl; fi" + "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"{\\\"timestamp\\\":\\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\",\\\"event\\\":\\\"evaluation-report-written\\\"}\" >> .evaluation-log.jsonl; fi" } ] } diff --git a/dist/kiro/skill-evaluator/LICENSE b/dist/kiro/skill-evaluator/LICENSE deleted file mode 100644 index cf29a8e..0000000 --- a/dist/kiro/skill-evaluator/LICENSE +++ /dev/null @@ -1,15 +0,0 @@ -ISC License - -Copyright (c) 2025 - -Permission to use, copy, modify, and/or distribute this software for any -purpose with or without fee is hereby granted, provided that the above -copyright notice and this permission notice appear in all copies. - -THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH -REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY -AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, -INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM -LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR -OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR -PERFORMANCE OF THIS SOFTWARE. diff --git a/dist/kiro/skill-evaluator/README.md b/dist/kiro/skill-evaluator/README.md deleted file mode 100644 index 2a2b201..0000000 --- a/dist/kiro/skill-evaluator/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# Skill Evaluator - -Evaluate AI skills across model tiers with blind testing and refinement recommendations. - -## What It Does - -Given a skill (SKILL.md) and a set of test cases (input/expected-outcome pairs), this plugin: - -1. Runs the skill with blind test-subject agents at different model tiers (opus → sonnet → haiku) -2. Compares outputs against expected outcomes -3. Identifies where skill clarity degrades at lower tiers -4. Generates actionable refinement recommendations - -## Usage - -### Claude Code / Cursor - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -### Gemini CLI - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -## Test Cases Format - -Create a JSON file with an array of test cases: - -```json -[ - { - "input": "Description of the input scenario", - "expectedOutcome": "Description of what the skill should produce" - }, - { - "input": "Another scenario", - "expectedOutcome": "Expected result for this scenario" - } -] -``` - -## Platform Support - -| Platform | Install Method | -|----------|---------------| -| Claude Code | `claude plugin add --path plugins/skill-evaluator` | -| Cursor | `cursor plugin add --path plugins/skill-evaluator` | -| Gemini CLI | `gemini extensions install` (use standalone build in `dist/gemini/skill-evaluator`) | -| Kiro | Install as power (use standalone build in `dist/kiro/skill-evaluator`) | -| Skills CLI | `npx skills add ` (auto-discovers SKILL.md files) | - -## Architecture - -- **evaluate-skill** (skill): Main entry point — orchestrates the evaluation -- **experimenter** (agent): Sets up blind tests, compares outcomes, generates report -- **test-subject** (agent): Executes skills blindly at various model tiers -- **evaluation-protocol** (rule): Ensures structured protocol adherence - -## License - -ISC diff --git a/gemini-extension.json b/gemini-extension.json new file mode 100644 index 0000000..72a094b --- /dev/null +++ b/gemini-extension.json @@ -0,0 +1,8 @@ +{ + "schemaVersion": "0.1.0", + "name": "skill-evaluator", + "version": "0.0.1", + "description": "Evaluate AI skills across model tiers with blind testing and refinement recommendations", + "contextFileName": "GEMINI.md", + "mcpServers": {} +} diff --git a/hooks/hooks.json b/hooks/hooks.json new file mode 100644 index 0000000..9cdff81 --- /dev/null +++ b/hooks/hooks.json @@ -0,0 +1,16 @@ +{ + "hooks": { + "PostToolUse": [ + { + "matcher": "write_file", + "description": "Append one JSON object per evaluation-report write to a JSON Lines log", + "hooks": [ + { + "type": "command", + "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"{\\\"timestamp\\\":\\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\",\\\"event\\\":\\\"evaluation-report-written\\\"}\" >> .evaluation-log.jsonl; fi" + } + ] + } + ] + } +} diff --git a/mcp.json b/mcp.json new file mode 100644 index 0000000..da39e4f --- /dev/null +++ b/mcp.json @@ -0,0 +1,3 @@ +{ + "mcpServers": {} +} diff --git a/package.json b/package.json index f1780df..b65f82f 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "scaffold": "aipm scaffold" }, "devDependencies": { - "@ai-plugin-marketplace/cli": "^0.1.0", - "@ai-plugin-marketplace/core": "^0.1.0" + "@ai-plugin-marketplace/cli": "^0.1.1", + "@ai-plugin-marketplace/core": "^0.2.0" } } diff --git a/plugins/skill-evaluator/LICENSE b/plugins/skill-evaluator/LICENSE deleted file mode 100644 index cf29a8e..0000000 --- a/plugins/skill-evaluator/LICENSE +++ /dev/null @@ -1,15 +0,0 @@ -ISC License - -Copyright (c) 2025 - -Permission to use, copy, modify, and/or distribute this software for any -purpose with or without fee is hereby granted, provided that the above -copyright notice and this permission notice appear in all copies. - -THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH -REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY -AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, -INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM -LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR -OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR -PERFORMANCE OF THIS SOFTWARE. diff --git a/plugins/skill-evaluator/README.md b/plugins/skill-evaluator/README.md deleted file mode 100644 index 2a2b201..0000000 --- a/plugins/skill-evaluator/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# Skill Evaluator - -Evaluate AI skills across model tiers with blind testing and refinement recommendations. - -## What It Does - -Given a skill (SKILL.md) and a set of test cases (input/expected-outcome pairs), this plugin: - -1. Runs the skill with blind test-subject agents at different model tiers (opus → sonnet → haiku) -2. Compares outputs against expected outcomes -3. Identifies where skill clarity degrades at lower tiers -4. Generates actionable refinement recommendations - -## Usage - -### Claude Code / Cursor - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -### Gemini CLI - -``` -/evaluate path/to/SKILL.md path/to/test-cases.json -``` - -## Test Cases Format - -Create a JSON file with an array of test cases: - -```json -[ - { - "input": "Description of the input scenario", - "expectedOutcome": "Description of what the skill should produce" - }, - { - "input": "Another scenario", - "expectedOutcome": "Expected result for this scenario" - } -] -``` - -## Platform Support - -| Platform | Install Method | -|----------|---------------| -| Claude Code | `claude plugin add --path plugins/skill-evaluator` | -| Cursor | `cursor plugin add --path plugins/skill-evaluator` | -| Gemini CLI | `gemini extensions install` (use standalone build in `dist/gemini/skill-evaluator`) | -| Kiro | Install as power (use standalone build in `dist/kiro/skill-evaluator`) | -| Skills CLI | `npx skills add ` (auto-discovers SKILL.md files) | - -## Architecture - -- **evaluate-skill** (skill): Main entry point — orchestrates the evaluation -- **experimenter** (agent): Sets up blind tests, compares outcomes, generates report -- **test-subject** (agent): Executes skills blindly at various model tiers -- **evaluation-protocol** (rule): Ensures structured protocol adherence - -## License - -ISC diff --git a/plugins/skill-evaluator/aipm.config.ts b/plugins/skill-evaluator/aipm.config.ts index 30f5608..a5dd10f 100644 --- a/plugins/skill-evaluator/aipm.config.ts +++ b/plugins/skill-evaluator/aipm.config.ts @@ -3,4 +3,7 @@ import { defineConfig } from '@ai-plugin-marketplace/core'; export default defineConfig({ version: '0.1.0', targets: ['claude', 'cursor', 'gemini', 'kiro', 'vercel'], + description: + 'Evaluate AI skills across model tiers with blind testing and refinement recommendations', + keywords: ['evaluation', 'testing', 'skills', 'model-tiers'], }); diff --git a/plugins/skill-evaluator/hooks/claude.json b/plugins/skill-evaluator/hooks/claude.json index 1716693..83570ac 100644 --- a/plugins/skill-evaluator/hooks/claude.json +++ b/plugins/skill-evaluator/hooks/claude.json @@ -7,11 +7,11 @@ "PostToolUse": [ { "matcher": "Write", - "description": "Log evaluation report writes to a structured log file", + "description": "Append one JSON object per evaluation-report write to a JSON Lines log", "hooks": [ { "type": "command", - "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Evaluation report written\" >> .evaluation-log.jsonl; fi" + "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"{\\\"timestamp\\\":\\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\",\\\"event\\\":\\\"evaluation-report-written\\\"}\" >> .evaluation-log.jsonl; fi" } ] } diff --git a/plugins/skill-evaluator/hooks/claude.yaml b/plugins/skill-evaluator/hooks/claude.yaml index 60cef66..97ed3e0 100644 --- a/plugins/skill-evaluator/hooks/claude.yaml +++ b/plugins/skill-evaluator/hooks/claude.yaml @@ -1,10 +1,10 @@ hooks: PostToolUse: - matcher: Write - description: Log evaluation report writes to a structured log file + description: Append one JSON object per evaluation-report write to a JSON Lines log hooks: - type: command command: >- if echo "$TOOL_INPUT" | grep -q 'evaluation-report'; - then echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Evaluation report written" + then echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"event\":\"evaluation-report-written\"}" >> .evaluation-log.jsonl; fi diff --git a/plugins/skill-evaluator/hooks/hooks.json b/plugins/skill-evaluator/hooks/hooks.json index b011acb..af841fc 100644 --- a/plugins/skill-evaluator/hooks/hooks.json +++ b/plugins/skill-evaluator/hooks/hooks.json @@ -7,11 +7,11 @@ "PostToolUse": [ { "matcher": "write_file", - "description": "Log evaluation report writes to a structured log file", + "description": "Append one JSON object per evaluation-report write to a JSON Lines log", "hooks": [ { "type": "command", - "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Evaluation report written\" >> .evaluation-log.jsonl; fi" + "command": "if echo \"$TOOL_INPUT\" | grep -q 'evaluation-report'; then echo \"{\\\"timestamp\\\":\\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\",\\\"event\\\":\\\"evaluation-report-written\\\"}\" >> .evaluation-log.jsonl; fi" } ] } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3933624..d34d118 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9,21 +9,21 @@ importers: .: devDependencies: '@ai-plugin-marketplace/cli': - specifier: ^0.1.0 - version: 0.1.0 + specifier: ^0.1.1 + version: 0.1.1 '@ai-plugin-marketplace/core': - specifier: ^0.1.0 - version: 0.1.0 + specifier: ^0.2.0 + version: 0.2.0 packages: - '@ai-plugin-marketplace/cli@0.1.0': - resolution: {integrity: sha512-DOoDFaDSZqsGTvxjiByIgQdTFbYBOeet+dee6FDavW8FtDBWKUs9xYc+FuGQPu1G9+ztT5uS9dIcQpdGZbEGaQ==} + '@ai-plugin-marketplace/cli@0.1.1': + resolution: {integrity: sha512-mEHq89uoplj4LT7JvxGAweR3qnb+6sHk97w+6l02avArJ4+cVUr2K+Fb2WOwVjGYOGO1DD3y8r+hTiJWg6JuJQ==} engines: {node: '>=20.0.0'} hasBin: true - '@ai-plugin-marketplace/core@0.1.0': - resolution: {integrity: sha512-nUlTv2M/PIq8HtyLgeX9K8boLZJslS/4XrwHyamrgxINAxJOXyljelzYUFP8KqLypZPxfdlsuLILG8wW9LSeiQ==} + '@ai-plugin-marketplace/core@0.2.0': + resolution: {integrity: sha512-nnD9BZHZnlcw1dm/BzwC98T2QdBxiaeHCIGGvpT5/9DL4BHvrtUaG0KtoPUCLOLtGKgpaZfoulMO9s/DfWlMSA==} engines: {node: '>=20.0.0'} jiti@2.7.0: @@ -40,11 +40,11 @@ packages: snapshots: - '@ai-plugin-marketplace/cli@0.1.0': + '@ai-plugin-marketplace/cli@0.1.1': dependencies: - '@ai-plugin-marketplace/core': 0.1.0 + '@ai-plugin-marketplace/core': 0.2.0 - '@ai-plugin-marketplace/core@0.1.0': + '@ai-plugin-marketplace/core@0.2.0': dependencies: jiti: 2.7.0 yaml: 2.9.0 diff --git a/skills/evaluate-skill/SKILL.md b/skills/evaluate-skill/SKILL.md new file mode 100644 index 0000000..89d32bc --- /dev/null +++ b/skills/evaluate-skill/SKILL.md @@ -0,0 +1,51 @@ +--- +name: evaluate-skill +description: Evaluate an AI skill across model tiers using blind sub-agent testing +arguments: + - name: skill-path + description: Path to the SKILL.md file to evaluate + required: true + - name: test-cases-path + description: Path to a JSON file containing test cases (array of { input, expectedOutcome }) + required: true +--- + +# Evaluate Skill + +Orchestrate a cross-tier evaluation of an AI skill to determine its clarity and robustness. + +## Procedure + +1. **Load inputs** + - Read the skill file at `{{ skill-path }}` + - Read the test cases file at `{{ test-cases-path }}` + - Validate that test cases is a JSON array of objects with `input` and `expectedOutcome` fields + +2. **Set up evaluation matrix** + - Model tiers to test: `opus`, `sonnet`, `haiku` + - For each tier, for each test case: plan one blind test run + +3. **Execute blind tests** (highest tier first) + - For each model tier (opus → sonnet → haiku): + - For each test case: + - Spawn a **test-subject** agent at the current tier + - Provide it ONLY the skill content and the test case `input` + - Do NOT provide the `expectedOutcome` to the test subject + - Collect the test subject's output + +4. **Evaluate results** + - For each test run, compare the test subject's output against the `expectedOutcome` + - Determine pass/fail using semantic similarity (the output need not be identical, but must achieve the same goal) + - Record: tier, test case index, pass/fail, output summary + +5. **Generate refinement report** + - Identify the lowest tier where all test cases pass ("clarity floor") + - For each failure, analyze WHY the lower-tier agent failed: + - Ambiguous instructions? + - Missing context or assumptions? + - Overly complex multi-step reasoning? + - Implicit knowledge requirements? + - Produce specific, actionable recommendations to improve the skill + - Format as a structured report with sections: Summary, Per-Tier Results, Failure Analysis, Recommendations + +6. **Output the report** to the user diff --git a/steering/evaluation-workflow.md b/steering/evaluation-workflow.md new file mode 100644 index 0000000..76ef06a --- /dev/null +++ b/steering/evaluation-workflow.md @@ -0,0 +1,34 @@ +--- +inclusion: manual +--- + +# Evaluation Workflow + +## Purpose + +Guide the evaluation of AI skills across model tiers to identify clarity breakpoints and generate refinement recommendations. + +## Steps + +1. **Load the skill and test cases** + - Read the target SKILL.md file + - Parse the test cases JSON (array of `{ input, expectedOutcome }`) + +2. **Execute blind tests per tier** + - Tiers: opus → sonnet → haiku + - For each tier and test case, run the skill blind (no expected outcome provided) + - Collect structured output from each run + +3. **Compare against expected outcomes** + - Evaluate semantic equivalence (not exact match) + - Record pass/fail per tier per test case + +4. **Analyze failures** + - Categorize: ambiguous instructions, missing context, complex reasoning, implicit knowledge + - Map each failure to a specific part of the skill + +5. **Generate refinement report** + - Summary with clarity floor (lowest tier where all tests pass) + - Per-tier result tables + - Failure analysis with root causes + - Ordered recommendations for skill improvement