Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .aipm/generated-root.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"version": 1,
"paths": [
".kiro/agents/experimenter.json",
".kiro/agents/test-subject.json",
"GEMINI.md",
"POWER.md",
"agents/experimenter.md",
"agents/test-subject.md",
"commands/evaluate.toml",
"gemini-extension.json",
"hooks/hooks.json",
"mcp.json",
"skills/evaluate-skill/SKILL.md",
"steering/evaluation-workflow.md"
]
}
8 changes: 6 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@
"name": "AI Plugin Marketplace Template"
},
"metadata": {
"version": "0.0.1",
"description": "Universal AI Plugin Marketplace — author once, distribute to all platforms"
},
"plugins": [
{
"name": "skill-evaluator",
"source": "./plugins/skill-evaluator",
"description": "Evaluate AI skills across model tiers with blind testing and refinement recommendations",
"tags": ["evaluation", "testing", "skills", "model-tiers"]
"tags": [
"evaluation",
"testing",
"skills",
"model-tiers"
]
}
]
}
8 changes: 6 additions & 2 deletions .cursor-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@
"name": "AI Plugin Marketplace Template"
},
"metadata": {
"version": "0.0.1",
"description": "Universal AI Plugin Marketplace — author once, distribute to all platforms"
},
"plugins": [
{
"name": "skill-evaluator",
"source": "./plugins/skill-evaluator",
"description": "Evaluate AI skills across model tiers with blind testing and refinement recommendations",
"tags": ["evaluation", "testing", "skills", "model-tiers"]
"tags": [
"evaluation",
"testing",
"skills",
"model-tiers"
]
}
]
}
21 changes: 21 additions & 0 deletions .kiro/agents/experimenter.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "experimenter",
"description": "Orchestrates blind skill evaluation across model tiers",
"prompt": "# Experimenter Agent\n\nYou are the experimenter in a blind skill evaluation. Your job is to orchestrate test runs of a skill across model tiers and produce a refinement report.\n\n## Principles\n\n- **Blind testing**: Never reveal expected outcomes to test subjects\n- **Structured protocol**: Define pass/fail criteria BEFORE running tests\n- **Systematic comparison**: Evaluate each tier independently before comparing across tiers\n- **Actionable output**: Every identified failure must include a specific recommendation\n\n## Workflow\n\n1. Receive the skill content and test cases from the evaluate-skill skill\n2. For each model tier (opus, sonnet, haiku):\n a. For each test case, spawn a test-subject agent at the appropriate tier\n b. Provide only the skill content and the input — never the expected outcome\n c. Collect and store the output\n3. Compare outputs against expected outcomes\n4. Generate a structured refinement report\n\n## Report Format\n\n```\n# Skill Evaluation Report\n\n## Summary\n- Skill: [name]\n- Clarity Floor: [lowest passing tier]\n- Overall Pass Rate: [X/Y]\n\n## Per-Tier Results\n### Opus\n| Test Case | Pass/Fail | Notes |\n|-----------|-----------|-------|\n| ... | ... | ... |\n\n### Sonnet\n...\n\n### Haiku\n...\n\n## Failure Analysis\n### [Test Case N at Tier X]\n- **Symptom**: [what went wrong]\n- **Root Cause**: [why the lower-tier agent failed]\n- **Recommendation**: [specific improvement to the skill]\n\n## Recommendations\n1. [Ordered list of improvements, highest impact first]\n```",
"mcpServers": {},
"tools": [
"delegate",
"read",
"write",
"glob",
"grep",
"shell"
],
"toolAliases": {},
"allowedTools": [],
"resources": [],
"hooks": {},
"toolsSettings": {},
"includeMcpJson": true,
"model": null
}
20 changes: 20 additions & 0 deletions .kiro/agents/test-subject.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "test-subject",
"description": "Blind agent that executes a skill and produces output without knowledge of expected outcomes",
"prompt": "# Test Subject Agent\n\n<!-- The model tier above (`sonnet`) is a default. The experimenter agent typically overrides this when spawning test subjects at different tiers (opus → sonnet → haiku). -->\n\nYou are a test subject in a blind skill evaluation. You will receive a skill and an input. Execute the skill to the best of your ability and produce your output.\n\n## Rules\n\n- You do NOT know what the expected outcome is — just do your best\n- Follow the skill instructions exactly as written\n- If the skill instructions are ambiguous, make your best interpretation and note the ambiguity\n- Do not ask clarifying questions — work with what you have\n- Produce your output in a clear, structured format\n\n## Process\n\n1. Read the skill content provided to you\n2. Read the input provided to you\n3. Execute the skill's instructions against the input\n4. Produce your output\n\n## Output Format\n\nProduce your result in this format:\n\n```\n## Output\n\n[Your skill execution output here]\n\n## Notes\n\n- [Any ambiguities encountered]\n- [Assumptions made]\n- [Difficulties faced]\n```",
"mcpServers": {},
"tools": [
"read",
"write",
"shell",
"glob",
"grep"
],
"toolAliases": {},
"allowedTools": [],
"resources": [],
"hooks": {},
"toolsSettings": {},
"includeMcpJson": true,
"model": null
}
12 changes: 7 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,19 @@ cloning, run:
pnpm install
```

This requires `@ai-plugin-marketplace/cli` and `@ai-plugin-marketplace/core` to
be **published to npm**. A fresh `pnpm install` regenerates `pnpm-lock.yaml`
against the published versions.
`@ai-plugin-marketplace/cli` and `@ai-plugin-marketplace/core` are published to
npm, so a fresh `pnpm install` resolves them and writes `pnpm-lock.yaml` against
the published versions.

## Workflow

1. Add or edit a plugin under `plugins/<name>/`. Use `aipm scaffold <name>` to
create a new one or `aipm add-target <plugin> <target>` to expand an
existing plugin's support envelope.
2. Run `aipm build` to regenerate toolkit-owned artifacts (hook JSON and the
`dist/` bundles). Commit both your authored sources and the regenerated
2. Run `aipm build` to regenerate toolkit-owned artifacts: hook JSON, the
`dist/` bundles, the marketplace registries (generated from
`aipm.workspace.ts`), and — for a single-plugin marketplace — the repo-root
Gemini/Kiro artifacts. Commit both your authored sources and the regenerated
output — CI enforces that the tree is clean after a build (freshness).
3. Run `aipm validate` and resolve any hard findings.

Expand Down
26 changes: 26 additions & 0 deletions GEMINI.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Skill Evaluator

Evaluate AI skills across model tiers with blind testing and refinement recommendations.

## Overview

This extension tests AI skills across model tiers (opus → sonnet → haiku) using blind sub-agent testing. It identifies where skill clarity degrades and provides actionable refinement recommendations.

## Usage

Use the `/evaluate` command to start an evaluation:

```
/evaluate <skill-path> <test-cases-path>
```

- `skill-path`: Path to the SKILL.md file to evaluate
- `test-cases-path`: Path to a JSON file with test cases (array of `{ input, expectedOutcome }`)

## How It Works

1. The experimenter agent reads the skill and test cases
2. Separate blind test-subject agents are spawned per test case at the highest tier (opus)
3. The process repeats at lower tiers (sonnet, then haiku)
4. Results are compared against expected outcomes per tier
5. A refinement report identifies where clarity breaks down and recommends improvements
28 changes: 28 additions & 0 deletions POWER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
schemaVersion: 0.1.0
name: skill-evaluator
description: Evaluate AI skills across model tiers with blind testing and refinement recommendations
version: 0.0.1
---

# Skill Evaluator

This power evaluates AI skills across model tiers (opus → sonnet → haiku) using blind sub-agent testing.

## Capabilities

- **Blind evaluation**: Test subjects execute skills without knowledge of expected outcomes
- **Multi-tier testing**: Run skills at progressively lower model tiers to find clarity breakpoints
- **Refinement recommendations**: Generate specific suggestions to improve skill clarity

## Workflow

1. Provide a skill path and test cases (input/expected-outcome pairs)
2. The experimenter agent orchestrates blind test runs across model tiers
3. Results are compared against expected outcomes
4. A refinement report is generated with specific recommendations

## Related Files

- `steering/evaluation-workflow.md` — Evaluation workflow steering
- `mcp.json` — MCP server configuration (empty — this power is skill/agent-based)
104 changes: 87 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ then:
pnpm install
```

> **Note:** `pnpm install` requires the `@ai-plugin-marketplace/*` packages to be
> published to npm. See [CONTRIBUTING.md](CONTRIBUTING.md).
The `@ai-plugin-marketplace/*` packages are published to npm, so a fresh
`pnpm install` resolves them directly. See [CONTRIBUTING.md](CONTRIBUTING.md).

## Authoring plugins

Expand All @@ -43,11 +43,20 @@ repo-root marketplace registries.
aipm build
```

`aipm build` regenerates every toolkit-owned artifact — per-plugin hook JSON
(`hooks/claude.json`, `hooks/hooks.json`) and the standalone bundles under
`dist/gemini/` and `dist/kiro/`. Both authored sources and generated outputs are
committed, so plugins stay browsable on GitHub and pull-request diffs stay
honest.
`aipm build` regenerates every toolkit-owned artifact:

- per-plugin hook JSON (`hooks/claude.json`, `hooks/hooks.json`);
- the standalone bundles under `dist/gemini/` and `dist/kiro/`;
- the marketplace registries (`.claude-plugin/marketplace.json`,
`.cursor-plugin/marketplace.json`) — generated from `aipm.workspace.ts` plus
each plugin's `aipm.config.ts` (see [Marketplace metadata](#marketplace-metadata-and-generated-registries));
- for a single-plugin marketplace, the **repo-root Gemini extension and Kiro
power** (`gemini-extension.json`, `GEMINI.md`, `POWER.md`, `commands/`,
`skills/`, `steering/`, `.kiro/`, …) so the repo installs natively into hosts
that have no marketplace concept.

Both authored sources and generated outputs are committed, so plugins stay
browsable on GitHub and pull-request diffs stay honest.

### Validate

Expand All @@ -57,8 +66,8 @@ aipm validate

Validation checks the support envelope, every target manifest's schema,
cross-target name consistency, MCP server-key sync, marketplace registration,
and freshness (that the committed generated artifacts match what `aipm build`
would produce).
and freshness that the committed generated artifacts (registries, bundles,
and the repo-root Gemini/Kiro emission) match what `aipm build` would produce.

`package.json` exposes these as `pnpm build`, `pnpm check`, and
`pnpm scaffold` if you prefer the npm-script entry points.
Expand All @@ -81,24 +90,76 @@ plugin that carries files for a target outside its envelope. To expand a
plugin's envelope, run `aipm add-target <plugin> <target>` to scaffold the
skeleton files for a new target, then fill in the manifest fields.

## Marketplace metadata and generated registries

Repo-level marketplace metadata lives in `aipm.workspace.ts` at the repo root:

```ts
import { defineWorkspace } from '@ai-plugin-marketplace/core';

export default defineWorkspace({
marketplace: {
name: 'ai-plugin-marketplace',
owner: { name: 'AI Plugin Marketplace Template' },
description: 'Universal AI Plugin Marketplace — author once, distribute to all platforms',
},
});
```

The **presence of this file opts the repo into generated registries.** Instead
of hand-maintaining `.claude-plugin/marketplace.json` and
`.cursor-plugin/marketplace.json`, `aipm build` generates them from the
workspace metadata plus each plugin's `aipm.config.ts` (its `description` and
`keywords` become the registry entry's `description` and `tags`). The generated
registries are committed and freshness-checked like every other artifact.

### Gemini and Kiro: the single-plugin marketplace

Gemini CLI and Kiro have **no marketplace concept** — a repo is installed as one
extension (Gemini) or one power (Kiro) from its root. So when a marketplace
exposes **exactly one plugin**, `aipm build` additionally emits that plugin's
Gemini/Kiro artifacts at the **repo root** (`gemini-extension.json`, `GEMINI.md`,
`POWER.md`, `commands/`, `skills/`, `steering/`, `.kiro/`), letting the repo
install natively into those hosts. A sidecar at `.aipm/generated-root.json`
records exactly which root paths the toolkit owns.

If you add a **second plugin**, the repo can no longer be a single Gemini/Kiro
artifact: `aipm validate` reports a `single-artifact-host` finding, and you keep
full Claude/Cursor marketplace support (which can host many plugins) while
choosing one plugin to expose to Gemini/Kiro. Because the repo root is the
single Gemini/Kiro artifact, the plugin does **not** carry its own `LICENSE` or
`README.md` — the repo-root `LICENSE`/`README.md` are canonical and are never
overwritten by generation.

## Repository structure

```
.
├── aipm.workspace.ts # Marketplace metadata (opts into generated registries)
├── .claude-plugin/
│ └── marketplace.json # Claude Code marketplace registry
│ └── marketplace.json # Claude Code registry (GENERATED)
├── .cursor-plugin/
│ └── marketplace.json # Cursor marketplace registry
│ └── marketplace.json # Cursor registry (GENERATED)
├── plugins/
│ └── <plugin-name>/ # One directory per plugin (authored sources)
│ ├── aipm.config.ts # Support envelope + plugin version
│ ├── aipm.config.ts # Support envelope + version + description/keywords
│ └── ... # Manifests, agents, skills, commands, hooks, rules
├── dist/ # Generated standalone bundles (committed)
│ ├── gemini/
│ └── kiro/
│ # Repo-root Gemini/Kiro artifacts — emitted for a single-plugin marketplace
│ # (GENERATED; tracked in .aipm/generated-root.json):
├── gemini-extension.json
├── GEMINI.md
├── POWER.md
├── commands/ skills/ steering/ agents/ hooks/ mcp.json .kiro/
├── .aipm/
│ └── generated-root.json # Records which repo-root paths the toolkit owns
├── package.json # Depends on @ai-plugin-marketplace/cli + core
├── LICENSE
└── README.md
├── LICENSE # Canonical — also serves the Gemini/Kiro artifact
└── README.md # Canonical — also serves the Gemini/Kiro artifact
```

## Upgrading the toolkit
Expand All @@ -119,9 +180,18 @@ when.
## Example plugin: skill-evaluator

The included `skill-evaluator` plugin demonstrates the full multi-platform
pattern end-to-end — it evaluates AI skills across model tiers (opus → sonnet →
haiku) using blind sub-agent testing. See
[plugins/skill-evaluator/README.md](plugins/skill-evaluator/README.md).
pattern end-to-end. Given a skill (`SKILL.md`) and a set of test cases, it:

1. runs the skill with blind test-subject agents at different model tiers
(opus → sonnet → haiku);
2. compares outputs against expected outcomes;
3. identifies where skill clarity degrades at lower tiers;
4. generates actionable refinement recommendations.

Invoke it in any host with `/evaluate path/to/SKILL.md path/to/test-cases.json`,
where each test case is `{ "input": "...", "expectedOutcome": "..." }`. The skill
definition lives at
[plugins/skill-evaluator/skills/evaluate-skill/SKILL.md](plugins/skill-evaluator/skills/evaluate-skill/SKILL.md).

## License

Expand Down
Loading
Loading