Add system-prompt-optimizer and synthetic-dataset-v2 skills#1
Conversation
Implements RES-398 (automated prompt optimization via PO1/PO2 deployments) and RES-399 (deployment-based synthetic data generation with create and expand modes) as MCP-compatible agent skills. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RES-355 Implement MCP skills in orqkit for mcp usage
As reference, look at https://github.com/huggingface/skills Ticket to transform into skills: |
| ### 3. Preserve Intent | ||
| The optimizer should improve how the prompt is expressed, not change what it does. Always verify the optimized prompt preserves the original intent, persona, and constraints. | ||
|
|
||
| ### 4. Validate with Experiments |
There was a problem hiding this comment.
This is not able to be done with current mcp. Should we omit this?
|
|
||
| When presenting PO1/PO2 suggestions to the user, reference which guideline each suggestion targets to help them understand the reasoning. | ||
|
|
||
| ### Model Configuration |
There was a problem hiding this comment.
this feels unnecessary to include
|
|
||
| # System Prompt Optimizer | ||
|
|
||
| Automated prompt optimization using orq.ai's PO1 (analyzer) and PO2 (rewriter) deployments — get AI-powered analysis and rewriting of system prompts without manual trace analysis. |
There was a problem hiding this comment.
As mentioned, we should do this all within the coding agent, and not use deployments externally for this. Simplifies the solution and imo makes more sense.
| - `run-experiment` — validate optimized prompts with A/B experiments | ||
| - `manage-deployment` — configure deployments with the optimized prompt | ||
|
|
||
| ## When to use |
There was a problem hiding this comment.
not sure we need this. It's invoked by the user themselves.
| allowed-tools: Bash, Read, Write, Edit, Grep, Glob, WebFetch, Task, AskUserQuestion, mcp__linear-server__*, orq* | ||
| --- | ||
|
|
||
| # System Prompt Optimizer |
There was a problem hiding this comment.
PO1 is essentially an analyzer, and po2 which optimizes a prompt given some instructions or suggestions. To be used one after another.
In the context of this skill, let's just make it a 2 step process. Step 1 can be somewhat skipped if the user already gave specific instructions in what way they want the prompt to be changed.
Examples
/prompt-optimizer - start with analysis, applies the findings
/prompt-optimizer make this way more assertive - analysis not needed, instructions already given. Start directly with step 2.
| --- | ||
| name: generate-synthetic-dataset-v2 | ||
| description: Deployment-based synthetic data generation — create datasets from descriptions or expand existing datasets with few-shot examples | ||
| allowed-tools: Bash, Read, Write, Edit, Grep, Glob, WebFetch, Task, AskUserQuestion, mcp__linear-server__*, orq* |
There was a problem hiding this comment.
this needs access to linear?
|
|
||
| ### orq MCP Tools | ||
|
|
||
| Use the orq MCP server (`https://my.orq.ai/v2/mcp`) as the primary interface. For operations not yet available via MCP, use the HTTP API as fallback. |
There was a problem hiding this comment.
perhaps also have it use the api if the data is really big. mcp is not ideal for that.
| - evaluate-rag -> "skills/evaluate-rag/SKILL.md" | ||
| - feedback-loop -> "skills/feedback-loop/SKILL.md" | ||
| - generate-synthetic-dataset -> "skills/generate-synthetic-dataset/SKILL.md" | ||
| - generate-synthetic-dataset-v2 -> "skills/generate-synthetic-dataset-v2/SKILL.md" |
There was a problem hiding this comment.
as mentioned, let's see if can deprecate v1 so we only have 1 version. If you think there are things missing in v2 that are in v1, please add them.
…thetic dataset skills - Rename system-prompt-optimizer → prompt-optimizer with agent-native analysis/rewriting (no external deployments) - Merge generate-synthetic-dataset v1+v2 into single skill with 3 modes (structured, quick, expand) - Remove deployment dependencies per reviewer feedback - Add HTTP API bulk fallback for large datasets - Update AGENTS.md, README.md, and optimize-prompt companion refs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@Baukebrenninkmeijer I made the corrections based on yout feedback. consolidated into 1 prompt-optimizer skill and one dataset generation skill. I will leave the other structural changes for another PR. |
- Remove invalid `metadata` field from memory creation examples (issues #1) - Fix memory document field name from `content` to `text` (issue #2) - Add required `path` field to tool creation example (issue #3) - Fix KB search param from `limit` to `top_k` (issue #4) - Correct MCP URL to `https://my.orq.ai/v2/mcp` in run-experiment and generate-synthetic-dataset (issue #5) - Replace non-existent `setup-observability` with `analyze-trace-failures` in compare-agents companion skills (issue #6) - Update stale TOC entry in knowledge-base-management (issue #7) - Add missing `path` and `type` to KB creation in run-experiment (issue #8) - Add explicit `-X POST` to KB search curl commands (issue #9) - Fix "two things" wording when listing three items (issue #10) - Standardize agent model format to object style (issue #11) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 participants
Summary
system-prompt-optimizerskill — automated prompt optimization using orq.ai's PO1 (analyzer) and PO2 (rewriter) deployments with Quick Optimize and Advanced Optimize workflowsgenerate-synthetic-dataset-v2skill — deployment-based synthetic data generation with two modes: create from scratch via description, and expand existing datasets with few-shot examplesAGENTS.md,README.md)Test plan
./scripts/publish.sh --checkpasses (no stale artifacts)system-prompt-optimizer/SKILL.mdfor PO1/PO2 schema accuracy against deployed configurationsgenerate-synthetic-dataset-v2/SKILL.mdfor deployment I/O alignment with RES-33🤖 Generated with Claude Code