Skill to construct configuration JSON files powering the lemonade router - #105

.claude-plugin/marketplace.json

@@ -17,7 +17,8 @@
         "./skills/local-ai-use",
         "./skills/local-ai-app-integration",
         "./skills/serving-llms-on-instinct",
-        "./skills/tracelens-analysis-orchestrator"
+        "./skills/tracelens-analysis-orchestrator",
+        "./skills/build-lemonade-router"
       ],
       "description": "AMD's verified Agent Skills in one plugin: route image/audio through local AI on Ryzen AI, serve LLMs on AMD Instinct GPUs with vLLM, and analyze GPU kernel and PyTorch trace performance."
     }

.codex-plugin/plugin.json

@@ -21,7 +21,8 @@
     "./skills/local-ai-use",
     "./skills/local-ai-app-integration",
     "./skills/serving-llms-on-instinct",
-    "./skills/tracelens-analysis-orchestrator"
+    "./skills/tracelens-analysis-orchestrator",
+    "./skills/build-lemonade-router"
   ],
   "interface": {
     "displayName": "AMD Skills",

.cursor-plugin/marketplace.json

@@ -16,7 +16,8 @@
         "./skills/local-ai-use",
         "./skills/local-ai-app-integration",
         "./skills/serving-llms-on-instinct",
-        "./skills/tracelens-analysis-orchestrator"
+        "./skills/tracelens-analysis-orchestrator",
+        "./skills/build-lemonade-router"
       ],
       "description": "AMD's verified Agent Skills in one plugin: route image/audio through local AI on Ryzen AI, serve LLMs on AMD Instinct GPUs with vLLM, and analyze GPU kernel and PyTorch trace performance.",
       "author": {

skills/build-lemonade-router/SKILL.md

@@ -0,0 +1,256 @@
+---
+name: build-lemonade-router
+description: >-
+  Turns a natural-language description of routing intent into a valid Lemonade
+  `collection.router` policy JSON. The skill generates and validates the JSON
+  only - it does not register it or call the live server.
+  Use when the user wants to route requests between models ("route sensitive
+  queries to X and everything else to Y"), generate a router/hybrid-router
+  config or policy, author a collection.router JSON, split traffic between a
+  small local model and a big/cloud model, add PII/jailbreak/topic classifiers
+  to routing, or mentions Lemonade Router, routing rules, routing.router,
+  candidates/default_model, keywords_any, semantic_similarity, or LLM-as-router.
+  Fills every field the user did not specify with safe defaults.
+---
+
+# Lemonade Router Config Generator
+
+Generate a **`collection.router` policy JSON** from a plain-English description
+of how requests should be routed. The skill produces and validates the JSON
+only - it does not call the live server, register the policy, or run requests
+through it. The JSON is accepted by the strict server-side parser on the first
+try and stays editable in the desktop app's Hybrid Router editor.
+
+## Prerequisites
+
+- **Lemonade Server v10.1.0+** running locally (`lemonade server start`).
+  Required only to register and test the generated policy - the skill itself
+  (JSON generation + offline validation) works without a live server.
+- **No GPU or ROCm dependency** for authoring. The router policy is a JSON
+  document; no hardware is needed to generate or validate it.
+- **Python** (any 3.x) in PATH - used by the bundled offline validator
+  (`scripts/validate.py`). No extra packages required.
+
+The router picks one **candidate** model per request. Two authoring modes
+exist, and choosing the right one is the first decision:
+
+| Mode | JSON shape | When |
+|------|-----------|------|
+| **LLM-as-router** | `routing.router` block | The user describes intent only by *meaning* ("sensitive", "hard questions", "creative writing") with no concrete signals. A small LLM reads each prompt and picks the candidate. |
+| **Rules** | `routing.rules` (+ optional `routing.classifiers`) | The user names any concrete signal: keywords, regex, length, tools, images, metadata, PII/topic classifiers, thresholds, "first match", fallback logic. Deterministic, no extra LLM call for simple conditions. |
+
+`routing.router` is **mutually exclusive** with `routing.rules` and
+`routing.classifiers` - never emit both.
+
+## Step 1 - Extract from the user's words
+
+- **Candidates**: the models that may *answer* requests. Verbatim model names
+  (e.g. `Gemma-3-4b-it-GGUF`). If the user names none, ask - never invent
+  model names. `lemonade list` or `GET /api/v1/models` shows what's available.
+- **Default / fallback**: which candidate gets everything that matches nothing.
+  If unstated, use the model the user framed as "local", "small", or "safe";
+  otherwise the first candidate mentioned.
+- **Signals**: every condition mentioned (keywords, patterns, length, images,
+  tools, topics, PII, safety) and which model each one routes to.
+- **Classifier models**: models named for *detection* rather than answering
+  (BERT-style encoders, embedding models, an LLM used as judge).
+
+## Step 2 - Scaffold
+
+Always exactly this envelope (the parser rejects unknown or missing keys):
+
+```json
+{
+  "version": "1",
+  "model_name": "user.MyHybridRouter",
+  "recipe": "collection.router",
+  "components": [],
+  "routing": { }
+}
+```
+
+- `version` is the literal string `"1"`.
+- `model_name` must start with `user.`; slug from the user's description if
+  they gave a name (`user.<Name>` using only `[A-Za-z0-9._-]`). If they didn't
+  name it, derive one from context instead of a fixed literal - e.g.
+  `user.<slug-of-default-candidate>-Router` - so two different policies don't
+  collide by default. **`/pull` is idempotent per `model_name`: registering a
+  second policy under the same name silently overwrites the first.** If this
+  conversation already produced an unnamed router, don't reuse the same
+  derived name for the next one - ask, or pick a visibly different name.
+
+## Step 3 - Candidates and default
+
+```json
+"candidates": ["<answering models>"],
+"default_model": "<one of candidates>"
+```
+
+`default_model` MUST be listed in `candidates`. Candidates should be
+chat-capable LLMs - not embedding, classification, or image models.
+
+## Step 4 - Mode A: LLM-as-router
+
+```json
+"router": {
+  "type": "llm",
+  "model": "<small chat LLM>",
+  "prompt": "You route user requests to the best model. <one sentence per candidate: when to pick it, using the exact model name>."
+}
+```
+
+- `model` defaults to the smallest candidate (it may be a candidate; it also
+  works as a separate small model).
+- **Write intent only - never specify a reply format.** The engine
+  unconditionally appends its own contract after your prompt: it lists the
+  candidate names and demands a strict JSON reply `{"model": "<name>",
+  "rationale": "<one sentence>"}`, then falls back to `default_model` on any
+  deviation. A prompt that also says "reply with ONLY the model name" (or
+  similar) is not just redundant, it's wrong about the wire format and can
+  confuse weaker judge models into replying with a bare string that then
+  fails to parse - silently falling back to `default_model` on every request,
+  with no visible error. Only describe *when to pick which candidate*.
+- Emit **no** `rules` and **no** `classifiers` key in this mode.
+
+## Step 5 - Mode B: classifiers
+
+Only declare classifiers the rules actually reference. Three types:
+
+```json
+{ "id": "clf-1", "type": "classifier", "model": "<classification model>",
+  "labels": ["PII", "Jailbreak"], "default_label": "PII", "on_error": "match_false" }
+
+{ "id": "clf-2", "type": "semantic_similarity", "model": "<embedding model>",
+  "reference_phrases": { "shopping": ["I want to shop for pants", "add to cart"] },
+  "default_label": "shopping", "on_error": "match_false" }
+
+{ "id": "clf-3", "type": "llm", "model": "<chat LLM>",
+  "prompt": "Classify the request into only labels SAFE, RISKY",
+  "labels": ["SAFE", "RISKY"], "default_label": "SAFE", "on_error": "match_false" }
+```
+
+Hard constraints (parser-enforced - see `reference.md` for the full matrix):
+
+- `classifier` type: model should be a text-classification model (an
+  `onnxruntime` encoder like `Bert-Phishing-ONNX`); `labels` must match the
+  model's actual output labels. A chat LLM here is legal (LLM-as-classifier
+  via chat) but prefer `type: "llm"` for that - it is explicit and prompted.
+- `semantic_similarity`: `reference_phrases` is `{concept: [phrases...]}`,
+  at least one concept, each with at least one phrase. Concept names ARE the
+  labels - a `labels` key is **rejected** for this type. Model must be an
+  embedding model. Give 3–5 varied phrases per concept when inventing them.
+- `llm`: `prompt` AND non-empty `labels` are both required. **Write intent
+  only - never tell the model how to format its reply.** The engine appends
+  its own `{"model": "<chosen_label>", "rationale": "..."}` contract after
+  your prompt (the same contract as `routing.router`). An authored line like
+  "Reply with exactly one label: SAFE or RISKY" causes weaker models to output
+  bare `SAFE`, which the parser rejects - the score comes back empty and the
+  rule silently never fires. Describe what makes a request belong to each
+  label; leave the reply format to the engine.
+- `default_label`, when present, must be one of the labels/concepts.
+- Defaults when unspecified: `id` = `clf-1`, `clf-2`, …; `on_error` =
+  `"match_false"` (fail-open: a broken classifier doesn't match, so requests
+  fall through - use `"match_true"` only when the user wants fail-closed
+  safety); `default_label` = the first label.
+
+## Step 6 - Mode B: rules
+
+```json
+"rules": [
+  { "id": "rule-1", "match": { ... }, "route_to": "<candidate>",
+    "outputs": { "reason": "<optional free-form>" } }
+]
+```
+
+- **Order matters - first match wins.** Put the most specific /
+  privacy-critical rules first (a "sensitive stays local" rule must precede a
+  "code goes to the big model" rule, or coding prompts with PII leak).
+- `route_to` MUST be a candidate. `id` uses only `[A-Za-z0-9._-]`; default
+  `rule-1`, `rule-2`, ….
+- No rule for the "everything else" case - that is `default_model`.
+
+**Match conditions** - combine with `all` (AND), `any` (OR), `not`; one
+condition per leaf object; nesting is allowed:
+
+| Leaf | Example | Notes |
+|------|---------|-------|
+| `keywords_any` / `keywords_all` | `{ "keywords_any": ["SSN", "Email"] }` | case-insensitive substring - `"hi"` matches inside `"this"`, `"shipping"`, `"high"`, etc. Use `regex` with `\b...\b` when word-boundary precision is needed |
+| `regex` | `{ "regex": "\\b\\d{3}-?\\d{2}-?\\d{4}\\b" }` | ECMAScript flavor |
+| `min_chars` / `max_chars` | `{ "min_chars": 4000 }` | input length, UTF-8 bytes, non-negative integer |
+| `has_tools` / `has_images` | `{ "has_images": true }` | booleans |
+| `classifier` | `{ "classifier": "clf-1", "label": "PII", "min_score": 0.5 }` | band test; `min_score`/`max_score` in [0,1]; default `min_score` 0.5; omit `label` only if the classifier has `default_label` |
+| `metadata` | `{ "metadata": { "key": "consent", "equals": "denied" } }` | exactly one of `equals` / `any` / `exists`; note: not editable in the desktop UI yet - use only when the user asks for metadata routing |
+
+## Step 7 - Components
+
+`components` = union of: all `candidates` + every classifier `model` + the
+`router.model` (Mode A). Deduplicate, keep order stable. The parser rejects
+any referenced model that is not declared here.
+
+## Step 8 - Validate
+
+Write the JSON to a file, then **run the bundled offline validator** before
+presenting anything to the user - it mechanically re-checks every rule in
+`reference.md`'s validation checklist (score ranges, non-negative integer
+char counts, mode exclusivity, label/classifier references, the `model_name`
+collision, the router-prompt-contract mistake above, etc.) so you don't have
+to re-derive them by eye every time:
+
+```bash
+python scripts/validate.py router.json    # Windows
+python3 scripts/validate.py router.json   # macOS/Linux
+```
+
+It exits 0 with `"ready": true` when there are no errors (warnings/advisories
+may still be worth mentioning to the user). If it reports errors, fix the
+JSON and re-run - don't present JSON that fails this check. It is pure
+offline structural/numeric validation; it cannot verify a named model
+actually exists or has the right capability (chat/embedding/classification).
+
+## Step 9 - Output curl commands
+
+**This step is mandatory — do not skip it even if the user did not ask.**
+
+Do not call the live server yourself — not even to check models or register
+the policy. Instead, print the three curl commands below as plain text for
+the user to copy and run. Fill in `<model-id>` and `<model_name>` from the
+policy, and a short `<test prompt>` that should hit the first rule. Do not
+execute these commands with Bash or any other tool.
+
+```bash
+# 1. Check a model exists before registering
+curl http://localhost:13305/api/v1/models/<model-id>
+
+# 2. Register the policy (idempotent - re-POST to update)
+curl -X POST http://localhost:13305/api/v1/pull \
+     -H "Content-Type: application/json" --data-binary @router.json
+
+# 3. Route a request and inspect the decision
+curl -X POST http://localhost:13305/api/v1/chat/completions \
+     -H "Content-Type: application/json" \
+     -d '{"model": "<model_name>", "route_trace": true,
+          "messages": [{"role": "user", "content": "<test prompt>"}]}'
+```
+
+The `x-lemonade-route` response header carries the matched rule id (or
+`default`). With `"route_trace": true` the body also carries
+`x_lemonade_route`: `{ route_to, matched_rule, default_used, outputs,
+trace[] }` - useful for verifying each rule fires as expected.
+
+## Defaults summary
+
+| Field | Default when the user doesn't say |
+|-------|-----------------------------------|
+| `model_name` | `user.<default-candidate-slug>-Router` (never reuse a name already used earlier in this conversation) |
+| `default_model` | the "small/local/safe" candidate, else first mentioned |
+| mode | rules if any concrete signal is named, else LLM-as-router |
+| classifier `id` / rule `id` | `clf-N` / `rule-N` |
+| `on_error` | `match_false` |
+| `default_label` | first label / concept |
+| `min_score` | `0.5` |
+| `outputs` | omit |
+| router prompt | intent only - no reply-format instruction (Step 4) |
+
+Worked NL → JSON pairs live in `examples.md`; the full schema, parser error
+matrix, and model-capability table live in `reference.md`; the offline
+validator is `scripts/validate.py` (run it - see Steps 8–9).

skills/build-lemonade-router/evals/evals.py

@@ -0,0 +1,111 @@
+# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+#
+# See LICENSE for license information.
+
+"""Behavioral tests for the `build-lemonade-router` skill.
+
+Run locally (needs the `claude` CLI authenticated):
+
+    cd eval/behavioral
+    python -m pytest -c pytest.ini -p conftest ../../skills/build-lemonade-router/evals/evals.py
+
+Each check on `run` prints a `[PASS]`/`[FAIL]` line and raises on failure.
+`logs_contains` / `workspace_contains` are deterministic; `should` /
+`should_not` are graded by an LLM judge over the captured evidence.
+"""
+
+from harness import claude
+
+
+def test_keyword_router_generation():
+    """Trigger: user asks to route coding questions to a bigger model."""
+    agent_configs = [(claude, "opus")]
+    for agent, model in agent_configs:
+        with agent(model, skill="build-lemonade-router") as run:
+            run = run.prompt(
+                "Route coding questions - anything mentioning functions, bugs, "
+                "or stack traces - to Qwen3.5-9B-GGUF. Everything else goes to "
+                "Qwen3.5-2B-GGUF."
+            )
+
+            # Deterministic: skill must produce a JSON file and validate it
+            run.logs_contains("build-lemonade-router")
+            run.workspace_contains("router.json")
+
+            # Positive behavioral expectations
+            run.should("Produce a collection.router JSON with recipe collection.router")
+            run.should("Include keywords_any condition matching coding-related terms")
+            run.should("Set Qwen3.5-2B-GGUF as default_model")
+            run.should("Run the offline validator scripts/validate.py and report it passed")
+            run.should("Output curl commands for the user to register and test the policy")
+
+            # Negative behavioral expectations
+            run.should_not("Execute POST /api/v1/pull by making an actual HTTP call with a tool")
+            run.should_not("Invent model names not provided by the user")
+            run.should_not("Include both router and rules keys in the same policy")
+
+
+def test_pii_regex_router_generation():
+    """Trigger: user asks to keep PII local using regex patterns."""
+    agent_configs = [(claude, "opus")]
+    for agent, model in agent_configs:
+        with agent(model, skill="build-lemonade-router") as run:
+            run = run.prompt(
+                "Any message with a Social Security number or email address must "
+                "stay on Qwen3.5-9B-GGUF. Everything else can go to "
+                "Qwen3.5-9B-NoThinking."
+            )
+
+            run.logs_contains("build-lemonade-router")
+            run.workspace_contains("router.json")
+
+            run.should("Include a regex condition matching SSN patterns")
+            run.should("Include a regex condition matching email addresses")
+            run.should("Place the PII rule before any other rules")
+            run.should("Run the offline validator and confirm the JSON is ready")
+
+            run.should_not("Execute any live Lemonade server API call by making an actual HTTP call with a tool")
+            run.should_not("Use routing.router instead of routing.rules for this request")
+
+
+def test_llm_as_router_generation():
+    """Trigger: user describes intent only by meaning with no concrete signals."""
+    agent_configs = [(claude, "opus")]
+    for agent, model in agent_configs:
+        with agent(model, skill="build-lemonade-router") as run:
+            run = run.prompt(
+                "I want sensitive queries to go to Qwen3.5-9B-GGUF and everything "
+                "else to Qwen3.5-9B-NoThinking. Use the local model as the router."
+            )
+
+            run.logs_contains("build-lemonade-router")
+            run.workspace_contains("router.json")
+
+            run.should("Use routing.router block instead of routing.rules")
+            run.should("Set type to llm inside the router block")
+            run.should("Write a prompt that describes routing intent only, not reply format")
+            run.should_not("Include both routing.router and routing.rules in the output")
+            run.should_not("Tell the router model to reply with only the model name")
+
+
+def test_non_trigger_general_question():
+    """Non-trigger: general question unrelated to routing should not activate skill."""
+    agent_configs = [(claude, "opus")]
+    for agent, model in agent_configs:
+        with agent(model, skill="build-lemonade-router") as run:
+            run = run.prompt("What is the capital of France?")
+
+            run.should_not("Produce a collection.router JSON")
+            run.should_not("Run scripts/validate.py")
+            run.should_not("Mention routing.rules or routing.candidates")
+
+
+def test_non_trigger_unrelated_code_task():
+    """Non-trigger: coding task unrelated to Lemonade routing."""
+    agent_configs = [(claude, "opus")]
+    for agent, model in agent_configs:
+        with agent(model, skill="build-lemonade-router") as run:
+            run = run.prompt("Write a Python function to reverse a linked list.")
+
+            run.should_not("Generate a collection.router policy")
+            run.should_not("Ask about Lemonade model candidates")