Skill to construct configuration JSON files powering the lemonade router - #105

[sdevinenamd]

Adds a new skill that converts plain-English routing intent into a valid Lemonade collection.router policy JSON, ready for the user to register via POST /api/v1/pull.

Router reference: https://lemonade-server.ai/docs/dev/router-policy/

What the skill does

• Supports all three authoring levels: deterministic rules (keywords, regex, length, tools, images, metadata), semantic similarity classifiers (L2), and LLM-as-router (L0a)
• Generates and offline-validates the JSON before showing it
• Fills all unspecified fields with safe defaults (mode selection, ids, thresholds, on_error, model_name derivation)
• Hands the user copy-paste curl commands to register and trace-test on their own

Files

• SKILL.md: Generation steps 1–8, mode selection logic, condition reference table
• reference.m: Full schema contract, classifier type matrix, validation checklist
• examples.md: 7 worked NL→JSON pairs (examples 5–7 are production configs from an enterprise HR/Benefits/Finance deployment)
• scripts/validate.py: Offline structural validator

Example Prompts:

• Any message containing a credit card number pattern (four groups of four digits, optionally separated by spaces or dashes) must stay on Qwen3.5-9B-GGUF. Everything else goes to Qwen3.6-35B-A3B.
• Requests tagged with either "legal" or "compliance" or "audit" must stay on the local model Qwen3.5-9B-GGUF. Everything else goes to Qwen3.6-35B-A3B.
• Use an embedding model to detect whether a request is about cooking. High confidence cooking questions (above 0.8) go to Qwen3-1.7B-GGUF. Medium confidence ones (between 0.5 and 0.8) go to Qwen3.5-9B-GGUF but tag them as uncertain. Anything below 0.5 falls through to Qwen3.6-35B-A3B.
• I want image requests to stay on Qwen3.5-9B-GGUF since it supports vision. Requests without any images go to Qwen3-1.7B-GGUF. Default to the vision model.

[ramkrishna2910]

Some comments:

1. Example 7 embeds the exact mistake the skill warns against. The complexity llm classifier's prompt ends with "Reply with exactly one label: COMPLEX or SIMPLE." The engine's {"model", "rationale"} contract applies to all llm classifiers, not just routing.router — it's the same LlmClassifier class either way. A weaker model obeying that authored line replies bare COMPLEX, the strict parser rejects it, the score comes back empty, and the rule silently never fires. That's the identical silent-fall-through failure described in checklist item 12 — but the skill scopes the warning (and validate.py's BAD_ROUTER_PROMPT_RE) to router.prompt only. Fix: delete that sentence from example 7, and extend item 12 + the validator regex to cover llm classifier prompts too.

2. The reference never documents that hand-authored llm classifiers speak the router's wire contract — that their score is binary (chosen label = 1.0, everything else absent). Authors will naturally write "reply with the label" prompts unless told; this is the doc gap that produced problem 1.

3. Minor gap in validate.py: the server rejects regexes with nested unbounded quantifiers (e.g. (X+)+) at policy load as a ReDoS safeguard (documented in route_policy.schema.json); the validator's Python re.compile check can't catch that, so a policy can pass validation and still fail /pull. A cheap pattern check would close it.

4. Nits: the decision object also carries version: "1" (skill omits it); metadata comparisons are case-sensitive with comma-decoded token sets for any (unmentioned); and example 3's keywords_all: ["apparel", "clothes"] demands both words for a user who said "clothing or apparel" — should be keywords_any.

[sdevinenamd]

@ramkrishna2910 Thanks for the feedback! Addressed them in the latest commit

[eddierichter-amd]

High @meghsat thanks for putting this together. Just a couple small changes:

1. Unknown keys pass validation. The reference says the server rejects unknown keys at every level, but scripts/validate.py does not reject them at the root, routing, classifier, or nested metadata levels. I reproduced this by adding "typo": true at each level; every policy still returned ready: true. This could allow a policy to pass offline
   validation and then fail during /pull.

2. Regex validation uses the wrong dialect. Policies use ECMAScript regex, but the validator checks patterns with Python re. For example, (?Pcode) is valid Python regex but invalid ECMAScript, yet the validator returns ready: true.

Non-blocking: the new skill is published in the plugin manifests but is missing from the main README catalog.

Validation performed:

• ./.github/scripts/check.sh passed.
• Python compilation passed.
• All seven example policies passed the bundled validator.
• git diff --check passed.
• Behavioral tests were not run because they require an authenticated Claude CLI.

[sdevinenamd]

@eddierichter-amd Thanks for the review! Fixed the bugs.
Will check with @danielholanda on editing the readme

[eddierichter-amd]

@eddierichter-amd Thanks for the review! Fixed the bugs. Will check with @danielholanda on editing the readme

Thanks for the updates. Root, routing, and classifier typos are now rejected, and the original Python named-group regex case is handled. Two blocking gaps remain:

1. Unknown keys inside a metadata object still pass with ready: true. Also, the new root whitelist rejects the optional models field even though reference.md lists it as allowed.
2. Node’s regex engine is not equivalent to the server’s stated C++ std::regex::ECMAScript engine. For example, (?<=foo)bar is accepted by Node and the validator but rejected by std::regex. The fallback detects lookbehind, but that check is bypassed whenever Node is installed.

Non-blocking: the skill is still missing from the main README catalog.

./.github/scripts/check.sh, Python compilation, git diff --check, and all seven example policies passed.

[sdevinenamd]

@eddierichter-amd fixed the issues. For the 2nd gap, the only remaining issue is exotic patterns that are valid ES5.1 syntax but where std::regex's specific implementation diverges from Node on edge cases. That's a much smaller and less predictable class than the ES2018 gap we already closed.