Quality is checked on separate layers so one passing metric cannot hide another failure.
npm run validate checks:
- skill structure, metadata, invocation policies, local links, and progressive-reference reachability
- canonical rule schema, source IDs, target skills, and generated-reference drift
- plugin manifests, version synchronization, required public documentation, and lockfile state
- routing-case balance, explicit-only boundaries, scenario schemas, fixture containment, and golden expectations
No model token, authenticated CLI, or repository secret is needed.
Each skill has at least twenty natural requests split across positive and negative cases. Near-misses distinguish plan from build, audit from critique, operational apps from marketing pages, and broad workflows from isolated changes. Cases include English, German, and realistic typos.
For explicit-only skills, a semantically perfect but unnamed request remains negative. A positive case names $completeness-audit or $consistency-critique.
Candidate providers stage all four skills, including their invocation metadata. The shared routing prompt names the candidate set and output shape but does not restate the skills' trigger policy. The model must choose from the staged skills' own discoverable metadata and instructions. Client adapters translate exact native or manual invocation markers; malformed-marker cases test that deterministic boundary separately. Routing accuracy therefore measures the staged metadata, adapter, and model path. Native discovery outside this isolated harness remains a separate smoke test.
Each skill has at least three synthetic fixture cases:
- a normal incomplete operational workflow
- a scale, error, or state-consistency failure
- a bounded case that penalizes generic overbuilding
Golden files list expected stable rule IDs, critical expectations, and forbidden findings. Build cases run only in disposable copies.
The deterministic assertions verify structural contracts: canonical rule/status coverage, bounded rule references, declared verification methods, valid fixture evidence locations, and conservative nontrivial-text floors. They do not prove that free-form recommendations, target clauses, or exit criteria are semantically good. Provider-blind human review owns correctness, actionability, and prose quality.
Promptfoo drives pinned Codex and Claude CLI providers. The authoritative v0.1
workflow has six matrix shards: Codex and Claude Code each run separate routing,
objective, and critical shards. At most two shards run concurrently. Every shard
has a 360-minute job timeout and a 300,000 ms provider timeout, then runs its
required no-skill baseline before its candidate, three times per case with caching
and result sharing disabled.
Candidate thresholds are:
- at least 90% routing accuracy per gated client
- at least 90% objective scenario expectations
- every critical expectation satisfied
From the second release onward, the release decision also compares the previous tag; v0.1 has no previous-tag gate. Candidate-only local runs are calibration only and cannot replace an authoritative baseline-first workflow run; in particular, candidate-only provider filters are local calibration only.
Each result export is treated as untrusted evidence. The fail-closed integrity validator checks the expected case-by-repeat count, exact provider and generated metadata, unique result IDs, provider/assertion errors, parseable schema-valid output, selected-skill behavior for candidate versus baseline, and fixture evidence paths and line ranges. Missing, partial, malformed, mislabeled, or error-bearing exports cannot serve as release evidence. Artifact uploads also fail when required files are absent.
Blind A/B comparison is supporting evidence, not a sole blocker. The release workflow creates packets only in the two objective shards, after pairing the candidate and no-skill objective outputs. A cryptographically random 32-byte seed is generated at runtime with shell tracing disabled. It is not logged or derived from a public workflow value; the persisted seed and assignments exist only in the separate answer-key artifact. A fixed seed supplied to the local packet command is for reproduction only and is never an authoritative release assignment. Packet creation rejects provider/assertion errors, non-JSON or schema-invalid responses, wrong treatment routing, partial pairing, and unsafe redaction. Reviewers receive only the packet and score correctness, scope discipline, evidence/actionability, and overall preference. The resulting report is descriptive and contains no pass/fail field. Promptfoo result exports and tool versions are retained as shard artifacts for 30 days; review packets and answer keys are separate artifacts.
The workflow is manual and runs only for refs/heads/main through the protected
release-evals GitHub Environment. Model credentials are injected only into the
baseline and candidate eval steps; install, validation, packet, and upload steps do
not receive them. Third-party actions are pinned to immutable commit SHAs.
Maintainers must configure the environment with a required reviewer and a
main-only deployment-branch rule. The in-repository guard is validated, while
the environment protection itself remains an administrator setting.
The Cursor adapter and canary are included but advisory in v0.1 because cursor-agent is not calibrated in the release environment.