docs+skill: onboarding UX (Node 22 prereq, version pin, explicit model selection) - #5
Open
tangym wants to merge 1 commit into
Open
docs+skill: onboarding UX (Node 22 prereq, version pin, explicit model selection)#5tangym wants to merge 1 commit into
tangym wants to merge 1 commit into
Conversation
…l selection) Three small onboarding-experience improvements grouped together because they all fix places where the current flow silently fails or silently picks a suboptimal default: README: - Option 1 (npx skills add ...) now states the Node.js >= 22 prerequisite up front and points users to Option 2 as the no-Node fallback. The underlying skills CLI imports styleText from node:util which is Node-22-only; running under Node 18/20 fails with a SyntaxError that looks unrelated to Node. wire-assert-ci skill (SKILL.md + prompt + mdc mirrors): - 'assert-ai version pin' section: at workflow-authoring time (not every CI run), resolve the latest stable assert-ai from PyPI and write that exact version into assert-ai-version. Pinning is required so the paired-binary McNemar test compares like against like across baseline and PR runs; a floating install would attribute version drift to code drift and produce false regressions. Users still own subsequent bumps. - 'Model selection' section: available models are user-specific (Azure deployment names, provider access), so the skill must ask the user before generating any behavior YAML. Ask for three roles in one prompt (judge, tester, default), use LiteLLM-style names, verify provider-env secrets match the chosen provider, and write every role explicitly. Never let pipeline.judge.model silently fall back to default_model (which is typically small/cheap and downgrades gate fidelity), and never write tester: null without explicit single-turn confirmation. Replaces the earlier draft that let the skill guess a default model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three small onboarding-experience fixes for
wire-assert-ci. All three cover places where the current flow silently fails or silently picks a suboptimal default.Changes
README.md— Option 1 installRequires Node.js ≥ 22note and cross-link to Option 2 as fallbackskillsCLI (skills@1.5.22) importsstyleTextfromnode:util, added in Node 22. On fresh devcontainers running Node 18/20 (Debian bookworm default) users hitSyntaxError: The requested module 'node:util' does not provide an export named 'styleText', which reads as unrelated to Node version.wire-assert-ciskill — new## assert-ai version pinsection (all 3 mirrors)curl -fsSL https://pypi.org/pypi/assert-ai/json | python -c ...and pin the exact version intoassert-ai-version. Users own subsequent bumps.assert-ai(prompt formatting, judge behavior, tokenization) doesn't get attributed to code drift and fire false regressions.wire-assert-ciskill — new### Model selectionsection (all 3 mirrors)azure/<deployment>,openai/gpt-5.4,anthropic/claude-4-sonnet). Verifyprovider-envsecrets match. Write every role explicitly. Never letpipeline.judge.modelsilently fall back todefault_model. Never writetester: nullwithout explicit single-turn confirmation.Skill edits are applied identically across the three mirror files:
SKILL.md(Claude),wire-assert-ci.prompt.md(Copilot),assert-ci.mdc(Cursor).Not included
Earlier iterations of the skill also carried a YAML-emission mitigation (instructing the skill to hand-format multi-line strings with
|block scalars). That's dropped here — the correct fix lives upstream inassert-ai inititself: responsibleai/ASSERT#314. Once that lands and the pinned version rolls forward, behavior YAMLs will be emitted with block scalars and verbatim unicode automatically.Testing
judge.modelin every behavior YAML, writes pinnedassert-ai-versionin the generated workflow.Notes
<LATEST_STABLE_AT_AUTHORING_TIME>in the skill examples with an actual pin when running the skill. That's the point — it's meant to be filled in per authoring.