Skip to content

docs+skill: onboarding UX (Node 22 prereq, version pin, explicit model selection) - #5

Open
tangym wants to merge 1 commit into
mainfrom
tangym/skill-onboarding-updates
Open

docs+skill: onboarding UX (Node 22 prereq, version pin, explicit model selection)#5
tangym wants to merge 1 commit into
mainfrom
tangym/skill-onboarding-updates

Conversation

@tangym

@tangym tangym commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Three small onboarding-experience fixes for wire-assert-ci. All three cover places where the current flow silently fails or silently picks a suboptimal default.

Changes

Area What changed Why
README.md — Option 1 install Add Requires Node.js ≥ 22 note and cross-link to Option 2 as fallback The skills CLI (skills@1.5.22) imports styleText from node:util, added in Node 22. On fresh devcontainers running Node 18/20 (Debian bookworm default) users hit SyntaxError: The requested module 'node:util' does not provide an export named 'styleText', which reads as unrelated to Node version.
wire-assert-ci skill — new ## assert-ai version pin section (all 3 mirrors) At workflow-authoring time, resolve latest stable from PyPI via curl -fsSL https://pypi.org/pypi/assert-ai/json | python -c ... and pin the exact version into assert-ai-version. Users own subsequent bumps. The action's gate uses paired-binary McNemar across baseline and PR runs. Pinning is required so drift in assert-ai (prompt formatting, judge behavior, tokenization) doesn't get attributed to code drift and fire false regressions.
wire-assert-ci skill — new ### Model selection section (all 3 mirrors) Skill must ask the user for judge / tester / default roles in one prompt before writing any behavior YAML. Use LiteLLM-style names (azure/<deployment>, openai/gpt-5.4, anthropic/claude-4-sonnet). Verify provider-env secrets match. Write every role explicitly. Never let pipeline.judge.model silently fall back to default_model. Never write tester: null without explicit single-turn confirmation. Available models are user-specific (Azure deployment names, provider access). Guessing a default silently downgrades the paid-provider gate to whatever small model happens to be the fallback.

Skill edits are applied identically across the three mirror files: SKILL.md (Claude), wire-assert-ci.prompt.md (Copilot), assert-ci.mdc (Cursor).

Not included

Earlier iterations of the skill also carried a YAML-emission mitigation (instructing the skill to hand-format multi-line strings with | block scalars). That's dropped here — the correct fix lives upstream in assert-ai init itself: responsibleai/ASSERT#314. Once that lands and the pinned version rolls forward, behavior YAMLs will be emitted with block scalars and verbatim unicode automatically.

Testing

  • Ran the updated Option 1 instructions on a fresh Node-18 devcontainer: hit the styleText error as expected, then Node-22 install path worked.
  • Walked the skill end-to-end via Copilot chat against the demo repo: asks for model choices before writing behaviors, writes explicit judge.model in every behavior YAML, writes pinned assert-ai-version in the generated workflow.

Notes

  • Assumes the reviewer replaces <LATEST_STABLE_AT_AUTHORING_TIME> in the skill examples with an actual pin when running the skill. That's the point — it's meant to be filled in per authoring.

…l selection)

Three small onboarding-experience improvements grouped together because
they all fix places where the current flow silently fails or silently
picks a suboptimal default:

README:
- Option 1 (npx skills add ...) now states the Node.js >= 22 prerequisite
  up front and points users to Option 2 as the no-Node fallback. The
  underlying skills CLI imports styleText from node:util which is
  Node-22-only; running under Node 18/20 fails with a SyntaxError that
  looks unrelated to Node.

wire-assert-ci skill (SKILL.md + prompt + mdc mirrors):

- 'assert-ai version pin' section: at workflow-authoring time (not every
  CI run), resolve the latest stable assert-ai from PyPI and write that
  exact version into assert-ai-version. Pinning is required so the
  paired-binary McNemar test compares like against like across baseline
  and PR runs; a floating install would attribute version drift to code
  drift and produce false regressions. Users still own subsequent bumps.

- 'Model selection' section: available models are user-specific (Azure
  deployment names, provider access), so the skill must ask the user
  before generating any behavior YAML. Ask for three roles in one prompt
  (judge, tester, default), use LiteLLM-style names, verify provider-env
  secrets match the chosen provider, and write every role explicitly.
  Never let pipeline.judge.model silently fall back to default_model
  (which is typically small/cheap and downgrades gate fidelity), and
  never write tester: null without explicit single-turn confirmation.
  Replaces the earlier draft that let the skill guess a default model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant