Repository navigation
Description
Parent
Refs #15703
Source: #16029, part 2 of #15729. The list recorded in #16602 gives /testing:setup the route "call /corrections:codify, presence-gated, with the strongest-first fallback".
Depends on
- none open. The testing cut (Cut the model-read ecosystem specifics from testing #16755, no pull request yet) lists no row in
skills/setup/SKILL.md; its setup-adjacent row (D22) is in the plugin manifest, schema and resolver.
Outcome
/testing:setup check reports a fixed, script-backed set of test-lint rules per language and, for a language it has no rule for, prints "setup has no lint check for this language"; it also offers an instruction line to paste. Those two outputs are the skill's own choice of mechanism for a can't-fail test pattern: an installed analyzer rule, or a line in an instruction file. After this change, a detected language with no shipped rule, and the instruction-line offer, name /corrections:codify for the mechanism when it is available, with the shared strongest-first fallback when it is not. The script-backed rule rows and the config cascade do not change, and setup still edits no instruction file or lint config.
Demo: with a corrections stand-in plugin loaded, run /testing:setup check in a repository whose tests are in a language setup has no lint row for. The report names that language and names /corrections:codify for the mechanism, instead of only "setup has no lint check for this language".
Where the text changes
Line numbers as read on main at a483a2c1; they may move. Owner decision of 2026-11-18 on #16029 (option A): the codify call covers only the choice setup makes beyond its fixed deliverable, which stays as it is.
| File and lines | Present wording | Change |
|---|---|---|
plugins/testing/skills/setup/SKILL.md:103-104 (check, section 2 "lint") |
"Any other detected language: named with setup has no lint check for this language, not a finding." |
Keep "not a finding"; add the gated codify call plus the shared fallback for a can't-fail pattern in that language |
same file, :105-108 and :139-143 (section 3 "instruction", apply step 4) |
"the optional line, naming one file Claude Code loads at the repository root ... Offer it; do not paste it anywhere." | The offer says the line is a pointer to the team file, and a rule a check could decide goes to /corrections:codify instead of the line |
Call form: /corrections:codify "<the test pattern>". Setup is user-invoked (disable-model-invocation: true); it can still name or invoke codify, which is model-invocable.
Decisions
-
Design Q21 (Resolve ecosystem specifics from the consumer repository and align the enforcement ladder with the talk's pyramid #15703): a skill that carries no ladder copy reads the ladder by calling codify; only a skill that applies its own selection rule holds a synced copy and a Selection rules row.
-
docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602 recorded this skill under "call
/corrections:codify, presence-gated, with the strongest-first fallback (12; change, filed as follow-on items by part 2, List enforcement-placing skills and give each one route to the ladder - part 2 of 2 #16029)", and placed every other:setupskill outside the scope test. -
The fallback wording is shared by all nine follow-ons of List enforcement-placing skills and give each one route to the ladder - part 2 of 2 #16029. After the gate sentence ("when
/corrections:codifyis among the available skills, hand it the choice of mechanism; it picks the strongest rung of the enforcement ladder that can decide the rule and proves the check"), the fallback is exactly:Otherwise, propose the strongest mechanism that can decide the rule: a change that makes the mistake impossible first, then a check that blocks it, then a test, and an instruction only when nothing a machine runs can decide it. Say that
/corrections:codifywas not available, so the proposal is unproven.
Key interfaces
- "A skill that carries no ladder copy calls
/corrections:codify, presence-gated, with a two-sentence strongest-first fallback. Only a skill that applies its own selection rule holds a synced copy and gets a row in the ladder's Selection rules table." (List enforcement-placing skills and give each one route to the ladder #15729) - The seam-phrasing convention: an availability gate, a stated fallback beside it, the plugin named and never the marketplace.
- ADR 0101: the fallback names no tool; the script-backed rule rows are an executed adapter and stay.
Done when
- The "any other detected language" row and the instruction-line offer hand the mechanism choice to
/corrections:codifybehind the availability gate, followed by the shared fallback. -
setup.sh, its rule rows, exit codes and the "does not do" list are unchanged. - The skill gains no ladder copy or Selection rules row, and no rung list is restated.
- The pull request body quotes the sentence that picked a mechanism from a private list before the change. If the recheck finds none, the skill is left unchanged and the finding is reported on List enforcement-placing skills and give each one route to the ladder - part 2 of 2 #16029.
- One new eval case loads a corrections stand-in plugin, in the shape of session-flow's
re-mechanical-finding-check-proposalcase, and grades that a language with no shipped rule is named with/corrections:codifyfor the mechanism and that setup writes no file duringcheck. - The pull request body names the existing case whose run reads the changed text (candidates:
setup-check-reports-missing-lint-rule,setup-claude-md-line-printed), or says none does. The lane runs no eval. - A changelog fragment under
.changes/testing/describes the change. - AC10: no new enforcement-placing skill is added here.
- Open the pull request with
Closesthis issue in its body, then stop: the lane adds no label.
After the pull request opens (host session, not the lane)
The pull request stays held with do-not-merge for the eval; a host session runs only the eval and posts its record; the merge lane releases the hold when the record matches the head.
- Hold line: the hub's sweep (or a host session) adds
do-not-mergeand posts, from a listed human's account, a comment with the line<!-- pr-hold reason=eval -->on a line of its own. - Run budget. List enforcement-placing skills and give each one route to the ladder #15729 asks for "that skill's cases on the
opusandsonnetaliases, three runs per side, and no more than 240 runs per plugin". The owner decision of 2026-11-16 in the owner-decisions record (hop 4 Q15, which names List enforcement-placing skills and give each one route to the ladder #15729, amended by hop 7 Q3) replaces the run counts: per changed skill, one targeted before-and-after case onsonnetat 3 runs a side;opusruns only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first. Select cases with--case, use--runs 3and a--max-cost-usdceiling, and start no batch after a usage-limit error. Ceiling for this plugin: 1 skill x (3 smoke + 6 sonnet + 6 opus confirm) = 15 runs, under the parent's 240. - Control: the replaced wording is a partial cut, so the pull request body carries a detection-control result. No paired no-loss compare (#15703 comment 6906800940).
- On a LOSS reading: a trace read of each dropped case and an
opusconfirm; only a loss traced to the changed text counts. - Record: the verdict and the control result go into the pull request body. When the run is VALID and shows no loss, the host session posts, from a listed human's account, a comment with the line
<!-- eval-pair verdict=no-loss validity=VALID head=<40-hex head SHA> -->naming the current head, keeping "do not merge" anddo-not-mergeout of that comment (docs/contracts/loop-lane/README.md, "Cross-lane PR hold"). A loss or a run that is not VALID leaves the hold in place; say so on the pull request. - Release: pr-merge's scheduled step "Release eval holds whose record matches the head" removes
do-not-mergeonce that record names the current head and no other hold stands. A push after the record needs a new record.
Out of scope
- The shipped per-language rule rows,
setup.sh, and thetest-weakenhook. - The testing ecosystem cut (Cut the model-read ecosystem specifics from testing #16755).
Related
- Resolve ecosystem specifics from the consumer repository and align the enforcement ladder with the talk's pyramid #15703 (container)
- List enforcement-placing skills and give each one route to the ladder #15729
- List enforcement-placing skills and give each one route to the ladder - part 2 of 2 #16029
- docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Priority
Activity
github-actions commentedon Nov 18, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Blast radius: the testing plugin's setup skill text (the "any other detected language" row and the instruction-line offer), one new eval case and a changelog fragment; people who run
/testing:setup checkread it.Reversible: yes; revert the skill text, eval case and fragment.
Open questions: none
Understood: when
/testing:setup checkmeets a language with no shipped lint rule, or offers the instruction line, it should hand the choice of mechanism to/corrections:codifywhen that skill is available. When it is not available, setup gives the shared strongest-first fallback.setup.sh, the rule rows and the config cascade stay unchanged, and setup still edits no files.Done-when items 4 and 5 (the PR-body quote and the new eval case) need a host run. They do not hold this issue. I found no open duplicate. #17158 and #17159 are sibling follow-ons for other plugins, not duplicates. I found no merged PR that already makes this change.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 18, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 4 (tracker-output: "The pull request body quotes the sentence that picked a mechanism from a priv..."), item 5 (eval: "One new eval case loads a corrections stand-in plugin, in the shape of sessio..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 18, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: The setup skill's "any other detected language" row and instruction-line offer hand the can't-fail test pattern's mechanism choice to /corrections:codify behind an availability gate with the shared strongest-first fallback, with setup.sh and the rule rows unchanged.
Split. Done-when items 4, 5 of the issue body need a person: they are checks a person runs, listed below, and the lane does not do them. This issue keeps the rest of its done-when.
Not done by the lane: checks a person runs
External state: none
Dependencies
Related work
Sources read
Automated refinement by the lanes App. Intake triages the issue again with it.
melodic-automation-lanes commentedon Nov 18, 2026
{"v":1,"actor_id":337595936,"title_sha256":"a38598cdccca830fefa5b958bd977dee8c283a77fe68ca7f54f3eab1bab974fd","body_sha256":"bae7b1d8eea1b4bed227b5ae2cb58fd3002550e65f9eb1ed03f079cfc8b9a67e"}
melodic-automation-lanes commentedon Nov 18, 2026
{"v":1,"actor_id":337595936,"title_sha256":"a38598cdccca830fefa5b958bd977dee8c283a77fe68ca7f54f3eab1bab974fd","body_sha256":"bae7b1d8eea1b4bed227b5ae2cb58fd3002550e65f9eb1ed03f079cfc8b9a67e"}
github-actions commentedon Nov 18, 2026
melodic-automation-lanes commentedon Nov 19, 2026
This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 44324786543, the work-resolve-issue run that
agent-readystarted, before it finished; that run ended at 2026-11-18T15:37:14Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits the same limits.stranded-backoff-hourskill-switchhours after each refused run before re-firing it, never longer.melodic-automation-lanes commentedon Nov 19, 2026
This records a re-fire of an earlier stop, not a new one. The start gate stopped run 44553288839, the work-resolve-issue run that
agent-readystarted, before any model turn, because the GITHUB_TOKEN budget was under its start-gate share, or the window was projected to have under its start-project share of the header's limit left by the run's expected end (github-token start-gate, start-project and start-run-minutes in rate-limits.json; the run's start-gate notice names which); that run ended at 2026-11-19T15:20:19Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A budget stop is the lanes' load, not a fault in this issue: it never counts toward limits.limit-stops-per-issue or thekill-switchwait.github-actions commentedon Nov 19, 2026
feat(testing): name corrections:codify on setup's no-maintained-rule …
test(testing): add the setup check codify hand-off case for PowerShell