You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Run only stated or inferred commands unattended and report the unresolved cell - part 1 of 3 #16097
Part 1 of 3 of #15731, split at intake (owner Q93). The done-when below is copied from #15731 as written.
Done when
AC5: an unattended run executes only stated or inferred commands, logs each as assumed with its source, installs nothing, writes no record, and reports a tie as unresolved; every unattended case's verdict is pass in the compare record.
AC4 (second clause): a repository with no record and nothing inferable shows a visible unresolved skip.
The check skill names the --unattended flag and the unresolved cell, and its install allowlist sentence is unchanged.
This records a re-fire of an earlier stop, not a new one. The start gate stopped run 44762156766, the work-resolve-issue run that agent-ready started, before any model turn, because the GITHUB_TOKEN budget was under its start-gate share, or the window was projected to have under its start-project share of the header's limit left by the run's expected end (github-token start-gate, start-project and start-run-minutes in rate-limits.json; the run's start-gate notice names which); that run ended at 2026-11-20T15:22:05Z. work-unblock has now removed and added agent-ready back to start a fresh run. A budget stop is the lanes' load, not a fault in this issue: it never counts toward limits.limit-stops-per-issue or the kill-switch wait.
Pull request #17666 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.
Item (b): inferred-runs on the plugin-eval CI workflow. Do not close yet
I ran (b), the owner's remaining item (issuecomment-7041405051), on the CI route the owner named: run 44803901363. Two of the three unattended cases now pass every grader with the plugin. ck-unattended-inferred-runs lost, and for a reason other than npm: the skill never ran npm test. AC5 is not met, so this issue stays open.
What was run
Workflow: pr-run-plugin-evals.yml, dispatched on main at 8c5f0af897. Its plugins/toolchain, plugins/evals and eval-workflow files have no diff from the commit the run was registered on.
npm did not fail to start. No run with the plugin called npm, so .calls.log was never written (grader: path ".calls.log" does not exist), and logged-assumed was 0/3. The Bash error excerpts show no npm error.
The run's npm-start check passed only because npm was never called. So the question (b) asked, whether npm starts in the CI sandbox, is still unanswered.
In all 3 runs with the plugin, the Overall line lists typescript keys as unresolved, starting typescript.build-cmd, typescript.test-c…. The excerpt is cut at 300 characters.
One run listed its workspace. Beside the fixture's files it held files that appear at run time: package-lock.json, pnpm-lock.yaml, yarn.lock, bunfig.toml, node_modules, .npmrc, .yarnrc.yml. The fixture writes none of these; its own comment says "The workspace holds harness dotfiles" (plugins/toolchain/evals/fixtures/eval-repo.sh:61). The same listing shows .claude/agents, .claude/commands and .claude/.cc-writes, but the scaffold refuses to run when .claude already exists (eval-repo.sh:48-53). So these entries were made after the scaffold, when the session ran. That points to the sandbox's protected-path mount points, which would also leave the lockfiles untracked.
The resolution ladder runs scripts.test "through the package manager its lockfile names, npm with no lockfile" (plugins/toolchain/reference/resolution-ladder.md:119-120). With three lockfiles present, the package manager is a tie. A tie is reported unresolved and nothing runs (:138-140).
Likely cause: the case's workspace, not the skill. This is unverified: the run uploads no traces, so I could not read the reply. The case assumes a package with no lockfile. On this runner, the lockfiles that appear at run time make "unresolved" the skill's correct answer.
The earlier local run (#16097 record of 2026-11-20 09:38Z) did log assumed: typescript.test-cmd=npm test (package.json:5). Its kept workspaces are gone, so whether that run saw the same lockfiles cannot be checked.
Done-when, item by item, on main
AC5: not met. Every graded clause passed in stated and tie: stated or inferred commands only, logged assumed, installs nothing, writes no record, tie unresolved. The last clause, "every unattended case's verdict is pass", fails on inferred-runs.
AC4, second clause: met. "a repository with no record and nothing inferable shows a visible unresolved skip": stated-unresolved's markdown-unresolved passed 3/3 with the plugin and 0/3 without. The rule is at resolution-ladder.md:133-135: "Its cell reads skip (unresolved: <ecosystem>.<key>)".
The check skill names the --unattended flag and the unresolved cell: met.
--unattended is at plugins/toolchain/skills/check/SKILL.md:5 (argument-hint: "[ecosystem] [--unattended]") and :32.
The unresolved cell row is at :164: "skip (unresolved: <ecosystem>.<key>) | environment: an unattended run found no recorded, stated or single inferred command for the key".
Its install allowlist sentence is unchanged: met.SKILL.md:120 on main ("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands: npm ci --ignore-scripts, uv sync --frozen --no-build --no-install-local, dotnet restore --locked-mode. …") is the same text as line 114 at f1cbd8c296, the last commit to that file before this issue was opened (2026-11-14).
Remaining
An owner decision on how inferred-runs gets one unambiguous package manager. Then rerun inferred-runs once on this workflow, which also answers whether npm starts there.
Recommended: the ladder counts only tracked lockfiles (git ls-files) when it picks the package manager. Basis: the .claude entries above show the extra files are made at run time, so they are untracked. This changes the rule for every repository the skill checks.
Alternative: change only the case, for example by committing a package-lock.json in its scaffold. This likely does not help: the untracked pnpm-lock.yaml and yarn.lock would still make a tie under resolution-ladder.md:119-120.
Item (b), re-run after the tracked-lockfile rule: inferred-runs passes
The earlier CI run, 44803901363, lost on inferred-runs. Its workspace held untracked lockfiles that the run created, and they made the package manager a tie. After #17692 (owner Q114 A), the ladder counts a lockfile only when git ls-files lists it (plugins/toolchain/reference/resolution-ladder.md:122-125). I re-ran the case under a new registration: run 44823751095.
Model: claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329.
Fixed before the run:
Pass rule: case score with the plugin at least 0.80, and ran-inferred-test, logged-assumed, no-bundled-typescript and no-install each passing 3 of 3.
No rerun to turn a loss into a pass.
Validity: verdict: VALID (warnings: skill fired). Bash ran in 3 of 3 runs in each arm.
Cost: 0.61 USD (list-price estimate).
Result (runs that passed, out of 3)
Grader
With the plugin
Without
ran-inferred-test
3/3
3/3
logged-assumed
3/3
0/3
build-unresolved
3/3
0/3
no-bundled-typescript
3/3
3/3
no-install
3/3
3/3
case score
1.00 PASS
0.60
npm started inside the CI sandbox. All three runs with the plugin logged assumed: typescript.test-cmd=npm test (package.json:…), which is what logged-assumed matches. The package's test script then wrote typescript test to .calls.log, which is what ran-inferred-test reads.
The Bash error excerpts show no npm error in any of the 6 runs.
AC5 ("an unattended run executes only stated or inferred commands, logs each as assumed with its source, installs nothing, writes no record, and reports a tie as unresolved; every unattended case's verdict is pass in the compare record").
The rules are at resolution-ladder.md:105-125 (sources: recorded, stated, inferred only), :133 ("Log each command as assumed"), :141 ("A tie is unresolved") and :144 ("Install nothing, write nothing").
ck-unattended-inferred-runs: PASS, 1.00, with every grader 3/3 (this run).
ck-unattended-stated-unresolved: PASS, 1.00, with every grader 3/3 (run 44803901363, head 8c5f0af).
ck-unattended-tie-unresolved: PASS, 1.00, with every grader 3/3, under the widened names-both-candidates grader (same run).
ck-unsure-run-is-unattended: 0.83 with the plugin, against 0.75 without, on the 2026-11-20 09:38Z local run (record). That clears the 0.80 case threshold, but markdown-unresolved passed only 1 of 3.
The owner put that fix outside this item (issuecomment-7041405051).
AC4, second clause ("a repository with no record and nothing inferable shows a visible unresolved skip"): met. The rule is at resolution-ladder.md:137: "Its cell reads skip (unresolved: <ecosystem>.<key>)". In the stated case, markdown-unresolved passed 3/3 with the plugin and 0/3 without. In this run, build-unresolved passed 3/3 with the plugin and 0/3 without.
"The check skill names the --unattended flag and the unresolved cell, and its install allowlist sentence is unchanged": met.
The flag: plugins/toolchain/skills/check/SKILL.md:5 reads argument-hint: "[ecosystem] [--unattended]", and :32 reads "--unattended marks the run unattended; …".
The unresolved cell: :164 reads "skip (unresolved: <ecosystem>.<key>) | environment: an unattended run found no recorded, stated or single inferred command for the key".
The install allowlist sentence, at :120 ("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands: npm ci --ignore-scripts, uv sync --frozen --no-build --no-install-local, dotnet restore --locked-mode. …"), is word for word the same as line 114 at f1cbd8c296. That commit is the last change to the file before this issue was opened (2026-11-14).
ck-unsure-run-is-unattended: passes on a second CI run; closing
Run 44826315318 ran on main at 6b6cde4f01, which contains the rung-2 fix #17641 (c9eec4baeb). Model claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329, cost 0.65 USD (list-price estimate). The inputs are the same as run 44824924921.
With the plugin the case scores 1.00, and every grader passes 3 of 3: markdown-unresolved, no-bundled-markdownlint, no-record-write, no-write. Without the plugin it scores 0.75, and markdown-unresolved passes 0 of 3.
Both runs agree. 44824924921 also scored 1.00 against 0.75.
Validity: INVALID (2 permission denials in the traces: Bash ls ~/.claude/ecosystems …, Read ~/work/claude-code-plugins/claude-code-plugins/.base/plugins/toolchain/context/bash.md). The run before had the same Read denial. No grader of this case reads either call, so under the owner's counted-denial decision (a denied call no grader reads is counted, with the calls listed) the run counts.
Done-when, all met on main:
AC5, "every unattended case's verdict is pass":
inferred-runs: issuecomment-7051601811
stated-unresolved and tie-unresolved: issuecomment-7049175450
unsure: this run
AC4, second clause; the --unattended flag and the unresolved cell; the unchanged install allowlist sentence: as recorded in issuecomment-7051601811.
Activity
36 remaining items
melodic-automation-lanes commentedon Nov 20, 2026
This records a re-fire of an earlier stop, not a new one. The start gate stopped run 44762156766, the work-resolve-issue run that
agent-readystarted, before any model turn, because the GITHUB_TOKEN budget was under its start-gate share, or the window was projected to have under its start-project share of the header's limit left by the run's expected end (github-token start-gate, start-project and start-run-minutes in rate-limits.json; the run's start-gate notice names which); that run ended at 2026-11-20T15:22:05Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A budget stop is the lanes' load, not a fault in this issue: it never counts toward limits.limit-stops-per-issue or thekill-switchwait.github-actions commentedon Nov 20, 2026
github-actions commentedon Nov 20, 2026
melodic-automation-lanes commentedon Nov 20, 2026
Pull request #17666 merged with
Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removedstatus: readyandagent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it atstatus: confirmedfor a check of what that pull request left, until a**Decision recorded.**comment says what is left or someone closes it.kyle-sexton commentedon Nov 20, 2026
Item (b): inferred-runs on the plugin-eval CI workflow. Do not close yet
I ran (b), the owner's remaining item (issuecomment-7041405051), on the CI route the owner named: run 44803901363. Two of the three unattended cases now pass every grader with the plugin. ck-unattended-inferred-runs lost, and for a reason other than npm: the skill never ran
npm test. AC5 is not met, so this issue stays open.What was run
pr-run-plugin-evals.yml, dispatched on main at8c5f0af897. Itsplugins/toolchain,plugins/evalsand eval-workflow files have no diff from the commit the run was registered on.target=plugins/toolchain,case=ck-unattended-*,runs=3,allow-tools=Bash,env-scrub=true,debug-errors=true,max-cost-usd=5.verdict: VALID (warnings: skill fired). Cost: 1.81 USD (list-price estimate).--threshold, so the default 1.00 applies. The verdicts below follow the registered rule, not the job colour.Per case (runs that passed, out of 3, with the plugin / without it)
Why inferred-runs lost
.calls.logwas never written (grader:path ".calls.log" does not exist), and logged-assumed was 0/3. The Bash error excerpts show no npm error.typescript.build-cmd, typescript.test-c…. The excerpt is cut at 300 characters.package-lock.json,pnpm-lock.yaml,yarn.lock,bunfig.toml,node_modules,.npmrc,.yarnrc.yml. The fixture writes none of these; its own comment says "The workspace holds harness dotfiles" (plugins/toolchain/evals/fixtures/eval-repo.sh:61). The same listing shows.claude/agents,.claude/commandsand.claude/.cc-writes, but the scaffold refuses to run when.claudealready exists (eval-repo.sh:48-53). So these entries were made after the scaffold, when the session ran. That points to the sandbox's protected-path mount points, which would also leave the lockfiles untracked.scripts.test"through the package manager its lockfile names,npmwith no lockfile" (plugins/toolchain/reference/resolution-ladder.md:119-120). With three lockfiles present, the package manager is a tie. A tie is reported unresolved and nothing runs (:138-140).assumed: typescript.test-cmd=npm test (package.json:5). Its kept workspaces are gone, so whether that run saw the same lockfiles cannot be checked.Done-when, item by item, on main
markdown-unresolvedpassed 3/3 with the plugin and 0/3 without. The rule is atresolution-ladder.md:133-135: "Its cell readsskip (unresolved: <ecosystem>.<key>)".--unattendedflag and the unresolved cell: met.--unattendedis atplugins/toolchain/skills/check/SKILL.md:5(argument-hint: "[ecosystem] [--unattended]") and:32.:164: "skip (unresolved: <ecosystem>.<key>)| environment: an unattended run found no recorded, stated or single inferred command for the key".SKILL.md:120on main ("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands:npm ci --ignore-scripts,uv sync --frozen --no-build --no-install-local,dotnet restore --locked-mode. …") is the same text as line 114 atf1cbd8c296, the last commit to that file before this issue was opened (2026-11-14).Remaining
git ls-files) when it picks the package manager. Basis: the.claudeentries above show the extra files are made at run time, so they are untracked. This changes the rule for every repository the skill checks.package-lock.jsonin its scaffold. This likely does not help: the untrackedpnpm-lock.yamlandyarn.lockwould still make a tie underresolution-ladder.md:119-120.kyle-sexton commentedon Nov 20, 2026
Item (b), re-run after the tracked-lockfile rule: inferred-runs passes
The earlier CI run, 44803901363, lost on inferred-runs. Its workspace held untracked lockfiles that the run created, and they made the package manager a tie. After #17692 (owner Q114 A), the ladder counts a lockfile only when
git ls-fileslists it (plugins/toolchain/reference/resolution-ladder.md:122-125). I re-ran the case under a new registration: run 44823751095.What was run
pr-run-plugin-evals.yml, dispatched on main atf67ad7ec11, the fix(toolchain): count only tracked lockfiles when the ladder picks a package manager #17692 merge.case=ck-unattended-inferred-runs,runs=3,allow-tools=Bash,env-scrub=true,debug-errors=true,max-cost-usd=5.verdict: VALID (warnings: skill fired). Bash ran in 3 of 3 runs in each arm.Result (runs that passed, out of 3)
assumed: typescript.test-cmd=npm test (package.json:…), which is what logged-assumed matches. The package's test script then wrotetypescript testto.calls.log, which is what ran-inferred-test reads.Done-when, item by item, on main (
be062c23f2)resolution-ladder.md:105-125(sources: recorded, stated, inferred only),:133("Log each command as assumed"),:141("A tie is unresolved") and:144("Install nothing, write nothing").resolution-ladder.md:137: "Its cell readsskip (unresolved: <ecosystem>.<key>)". In the stated case, markdown-unresolved passed 3/3 with the plugin and 0/3 without. In this run, build-unresolved passed 3/3 with the plugin and 0/3 without.--unattendedflag and the unresolved cell, and its install allowlist sentence is unchanged": met.plugins/toolchain/skills/check/SKILL.md:5readsargument-hint: "[ecosystem] [--unattended]", and:32reads "--unattendedmarks the run unattended; …".:164reads "skip (unresolved: <ecosystem>.<key>)| environment: an unattended run found no recorded, stated or single inferred command for the key".:120("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands:npm ci --ignore-scripts,uv sync --frozen --no-build --no-install-local,dotnet restore --locked-mode. …"), is word for word the same as line 114 atf1cbd8c296. That commit is the last change to the file before this issue was opened (2026-11-14).kyle-sexton commentedon Nov 20, 2026
ck-unsure-run-is-unattended: passes on a second CI run; closing
Run 44826315318 ran on main at
6b6cde4f01, which contains the rung-2 fix #17641 (c9eec4baeb). Model claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329, cost 0.65 USD (list-price estimate). The inputs are the same as run 44824924921.INVALID (2 permission denials in the traces: Bash ls ~/.claude/ecosystems …, Read ~/work/claude-code-plugins/claude-code-plugins/.base/plugins/toolchain/context/bash.md). The run before had the same Read denial. No grader of this case reads either call, so under the owner's counted-denial decision (a denied call no grader reads is counted, with the calls listed) the run counts.Done-when, all met on main:
--unattendedflag and the unresolved cell; the unchanged install allowlist sentence: as recorded in issuecomment-7051601811.Closing as completed.