Skip to content

Run only stated or inferred commands unattended and report the unresolved cell - part 1 of 3 #16097

Closed

Description

@melodic-automation-lanes

Part 1 of 3 of #15731, split at intake (owner Q93). The done-when below is copied from #15731 as written.

Done when

  • AC5: an unattended run executes only stated or inferred commands, logs each as assumed with its source, installs nothing, writes no record, and reports a tie as unresolved; every unattended case's verdict is pass in the compare record.
  • AC4 (second clause): a repository with no record and nothing inferable shows a visible unresolved skip.
  • The check skill names the --unattended flag and the unresolved cell, and its install allowlist sentence is unchanged.

Activity

  1. 36 remaining items

  2. melodic-automation-lanes commented on Nov 20, 2026

    @melodic-automation-lanes
    ContributorAuthor

    This records a re-fire of an earlier stop, not a new one. The start gate stopped run 44762156766, the work-resolve-issue run that agent-ready started, before any model turn, because the GITHUB_TOKEN budget was under its start-gate share, or the window was projected to have under its start-project share of the header's limit left by the run's expected end (github-token start-gate, start-project and start-run-minutes in rate-limits.json; the run's start-gate notice names which); that run ended at 2026-11-20T15:22:05Z. work-unblock has now removed and added agent-ready back to start a fresh run. A budget stop is the lanes' load, not a fault in this issue: it never counts toward limits.limit-stops-per-issue or the kill-switch wait.

  3. github-actions commented on Nov 20, 2026

    @github-actions
    Contributor
    No description provided.
  4. github-actions commented on Nov 20, 2026

    @github-actions
    Contributor
    No description provided.
  5. melodic-automation-lanes commented on Nov 20, 2026

    @melodic-automation-lanes
    ContributorAuthor

    Pull request #17666 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  6. kyle-sexton commented on Nov 20, 2026

    @kyle-sexton
    Member

    Item (b): inferred-runs on the plugin-eval CI workflow. Do not close yet

    I ran (b), the owner's remaining item (issuecomment-7041405051), on the CI route the owner named: run 44803901363. Two of the three unattended cases now pass every grader with the plugin. ck-unattended-inferred-runs lost, and for a reason other than npm: the skill never ran npm test. AC5 is not met, so this issue stays open.

    What was run

    • Workflow: pr-run-plugin-evals.yml, dispatched on main at 8c5f0af897. Its plugins/toolchain, plugins/evals and eval-workflow files have no diff from the commit the run was registered on.
    • Inputs: target=plugins/toolchain, case=ck-unattended-*, runs=3, allow-tools=Bash, env-scrub=true, debug-errors=true, max-cost-usd=5.
      • The glob matches three cases: inferred-runs, stated-unresolved and tie-unresolved.
      • Tie-unresolved is included because the #17666 re-grade counts only after this run.
      • ck-unsure-run-is-unattended was not run, because its fix is #17636.
    • Model: claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329.
    • Fixed before the run: the pass rule (with-plugin case score at least 0.80, and every AC5-clause grader passing 3 of 3), and that a loss is not rerun.
    • Validity: verdict: VALID (warnings: skill fired). Cost: 1.81 USD (list-price estimate).
    • The job is red because CI passes no --threshold, so the default 1.00 applies. The verdicts below follow the registered rule, not the job colour.

    Per case (runs that passed, out of 3, with the plugin / without it)

    Case Grader With Without
    stated-unresolved ran-stated-check, logged-assumed, markdown-unresolved, overall-incomplete 3/3 each 1/3, 0/3, 0/3, 0/3
    no-bundled-markdownlint, no-record-write, no-edit, no-write, no-install 3/3 each 3/3 each
    case score / verdict 1.00 PASS 0.59
    tie-unresolved test-unresolved, names-both-candidates, neither-ran, overall-incomplete 3/3 each 0/3, 0/3, 2/3, 0/3
    case score / verdict 1.00 PASS 0.17
    inferred-runs ran-inferred-test 0/3 2/3
    logged-assumed 0/3 0/3
    build-unresolved 3/3 0/3
    no-bundled-typescript, no-install 3/3 each 3/3 each
    case score / verdict 0.60 LOSS 0.53

    Why inferred-runs lost

    • npm did not fail to start. No run with the plugin called npm, so .calls.log was never written (grader: path ".calls.log" does not exist), and logged-assumed was 0/3. The Bash error excerpts show no npm error.
    • The run's npm-start check passed only because npm was never called. So the question (b) asked, whether npm starts in the CI sandbox, is still unanswered.
    • In all 3 runs with the plugin, the Overall line lists typescript keys as unresolved, starting typescript.build-cmd, typescript.test-c…. The excerpt is cut at 300 characters.
    • One run listed its workspace. Beside the fixture's files it held files that appear at run time: package-lock.json, pnpm-lock.yaml, yarn.lock, bunfig.toml, node_modules, .npmrc, .yarnrc.yml. The fixture writes none of these; its own comment says "The workspace holds harness dotfiles" (plugins/toolchain/evals/fixtures/eval-repo.sh:61). The same listing shows .claude/agents, .claude/commands and .claude/.cc-writes, but the scaffold refuses to run when .claude already exists (eval-repo.sh:48-53). So these entries were made after the scaffold, when the session ran. That points to the sandbox's protected-path mount points, which would also leave the lockfiles untracked.
    • The resolution ladder runs scripts.test "through the package manager its lockfile names, npm with no lockfile" (plugins/toolchain/reference/resolution-ladder.md:119-120). With three lockfiles present, the package manager is a tie. A tie is reported unresolved and nothing runs (:138-140).
    • Likely cause: the case's workspace, not the skill. This is unverified: the run uploads no traces, so I could not read the reply. The case assumes a package with no lockfile. On this runner, the lockfiles that appear at run time make "unresolved" the skill's correct answer.
    • The earlier local run (#16097 record of 2026-11-20 09:38Z) did log assumed: typescript.test-cmd=npm test (package.json:5). Its kept workspaces are gone, so whether that run saw the same lockfiles cannot be checked.

    Done-when, item by item, on main

    • AC5: not met. Every graded clause passed in stated and tie: stated or inferred commands only, logged assumed, installs nothing, writes no record, tie unresolved. The last clause, "every unattended case's verdict is pass", fails on inferred-runs.
    • AC4, second clause: met. "a repository with no record and nothing inferable shows a visible unresolved skip": stated-unresolved's markdown-unresolved passed 3/3 with the plugin and 0/3 without. The rule is at resolution-ladder.md:133-135: "Its cell reads skip (unresolved: <ecosystem>.<key>)".
    • The check skill names the --unattended flag and the unresolved cell: met.
      • --unattended is at plugins/toolchain/skills/check/SKILL.md:5 (argument-hint: "[ecosystem] [--unattended]") and :32.
      • The unresolved cell row is at :164: "skip (unresolved: <ecosystem>.<key>) | environment: an unattended run found no recorded, stated or single inferred command for the key".
    • Its install allowlist sentence is unchanged: met. SKILL.md:120 on main ("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands: npm ci --ignore-scripts, uv sync --frozen --no-build --no-install-local, dotnet restore --locked-mode. …") is the same text as line 114 at f1cbd8c296, the last commit to that file before this issue was opened (2026-11-14).

    Remaining

    • An owner decision on how inferred-runs gets one unambiguous package manager. Then rerun inferred-runs once on this workflow, which also answers whether npm starts there.
      • Recommended: the ladder counts only tracked lockfiles (git ls-files) when it picks the package manager. Basis: the .claude entries above show the extra files are made at run time, so they are untracked. This changes the rule for every repository the skill checks.
      • Alternative: change only the case, for example by committing a package-lock.json in its scaffold. This likely does not help: the untracked pnpm-lock.yaml and yarn.lock would still make a tie under resolution-ladder.md:119-120.
  7. kyle-sexton commented on Nov 20, 2026

    @kyle-sexton
    Member

    Item (b), re-run after the tracked-lockfile rule: inferred-runs passes

    The earlier CI run, 44803901363, lost on inferred-runs. Its workspace held untracked lockfiles that the run created, and they made the package manager a tie. After #17692 (owner Q114 A), the ladder counts a lockfile only when git ls-files lists it (plugins/toolchain/reference/resolution-ladder.md:122-125). I re-ran the case under a new registration: run 44823751095.

    What was run

    • Workflow: pr-run-plugin-evals.yml, dispatched on main at f67ad7ec11, the fix(toolchain): count only tracked lockfiles when the ladder picks a package manager #17692 merge.
    • Inputs: case=ck-unattended-inferred-runs, runs=3, allow-tools=Bash, env-scrub=true, debug-errors=true, max-cost-usd=5.
    • Model: claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329.
    • Fixed before the run:
      • Pass rule: case score with the plugin at least 0.80, and ran-inferred-test, logged-assumed, no-bundled-typescript and no-install each passing 3 of 3.
      • No rerun to turn a loss into a pass.
    • Validity: verdict: VALID (warnings: skill fired). Bash ran in 3 of 3 runs in each arm.
    • Cost: 0.61 USD (list-price estimate).

    Result (runs that passed, out of 3)

    Grader With the plugin Without
    ran-inferred-test 3/3 3/3
    logged-assumed 3/3 0/3
    build-unresolved 3/3 0/3
    no-bundled-typescript 3/3 3/3
    no-install 3/3 3/3
    case score 1.00 PASS 0.60
    • npm started inside the CI sandbox. All three runs with the plugin logged assumed: typescript.test-cmd=npm test (package.json:…), which is what logged-assumed matches. The package's test script then wrote typescript test to .calls.log, which is what ran-inferred-test reads.
    • The Bash error excerpts show no npm error in any of the 6 runs.
    • (b) is met.

    Done-when, item by item, on main (be062c23f2)

    • AC5 ("an unattended run executes only stated or inferred commands, logs each as assumed with its source, installs nothing, writes no record, and reports a tie as unresolved; every unattended case's verdict is pass in the compare record").
      • The rules are at resolution-ladder.md:105-125 (sources: recorded, stated, inferred only), :133 ("Log each command as assumed"), :141 ("A tie is unresolved") and :144 ("Install nothing, write nothing").
      • ck-unattended-inferred-runs: PASS, 1.00, with every grader 3/3 (this run).
      • ck-unattended-stated-unresolved: PASS, 1.00, with every grader 3/3 (run 44803901363, head 8c5f0af).
      • ck-unattended-tie-unresolved: PASS, 1.00, with every grader 3/3, under the widened names-both-candidates grader (same run).
      • About the two cases above: they ran before fix(toolchain): count only tracked lockfiles when the ladder picks a package manager #17692. That change only narrows which lockfiles count. Neither case's graders read a lockfile choice: stated runs a stated command, and tie is package.json against Makefile.
      • ck-unsure-run-is-unattended: 0.83 with the plugin, against 0.75 without, on the 2026-11-20 09:38Z local run (record). That clears the 0.80 case threshold, but markdown-unresolved passed only 1 of 3.
    • AC4, second clause ("a repository with no record and nothing inferable shows a visible unresolved skip"): met. The rule is at resolution-ladder.md:137: "Its cell reads skip (unresolved: <ecosystem>.<key>)". In the stated case, markdown-unresolved passed 3/3 with the plugin and 0/3 without. In this run, build-unresolved passed 3/3 with the plugin and 0/3 without.
    • "The check skill names the --unattended flag and the unresolved cell, and its install allowlist sentence is unchanged": met.
      • The flag: plugins/toolchain/skills/check/SKILL.md:5 reads argument-hint: "[ecosystem] [--unattended]", and :32 reads "--unattended marks the run unattended; …".
      • The unresolved cell: :164 reads "skip (unresolved: <ecosystem>.<key>) | environment: an unattended run found no recorded, stated or single inferred command for the key".
      • The install allowlist sentence, at :120 ("Install only from the committed lockfile, with install scripts disabled, and only with one of these commands: npm ci --ignore-scripts, uv sync --frozen --no-build --no-install-local, dotnet restore --locked-mode. …"), is word for word the same as line 114 at f1cbd8c296. That commit is the last change to the file before this issue was opened (2026-11-14).
  8. kyle-sexton commented on Nov 20, 2026

    @kyle-sexton
    Member

    ck-unsure-run-is-unattended: passes on a second CI run; closing

    Run 44826315318 ran on main at 6b6cde4f01, which contains the rung-2 fix #17641 (c9eec4baeb). Model claude-sonnet-5-5, 3 runs per arm, Claude Code 2.1.329, cost 0.65 USD (list-price estimate). The inputs are the same as run 44824924921.

    • With the plugin the case scores 1.00, and every grader passes 3 of 3: markdown-unresolved, no-bundled-markdownlint, no-record-write, no-write. Without the plugin it scores 0.75, and markdown-unresolved passes 0 of 3.
    • Both runs agree. 44824924921 also scored 1.00 against 0.75.
    • Validity: INVALID (2 permission denials in the traces: Bash ls ~/.claude/ecosystems …, Read ~/work/claude-code-plugins/claude-code-plugins/.base/plugins/toolchain/context/bash.md). The run before had the same Read denial. No grader of this case reads either call, so under the owner's counted-denial decision (a denied call no grader reads is counted, with the calls listed) the run counts.

    Done-when, all met on main:

    • AC5, "every unattended case's verdict is pass":
      • inferred-runs: issuecomment-7051601811
      • stated-unresolved and tie-unresolved: issuecomment-7049175450
      • unsure: this run
    • AC4, second clause; the --unattended flag and the unresolved cell; the unchanged install allowlist sentence: as recorded in issuecomment-7051601811.

    Closing as completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions