fix(cli): exit 1 when a denied sandbox boundary is never retried - #4506
fix(cli): exit 1 when a denied sandbox boundary is never retried#4506ggbdpq wants to merge 2 commits into
Conversation
820d8d9 to
31492a9
Compare
jackwener
left a comment
jackwener
left a comment
There was a problem hiding this comment.
[P1] Correlate the retry itself, not just the tool name
packages/cli/src/runtime-host-run-command.ts:644 treats any later successful call with the same toolName as proof that the blocked call recovered. A tool name does not identify an invocation or its target. For example, a denied Read of /outside/secret.txt followed in a later step by a successful Read of /workspace/README.md still clears the unresolved boundary failure. run-command-core.ts then removes the boundary diagnostic from its failure set, prints the final apology as ordinary output, and exits 0, so scripted callers can still accept an operation that never completed.
I reproduced this through the production runFixtureCommand seam on exact head 31492a9dba447e2e2566ae539a81e7561f8f60ae: two Read starts carried different args.path values, the first result carried sandbox_boundary_required, and the second succeeded in a later step. The expected exit code was 1; the command returned 0. The new tests only vary toolName, while the existing positive recovery fixture gives both calls the default Read name and empty args, so neither case distinguishes a retry from a different invocation of the same tool.
Please correlate a stable retry identity, or compare a stable semantic target derived consistently for live and durable observations. If the available wire data cannot prove that the success belongs to the blocked operation, keep the failure unresolved. Add live, durable, and command-exit regressions with two same-named calls whose targets differ.
The exact-head CLI build succeeded, the complete CLI suite passed 723 tests with 3 platform skips, the changed files pass Biome and git diff --check, the hosted test job is successful on this head, and the current-main merge is clean. I am withholding approval for the false-success path above.
Automated review notice: This comment was posted by an automated review agent operated by WAWQAQ. It is not an independent human review and does not replace one.
|
Fixed in 6ee47cd — the recovery correlation now compares the retried call's semantic target, not just its name:
Local: target suite 40/40 pass on Windows; the full CLI suite shows only the pre-existing platform failures (LaunchAgent/EPERM fsync/symlink) also present without this change. |
Astro-Han
left a comment
Astro-Han
left a comment
There was a problem hiding this comment.
The problem is #4388 and it is real; the classifier change is on the right owner and the four new tests do fail on the old code (I short-circuited the name and args checks in dist and all four went red). CI, lint, format and the 40-case file are green on 6ee47cd3, merge-tree against main is clean.
The args half does not work on the path maka run actually uses. The Guest projector (session-projector.ts:619) sets args: undefined on every live tool_start, by design: session-continuity.ts:69 says live frames carry a bounded, redacted argsPreview and never the full args, and shell runs carry only shellRunRef. The TUI already reads event.args ?? event.argsPreview for that reason. So in production stableArgsJson(event.args) always returns a fresh unprovable:N, argsJson never equals failedArgsJson, and recovered is unreachable on live observation: a sandbox failure that really was widened and retried still classifies as unresolved, run-command-core.ts returns 1 and drops finalOutput even on a completed Turn. The tests pass because the fixture hands the classifier args: {}, a shape the producer never emits. The body's "both live tool_start events and durable tool_call messages carry args" is not true for live.
That leaves a decision I would rather make now than after another round. #4389 (jsiu93, approved, green) fixes the same issue on the same hunk with the tool-name check alone, which live events do carry, and its tests also pin empty stdout and the exact stderr diagnostic. Your first commit 31492a9d is semantically that change; the second adds args on top. I intend to merge #4389 first. If you want to keep this PR, rebase it on #4389 and reduce it to what live observation can prove: either compare argsPreview / shellRunRef and fail closed when the preview is truncated, saying so in the body, or keep args only for the durable path and add one live regression built from the projector's real shape (args: undefined plus argsPreview). If neither is worth it, closing in favour of #4389 is fine.
Two simplifications that survive either way, both tested by me against the 40 cases with no change in result: @maka/core/tool-args-identity already exports stableJsonStringify, which is what tool-ledger-scanner.ts and tool-recovery-bundle.ts use to say "same call", so the hand-written normaliser can go; and unprovableArgsSeq, a process-wide mutable counter whose only job is to be unequal, is redundant with the failedArgsJson !== undefined guard three lines below, so undefined is the sentinel.
One more for whichever PR lands: the sandbox-boundary scenario in run-command-fixture.ts is the only one that emits sandbox_boundary_request, which is the sequence #4388 actually reports, and it still hard-codes sandboxBoundary: 'none'; a recovered variant asserting exit 1 would pin the original report. And run-command-core.ts:314 lets recovered clear unclassifiedBoundaryFailure, which line 406 set from a hard fact (this run denied a widening); a heuristic should not erase that, but I could not build a false success after this PR, so noting it, not blocking on it.
Evidence boundary: static read against main 61224f51; cli build, the run-command test file, lint and format run locally; the live-path claim is from the projector and continuity protocol source, not a live Host run; durable StoredMessage.args not traced through the transcript RPC.
AI-assisted review: drafted with Maka; I verified the projector's args: undefined, the overlap with #4389 and the two dist substitutions myself.
简体中文
问题真实、owner 正确、新测试在旧代码上确实会红。但 args 这半在 maka run 实际走的 live 路径上不生效：Guest projector 把每个 live tool_start 的 args 写死为 undefined（协议注释明确 live 只带有损的 argsPreview），于是生产中 recovered 不可达，真正扩权重试成功的情况也会 exit 1 并吞掉 stdout；测试全绿是因为 fixture 给了生产不会产出的 args: {}。正文「live 事件携带 args」不成立。#4389 已 approve 且用 tool name 就解决了同一 hunk、同一 issue，我打算先合它；本 PR 要留就 rebase 到 #4389 上并只保留 live 能证明的部分（比较 argsPreview/shellRunRef 并 fail-closed，或 args 只用于 durable 路径并补真实形状的 live 回归），否则关掉也可以。两处简化：复用 @maka/core/tool-args-identity 的 stableJsonStringify，删掉 unprovableArgsSeq 计数器，均已实测等价。
| * keeping the failure unresolved (fail-closed). | ||
| */ | ||
| let unprovableArgsSeq = 0; | ||
|
|
There was a problem hiding this comment.
On the live path this branch is the only one ever taken: session-projector.ts:619 sets args: undefined on every tool_start (session-continuity.ts:69 says live frames carry only argsPreview). So recovered is unreachable in production and a genuinely widened retry still exits 1. Also @maka/core/tool-args-identity already has stableJsonStringify, and undefined works as the sentinel because of the guard on line 770.
|
On the Host path the denial never reaches the exit decision: |
6ee47cd to
8d8d482
Compare
|
Pushed 8d8d482 addressing both follow-ups:
Red/green would have been welcome - the added regressions cover the three shapes: live and durable same-name/different-target (exit 1), same-target retry still recovers (exit 0), and denied-widening-then-success (exit 1 with the diagnostic). CI is running on 8d8d482. |
A follow-up success with the same toolName cleared an unresolved sandbox boundary failure even when it targeted a different semantic target: a denied Read of /outside/secret.txt followed by a successful Read of /workspace/README.md still reported the boundary as recovered and exited 0. Tool names do not identify an invocation. The wire has no retry identity, but both live tool_start events and durable tool_call messages carry args. Serialize them with sorted keys (live and durable observations of the same call produce identical text) and require an exact match with the blocked call's args before clearing the failure. Missing or unserializable args yield a unique unprovable marker that never matches, keeping the failure unresolved. Adds live and durable regressions with two same-named Read calls whose targets differ (both must exit 1) and a same-target retry that must still report recovery and exit 0. Generated-by: GLM-5.3-Flash (ZCode)
Non-interactive runs cannot grant a sandbox widening request, and a denied one cannot be retried. Once a Turn contains a request_sandbox_boundary call, pin the failure set so no later success (even a same-target retry of an earlier blocked call) can report recovery and mask the denied expansion with exit 0. The existing request_sandbox_boundary exclusion in the clear condition keeps the pinned entry from being cleared by its own result, and the toolName + args match requirements keep other tools' successes from clearing it. Covers the Host-path gap in yihanzhu's review: #observeTurn drops the sandbox_boundary_request interaction event, so the denial itself never reaches the classifier; the tool_start of the widening call does, and is now enough to pin the outcome. Generated-by: GLM-5.3-Flash (ZCode)
8d8d482 to
f1130cb
Compare
4 participants
Summary
request_sandbox_boundarycall pins the boundary to unresolved for the rest of the Turn - no later success can report recovery, including the Host path where#observeTurndrops the interaction event but not the widening call'stool_start.Verification
#4388casesnode --test dist/__tests__/runtime-host-run-command.test.js(packages/cli)npm run format:checknpm run check:asf-headersnode --test "dist/**/*.test.js"AI use
Analysis, patch, and tests were produced with GLM-5.3-Flash (ZCode) under the contributor's direction; the contributor reviewed and is the human contributor of record.
Checklist
1instead of0