Skip to content

fix(subagent): re-dispatch a read-only child once when its stream is cut with zero output - #796

Open
griffinwork40 wants to merge 3 commits into
mainfrom
afk/subagent-stream-cut-retry
Open

fix(subagent): re-dispatch a read-only child once when its stream is cut with zero output#796
griffinwork40 wants to merge 3 commits into
mainfrom
afk/subagent-stream-cut-retry

Conversation

@griffinwork40

Copy link
Copy Markdown
Owner

Why

A forked sub-agent whose model stream gets closed mid-generation by an intermediary dies for good. The provider loop re-drives that cut twice per tool-use round (STREAM_INCOMPLETE_MAX_RETRIES, loop.ts:128), but once the budget is spent nothing above it retries — handle.ts:555 throws StreamIncompleteError whose own message states the intended contract ("the parent should retry or fall back") and no layer implemented it. dag.ts has no retry at all. The parent just gets an isError payload; only the parent model can choose to re-call.

The error message itself is not a bug — it is the deliberate anti-truncation guard from #635/#628 (translate.ts:349) correctly refusing to present a partial as a finished answer.

Forensics

12,478 witness traces: 115 failure events across 74 traces, against a 2,997-child control set. Every correlation base-rated.

finding failing set control
ran >= 360s 100% 30.7%
median lifetime 1119s (~19 min) 184s
was a nested orchestrator 40.9% 8.2%
  • Duration is the discriminator — no failure has ever occurred under 6 minutes. It is a duration-exposure effect: the longer a stream stays open, the likelier an intermediary reaps it.
  • Nested orchestrators are 5x enriched (each absorbing up to MODEL_CAP_BYTES=100_000 per nested return, _output-cap.ts:47) — but 59% of failures are leaves, so that is a multiplier, not the mechanism.
  • Rate rose ~6.7x per-dispatch on top of a 6.5x volume increase, while the orchestrator share declined and no commit touched providers/ or subagent* in that window — consistent with external gateway behaviour, not a regression.
  • Model is not a factor: the "95% opus_1m" signal was an artifact of reading the parent session's model field. Failing children are 87% sonnet.

What this does

Adds src/agent/subagent/stream-cut-retry.ts — a pure decision module plus a wrapper that re-runs the full executeOnce body, so a re-dispatch forks a genuinely fresh child (new session, trace, worktree) rather than re-awaiting a spent handle. forkSubagent call-count is the load-bearing test assertion.

Safety: "zero output" is NOT "zero side effects"

The naive version of this fix — retry whenever partialOutputBytes == 0would have corrupted user state, and an adversarial review caught it before merge. handle.ts accumulates streamed text (:413) and tool calls (:415) separately, and :513-515 explicitly names tool-loops-without-a-final-turn as reaching the zero-output branch. So a write-capable child can run dozens of tool calls — file writes, git commit, HTTP POSTs — emit no text, and report zero output. Re-running its prompt would double-fire all of it.

Two gates therefore guard every re-dispatch:

  1. childWriteCapable === false (child-config.ts:204, which already folds in the agent's tool allowlist, the parent cage, and readOnlyBash). Only provably side-effect-free children are retried. This also removes the stranded-dirty-worktree hazard, since isolated worktrees exist only for write-capable children.
  2. A monotonic cancel generation must not have moved. Across the retry gap both in-flight handle maps are empty, so cancelActiveForeground() finds nothing to cancel and never aborts call.signal — without the counter a user cancel during the settle delay would be silently ignored and a fresh child forked anyway.

Also excluded from retry: buffered partials (status:'succeeded' — retrying would destroy salvaged findings) and tool_use_loop_capped (a requested ceiling, not a transport failure).

Budget is 1, deliberately below the provider loop's 2: by this layer the provider has already spent its per-round budget, so a re-dispatch re-runs a whole multi-minute prompt.

Accepted, documented costs

  • SubagentStop fires once per attempt; attempt 1's injectContext is discarded.
  • Telemetry emits one subagent.failed row per attempt with no attempt dimension.

Both are inherent to a retry genuinely being a second child. Attempt-scoped trace events are kept on purpose so a rescued failure stays visible in afk trace show.

Not in scope

dag.ts nodes and the skill/mint dispatch paths still do not retry. The module is reusable; they can adopt it incrementally.

Verification

  • pnpm test:coverage723 files / 13,774 tests pass, coverage thresholds enforced (74/79/82/74).
  • pnpm lint (tsc --noEmit) clean.
  • New module: 100% statements / branches / functions / lines.
  • 29 new tests, including a write-capable child refusing to retry, a cancel-during-gap regression guard, and an anti-drift test asserting the predicate against the real incompleteToolResultFields producer rather than a hand-rolled fixture.

Full diagnosis: ~/.afk/workspace/reports/subagent-stream-cut-2026-07-31.md

…cut with zero output

A forked sub-agent whose model stream is closed mid-generation by an
intermediary dies for good. The provider loop re-drives such a cut twice per
tool-use round (STREAM_INCOMPLETE_MAX_RETRIES), but once that budget is spent
nothing above it retries: handle.ts:555 throws StreamIncompleteError whose own
message states the intended contract ("the parent should retry or fall back"),
and no layer implemented it. dag.ts has no retry at all. The parent just
receives an isError payload and only the parent MODEL can choose to re-call.

Forensics over 12,478 witness traces (115 failures / 74 traces, against a
2,997-child control set):

  - 100% of failures ran >=360s (median 1119s ~19min); succeeding children run
    a median of 184s. No failure has ever occurred under 6 minutes — this is a
    duration-exposure effect, not a model or concurrency effect.
  - Nested orchestrator children are 5x enriched (40.9% of failures vs 8.2% of
    successes), each absorbing up to MODEL_CAP_BYTES=100_000 per nested return.
    But 59% of failures are leaves, so that is a multiplier, not the mechanism.
  - The rate rose ~6.7x per-dispatch on top of a 6.5x volume increase while the
    orchestrator share DECLINED, and no commit touched providers/ or subagent*
    in that window — consistent with an external gateway, not a regression.

Adds subagent/stream-cut-retry.ts: a pure decision module plus a wrapper that
re-runs the full executeOnce body, so a re-dispatch forks a genuinely fresh
child (new session, trace, worktree) rather than re-awaiting a spent handle.

Safety is gated twice, because "zero output" is NOT "zero side effects":
handle.ts accumulates streamed text (:413) and tool calls (:415) separately,
and :513-515 names tool-loops-without-a-final-turn as reaching this branch. A
write-capable child could therefore have written files, committed, or POSTed
before the cut while reporting zero output, and re-running its prompt would
double-fire all of it. So:

  1. childWriteCapable must be false (child-config.ts:204 — already folds in
     the agent tool allowlist, the parent cage, and readOnlyBash). Only
     provably side-effect-free children are re-dispatched. This also removes
     the stranded-dirty-worktree hazard, since isolated worktrees are created
     only for write-capable children.
  2. A monotonic cancel generation must not have moved. Across the retry gap
     both in-flight handle maps are empty, so cancelActiveForeground() finds
     nothing to cancel and never aborts call.signal; without the counter a
     cancel during the settle delay would be ignored and a child forked anyway.

Buffered partials (status:'succeeded' — real findings salvaged) and
tool_use_loop_capped (a requested ceiling) are both excluded from retry.

Budget is 1, deliberately below the provider loop's 2: by this layer the
provider has already spent its per-round budget, so a re-dispatch re-runs a
whole multi-minute prompt and costs far more than a single-round re-drive.

Accepted, documented costs: SubagentStop fires once per attempt and attempt 1's
injectContext is discarded; telemetry emits one subagent.failed row per attempt
with no attempt dimension. Both are inherent to a retry being a second child.

Verified: 723 test files / 13,774 tests pass with coverage thresholds enforced;
tsc --noEmit clean; new module at 100% statements/branches/functions/lines.
@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agent-afk-docs Ready Ready Preview Aug 1, 2026 12:36pm

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19e0ae2219

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agent/tools/subagent-executor.ts Outdated
// whole prompt, so it is only safe when the child cannot mutate anything.
// `childWriteCapable` is the authoritative answer — it already accounts for
// the named agent's tool allowlist, the parent's cage, and readOnlyBash.
if (probe !== undefined) probe.sideEffectFree = !childWriteCapable;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Gate retries on every side-effecting tool

When a restricted or named child is allowed a non-file mutator such as send_telegram, config_set, create_schedule, or browser_act, it can perform that action, emit no assistant text, and then end with stream_incomplete. This assignment treats !childWriteCapable as side-effect-free, but childWriteCapable only checks write_file, edit_file, and mutating bash; the retry therefore reruns the whole prompt and can duplicate messages, remote actions, schedules, or configuration changes.

AGENTS.md reference: AGENTS.md:L79-L79

Useful? React with 👍 / 👎.

Comment on lines +444 to +445
canRedispatch: () =>
probe.sideEffectFree && this.cancelGeneration === cancelGenerationAtStart,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recheck cancellation after the fresh fork

If cancelActiveForeground() is called after this gate passes but while the second executeOnce() is awaiting forkSubagent()—which can await hooks and read-scope resolution—the handle maps are still empty, so the cancellation only increments cancelGeneration. Nothing checks that generation again after the fork resolves, and the newly created child is registered and run despite the user's cancellation; recheck the generation before starting the foreground branch or make the pending fork cancellable.

AGENTS.md reference: AGENTS.md:L83-L83

Useful? React with 👍 / 👎.

@griffinwork40

Copy link
Copy Markdown
Owner Author

@codex fix comments

@chatgpt-codex-connector

Copy link
Copy Markdown

Summary

Testing

  • pnpm test:file src/agent/tools/subagent/child-config.test.ts src/agent/tools/subagent/subagent-executor.stream-cut-retry.test.ts — 67 tests passed.
  • pnpm lint
  • git diff --check
  • git status --short --branch — working tree clean after commit.

View task →

@griffinwork40

Copy link
Copy Markdown
Owner Author

Wave 1.5 + shadow-verify complete. All pivotal citations re-derived at the ref; one high finding inverted and one narrowed.


Code Review — PR #796

fix(subagent): re-dispatch a read-only child once when its stream is cut with zero output
afk/subagent-stream-cut-retrymain · ref b90271e · 6 files, +801/−2 · regime full

Blocking

1 · high · confidence high · correctness + spec-compliance — defeats stated purpose

src/agent/tools/subagent/child-config.ts:213-218 · ref:b90271e · citation-type: diff-context

The retry never fires for research-agent, the dominant read-only dispatch surface in this codebase — so the fix does not cover the case its own forensics describe.

const childSideEffectFree =
  effectiveAllowedTools !== undefined &&
  effectiveAllowedTools.every((tool) =>
    SIDE_EFFECT_FREE_TOOLS.has(tool) || (tool === 'bash' && effectiveReadOnlyBash === true));

research-agent's registry surface is ['Read','Grep','Glob','WebFetch','WebSearch'] (src/skills/_agents/research-agent.ts:17) plus 'Agent(git-investigator)' (src/agent/agents/builtins.ts:119). The normalizer (src/agent/agents/resolve.ts:70-88src/agent/plugins/tool-injector.ts:81-93; resolve.ts:85 for the dispatch alias) maps that to ['read_file','grep','glob','web_scrape','agent']. Both web_scrape and agent are explicitly excluded from READ_ONLY_PHASE_TOOLSsrc/agent/tool-category.ts:186 (send_telegram, web_scrape — outbound network) and :182-184 (agent, skill, compose — dispatch grandchildren). So childSideEffectFree === false, probe.sideEffectFree === false (subagent-executor.ts:602), canRedispatch() short-circuits (:445-446), and stream-cut-retry.ts:148 returns the cut unretried.

The mismatch is documented too: the PR body names gate (1) as childWriteCapable === false at child-config.ts:204 — which would include research-agent — while the shipped gate is the stricter :213-218. The load-bearing safety gate described in the PR is not the one that shipped.

Scope, precisely (a broader claim was refuted — see dropped manifest): the retry does fire for git-investigator (builtins.ts:146 bashReadOnly: true) and Explore (builtins.ts:175), and for any dispatch whose parent cage intersects down to pure-read (child-config.ts:189-192). It is not a no-op — it misses the primary case.

suggestion — Introduce a dedicated RETRY_SAFE_TOOLS set in tool-category.ts that adds web_scrape and agent to the READ_ONLY_PHASE_TOOLS members, and key childSideEffectFree on it; reusing a set whose contract is pre-approval phase gating for retry authorization couples two unrelated policies.

Non-blocking

2 · medium · high · security

src/agent/tools/subagent/child-config.ts:217 · ref:b90271e · diff-context

The predicate admits bash — a tool READ_ONLY_PHASE_TOOLS deliberately excludes at tool-category.ts:181 — on the strength of classifyBashCommand, whose own header declares it "a BEST-EFFORT, default-ALLOW defense-in-depth layer — NOT a security boundary" and states "no allowlist or denylist can be exhaustive — a determined model could craft an obfuscated mutation this misses (e.g. eval "$(printf ...)")" (src/agent/tools/readonly-bash.ts:5-6,15-17, file-state). A mutation the classifier missed on attempt 1 re-fires on attempt 2 under a field documented as "proven free of persistent side effects" (child-config.ts:128).
suggestion — Require literal membership in the retry set; drop the bash carve-out.

3 · medium · high · security

src/agent/tools/subagent/foreground-promotion.ts:380-384 · ref:b90271e · file-state → verified at ref

handle.teardown() fires SubagentStop ("teardown() fires SubagentStop"), once per attempt. The childSideEffectFree gate constrains the child's tools, not the parent's hook config, so a non-idempotent user SubagentStop hook (notification POST, file append) double-fires on every rescued dispatch. The PR documents this as accepted, but frames the cost as attempt 1's injectContext being discarded — not as user hook code re-executing.
suggestion — Pass an attempt field in the SubagentStop payload so hooks can dedupe.

4 · medium · high · perf-observability

src/agent/tools/subagent-executor.ts:447-451 · ref:b90271e · diff-context

A re-dispatch doubles a multi-minute child's wall-clock and token spend with no durable record: the only re-dispatch-specific signal is debugLog, which emits solely under AFK_DEBUG=1/DEBUG=1 (src/utils/debug.ts:13-14). Absence confirmed at ref: git grep -i redispatch over non-test src/**/*.ts returns zero emitTelemetry and zero trace-event calls; attempt in failure-payload.ts → zero matches. The two subagent.failed rows an operator sees (foreground-promotion.ts:301) are indistinguishable from two unrelated failures.
suggestion — Add attempt as a dimension on the subagent.failed telemetry emit.

5 · medium · high · test-coverage

src/agent/tools/subagent/subagent-executor.stream-cut-retry.test.ts:106-118 · ref:b90271e · diff-context

No test dispatches a real registered agent through the gate — every positive-path test hand-rolls allowedTools: ['read_file','grep','glob'], a set no shipped agent uses. That is why finding #1 shipped invisibly. The suite's own comment asserts the wrong mechanism: it says readOnly "drives childWriteCapable === false … and hence probe.sideEffectFree === true". Absence confirmed: research-agent|git-investigator|namedAgent0 matches in both new test files.
suggestion — Add a child-config.test.ts case asserting childSideEffectFree for buildChildConfig({ namedAgent: <research-agent registry entry> }).

6 · low · high · correctness — inert mechanism, misleading invariant

src/agent/tools/subagent-executor.ts:342-353 + :434-440 · ref:b90271e · diff-context

cancelGeneration cannot activate in the window its comments describe. It is written only at :353, inside cancelActiveForeground(), whose sole production call site is guarded: if (ctrl?.hasActiveForeground()) at src/cli/commands/interactive/turn-handler.ts:297, reading a map that is empty across the retry gap (activeForegroundHandles.set at foreground-promotion.ts:164, delete in the finally at :371). The window is already covered by a different mechanism — see the dropped manifest. Its regression test (subagent-executor.stream-cut-retry.test.ts:179-195) passes only because it calls executor.cancelActiveForeground() directly at :187, bypassing the hasActiveForeground() guard that is the sole production gate, so the guard validates a path production cannot reach.
suggestion — Either bump the counter from an unconditional SubagentControl method the soft-stop calls before its guard, or delete the counter and correct the comments at :342-345, :350-352, :434-440 to name call.signal as the actual protection.

7 · low · high · security/observability

src/agent/tools/subagent-executor.ts:705-709 · ref:b90271e · diff-context

The retry-cancel early return tears down and returns without emitting telemetry, while every sibling path does (:712-721 fork-failure, foreground-promotion.ts:301-325). It also calls handle.teardown() without { deferInjectContextToCaller: true }, unlike the normal path (foreground-promotion.ts:384), so a cancelled retry fork's SubagentStop note routes to the deferred queue instead of being suppressed.
suggestion — Emit subagent.failed with status:'cancelled' before the return, and match the teardown options of the normal path.

8 · low · high · perf-observability

src/agent/subagent/stream-cut-retry.ts:150-153 · ref:b90271e · diff-context

onRedispatch fires before the settle delay and before the post-delay gate re-check, so the debug line claims a re-dispatch that :152-153 then suppresses. The module's own tests at stream-cut-retry.test.ts:139-153 and :196-210 pin this ordering by using onRedispatch as their mutation trigger, proving the hook fires on paths that never fork.
suggestion — Move opts.onRedispatch?.(attempt) to immediately before result = await opts.dispatch(attempt) at :155.

9 · low · high · test-coverage

ref:b90271e · diff-context — absence confirmed at ref (0 matches for delayMs|Promise.all|concurrent in the integration suite)

No test covers: two concurrent execute() calls sharing cancelGeneration (the counter's stated reason for being a counter rather than a boolean, subagent-executor.ts:438-439); budget exhaustion with a non-zero delayMs, so STREAM_CUT_REDISPATCH_DELAY_MS = 1_000 is never exercised end-to-end.

10 · nit · high · correctness

src/agent/tools/subagent/child-config.ts:47 · ref:b90271e · diff-context

No test pins SIDE_EFFECT_FREE_TOOLS ⊆ side-effect-free (grep confirmed: the only SIDE_EFFECT_FREE_TOOLS reference outside child-config.ts is none). The set is shared with nesting.ts:473 and subagent.ts:287,966 for phase gating, so a future member added for a read-only research phase silently widens retry authorization.

Clean reads

  • api-compat — no issues found. BuildChildConfigResult.childSideEffectFree is a new required field on a return type; buildChildConfig has exactly one production importer (subagent-executor.ts:31), updated in this PR. Not re-exported from src/index.ts. cancelActiveForeground(): Promise<number> keeps its signature and 0-on-empty return (:362); cancelGeneration is private; execute(call) unchanged.
  • Retry loop control flow — no bug. Walked stream-cut-retry.ts:132-159 at budget 0/1/3: 0 disables, a rescued attempt returns immediately, and the trailing return result correctly returns the final failure unchecked.
  • probe staleness — not reachable. probe is a per-execute() local (:433), never on this, only ever assigned the deterministic childSideEffectFree (:602).
  • Worktree hazard — conclusion holds. sideEffectFree ⟹ !writeCapable, so no worktree path is reachable on a retry (all four combinations enumerated) — though not for the reason the PR body gives, which keys on the looser field.
  • "Purely additive" contract holds. sleepWithAbort resolves early, it does not reject (src/agent/providers/shared/sleep-with-abort.ts:19-26new Promise((resolve) => with no reject), so it never throws out of the wrapper.

Dropped / corrected in verification

claim status resolution
"ESC during the retry gap forks a fresh child anyway" (raised high + medium by two agents independently) REFUTED turn-handler.ts:281 fires session.interrupt() unconditionally, before the guard at :297. Chain: agent-session.ts:763-767anthropic-direct/query.ts:621-623 abort.requestAbort('interrupted')abort-coordinator.ts:114-121 controller.abort() — on the same per-turn controller that owns call.signal (query.ts:359 abort.begin():428 signal: controller.signalloop.ts:1014-1019 signal: input.signal). stream-cut-retry.ts:145/152 and subagent-executor.ts:462 all suppress. Residual doc/test defect kept as finding #6 at low.
"retry never fires for any shipped read-only agent" REFUTED (narrowed) git-investigator (builtins.ts:146) and Explore (builtins.ts:175) both pass the predicate. Finding #1 restated as research-agent-specific.
agent-session.ts:766 path-corrected File absent at ref; real path src/agent/session/agent-session.ts:763-767. Content correct — not fabricated, finding retained.
sleepWithAbort rejects on abort disproven Resolves early; no finding.

What was not checked

Merge decision

DO NOT MERGE

One high finding defeats the change's stated purpose for its primary case: research-agent — named in child-config.ts:200-201 as the canonical read-only agent and mandated as subagent_type by this repo's own skills — is silently ineligible for the retry, because the authorization set is READ_ONLY_PHASE_TOOLS, whose contract is pre-approval phase gating and which excludes web_scrape and agent by design. The PR body documents a different, looser gate than the one that shipped. The fix is small and local: a dedicated retry-safety set, plus a test that runs a real registered agent through the predicate.


Done

  • Reviewed PR fix(subagent): re-dispatch a read-only child once when its stream is cut with zero output #796 across security, api-compat, correctness, spec-compliance, test-coverage, and perf-observability: 2 parallel Wave 1 agents → inline Wave 1.5 citation/absence verification at the ref → 2 shadow verifiers on the high claims; emitted 10 findings, 1 blocking.
  • Evidence: reviewed ref b90271e049cdf655c1891e3ce98b3dcf9d2fc548; diff artifact /Users/griffinlong/.afk/state/skill-preflight/0ebee9f3-404f-427b-ab67-4c3c5e25737e/pr-796.diff; every citation re-derived via git show b90271e:<path> / git grep b90271e; verifier chains cited to file:line above.
  • What changed in the world: nothing — this skill is read-only. No files written, no commits, no PR comments, no pushes. Working tree remains clean at b90271e. The review itself is transcript-delivered because the skill forbids file writes; the durable anchors are the ref SHA and the diff artifact path above.
  • Pending: the review is not posted to the PR and no fix was applied — both are deliberately out of scope. Findings test: hermetically lock AFK_INTERNAL tier in audience-gate tests #1 and fix(skills): enforce read-only tool gating for recon skills (ground-state) #5 are the two that gate a merge; verifying the PR's own test/lint/coverage claims requires a run I did not perform.

🤖 Posted by agent-afk /review --post github

@griffinwork40

Copy link
Copy Markdown
Owner Author

@codex fix comments

@chatgpt-codex-connector

Copy link
Copy Markdown

Summary

Testing

  • pnpm test:file src/agent/tools/subagent/child-config.test.ts src/agent/tools/subagent/subagent-executor.stream-cut-retry.test.ts src/agent/subagent/stream-cut-retry.test.ts — 87 tests passed across 3 test files.
  • pnpm lint
  • git diff --check
  • git status --short --branch — working tree clean after commit.

View task →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant