Skip to content

fix(gateway): bound subagent batch shutdown - #6434

Merged
WillemJiang merged 8 commits into
bytedance:mainfrom
poijygfdyy:fix/subagent-batch-shutdown-timeout
Oct 8, 2026
Merged

WillemJiang merged 8 commits into
bytedance:mainfrom
poijygfdyy:fix/subagent-batch-shutdown-timeout

Conversation

@poijygfdyy

@poijygfdyy poijygfdyy commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Why

Gateway shutdown can still hang indefinitely in the durable subagent-batch path. Current main calls await app.state.subagent_batch_service.stop() without a deadline. SubagentBatchService.stop() cancels and then gathers its poller and active execution tasks, so a cancellation-resistant task can hold the Gateway lifespan open forever.

This is the follow-up explicitly left out of #6409: MCP-task shutdown is internally bounded, but the subagent-batch service is not.

What changed

  • Add a dedicated bounded Gateway shutdown helper for SubagentBatchService.
  • Reuse the existing _SHUTDOWN_HOOK_TIMEOUT_SECONDS policy introduced for other lifecycle hooks.
  • Preserve the existing ordering: mark batch submission unavailable before stop, and always clear the global batch submitter afterward.
  • Add a deterministic regression with a hanging stop() plus a lightweight lifespan wiring assertion.

Surface area

  • Backend API — Gateway lifespan/runtime lifecycle
  • Agents / LangGraph — durable subagent-batch runtime shutdown boundary
  • Sandbox
  • Skills
  • Dependencies
  • Default behavior change — Gateway exit is no longer indefinitely blocked by a stuck batch-service stop
  • Docs / tests / CI only

Screenshots / Recording

Not applicable.

Bug fix verification

  • Regression: backend/tests/test_subagent_batch_shutdown_timeout.py::test_subagent_batch_service_shutdown_is_bounded
  • Current-main mechanism confirmed before publication: lifespan directly awaits subagent_batch_service.stop() with no deadline, while that method awaits cancelled poller/execution tasks via asyncio.gather(..., return_exceptions=True).
  • The regression supplies a deliberately hanging stop implementation and verifies the Gateway helper returns within the configured shutdown budget.

Collision audit

Repeated immediately before publication against current upstream main@1d3e68363c38f5c55fbb47a87400c8e3ba7d1fa1.

Validation

  • Branch: fix/subagent-batch-shutdown-timeout
  • Head: f930911c54ec6c3cb2a8b6d2972e94d16e5a26ed
  • Base: current upstream main@1d3e68363c38f5c55fbb47a87400c8e3ba7d1fa1
  • Reviewer follow-up added cancellation-safe ordering inside SubagentBatchService.stop() (all native/item cancellation requests are issued before the first await), a regression that pins that ordering, and synchronized shutdown-budget accounting/docs. The branch remains architecture-aligned with the Gateway bounded-hook policy.
  • Targeted pytest / lint / format / git diff --check: not run / blocked — connected validation device is offline.
  • GitHub CI on the reviewer follow-up head is fully green: backend unit-test shards, lint, blocking-I/O, frontend unit tests, Replay E2E, agent guidance, and Helm validation all passed.

AI assistance

Tool(s) used: ChatGPT

How you used it: Audited Gateway and subagent-batch shutdown ownership, confirmed the unbounded await on current main, implemented the minimal architecture-matching timeout helper and regression, and performed collision checks.

  • I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/S PR changes 20-100 lines labels Oct 7, 2026

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read-only static review of PR #6434 at a9a2892. Verified introduced findings below; no PR code or tests executed.

Comment thread backend/app/gateway/app.py

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at head a9a2892 (static review; no PR code executed). This is the follow-up #6409 left scoped, and it follows the established pattern correctly: dedicated helper reusing _SHUTDOWN_HOOK_TIMEOUT_SECONDS, preserved ordering (subagent_batches_available = False before stop, submitter cleared in finally), and the #6409-style tests (bounded-stop regression plus lifespan wiring assertion). One blocking gap: the new bounded site is not modeled in backend/tests/_gateway_shutdown_budget.py, so test_shutdown_budget_models_every_bounded_lifespan_hook fails deterministically (7 sites vs 6 modeled) and the deployment grace arithmetic under-counts the new sequential 5s hook. The deeper timeout-semantics concern (wait_for cancels stop() mid-orchestration, skipping native/item cancellation) was already flagged in the prior review thread and is not repeated here.

Comment thread backend/app/gateway/app.py
@github-actions github-actions Bot added area:agents Agents, subagents, graph wiring, prompts, langgraph.json area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ size/M PR changes 100-300 lines and removed size/S PR changes 20-100 lines labels Oct 7, 2026

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta re-review at f930911: both prior findings verified fixed (P1 shutdown-budget tuple + P2 cancel-before-drain; acks posted in-thread). One new residual found in the compose comment refresh (inline below).

Comment thread docker/docker-compose.yaml Outdated
# scheduled task service, subagent batch service, browser sessions,
# MCP session pool), the 1s retrieval-warm wait, the 5s
# run drain and the 30s memory flush (memory.shutdown_flush_timeout_seconds)
# -- about 61s -- plus slack. backend/tests/_gateway_shutdown_budget.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] The refreshed total is stale: "seven hooks bounded at 5s each" (35s) + 1s retrieval-warm + 5s run drain + 30s memory flush = about 71s, not 61s. The helm README was updated to "about 71s" in 34da587, so the two docs now disagree; 61s was only correct for the old five-hook list. stop_grace_period: 90s still covers 71s + the 10s --timeout-graceful-shutdown + slack, so this is comment-only - but the very next line claims backend/tests/_gateway_shutdown_budget.py pins the arithmetic, so the number should track the model.

@WillemJiang
WillemJiang merged commit fa3bb43 into bytedance:main Oct 8, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agents Agents, subagents, graph wiring, prompts, langgraph.json area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/M PR changes 100-300 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants