Skip to content

fix(runtime): recover once from output-free streams closed before completion - #4600

Open
jsiu93 wants to merge 1 commit into
apache:mainfrom
jsiu93:fix/output-free-stream-close-recovery
Open

fix(runtime): recover once from output-free streams closed before completion#4600
jsiu93 wants to merge 1 commit into
apache:mainfrom
jsiu93:fix/output-free-stream-close-recovery

Conversation

@jsiu93

@jsiu93 jsiu93 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Recover once from premature stream disconnections that occur before response.completed when the current provider attempt has produced no observable output.

When a provider stream closes unexpectedly before protocol completion without generating text, tool activity, thinking, completed steps, or continuation signatures, Maka initiates a single bounded recovery attempt. Any attempt with observable output triggers fail-closed termination, preserving durable tool side effects and preventing operation replays.

Fixes #4599

Changes

File-by-file changes across 13 files (+556, −23):

  • packages/runtime/src/provider-error-classification.ts: Identifies protocol-incomplete streams, prioritizing typed cause invalid_request_error and dual-phrase matching; enables dual-phrase fallback only when the cause chain lacks structured codes/types; preserves retryable semantics.
  • packages/runtime/src/model-protocol.ts: Adds optional recoveryReason: 'incomplete_stream' to ModelFailure.
  • packages/runtime/src/model-adapter.ts: Forwards bounded-recovery reasons across the adapter boundary to Runtime, preserving provider codes and terminal error messages.
  • packages/runtime/src/ai-sdk-backend.ts: Integrates the new reason into the existing single-recovery incomplete-stream counter, backoff, Stop abort, and step-budget paths; triggers fail-closed termination on observable output.
  • packages/core/src/events.ts: Exposes public telemetry reason incomplete_stream.
  • packages/runtime-host/src/protocol/turn.ts: Strictly decodes the new recovery reason.
  • packages/runtime-host/src/protocol/index.ts: Bumps RUNTIME_HOST_COMPATIBILITY_EPOCH from 101 to 102 to isolate closed-union enum decoding for older peers.
  • packages/ui/src/conversation-copy.ts: Adds bilingual copy for the new recovery reason.
  • packages/runtime/src/__tests__/provider-error-classification.test.ts: Covers typed causes, dual-phrase fallbacks, single-phrase rejection, and direct/nested conflicting structured errors.
  • packages/runtime/src/__tests__/model-adapter-onerror.test.ts: Covers reason propagation, retryable=false, providerCode, and terminal message boundary passing.
  • packages/runtime/src/__tests__/ai-sdk-backend.test.ts: Covers successful single recovery, budget exhaustion, five fail-closed guardrails, single execution of tool side effects, and Stop cancellation during backoff.
  • packages/runtime-host/src/__tests__/protocol.test.ts: Covers new reason decoding and compatibility epoch boundaries.
  • packages/ui/src/__tests__/live-turn-projection.test.ts: Covers bilingual distinct reason copy projections.

Compatibility

RUNTIME_HOST_COMPATIBILITY_EPOCH increases from 101 to 102 because adding incomplete_stream to the closed-union ProviderRetryReason causes older peers to reject the unknown enum value.

Recovery guards

Single automatic recovery activates when the current physical provider attempt meets all 9 conditions:

  1. Error is classified by adapter as incomplete_stream, or matches existing graceful truncated-stream branch.
  2. Current attempt produced no text (attemptSawText === false).
  3. Current attempt produced no thinking (attemptSawThinking === false).
  4. Current attempt produced no provider tool activity (attemptSawToolActivity === false).
  5. Current attempt contains no continuation metadata (stepStartedWithProviderContinuation === false).
  6. Current attempt reached no completed-step boundary (attemptSawCompletedStep === false).
  7. Current provider step retains remaining step budget.
  8. Current error is not a context overflow.
  9. Current provider step has not consumed its single incomplete-stream recovery budget.

Supplementary invariants:

  • retryable remains false; new path uses dedicated recoveryReason.
  • Provider code invalid_request_error and terminal error message remain preserved verbatim.
  • Exhausted recoveries follow standard terminal error handling.
  • Backoff honors turn abort signals; Stop cancels pending retry waits.
  • Persisted tool results reload from durable ledger; tool executions, tool_result, and function_response occur exactly once.
  • Public closed-union reason corresponds to Runtime Host compatibility epoch 102.

Verification

Targeted regression and quality checks:

  • Red tests: 260 items; 254 pass, 6 fail (failures covering classifier, adapter, single recovery, exhaustion, tool side-effects, Stop).
  • Green targeted test suite: 374/374 pass, 0 skip (covering 3 Runtime test files, Runtime Host protocol, and UI projection).
  • packages/runtime full suite: 3184 items; 3171 pass, 13 skip, 0 fail.
  • Workspace build: pass.
  • Format check: 1867 files, pass.
  • Lint check: 3138 files, pass.
  • Workspace typecheck: pass.
  • git diff --check: pass.
  • Diffstat: 13 files changed, 556 insertions, 23 deletions.
  • Independent review: OK — no remaining blocking findings.

Not covered

  • No local fake relay was spawned.
  • Did not reproduce the live upstream ~902 s stream close timing.
  • PR uses fake stream to deterministically simulate invalid_request_error and missing response.completed termination events.
  • Full version matrix of SDK-specific error constructors is not exhaustively simulated; tests cover nested typed causes and observed raw stream error objects.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: implementation drafted by pi running gpt-5.6-sol under human review; commit trailer Generated-by: pi (gpt-5.6-sol)

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Sep 3, 2026
@jsiu93
jsiu93 marked this pull request as ready for review September 3, 2026 06:27
@jsiu93
jsiu93 force-pushed the fix/output-free-stream-close-recovery branch from c56056f to 24721f3 Compare September 3, 2026 06:36
@jsiu93
jsiu93 force-pushed the fix/output-free-stream-close-recovery branch from 24721f3 to c833f48 Compare September 3, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(runtime): output-free streams closed before response.completed bypass bounded recovery

1 participant