fix(responses): recover encrypted output rejections without masking failures

devlog/_plan/260906_opaque_transport_finality/000_plan.md

@@ -0,0 +1,21 @@
+# Opaque preflight transport and terminal outcomes
+
+Class C4. Mandatory parent-PR review repair under the existing authorized release
+chain; work phase opaque-transport-finality, criterion c-2. Parent #3753 remains
+open/draft at b73809f7e, child #3754 remains open/draft at f5c88beb9 with its parent
+base restored. No parent merge occurred. The original #3535 was briefly closed
+by an out-of-order follow-up, immediately reopened, and its comment corrected.
+No completion, approval or release gate is waived.
+
+Public review references: PRRT_kwDOS-0Gi86fqEUo (preflight read failure escapes)
+and PRRT_kwDOS-0Gi86fqEUq (tee EOF reports incomplete despite failed client tail).
+The earlier full CI and independent reviews did not cover these paths. The
+unfinished combo cycle is preserved and must consume the repaired parent before
+its final verification. All execution remains hosted; no local suite/typecheck/
+build or live Kiro request.
+
+Implementation is one bounded failure-contract unit in 010_failure_boundaries.md.
+Update the existing parent PR, run exact-head CI, cascade its commit into #3754,
+and require fresh composed CI and review before bottom-up integration. Do not
+close an original or retarget a child as a side effect of an unverified merge:
+verify each preceding command and actual merged state before dependent actions.

devlog/_plan/260906_opaque_transport_finality/010_failure_boundaries.md

@@ -0,0 +1,67 @@
+# Preserve preflight read failures and inspection finality
+
+## Current ownership
+
+Core selects native encrypted-output candidates and awaits combo-stream-preflight
+before exposing headers. The preflight owns a bounded retained prefix and one
+reader; replayBufferedResponse already emits that prefix and forwards later read
+errors. Client relays own synthetic failed tails. consumeForInspection owns the
+independent tee terminal callback used by native account health. The shared SSE
+inspector reports real terminals and exposes parsed payload callbacks.
+
+## Planned change
+
+- src/server/responses/combo-stream-preflight.ts: native-only replayReadErrors
+  option, default false. Catch only reader.read rejection; opted-in callers get
+  an accepted reconstructed stream retaining the bounded prefix and the errored
+  reader. Never cancel that errored reader: its original rejection must survive
+  the replay into relay/inspection. Default combo callers preserve their prior throw behavior. Do not retry
+  or classify a read reset as a decrypt rejection, swallow it, or grow buffers.
+- src/server/responses/core.ts: enable that option only on the native opaque
+  preflight. After its await, caller abort takes the existing cancellation cleanup
+  path before any replay/rebuild. Other read failures reach the normal mid-stream
+  relay and inspection path, not a connect-phase error classifier.
+- src/server/relay.ts: reuse a bounded/redacted bare-error message helper at the
+  client boundary and within consumeForInspection's parsed-payload callback.
+  Keep that evidence local to this reader rather than borrowing stale log state.
+  At clean EOF without a real terminal, a witnessed bare error reports failed
+  using the shared terminal HTTP mapper; an error-free EOF remains incomplete.
+  Preserve the caller's parsed-payload callback. Real terminals and cancellation
+  retain precedence; no extra terminal callback or healthy-account reset.
+
+## Rejected alternatives and scope
+
+A blanket core catch mapped as a connect error can misclassify an already-started
+response's account outcome. Globally replaying all preflight errors changes combo
+behavior. Reporting failure at the first bare error would override a later real
+terminal. Borrowing the client relay's mutable state revives tee scheduling races.
+Use the existing preflight/relay ownership and callback seams instead; no new
+public inspector method, provider policy or retry budget.
+
+## Verification
+
+Existing native request fixtures add created-then-reset and created-then-caller-
+abort cases: no uncaught handleResponses rejection, no sanitize resend, normal
+failed stream or 499 cancellation and appropriate attempt/terminal metadata.
+Run tee/eager variants where selected by the existing harness. Preflight tests
+prove default read-error behavior is unchanged and native opt-in preserves prefix
+and exact failure. Inspection/account-health fixtures cover flat/nested bare
+errors at EOF, prior failure/avoidance not cleared, real-terminal precedence,
+error-free EOF compatibility and cancellation neutrality. Existing redaction,
+byte bounds, no-persistence and one-shot recovery tests remain.
+
+Independent plan/source/final review; exact parent and cascaded child hosted
+Linux/macOS/gates CI. Final Windows six-shard and release gates remain mandatory.
+
+## Usage-marker parity amendment
+
+Source review confirms the account-health blocker is closed by failed EOF. The
+existing eager callback still labels every synthetic failure as streamAborted,
+though a clean EOF after an explicit upstream error is a semantic failure, not a
+body-read reset (PersistedUsageAttempt documents that distinction). Criterion c-2
+also requires usage outcome parity, so include this small related correction:
+relay-eager passes optional upstream_error provenance only for that clean-EOF tail;
+core records its semantic failed status without streamAborted. Ordinary reset
+callbacks retain their one-argument shape, 502 and streamAborted. Add request-level
+tee/eager assertions for repeated bare errors versus actual reset; do not infer
+this marker from a stale log message or change real-terminal precedence.

devlog/_plan/260906_release_244_followups/040_opaque_recovery.md

@@ -22,3 +22,35 @@ Existing maintainer CHANGES_REQUESTED targeted older 2d90f9684 reader race; inde
 
 Owner explicitly requests stacked PR workflow; use this relay foundation before combo-recovery and Grok terminal integration as an integration-validation stack, even though fixes are independently useful. Each layer remains independently tested via exact-head ci.yml runtime/gates. Security analysis stays scratch until public diff; no live Kiro.
 
+
+## Current-dev carry amendment (2026-09-06)
+
+The carry starts at adb696197 after the task-input, Kiro and fixture layers.
+The source remains 2396829bd. The current core rewrite order also contains
+tool-search restoration and function completion repair; preserve both and the
+shared prompt-cache cohort field. Source review is not current-head approval.
+
+Default two-argument preflight callers keep their previous event classification.
+Only the explicitly supplied exact decrypt predicate may make a matching bare
+error replayable; unrelated errors still commit the stream, and an existing
+unrelated response.failed stays an SSE terminal. The retry predicate accepts
+only error/failed/incomplete envelopes, never output events carrying a message.
+The new failed tail uses existing redactSecretString before the 512-character
+limit. Test bounded synthesized messages in tee and eager paths with synthetic
+credential canaries; retain original upstream frame passthrough semantics.
+
+This cohesive carry exceeds the default 500-line review size because the source
+includes a large request-level regression matrix. Keep source and regression
+commits distinct inside this one layer, with independent protocol/security review;
+splitting the tests into a later PR would leave recovery unverified. Existing
+large core/relay files retain their current ownership for this bounded carry:
+no export moves or broad refactor amid replay/cancellation changes. A new generic
+retry abstraction or core extraction would enlarge the behavior under review.
+Remote CI verifies all source and tests together; no local suite/typecheck/build.
+
+The existing core recognizes successful streaming Responses without Content-Type.
+Keep that parity in the new preflight through an explicit fourth options argument
+allowMissingContentType, enabled only by the same core streaming condition; default
+combo callers still require text/event-stream. Add missing-header recovery and
+non-SSE refusal controls. This avoids a source-PR gap where core selected recovery
+but its preflight returned early solely because the header was absent.

devlog/_plan/260906_release_244_followups/041_opaque_recovery_implementation.md

@@ -0,0 +1,20 @@
+# Opaque recovery implementation evidence
+
+Source 3b8cf8a8f carries PR3535 with a narrowly scoped preflight opt-in. The default
+combo event classifier is unchanged; only a matched bare error supplied by the
+native decrypt caller is replayable. Headerless streaming is an explicit option
+under the existing core condition. Client-reader error evidence is redacted and
+bounded before a failed tail is synthesized; real terminals remain authoritative.
+
+Independent plan audit accepted the scoped seam. Independent source/security
+review passed: exact 502 gate, one sanitized rebuild, raw-body object identity,
+no replay after visible output, cancellation and current rewrite ordering remain.
+The source contributor is credited in the carry commit and PR.
+
+Regression commits cover native function and agent-message history, repeated
+flat/nested errors, both relay shapes, unrelated errors and default combo byte
+preservation, output commitment, missing-header and wrong-media-type controls,
+and bounded synthesized-message redaction. The headerless fixture uses bytes
+and asserts the absence of Content-Type because a string body supplies text/plain.
+No local test suite, typecheck, build or live Kiro request was run. Final evidence
+comes from hosted CI on the complete PR head and a fresh independent review.

docs-site/src/content/docs/guides/sub-agent-surface.md

@@ -172,6 +172,19 @@ for the full trust boundary and configuration.
 Combo routing remains unchanged and continues to consider only canonical native ChatGPT targets for
 encrypted tasks.
 
+## Rejected encrypted history
+
+An upstream Responses server can reject encrypted parts in earlier function/custom-tool
+output or `agent_message` content with `Encrypted function output content could not be decrypted or decoded.`. Before
+any output is committed, opencodex replaces those parts with `[encrypted content omitted]`
+and rebuilds the request once. The surrounding readable content stays intact; the
+omitted content is not decrypted or recovered by this retry.
+
+If the rebuilt request receives another bare SSE `error` followed by EOF, both relay
+modes preserve the error message in a `response.failed` terminal instead of reporting
+`adapter_eof`. Other upstream `response.failed` events remain SSE failures. This history
+recovery does not change the encrypted v2 task-delivery restrictions described above.
+
 ## Changing the mode
 
 ### GUI

src/lib/errors.ts

@@ -4,10 +4,15 @@ export interface OcxErrorPayload {
   code: string | null;
 }
 
+export const ENCRYPTED_FUNCTION_OUTPUT_REJECTION =
+  "Encrypted function output content could not be decrypted or decoded.";
+
 /** Canonical human-readable message paths used by Responses upstream failures. */
 export function upstreamErrorMessageFromPayload(payload: unknown): string | undefined {
   if (!payload || typeof payload !== "object" || Array.isArray(payload)) return undefined;
   const json = payload as {
+    type?: unknown;
+    message?: unknown;
     error?: { message?: unknown };
     last_error?: { message?: unknown };
     response?: {
@@ -18,7 +23,10 @@ export function upstreamErrorMessageFromPayload(payload: unknown): string | unde
   const message = json.error?.message
     ?? json.last_error?.message
     ?? json.response?.error?.message
-    ?? json.response?.incomplete_details?.message;
+    ?? json.response?.incomplete_details?.message
+    // The Responses stream error event carries a flat message (type/code/message),
+    // unlike the response.failed envelope the branches above already cover.
+    ?? (json.type === "error" ? json.message : undefined);
   return typeof message === "string" ? message : undefined;
 }
 

src/server/relay-eager.ts

@@ -29,6 +29,7 @@ import {
   createSseTerminalOutputBoundary,
   doneFrame,
   failedTailFrame,
+  upstreamErrorTailFrame,
 } from "./relay";
 import {
   nextSseBlock,
@@ -63,7 +64,7 @@ export type EagerRelayHooks = {
   /** True once inspection has reported a protocol terminal (inspector.reported). */
   sawTerminal: () => boolean;
   /** Record a synthetic terminal (caller decides incomplete vs failed-502). */
-  onSynthetic: (kind: "incomplete" | "failed") => void;
+  onSynthetic: (kind: "incomplete" | "failed", reason?: "upstream_error") => void;
   /** Client cancelled and NO terminal arrived within the drain bounds. */
   onClientCancel: () => void;
   /** Exactly once, after the producer fully stops (unregisterTurn parity). */
@@ -81,6 +82,8 @@ export type EagerRelayOptions = {
   postCancelDrainMs?: number;
   /** Post-cancel discard-drain byte bound. Default 32 MiB. */
   postCancelDrainBytes?: number;
+  /** Last known upstream failure to preserve when EOF would otherwise become adapter_eof. */
+  upstreamError?: string;
   /** Injectable clock for tests. */
   now?: () => number;
 };
@@ -239,6 +242,7 @@ export function relaySseEagerBounded(
 
   const producer = async () => {
     let syntheticKind: "incomplete" | "failed" | null = null;
+    let syntheticReason: "upstream_error" | undefined;
     let deliveryFallbackSent = false;
     let priorRewriteFailure = false;
     let priorRewriteError: unknown;
@@ -303,12 +307,17 @@ export function relaySseEagerBounded(
           } else if (!hooks.sawTerminal() && canDeliver()) {
             // A clean 200 EOF without a Responses terminal must be visible to
             // Codex as one incomplete turn, followed by the normal sentinel.
-            queuedBytes += adapterEofFrame.byteLength + terminalSentinel.byteLength;
+            const upstreamError = terminalBoundary.upstreamError() ?? opts?.upstreamError;
+            const upstreamErrorFrame = upstreamError === undefined
+              ? adapterEofFrame
+              : upstreamErrorTailFrame(terminalEncoder, upstreamError);
+            queuedBytes += upstreamErrorFrame.byteLength + terminalSentinel.byteLength;
             try {
-              controllerRef?.enqueue(adapterEofFrame);
+              controllerRef?.enqueue(upstreamErrorFrame);
               controllerRef?.enqueue(terminalSentinel);
             } catch { /* client already gone */ }
-            syntheticKind = "incomplete";
+            syntheticKind = upstreamError === undefined ? "incomplete" : "failed";
+            syntheticReason = upstreamError === undefined ? undefined : "upstream_error";
           }
           break;
         }
@@ -449,7 +458,10 @@ export function relaySseEagerBounded(
         frameBufferBytes = 0;
       }
       terminalBoundary.dispose();
-      if (syntheticKind && canDeliver()) hooks.onSynthetic(syntheticKind);
+      if (syntheticKind && canDeliver()) {
+        if (syntheticReason === undefined) hooks.onSynthetic(syntheticKind);
+        else hooks.onSynthetic(syntheticKind, syntheticReason);
+      }
       if (cancelled && !hooks.sawTerminal()) {
         hooks.onClientCancel();
       }

src/server/relay.ts

@@ -5,6 +5,7 @@ import {
   CYBER_POLICY_FALLBACK_MESSAGE,
   isCyberPolicyCode,
   isCyberPolicyMessage,
+  upstreamErrorMessageFromPayload,
 } from "../lib/errors";
 import { redactSecretString } from "../lib/redact";
 import { isTranslatorBudgetExceededError } from "../lib/translator-budget";
@@ -147,11 +148,31 @@ export function failedTailFrame(encoder: TextEncoder, err: unknown): Uint8Array
   return encoder.encode(`\n\nevent: response.failed\ndata: ${payload}\n\n${DONE_SSE_FRAME_TEXT}`);
 }
 
+export function upstreamErrorTailFrame(encoder: TextEncoder, message: string): Uint8Array {
+  const error = {
+    type: "upstream_error",
+    code: "upstream_server_error",
+    message: redactSecretString(message).slice(0, MAX_TAIL_ERROR_MESSAGE_CHARS),
+  };
+  return encoder.encode(`event: response.failed\ndata: ${JSON.stringify({
+    type: "response.failed",
+    response: { status: "failed", error, last_error: error },
+  })}\n\n`);
+}
+
+function boundedBareUpstreamErrorMessage(payload: unknown): string | undefined {
+  if (!payload || typeof payload !== "object" || Array.isArray(payload)
+    || (payload as { type?: unknown }).type !== "error") return undefined;
+  const message = upstreamErrorMessageFromPayload(payload);
+  return message ? redactSecretString(message).slice(0, MAX_TAIL_ERROR_MESSAGE_CHARS) : undefined;
+}
+
 export type SseTerminalOutputBoundary = {
   feed(chunk: Uint8Array): Uint8Array;
   finish(): Uint8Array;
   terminalSeen(): boolean;
   doneSeen(): boolean;
+  upstreamError(): string | undefined;
   dispose(): void;
 };
 
@@ -170,6 +191,7 @@ export function createSseTerminalOutputBoundary(): SseTerminalOutputBoundary {
   let done = false;
   let pendingDone: { block: Uint8Array; delimiter: Uint8Array } | null = null;
   let disposed = false;
+  let upstreamError: string | undefined;
 
   const processFrames = (
     frames: ReturnType<BoundedSseFrameBuffer["feed"]>,
@@ -181,6 +203,10 @@ export function createSseTerminalOutputBoundary(): SseTerminalOutputBoundary {
       const payload = sseDataPayload(decoder.decode(frame.block));
       const isDone = payload === "[DONE]";
       const parsed = payload === null ? undefined : parseSsePayload(payload);
+      // Observe on the client reader itself: a tee inspection branch may lag
+      // behind EOF, so its log context cannot determine the outgoing terminal.
+      const message = boundedBareUpstreamErrorMessage(parsed);
+      if (message !== undefined) upstreamError = message;
       const policyError = parsed !== undefined && isPolicyRewriteType(parsed)
         ? cyberPolicyTerminalError(parsed)
         : undefined;
@@ -239,6 +265,7 @@ export function createSseTerminalOutputBoundary(): SseTerminalOutputBoundary {
     },
     terminalSeen: () => terminal,
     doneSeen: () => done,
+    upstreamError: () => upstreamError,
     dispose() {
       if (disposed) return;
       disposed = true;
@@ -260,6 +287,7 @@ export function relaySseWithFailedTail(
   body: ReadableStream<Uint8Array>,
   upstream: AbortController,
   onClientGone?: (reason?: unknown) => void,
+  opts?: { upstreamError?: string },
 ): ReadableStream<Uint8Array> {
   const reader = body.getReader();
   const encoder = new TextEncoder();
@@ -306,8 +334,10 @@ export function relaySseWithFailedTail(
               // A clean upstream EOF is still a failed Responses turn when no
               // protocol terminal arrived. Make that state explicit so Codex
               // does not treat HTTP 200 + bare EOF as a retryable disconnect.
-              const incomplete = adapterEofIncompleteFrame(encoder);
-              controller.enqueue(incomplete);
+              const upstreamError = terminalBoundary.upstreamError() ?? opts?.upstreamError;
+              controller.enqueue(upstreamError === undefined
+                ? adapterEofIncompleteFrame(encoder)
+                : upstreamErrorTailFrame(encoder, upstreamError));
               controller.enqueue(doneFrame(encoder));
             }
             terminalBoundary.dispose();
@@ -1354,11 +1384,16 @@ export function consumeForInspection(
   options?: InspectionConsumerOptions,
 ): void {
   const reader = body.getReader();
+  let bareUpstreamError: string | undefined;
   const inspector = (options?.inspectorFactory ?? createSseInspector)({
     onTerminal,
     logCtx,
     onCompletedResponse,
-    onParsedPayload: options?.onParsedPayload,
+    onParsedPayload: payload => {
+      const message = boundedBareUpstreamErrorMessage(payload);
+      if (message !== undefined) bareUpstreamError = message;
+      options?.onParsedPayload?.(payload);
+    },
     onFirstOutput,
     pinCompletedResponseIdToFirstSeen: options?.pinCompletedResponseIdToFirstSeen,
   });
@@ -1372,7 +1407,11 @@ export function consumeForInspection(
     onCleanEof: () => {
       if (!inspector.reported()) {
         if (logCtx) logCtx.terminalSource = "synthetic";
-        onTerminal("incomplete");
+        if (bareUpstreamError !== undefined) {
+          onTerminal("failed", httpStatusForRequestLogTerminal("failed", logCtx));
+        } else {
+          onTerminal("incomplete");
+        }
       }
     },
     onReadError: () => {