Prevent oversized FIFO failures and preserve rejected drafts

[seanzqliang]

Summary

Fix two connected bugs: an oversized inline message can fail a durable session while writing its FIFO, and later rejected sends can remain forever pending in the portal/TUI.

Failure Evidence

A reported session's Duroxide history ended with:

OrchestrationFailed
KV value for key 'fifo.1' (590295 bytes) exceeds limit (...)

The session then refused new messages as a terminal orchestration, while the UI continued showing those requests as pending. The earlier idle affinity release and completed model turn were not the failure cause. Deployment identity and prompt contents are intentionally omitted.

Root Causes

1. FIFO rollover checked the combined bucket but wrote an oversized individual item into the next bucket without validating that item. It also counted JavaScript string length rather than serialized UTF-8 bytes. The buffer-capacity fallback could leave dequeued work unaccounted for.
2. SDK message-send paths had no inline-envelope admission limit. Browser-only validation would not protect direct SDK, CLI, MCP, or management callers.
3. The shared outbox treated terminal-session failures like transient transport errors, retained them as pending, and scheduled another send. It also needed a durable rejection receipt that wins over late enqueue acknowledgements.

Solution

• Add a shared 12 KiB serialized UTF-8 envelope limit for prompts and answers, including metadata, attachment references, and JSON escaping. Reject before enqueueing or marking the session running. Export MAX_MESSAGE_BYTES and MessageTooLargeError; HTTP/RPC preserve MESSAGE_TOO_LARGE and status 413 with artifact-reference guidance.
• Introduce orchestration 1.0.79. Validate individual FIFO items and combined buckets against their separate 14 KiB budget. Reserve buffer capacity while draining, leaving excess messages on the incoming durable queue. Preserve wait/child-digest state if runtime augmentation makes a prompt oversized.
• Persist session.message_rejected with the error and contributing message IDs when work is refused inside the orchestration. Rejected prompts are not acknowledged as processed or inserted into the duplicate-suppression receipt set. The session can process a later valid request.
• Stop retrying terminal/oversized refusals. Retain the original draft and display its Not sent reason. Reconcile rejection receipts through live and bulk history, and prevent a late enqueue acknowledgement from restoring queued state.
• Provide portal Recover rejected prompt, explicit resend with fresh IDs, and local dismissal; keep native TUI prompt recall and shared rejection semantics aligned. Surface durable rejection reasons in Activity.
• Update API, runtime, contributor, builder, sample, and Unreleased documentation.

Latest Follow-up (3bc2b11)

• Address both review comments: correct the frozen v1.0.78 entrypoint header and remove the import comma spacing. These are non-executable changes.
• Reclaim stranded FIFO head capacity before the cancellation sweep when tail capacity is exhausted. Compaction preserves item order, scans at most 20 buckets, and adds no external I/O.
• Validate oversized pre-dispatch prompts before tracking their client IDs, so rejected input cannot suppress a later valid envelope with a never-accepted ID.
• Guard outbox rollback against current state: failed cancellation restores only its still-cancelling item, preserving durable rejection/acknowledgement and newer drafts. Delayed send errors may change only the still-pending attempted item and cannot replace an authoritative rejection reason.
• Add seven regression cases in existing suites. The follow-up changes eight already-included files; total PR scope remains 40 files: 8 frozen replay modules, 9 SDK/runtime and version-wiring files, 4 UI files, 7 tests, and 12 documentation/maintenance files. See the full review and file-by-file rationale.

Compatibility And Safety

• Freeze 1.0.78 and keep it registered with its required CURRENT_ORCHESTRATION_VERSION pin. The follow-up changes only its header/import formatting. All eight frozen modules match the pre-PR executable implementation after that pin and checkout line-ending normalization. The new handler is 1.0.79; older historical handlers and shared activity serialization are unchanged.
• Existing in-flight executions retain their handler until normal versioned continuation. The new send-admission policy protects new public sends, but old executions do not retroactively gain the new FIFO code.
• Cancellation remains bounded by FIFO capacity and the pre-dispatch sweep limit. A fully occupied buffer with no reclaimable bucket, or a tombstone beyond available sweep capacity, can still dispatch work before reading that cancellation. This is best effort, not a priority control channel or cancellation of already-dispatched work.
• This is a PilotSwarm fix, not a Duroxide limit increase or a downstream application workaround.
• No silent prompt truncation, automatic artifact creation, splitting a single request into separate turns, database reset, or automatic revival of already-failed sessions.
• No dependency/package release-version changes, credential/config edits, downstream updates, tags, publishing, or deployment. Retained UI drafts remain local UI state; durable rejection diagnostics are stored as session events.

Validation

• Reproduced the original fifo.1 overflow and permanent-send pending-state failures before applying the fixes.
• Full workspace build passed: SDK, Horizon store, portal, and MCP. The SDK was rebuilt after the final FIFO correction. Vite reports empty-chunk and bundle-size warnings.
• Current follow-up (3bc2b11): 130/130 passed across all five changed SDK Vitest suites: cancel-pending-orchestration, orchestration-version-upgrade, outbox-bulk-reconcile, prompt-attachments, and session-refresh-ui. Normal parallelism and no retries; includes all seven added regression cases.
• Earlier PR head (5844117), not rerun in full for this follow-up: 135 passed across eight focused Vitest suites covering admission, FIFO byte boundaries/rollover, multibyte/escaped input, capacity backpressure, timer preservation, valid requests after rejection, frozen-version registration/upgrades, outbox recovery, live/bulk receipts, late acknowledgements, and history contracts.
• Earlier PR head (5844117), not rerun for this follow-up: 815 passed in the full shared UI unit suite. Four initial LF-sensitive source-extraction failures were resolved by local CRLF normalization, without weakening assertions or changing those tests.
• Current follow-up: 15/15 passed in the Web API router suite, including a real HTTP/API-client round-trip to the management send method: oversized input returns actionable 413 without enqueue/state mutation, then a short request succeeds. Legacy RPC mapping is covered too.
• Current follow-up: 6/6 passed in the full composer Playwright suite: desktop/mobile rejection recovery, fresh IDs, preserved drafts/reasons, geometry, and Chromium/WebKit composer resizing. Desktop/mobile screenshots were inspected. Browser tests ran on Windows because WSL lacks a Chromium system library.
• Editor diagnostics and git diff --check passed. Frozen-handler content was compared against the base commit.

Required Gate Still Blocked

The follow-up prerequisite check still found no .env, DATABASE_URL, or GITHUB_TOKEN in the test environment. GitHub currently reports no CI checks for this head; the passing results above are local validation.

The canonical full integration gate was attempted before this follow-up with ./scripts/run-tests.sh and stopped before tests with:

ERROR: .env not found. Create it with DATABASE_URL and GITHUB_TOKEN.

Credentialed PostgreSQL/Copilot integration, multi-worker, and chaos/replay execution therefore remain unverified in this environment. Run the full gate in an appropriately configured test environment before approving a release. The generator/HTTP/browser fixtures above are not a substitute for that gate. No credentials were read or created, and no test parallelism or retry policy was reduced.

Review Request

@affandar please review the inline admission policy, FIFO rejection/versioning behavior, and retained-draft UX. This submission is for review only: do not merge, tag, publish, or deploy as part of this request. The full credentialed gate must be completed and Affan's approval confirmed before a subsequent authorized release/rollout.

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

Review tier: Lite
Findings: 1 · 1

Open findings (2)

• The module-level docstring says v1.0.77 but this file defines… · New
• There is an extra space before the comma after validClientMessageIds (validClientMessageIds ,).… · New

What changed in this PR

Introduces explicit message-size admission limits and permanent rejection handling across the SDK, durable orchestration (v1.0.79), and UI, with updated docs and tests to ensure oversized/terminal-session sends are preserved for recovery rather than retried.

Changes:

• Enforce a 12 KiB serialized UTF-8 inline prompt/answer envelope limit (MESSAGE_TOO_LARGE, HTTP 413) in SDK send paths.
• Add orchestration v1.0.79 FIFO validation + durable session.message_rejected receipts and keep v1.0.78 frozen/registered for replay.
• Update portal/TUI outbox UX to retain rejected drafts, display “Not sent” reasons, and prevent automatic retry; expand test coverage and documentation.

File	Description
templates/​builder-agents/​skills/​pilotswarm-sdk-builder/​SKILL.md	Adds SDK builder guardrail guidance for inline message size limits and rejection handling.
templates/​builder-agents/​README.md	Documents the 12 KiB inline message limit and artifact-based large-input workflow.
packages/​sdk/​test/​local/​session-refresh-ui.test.js	Adds UI-controller tests for permanent outbox rejections and explicit recovery behavior.
packages/​sdk/​test/​local/​prompt-attachments.test.js	Adds tests for serialized UTF-8 payload measurement and pre-side-effect oversized rejection across client surfaces.
packages/​sdk/​test/​local/​outbox-bulk-reconcile.test.js	Ensures durable session.message_rejected reconciles correctly via live and bulk event paths.
packages/​sdk/​test/​local/​orchestration-version-upgrade.test.js	Verifies v1.0.78 handler remains frozen and v1.0.79 is registered as latest.
packages/​sdk/​test/​local/​cancel-pending-orchestration.test.js	Adds orchestration-level tests for FIFO-size rejection, state preservation, and queue capacity behavior.
packages/​sdk/​src/​orchestration_1_0_78/​utils.ts	Introduces frozen helpers for v1.0.78 (multi-writer attribution, context usage reduction, retry classification).
packages/​sdk/​src/​orchestration_1_0_78/​state.ts	Adds frozen orchestration state/types/constants for v1.0.78.
packages/​sdk/​src/​orchestration_1_0_78/​runtime.ts	Adds frozen runtime loop and startup gates for v1.0.78.
packages/​sdk/​src/​orchestration_1_0_78/​queue.ts	Adds frozen FIFO/drain/decide logic for v1.0.78.
packages/​sdk/​src/​orchestration_1_0_78/​lifecycle.ts	Adds frozen lifecycle utilities for v1.0.78 including CAN, status, regen pipeline.
packages/​sdk/​src/​orchestration_1_0_78/​index.ts	Adds frozen orchestration entrypoint for v1.0.78.
packages/​sdk/​src/​orchestration_1_0_78/​agents.ts	Adds frozen sub-agent tracking/actions and shutdown cascade for v1.0.78.
packages/​sdk/​src/​orchestration/​queue.ts	Adds FIFO item UTF-8 byte sizing, rejection receipts, and queue-capacity protection in v1.0.79.
packages/​sdk/​src/​orchestration/​index.ts	Bumps orchestration entrypoint to v1.0.79.
packages/​sdk/​src/​orchestration.ts	Re-exports latest durable orchestration entrypoint name/version.
packages/​sdk/​src/​orchestration-version.ts	Updates latest orchestration version constant to 1.0.79.
packages/​sdk/​src/​orchestration-registry.ts	Registers frozen v1.0.78 and latest v1.0.79 handlers.
packages/​sdk/​src/​message-size.ts	Adds shared 12 KiB serialized UTF-8 envelope measurement and MessageTooLargeError.
packages/​sdk/​src/​management-client.ts	Enforces serialized-size admission before enqueue/update and reuses shared serializer.
packages/​sdk/​src/​index.ts	Exports message-size constants/errors as part of the SDK public surface.
packages/​sdk/​src/​client.ts	Enforces serialized-size admission for sends/events and reuses shared serializer.
packages/​app/​web/​test/​e2e/​composer-resize.spec.mjs	Adds E2E coverage for rejected drafts retention/recovery across viewports.
packages/​app/​web/​test/​api-router.test.mjs	Ensures API-router surfaces actionable 413s and maps SDK errors into structured envelopes.
packages/​app/​ui/​react/​src/​web-app.js	Adds portal composer affordances for recovering/resending/dismissing rejected drafts.
packages/​app/​ui/​core/​src/​selectors.js	Adds rejected-reason rendering (“Not sent: …”) in outbox overlay lines.
packages/​app/​ui/​core/​src/​history.js	Surfaces durable session.message_rejected in activity feed.
packages/​app/​ui/​core/​src/​controller.js	Implements rejected outbox phase semantics, reconciliation, and local dismissal/recovery flows.
examples/​horizon-harvester/​README.md	Documents artifact-first workflow and inline message size restrictions.
examples/​devops-command-center/​README.md	Documents artifact-first workflow and inline message size restrictions.
docs/​user-guide/​keybindings.md	Documents rejected-draft behavior and recovery/dismiss UX.
docs/​developer/​building/​builder-agents.md	Adds maintenance guidance for builder templates re: inline message limit behavior.
docs/​architecture/​orchestration/​design.md	Documents FIFO byte-budget semantics and 1.0.79 rejection behavior.
docs/​api/​clients.md	Adds inline message limit specification and describes runtime FIFO rejection receipts.
docs/​api/​building-a-custom-ux.md	Documents recommended client UX for MESSAGE_TOO_LARGE / terminal rejections and durable receipts.
CHANGELOG.md	Summarizes unreleased message-admission + rejection-handling behavior across SDK/runtime/UI.
.github/​skills/​pilotswarm-tui/​SKILL.md	Updates TUI/portal alignment guidance to include rejected outbox state and recovery semantics.
.github/​copilot-instructions.md	Adds repo guidance around message admission and rejected-draft handling.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[seanzqliang]

Follow-up review: 3bc2b11

Pushed to the existing sean/fix-oversized-message-fifo branch. The follow-up changes 8 already-included files (116 additions, 10 deletions); the PR remains at 40 files. The starting worktree was clean and the PR head was rechecked before committing. Existing PR content and newer branch edits were preserved.

Findings and corrections

• Addressed both review comments: corrected the frozen entrypoint header to v1.0.78 and removed the import comma spacing. These edits produce identical executable JavaScript.
• Fixed stranded FIFO head capacity: after popping a head item, a full tail could prevent the cancellation sweep from reading an already-queued tombstone. The live 1.0.79 queue now compacts non-empty buckets, preserving order, only when tail capacity is exhausted.
• Fixed premature duplicate tracking: an oversized pre-dispatch prompt could reserve its client ID and suppress a later valid envelope with that never-accepted ID. Validation now precedes the pending-ID receipt.
• Fixed stale cancellation rollback: a delayed cancel failure restored an old whole-outbox snapshot, undoing durable rejection/acknowledgement and deleting newer drafts. Only the still-cancelling item is now rolled back from current state.
• Fixed delayed terminal send errors replacing an authoritative rejection reason. Send failures may now mutate only the still-pending attempted item. Added seven regression cases to the existing two suites; each newly exposed failure was reproduced before fixing it.

Replay, capacity, and performance

All 8 frozen 1.0.78 modules were compared with the pre-PR live implementation. Executable output matches after the required 1.0.78 version-constant pin and checkout CRLF/LF normalization. Older frozen versions and shared activity serialization were not changed. The registry/upgrade suite passes; this is source and harness verification, not a claim of live durable replay coverage.

Admission stays at 12 KiB of serialized UTF-8 per prompt/answer envelope; each FIFO bucket/item remains bounded at 14 KiB. No silent truncation or increased queue limits. Compaction scans at most 20 buckets, runs only under tail-capacity pressure, preserves item order, and adds no external I/O. Rejected sweep entries no longer consume pending-ID receipts.

Remaining boundary: cancellation is still bounded and best effort. A truly saturated FIFO with no reclaimable whole bucket, or a tombstone beyond available sweep capacity, can still dispatch work before that tombstone is read. This fixes stranded free capacity; it does not introduce a priority control channel or promise cancellation of already-dispatched work. The architecture doc states this explicitly.

Why all 40 files are justified

Paths below use braces to enumerate files under a common directory. The frozen snapshot accounts for most added lines. Documentation files are not runtime dependencies, but are required maintenance parity for this user/builder-facing behavior.

Purpose	Files	Necessity
Frozen replay snapshot (8)	packages/sdk/src/orchestration_1_0_78/{agents,index,lifecycle,queue,runtime,state,turn,utils}.ts	Keep the complete previous implementation and its local helpers available to existing executions.
Version wiring (4)	packages/sdk/src/{orchestration-registry,orchestration-version,orchestration}.ts; packages/sdk/src/orchestration/index.ts	Register frozen 1.0.78 and route new executions to 1.0.79.
SDK admission (4)	packages/sdk/src/{client,management-client,message-size,index}.ts	Apply one serialized-byte rule to SDK send paths, with a shared error and public exports.
Active FIFO (1)	packages/sdk/src/orchestration/queue.ts	Enforce byte-safe rollover, rejection receipts, backpressure, duplicate handling, and bounded cancellation scanning.
Shared outbox (3)	packages/app/ui/core/src/{controller,history,selectors}.js	TUI/portal parity for retained drafts, live/history reconciliation, recovery state, and visible rejection reasons.
Browser recovery (1)	packages/app/ui/react/src/web-app.js	Expose recover/resend/dismiss controls without changing native textarea navigation.
SDK regressions (5)	packages/sdk/test/local/{cancel-pending-orchestration,orchestration-version-upgrade,outbox-bulk-reconcile,prompt-attachments,session-refresh-ui}.test.js	Cover FIFO bytes/capacity/cancellation, registration, send-envelope hygiene, outbox races, and shared UI behavior.
API/browser regressions (2)	packages/app/web/test/api-router.test.mjs; packages/app/web/test/e2e/composer-resize.spec.mjs	Verify real HTTP 413 translation and desktop/mobile retained-draft recovery, new resend IDs, and composer behavior.
Contributor maintenance (2)	.github/copilot-instructions.md; .github/skills/pilotswarm-tui/SKILL.md	Preserve admission/replay requirements and shared UI rejection/rollback invariants in future edits.
Canonical docs (4)	docs/api/{building-a-custom-ux,clients}.md; docs/architecture/orchestration/design.md; docs/user-guide/keybindings.md	Document public limits, durable receipts, bounded cancellation, versions, and recovery gestures.
Builder guidance (3)	docs/developer/building/builder-agents.md; templates/builder-agents/README.md; templates/builder-agents/skills/pilotswarm-sdk-builder/SKILL.md	Keep generated app guidance aligned with artifact-based large inputs and permanent send refusal handling.
Maintained examples (2)	examples/{devops-command-center,horizon-harvester}/README.md	Explain the new input contract where large diagnostic/source documents are likely.
Release notes (1)	CHANGELOG.md	Record the shipped-facing behavior under Unreleased without tagging or publishing.

Validation

• Full monorepo build passed through WSL: SDK, Horizon store, portal, and MCP. Final SDK rebuild after the last queue correction also passed. Vite reports empty-chunk/large-chunk warnings.
• All five changed SDK suites: 130/130 tests passed with normal parallelism, no retries.
• Web API router suite: 15/15 tests passed, including 413 without enqueue/session mutation.
• Browser suite: 6/6 passed on Windows, including 1440px/390px rejection recovery and Chromium/WebKit composer checks. Desktop and mobile screenshots inspected. The initial WSL browser attempt failed before launch because libnspr4.so/WebKit were unavailable; Windows ran the same tests successfully.
• Frozen executable equivalence, touched-file diagnostics, and git diff --check passed.

Unverified integration gates: the full credentialed PostgreSQL/Copilot suite, including live replay/continue-as-new, durability, multi-worker handoff, and crash/chaos coverage. The standard .env is absent, and neither DATABASE_URL nor GITHUB_TOKEN is configured in the test environment. No credential files were created or modified. A configured maintainer environment still needs to run ./scripts/run-tests.sh with its normal parallelism before treating the integration gate as satisfied.

@affandar, please review the follow-up, the 40-file scope rationale, and the remaining saturation/integration caveats. No merge, main push, tag, publish, deploy, or Waldemort modification was performed.