fix(executor): account logical retained bytes instead of raw SSE line bytes in response budget (#288)

[Zheng-Lu]

Summary

Fixes #288.

Problem

Previously in crates/agentic-server-core/src/executor/upstream.rs, fetch_stream_payload charged raw wire bytes of every upstream SSE line (response_budget.consume(line.len())) before semantic accumulation. Because each line includes transport framing (data: , JSON delimiters, repeated IDs, and redundant completion snapshots), fine-grained chunking (e.g. 1-byte deltas) exhausted the default 1 MiB cumulative response budget after only ~7,655 bytes of answer text. In contrast, coarser deltas or non-streaming JSON succeeded with the identical answer text. Furthermore, the 256 KiB SSE line cap, 1 MiB JSON body limit, and 1 MiB stream event cap were tightly coupled and could cause late stream failures at the terminal boundary.

Solution

1. Logical Retained Accounting (Chunk-Invariant):

   • Removed raw wire-byte charging per SSE line in fetch_stream_payload.
   • Measured logical retained output in accumulator slots (SlotMap::apply / ActiveItem::apply_event) and container overhead across rounds.
   • For streaming deltas, charged incremental text/argument lengths immediately.
   • For done payloads (TextDone, ReasoningTextDone, FunctionCallArgsDone, etc.), charged unstreamed bytes immediately upon reception.
   • Preserved drain_output_with_budget as a final reconciliation safety net.
   • Verified that 1-byte chunks, 1024-byte chunks, and non-streaming JSON consume identical logical budget.

2. Decoupled & Configurable Resource Limits:

   • max_retained_bytes (AGENTIC_MAX_RETAINED_RESPONSE_BYTES, default 8 MiB): cumulative logical retained output across rounds.
   • max_upstream_json_bytes (AGENTIC_MAX_UPSTREAM_JSON_BYTES, default 16 MiB): bounds a single upstream JSON response body.
   • max_upstream_sse_line_bytes (AGENTIC_MAX_UPSTREAM_SSE_LINE_BYTES, default 16 MiB): bounds a single upstream SSE line buffer, scanned in linear $O(N)$ time.
   • max_stream_event_bytes (AGENTIC_MAX_STREAM_EVENT_BYTES, default 16 MiB): bounds a single serialized client stream event (accommodating full output_item.done and response.completed frames).
   • Enforced validation: wire limits must exceed max_retained_bytes by proportional wire headroom (max(64 KiB, max_retained_bytes / 4)) to account for JSON escaping, schemas, and message envelopes.

3. Synchronous Ingestion & State Machine Hardening:

   • Introduced typed pub(super) struct StreamedPart { text: String, streamed: bool } in slot.rs and MergeDone in completion.rs, cleanly separating multi-part stream tracking without data loss.
   • Preserved stream_lifecycle = AwaitingCreated on empty-ID response.created events to guard strict-mode transitions.
   • Accounted for WebSearchCall.action, McpCall.output/error, McpListTools.tools schemas, and ReasoningOutput.encrypted_content.
   • Introduced strongly typed ExecutorError::ResourceLimitExceeded { limit: ResourceLimit, max_bytes: usize }.
   • Documented the [responses] configuration and streaming in-flight memory sizing guide in README.md.

---

Test Plan

1. Reproduction Script & Live Binary E2E:

   • Verified all 4 scenarios from the issue's public_repro.py against the compiled agentic-server binary and mock upstream:
     • size=8000, chunk=1 (1-byte deltas): received all 8,000 bytes with response.completed and 0 errors (previously failed after 7,655 bytes with HTTP 500 budget exceeded).
     • size=8000, chunk=1024: passed with 8,000 bytes and response.completed.
     • size=220000, chunk=1024: received all 220,000 bytes with response.completed (previously rejected by terminal snapshot inflation).
     • size=270000, chunk=1024: passed under the new 16 MiB line limit (previously exceeded 256 KiB line cap).
     • 2,000,000 bytes non-streaming JSON: passed with HTTP 200 completed.

2. Unit & Invariance Tests:

   • small_and_large_sse_chunking_and_json_produce_identical_retained_bytes: asserts byte-identical budget.used() across 1-byte chunks, 1024-byte chunks, and JSON.
   • repeated_completion_snapshots_do_not_double_count_retained_bytes: verifies terminal snapshots do not inflate budget.
   • lenient_done_without_output_item_done_exceeding_budget_fails_promptly: asserts prompt failure directly on output_text.done.
   • two_part_done_only_stream_preserves_both_parts: asserts both parts are preserved and matches delta-stream budget.
   • responses_config_validation: tests valid configurations, boundary rejection, and proportional headroom limits.
   • retained_accounting_for_web_search_mcp_and_reasoning: asserts accurate estimation for extended items.

3. Workspace Suite & Linters:

   • cargo test --workspace (100% pass: 697 core library tests + 91 server tests + all replay cassettes).
   • cargo clippy --workspace --all-targets -- -D warnings (clean, 0 warnings).
   • cargo fmt --all -- --check (clean).

[maralbahari]

@Zheng-Lu please update the ARCHITECTURE.md based on the refactoring work.

[maralbahari]

Value::String("") currently contributes zero bytes, while Value::Array charges only one container plus its children’s measurements. An array containing 100,000 empty strings therefore measures just 32 bytes, despite retaining 100,000 Value entries.

this can be reproduced through both JSON and streaming: an annotation containing that array completed successfully with max_retained_bytes = 4096. The returned output was approximately 300 KB.

Please charge structural overhead per retained JSON value or collection entry, including empty strings, and add regression tests for this nested-array case on both ingestion paths. Final reconciliation currently uses the same measurement, so it cannot catch the bypass.

[maralbahari]

The RetainedSize documentation excludes type, role, and status because they are supposedly bounded by enums. However, fields such as OutputMessage.role and OutputTextContent.type_ are unrestricted Strings, and the live executor’s lenient ingestion accepts and retains arbitrary values.

this can be reproduced through both JSON and streaming: a message with a 100,000-byte role, and separately a content part with a 100,000-byte type, each completed successfully with max_retained_bytes = 4096.

Please include unrestricted strings in retained measurements, or enforce a bounded vocabulary during deserialization. Audit the other excluded fields for the same assumption and add regression tests on both ingestion paths. Accounting should only omit fields whose bounds are enforced by their owning types.

[maralbahari]

Each ReasoningTextDone calls RetainedAccount::grow with text_retained_bytes, which scans all accumulated content, summaries, and tracking maps before and after the mutation. ReasoningSummaryTextDone follows the same pattern.

Completing N separate parts therefore performs O(N²) accounting work, even when each completion simply appends a small part. This processing happens synchronously in ingestion.

A synthetic reproduction with sequential one-byte reasoning parts showed:

• 4,000 parts: 98 ms previously → 302 ms after this change.
• 8,000 parts: 197 ms previously → 956 ms after this change.
• 16,000 parts: 394 ms previously → 3,421 ms after this change.

These are debug-build measurements, including parsing; they demonstrate the scaling regression rather than production latency.

Please measure only the affected part, or maintain cached totals updated through the existing completion operation. Keep completion behavior in one place so accounting does not duplicate its reconciliation rules, and reserve full-item measurement for final reconciliation.