support Responses multi-agent orchestration over HTTP and WebSocket

[maralbahari]

Problem statement / motivation

Implement the Responses API multi-agent beta from the agentic-api gateway/server perspective. To the client, agentic-api must supply the hosted collaboration, continuation, streaming, and compaction behavior that OpenAI supplies. A root agent delegates to isolated subagents and synthesizes their output; the gateway owns that orchestration over its inference backend.

Today, agentic-api can execute multiple gateway-executed built-in tool calls concurrently and multiplex independent WebSocket responses. Neither mechanism supplies subagent contexts, collaboration actions, or an agent tree that survives continuation requests.

For example, a review task should let /root/correctness and /root/tests independently inspect code. When correctness requests a client-executed function, tests should continue. Over WebSocket, correctness should resume after its function output is injected, while tests is still working. Over HTTP, the response should return once the tree has finished or reached the client-tool continuation boundary, and the next request should resume the appropriate agents.

Existing concurrency versus subagents

Concern	Current parallel tool execution	Proposed multi-agent execution
Concurrent work	Calls emitted by one model inference round	Independent agent turns, each containing inference rounds and tool execution
Context	One enriched request/item history	Separate history and compaction state for each agent
Coordination	Execute a batch, collect outputs, then decide whether to infer again	Spawn, message, assign follow-up work, wait, interrupt, and inspect the tree
Scheduling	join_all and a semaphore local to one gateway round	A coordinator schedules agent turns across the whole tree; each turn reuses the gateway scheduler
Limits	Gateway execution concurrency and same-tool exclusion	Separate limits for active subagents, inference work, tools, mailboxes, and retained state
Client tools	The current engine returns unresolved client calls after processing the round's gateway calls	Only the calling agent pauses; response-level completion depends on all relevant agents and the transport continuation policy
Completion	One tool-loop decision finalizes the public response	Agent-turn completion and public-response completion are separate decisions
WebSocket lanes	Independent response requests, FIFO within a stream_id lane	Multiple agents inside one response; a lane is not an agent identity

parallel_tool_calls is a model-generation preference. In this checkout it defaults to false when omitted; it does not turn the gateway scheduler on or off. Keep it independent of multi_agent.enabled.

Proposed solution

Rust Cookbook design patterns

Apply message passing between state owners, illustrated by Pass data between two threads. The proposed Tokio adaptation sends a typed injection command through bounded mpsc and receives the coordinator's decision through oneshot. A control-response task delivers the decision through the shared relay while the socket reader continues.

Unlike the recipe's unbounded example channel, require entry/byte limits, cancellable waits, and defined overload behavior. Keep direct state mutation inside the coordinator. The acknowledgement protocol and terminal ordering are gateway requirements layered on the message-passing pattern, not features supplied by the Cookbook recipe.

Implement gateway multi-agent over HTTP JSON/SSE and OpenAI-compatible WebSocket injection over the same runtime, for store: true requests only. Each sub-issue includes reference recording, gateway recording, offline replay, tests, and client examples.

The guide's Python/JavaScript snippets are client request examples, not the gateway's orchestration implementation. Use them as scenario drivers and sources of prompts/tool fixtures for cassettes. The client executes its own function tools and submits outputs; the gateway executes hosted collaboration actions, maintains agent state, and compacts contexts. Adapt the examples to store: true with previous_response_id continuation and run that adapted workflow against both services. Multi-agent with store: false is intentionally unsupported by the gateway; document this difference from OpenAI explicitly.

Build on PR #274's merged unified pipeline and bounded delivery. #244 is still under development and is not in main; its proposed StreamRelay is a pending dependency for the multi-agent delivery integration. Existing parallel tools remain reusable within each agent turn.

Sub-issues

The following sub-issues will be opened after this parent issue.

Issue	Complete deliverable	Depends on
MA-01: Gateway multi-agent over HTTP	Typed protocol, coordinator/actions, shared limits, attributed JSON/SSE, durable tree continuation, automatic per-agent compaction, and paired HTTP cassettes	Merged #274; required #244 relay work is under development
MA-02: WebSocket injection and conformance	Persistent duplex recorder, live injection, completion/compaction races, cross-transport continuation, paired WS cassettes, and release qualification	MA-01; OpenAI WS characterization can start earlier

flowchart LR
    A[OpenAI HTTP reference recordings] --> B[MA-01: Complete gateway HTTP JSON/SSE]
    R[PR 274: merged pipeline] --> B
    P[Issue 244: relay under development] --> B
    W[OpenAI duplex WS reference recordings] --> C[MA-02: WebSocket injection and conformance]
    B --> C

Loading

Cassette recording is part of the implementation

Follow the repository's cassette recorder workflow. Extend the existing recorder/scenario scripts. Do not hand-author or edit captured YAML to manufacture expected behavior.

Each scenario needs:

1. A real OpenAI reference exchange: run the guide-derived beta client against a supported OpenAI model. Capture actual request headers/bodies, responses/events, status/error fields, and WS frame direction/close behavior where relevant. This establishes observable reference behavior.
2. A real gateway exchange: run the same public workflow through agentic-api over its intended model backend. Keep prompts, tool definitions, and deterministic client-function outputs equivalent; record intentional model/endpoint differences. This exercises the gateway implementation rather than passing the request to OpenAI's hosted coordinator.
3. Executable regression evidence: compare semantic contracts and replay captured model/tool dependencies while running the actual Rust gateway. Downstream gateway response cassettes alone are not an execution regression test. For concurrent inference, preserve enough per-agent/round association in the test harness to route recorded dependencies without depending on accidental completion order.

Compare supported store: true scenarios for parity. For store: false, record the gateway's explicit rejection as an intentional difference; do not require its result to equal OpenAI's.

The two runs need not produce identical natural-language text, agent choices, generated IDs, encrypted bytes, token counts, or cross-agent interleavings. Compare required shapes, identity relationships, item/event attribution, per-item lifecycle order, status/continuation rules, function-output routing, and supported errors. Normalize nondeterministic identifiers consistently so call/response references remain testable. Preserve captured event order; do not sort away ordering bugs or compare absolute token usage across different models.

Pin recording date, SDK/beta version, models, prompts, fixtures, callback implementation, transport, settings, and exact recorder command. Sanitize credentials using the recorder while preserving required beta/protocol fields and replayable synthetic-data payloads. Review re-recording diffs and explain changed behavior instead of accepting them automatically. Live recordings require actual service/model access; missing access is a tracked evidence gap, not permission to replace reference output with a mock.

Alternatives considered

No response

Additional context

Guide-derived scenarios

Scenario	Client driver / prompt basis	Required captures
H01: Delegated review	Quickstart's three review responsibilities with a fixed synthetic diff; verify root final_answer selection	OpenAI + gateway, JSON and SSE
H02: Proposal function tools	Adapt the HTTP guide's alpha/beta comparison and get_proposal callback to store: true and previous_response_id	All-agent calls, all pending outputs, subsequent stored requests, opaque items, final response
H03: Stored continuation	Adapt the same tool workflow to previous_response_id, with a separate branch scenario	Paired stored continuation and state-isolation evidence
H04: Collaboration and built-in tools	Focused prompts exercising nested delegation, messaging, follow-up, wait, interruption/listing, and available built-in tools	Verify actions actually occurred; a prompt requesting an action is not evidence that it happened
H05: Validation	Beta/configuration combinations, invalid function outputs, and unsupported store: false	Match supported reference errors; record the storage restriction as an intentional gateway difference
H06: Compaction	Extend H01/H02 with deterministic long context and supported threshold overrides	The compaction cases described below, with limits of observability documented
W01: Immediate injection	WebSocket guide's proposal tool loop; outputs returned as available	Persistent duplex exchange, acknowledgement, same-response agent resumption
W02: Late and invalid injection	Delayed output, completed/not-found response, malformed schema, and continuation fallback	Both frame directions, late acknowledgements, returned input, errors and close behavior
W03: Compaction with injection	Tool-output delays combined with long contexts and per-agent compaction scenarios	Observable state/event behavior plus deterministic gateway race tests

Start with the guide examples, then add fixed prompts that request concrete independent tasks when coverage requires delegation. Tool outputs should be deterministic functions of actual arguments, not fabricated model responses. No scenario is considered covered merely because the final answer looks plausible. Unreproducible internal races belong in deterministic gateway tests; keep those fixtures distinct from live cassettes.

Recorder gaps to close

Add beta multi-agent request/header support and stored previous_response_id continuation in MA-01. Preserve existing argument-aware function callbacks for proposal outputs. General stateless item-replay support is outside this RFC; offline cassette replay remains required.

The current WebSocket recorder opens one connection per request, forces store: true, and stops at the terminal response. MA-02 must preserve a persistent connection, send and record injections, and drain acknowledgements after response completion. Successful multi-agent scenarios use store: true. The recorder must also preserve an explicit store: false for rejection tests instead of silently changing it to true. A receive-only transcript or derived SSE projection cannot prove duplex injection behavior.

Compaction and compatibility boundaries

With multi_agent.enabled: true, the OpenAI contract enables automatic compaction even without explicit context-management settings and applies it separately to root and descendants. Explicit /responses/compact, reasoning.summary, and max_tool_calls are unsupported in that mode. Record exact accepted schemas and rejection envelopes; do not silently ignore those parameters.

Required compaction coverage includes default/overridden thresholds, independent contexts, pending calls, messages arriving during compaction, forks/interruption, repeated checkpoints and stored restoration, tool/instruction state, failed/oversized summaries, configuration changes, unsupported combinations, event ordering, and usage. WebSocket coverage also includes injection before/during/after compaction, late acknowledgements, disconnect, and durable cross-transport continuation.

Compaction must preserve the owning agent's effective state and pending relationships. A single global prune of a mixed-agent transcript can discard another agent's context. Preserve updates after a summary snapshot, commit the correct context generation, and retain the last valid checkpoint on failure. Preserve opaque public items and the stored canonical context needed to resume each agent; client-carried stateless checkpoint restoration is outside this RFC. Byte equality or cross-provider decryption is not a compatibility requirement. Different model tokenizers do not imply identical trigger timing.

Documented wire semantics and recorded observations define compatibility. Typed state machines, shared budgets, validated stored snapshots, snapshot generation checks, and task ownership are gateway implementation choices. Record uncertain behavior before treating an existing single-agent gateway rule as an OpenAI multi-agent rule. Any deliberate deployment restriction is a documented difference, not silently advertised parity.

The core admission path must reject effective multi-agent execution with store: false before starting inference, tools, or state mutation, including continuation of an existing stored tree. Apply the same rule to HTTP and WebSocket create requests; never silently force storage on or downgrade to single-agent execution. Requests with multi-agent disabled retain existing storage behavior.

Scope is gateway-executed orchestration for Responses over HTTP and WebSocket with store: true. Stateless multi-agent requests, client-carried full-history reconstruction, and transient multi-agent continuation are outside scope. Anthropic Messages orchestration, arbitrary per-agent model selection, background recovery after process failure, and cross-provider encrypted checkpoint interchange remain separate work.

Primary contract: OpenAI Responses multi-agent guide, rechecked September 14, 2026. General compaction context: OpenAI compaction guide; multi-agent-specific behavior takes precedence. Streaming foundation: PR #274, implementing #243 under #241, with #244 still under development and not in main. Each child issue retains cited Rust Cookbook patterns and concrete Tokio adaptations.