implement OpenAI-compatible multi-agent Responses in the gateway over HTTP

[maralbahari]

Problem statement / motivation

agentic-api is the Responses API server seen by the client. It must implement the hosted collaboration behavior supplied by OpenAI: agent creation and scheduling, isolated contexts, messages, tool routing, completion, state, and compaction. Returning collaboration actions for the client to execute does not provide this compatibility.

The guide's HTTP examples are client request/continuation drivers. Use their prompts, tool declarations, and deterministic tool callbacks to record real reference exchanges. The gateway implementation remains Rust code over the existing core executor.

Proposed solution

Rust Cookbook design patterns

Use scoped task ownership, inspired by Spawn a short-lived thread, for the coordinator's child lifetime. Adapt it with an owned Tokio JoinSet, explicit cancellation and joined teardown; Tokio tasks require owned/shared 'static inputs and do not inherit scoped threads' borrowing guarantees. Keep ingestion inline.

Use the bounded pipeline pattern for the common relay, with async channels and byte budgets. Apply the typed JSON decoding pattern to snapshot envelopes, followed by fallible domain validation. These are gateway implementation adaptations; serialization alone does not supply domain validation, transactional commit, or semantic compatibility.

Start with the observable contract and HTTP recorder

Extend the existing tests/cassettes/record_cassette.py and scenario-script workflow to send and capture the multi-agent beta header and typed request settings with store: true. Adapt the guide's proposal workflow to stored previous_response_id continuation and use the same adaptation against OpenAI and the gateway. General full-item stateless reconstruction is outside scope; offline cassette replay remains required.

Add a reproducible record_multi_agent_cassettes.sh scenario entry point, with separate OpenAI-reference and gateway targets. This is a proposed script to implement. Use the recorder's argument-aware Python callbacks for get_proposal, since two calls to that same function can require different outputs. A mapping based only on the function name is insufficient.

Record OpenAI first to pin the public request/item/event/error behavior, then run the same scenario through agentic-api. Pin the SDK/beta version, model, prompts, input/tool fixtures, transport, and recording command. Preserve all output items in recordings and stored history; client code handles every function_call regardless of originating agent and never supplies outputs for multi_agent_call.

Typed protocol and request admission

Add typed MultiAgentConfig, agent attribution, message phase, collaboration actions, and stored input/output forms of multi_agent_call, multi_agent_call_output, and agent_message. Preserve action/call IDs, author/recipient, and opaque content. For agent messages, attribution identifies the recipient. Preserve top-level agent attribution on applicable events and omit it on overall response lifecycle events.

Parse OpenAI-Beta: responses_multi_agent=v1 at HTTP admission and pass typed feature information into core. Multi-agent requires store: true and in-process execution. Reject store: false before inference, tools, or state mutation, including requests continuing a stored multi-agent tree; never silently force storage on or disable multi-agent. Validate the effective storage policy after resolving request/default semantics. Unsupported split-execution routes also fail explicitly. Keep parallel_tool_calls independent of agent admission and gateway tool scheduling. This storage restriction is an intentional difference from OpenAI.

Characterize omitted, null, disabled, invalid, and unsupported settings against OpenAI. Pin the exact beta schemas, parameter inheritance, function-output validation, and error envelopes before claiming parity. Core configuration may impose explicit deployment limits, but the public contract must not silently substitute gateway defaults for reference behavior.

Coordinator above the existing tool loop

Extract a resumable agent turn from EngineOrchestration; reuse inference, typed round ingestion, translation, tool normalization/registry, and GatewayScheduler. Do not recursively invoke public ExecuteRequest::run() for children, because it owns public-response finalization, session leasing, and persistence.

A core MultiAgentRun owns agent state, pending calls, mailboxes, run limits, usage, and public completion. Each agent owns its context and turn state. Use typed states for runnable, inferring, executing tools, waiting for client outputs/messages, idle, interrupted, and failed. These are internal states, not invented public status strings.

Implement spawn_agent, send_message, followup_task, wait_agent, interrupt_agent, and list_agents as gateway-hosted collaboration actions. Normalize model-visible schemas through the existing tool seam, reserve their names, and route them to the coordinator. Spawning returns after admission rather than waiting for the child's answer. Messaging does not start an idle turn; follow-up work does. Interrupt preserves context. Deliver child-turn completion to its parent once.

Inject the documented collaboration instructions into root/child context and implement the pinned fork semantics. Descendants inherit the request's model and tool availability, while post-fork histories and mutable tool-search visibility remain separate. max_concurrent_subagents defaults to three and counts descendants across the tree, excluding root. Record unresolved details of suspended-turn accounting; do not invent fixed protocol limits on depth or lifetime agent count.

Provide typed run-control and snapshot export/restore seams for MA-02. Core owns function-output acceptance and serialization against finalization. Register pending calls before making their completed call items visible; return an acceptance decision only after validation/state mutation. Stabilize globally unique call identities and owner mappings for JSON, SSE, persistence, and injection.

One attributed stream for HTTP SSE

Extend the single events/ → synchronous round ingestion → translation path with the new item kinds. Use separate instances of the existing state machine for concurrent agent rounds, with one implementation of lifecycle validation and delta folding.

Once the required #244 relay work lands, extend its delivery boundary for one public response. The StreamRelay interfaces below describe the proposed target, not the current main implementation. Map (agent, round, local output index) to stable public indexes and use the same positions in final output assembly. Stamp sequence_number at emission, after deferral. Preserve source-local item lifecycle order; cross-agent events may interleave. A source's hidden gateway call must not defer unrelated agents' output.

Keep item assembly in ingestion/engine and delivery/presentation in the relay. Preserve origin-specific rewriting, bounded client/deferred queues, size checks, and source-scoped teardown. Publish one overall terminal lifecycle event after state commit. Design relay lifetime so MA-02 can deliver late injection acknowledgements after that event. Retain #274's existing bounded delivery; this issue does not reopen worker placement under #245.

HTTP completion and continuation

When one agent calls a client function, pause that agent and let other runnable work continue. HTTP JSON/SSE ends when active agents have finished or reached the client-function boundary. Mailbox waits dependent on blocked children must not prevent completion indefinitely. Response completion with pending functions does not mean the overall task is finished.

Return all outstanding calls. On continuation, restore the tree and route function outputs by call_id to their owners. Follow the guide's client loop, which submits all pending outputs; capture reference behavior for partial, duplicate, unknown, or mismatched outputs instead of treating existing single-agent validation as automatically normative for multi-agent.

Extend checkpoints/storage through executor mode handlers with tree identity, effective configuration, per-agent histories, mailboxes, pending-call ownership, and effect-delivery cursors. Support durable previous_response_id for HTTP and MA-02's WebSocket continuation. Branching creates an independent stored tree snapshot; restoration must not repeat completed collaboration/tool effects. Multi-agent transient sessions and client-carried stateless reconstruction are outside scope.

Reserve retained-state capacity before writing. Persist response/tree state atomically, preserve conversation version checks and credential sanitization, and publish checkpoints before terminal delivery. Decode versioned snapshots into validated domain state before constructing live tasks. Retain canonical agent context in durable storage and preserve opaque public item representations. No client-facing stateless checkpoint codec is required. Provider ciphertext remains opaque; compatibility does not imply cross-provider decryption or identical encrypted bytes.

Alternatives considered

No response

Additional context

Automatic compaction when multi-agent is enabled

The multi-agent limitations establish that automatic compaction is implicit, applies independently to root and descendants, and permits an explicit threshold override. Explicit /responses/compact, reasoning.summary, and max_tool_calls are unsupported in this mode.

Implement those rules in the gateway. Do not depend on the client issuing compact requests, configuring a threshold, or maintaining child contexts. The existing compactor only runs with an explicit threshold and operates on one history; adding it unchanged to the tree is insufficient.

Pin the exact request schema and observable behavior through reference recordings. The general compaction guide provides useful context, but does not establish how every multi-agent compaction event or pruning operation behaves. In particular, never apply a single global “latest compaction item” cutoff to a mixed-agent transcript without proof that it preserves every agent's continuation state.

Case	Required gateway behavior / verification	Evidence
C01: Enabled with store: true and no context_management	Automatic compaction remains available for root and descendants; no client compact call is needed. Characterize empty/null policy values separately.	Reference and gateway long-context recordings; deterministic threshold tests
C02: Explicit threshold and invalid values	Apply the supported override independently. Record validation and default/inheritance behavior; test below/at/above the internal threshold without assuming identical token counts across models.	Paired request/error captures and gateway boundary tests
C03: Root only, child only, or several agents compact	Replace only the corresponding agent's context; retain parent/sibling state and source attribution. Concurrent compactions share resource budgets.	Targeted delegation recordings plus deterministic scheduler tests
C04: Pending client/gateway calls	Keep unresolved call identity, arguments, owner, and eventual output resolvable. Compact a safe prefix or defer that agent's compaction; do not summarize away a pending-call relationship.	HTTP continuation recordings; controlled pending-call tests
C05: Mail, child completion, or follow-up arrives during compaction	Preserve updates appended after the summary snapshot. Commit against the correct context generation and deliver messages once.	Deterministic race tests; externally visible reference observations where obtainable
C06: Fork, interrupt, resume, or branch around compaction	A fork sees one consistent effective window; cancellation cannot replace newer context with a stale summary. Parent/branch state stays isolated.	Fork/continuation recordings and cancellation tests
C07: Repeated compactions and stored restoration	Restore the right window per agent from durable state, preserve opaque items, and never rerun old spawn/tool effects. Continuation across requests/connections uses previous_response_id.	Paired stored multi-request recordings and snapshot validation
C08: Tool metadata, instructions, and reasoning	Retain effective tool-search/MCP state and required instructions without duplicating injected instructions; preserve supported opaque reasoning without requesting a reasoning summary.	Mixed-tool recordings and model-input assertions
C09: Summary fails, is oversized, times out, or commit fails	Preserve the last valid state, release permits, and surface a defined failure/incomplete result; no silent history truncation or successful terminal publication.	Fault injection; record reference wire behavior where observable
C10: Mode/configuration changes on continuation	Record enabled→omitted/disabled and changed threshold/model/settings behavior; match supported acceptance/rejection without merging incompatible histories. A continuation with store: false is rejected under the explicit gateway restriction before altering the stored tree.	OpenAI characterization and paired gateway errors
C11: Forbidden compact/summary/tool-limit combinations	Reject according to the recorded supported negative requests and error schema. Do not invent a multi_agent field on the standalone compact endpoint just to construct a test.	Reference negative cassettes and gateway validation
C12: Stream and usage accounting	Preserve the observed compaction item/event lifecycle and attribution; report current-response inference/compaction usage once. Previously reported continuation usage is not counted again.	HTTP SSE captures and usage/state assertions

Rows describing internal concurrency are gateway invariants, not claims that OpenAI exposes its internal scheduler. MA-02 repeats the relevant cases with function-output injection in flight. Unobserved compaction trigger timing and opaque contents must remain labeled as unobserved, not filled in with fabricated cassettes.

Resource and failure ownership

Share a run-level tool budget so separate round-local semaphores do not multiply effective concurrency without limit. Preserve materialization budgets and same-tool exclusion for shared resources. Bound agent admission, mailboxes, pending calls, controls, output, contexts, and total runtime independently of queue entry counts. Keep physical inference permits separate from logical agent admission; no inference/gateway permit is held across a mailbox wait.

Child failures become parent-visible outcomes under the pinned behavior; root/invariant/persistence failures terminate appropriately. Root finalization and disconnect cancel/settle and join remaining work, release leases, and prevent late publication. Interruption cannot roll back external tool side effects and must not trigger their automatic replay. Aggregate usage across agent inference and compaction once per response.