diff --git a/devlog/_plan/260914_provider_parity_stack/050_residuals.md b/devlog/_plan/260914_provider_parity_stack/050_residuals.md index 3efec8081c..72390a38c6 100644 --- a/devlog/_plan/260914_provider_parity_stack/050_residuals.md +++ b/devlog/_plan/260914_provider_parity_stack/050_residuals.md @@ -20,29 +20,19 @@ provider B, and a cache lifetime. `src/responses/reasoning-replay-cache.ts` already solves a narrower version of this inside one provider's session and is the natural starting point. It is a design unit, not a line change. -## R2 — real audio/file transport, and any adapter-level refusal (from F5) - -**F5 is PRESENCE-ONLY and is not fixed.** Phase 4 records that an audio attachment -existed and explicitly does not add audio support. `OcxContentPart` has no audio -member, no adapter consumes one, and per-provider audio capability is not recorded -anywhere in the catalog — `src/providers/registry.ts:1062` notes exactly this when it -omits audio from the Baseten hints. - -Two things are residual, not delivered: - -- **Transport.** A carrier type, capability data across the provider set, and a wire - mapping per vendor. Guessing any one of those produces a request that fails at call - time instead of a modality that works. -- **Refusal.** There is no adapter-level rejection of audio. By final dispatch the part - is already a text marker, so every adapter continues. Doing this properly needs a - typed unsupported-modality signal that survives to final adapter dispatch — including - `runTurn`, compaction and sidecar paths — while raw Responses passthrough stays - untouched. An early throw in the shared parser is not acceptable: raw passthrough - runs through `parseRequest` before the adapter forwards `_rawBody`. - -`input_file` keeps its existing filename-only marker, and Chat inbound has no file or -audio translation at all, so a Chat request can lose media before the Responses parser -sees it. Neither is addressed here. +## R2 — native audio/file transport (from F5) + +Layer 4 added presence markers but could still report successful translation after losing +an attachment. Layer 5 closes that silent-success gap: registered translated adapters inspect +the original content before dispatch, and Chat projection rejects recognized audio/file parts +before losing them. Build, stateful runTurn and local-completion hooks share that contract; +native Responses/Azure and native Chat retain their existing wire behavior. See the +[current registry contract](../../../structure/adapters/registry.md#untranslated-input-media). + +**Native audio/file transport through the normalized IR remains unimplemented.** This stack +does not add a carrier type, per-model capability data, file-ID resolution, URL fetching or +new upstream mappings. Unsupported translation now fails explicitly rather than pretending +to consume an attachment. A filename/audio marker alone is still not the attachment. ## R3 — Kiro remote images stay uninlined diff --git a/devlog/_plan/260914_provider_parity_stack/060_direct_media_guard.md b/devlog/_plan/260914_provider_parity_stack/060_direct_media_guard.md new file mode 100644 index 0000000000..68cffaf137 --- /dev/null +++ b/devlog/_plan/260914_provider_parity_stack/060_direct_media_guard.md @@ -0,0 +1,20 @@ +# Direct implementation: media admission + +This fifth layer follows PR #4539. ChatGPT authored the production changes and regression +tests directly in an isolated worktree, rather than handing this implementation to the prior +native authoring session. The outcome closes the silent-success part of F5, not native media +transport: recognized audio/file inputs either stay on an existing native wire or receive an +explicit conversion error. Legacy function-image conversion also refuses instead of losing +its result. The canonical current contract is in +[adapter registry](../../../structure/adapters/registry.md#untranslated-input-media). + +The pure scanner inspects typed content arrays only. The registry owns final translated +build/runTurn/local-completion admission, and Chat owns rejection before a lossy projection. +No new fetch, decoding, credential access, provider capability declarations or vendor CLI +permissions are introduced. Desired regression coverage includes unchanged native Responses +and Azure bodies, final hook ordering, typed runTurn error, legacy media failure, and real HTTP +rejection with zero upstream sends. Public Pi documentation records the pending behavior. + +The connected Mac runs no product verification by explicit user instruction. Tests are +written for hosted CI; their presence alone is not a passing result. This direct layer does +not reuse another session's PABCD identity or claim unperformed formal phase transitions. diff --git a/docs-site/src/content/docs/guides/pi.md b/docs-site/src/content/docs/guides/pi.md index f44e4be381..d7b9e3c3ed 100644 --- a/docs-site/src/content/docs/guides/pi.md +++ b/docs-site/src/content/docs/guides/pi.md @@ -129,6 +129,32 @@ through, translate it (wire aliases), clamp it to the configured ladder, emulate entirely (e.g. `noReasoningModels`). The boolean only controls whether Pi offers the control at all. +## Attachment and request compatibility + +:::note[Pending development behavior] +The provider-parity changes described here are on the development PR stack; an older installed +release may still have the previous conversion behavior. +::: + +OpenCodex normalizes Pi/MCP and Anthropic-shaped user images before choosing the native Chat +or translated route. Images returned by tools use a translated user-message carrier after the +paired tool results; ordinary user images and text-only tool results can keep the native path. +Use modern `tool_calls` and `role: "tool"` with `tool_call_id`: legacy `function`-result image +translation is rejected instead of silently discarding the result. + +An explicit reasoning effort of `none` survives Chat conversion. Output limits and sampling +controls are preserved for generic API-key Responses targets; the canonical ChatGPT target +still applies its own restrictions. This does not make all providers' controls equivalent. + +**Audio and files need a native input wire that supports them.** OpenCodex does not yet have +a lossless audio/file carrier for translated requests. When Chat requires projection, or a +Responses request targets a translated adapter, recognized audio/file attachments return an +explicit error rather than succeeding without the attachment. File-ID-only images have the +same restriction because translated adapters cannot resolve those IDs. Convert the attachment +to text first, or use a native wire and model that support it. Native Chat and raw Responses +(including Azure) retain their existing behavior; this is not a promise of every model's +upstream media support. Video conversion limits remain adapter-specific. + ## Schema status :::note[Unverified against a real install] diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index a0bc405e36..ecd8ca01a3 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1433,7 +1433,9 @@ "usage-log-ws-stage.test.ts": "usage", "main-device-reauth.test.ts": "codex-integration", "main-device-reauth-api.test.ts": "codex-integration", - "main-device-reauth-ui.test.ts": "gui" + "main-device-reauth-ui.test.ts": "gui", + "adapter-input-media-guard.test.ts": "adapters", + "chat-media-translation.test.ts": "responses" }, "migrated": [ "adapters", diff --git a/src/adapters/input-media-guard.ts b/src/adapters/input-media-guard.ts new file mode 100644 index 0000000000..ad8f468a95 --- /dev/null +++ b/src/adapters/input-media-guard.ts @@ -0,0 +1,45 @@ +import type { ProviderAdapter } from "./base"; +import { untranslatedInputMediaMessage, untranslatedResponsesInputMedia } from "../responses/input-media"; + +/** + * Refuse unrepresentable input at the final translated-adapter boundary. The registry + * applies this after wire resolution; Responses passthrough (including Azure) opts + * out because it uses the original body rather than the lossy normalized content. + */ +export function withInputMediaGuard(adapter: T): T { + const build = adapter.buildRequest.bind(adapter); + adapter.buildRequest = (parsed, incoming) => { + const kind = untranslatedResponsesInputMedia(parsed._rawBody); + if (kind) throw new Error(untranslatedInputMediaMessage(kind)); + return build(parsed, incoming); + }; + + const runTurn = adapter.runTurn?.bind(adapter); + if (runTurn) { + adapter.runTurn = async (parsed, incoming, emit) => { + const kind = untranslatedResponsesInputMedia(parsed._rawBody); + if (kind) { + emit({ + type: "error", + status: 400, + errorType: "invalid_request_error", + code: "unsupported_input_modality", + retryable: false, + message: untranslatedInputMediaMessage(kind), + }); + return; + } + await runTurn(parsed, incoming, emit); + }; + } + + const localTerminal = adapter.localTerminal?.bind(adapter); + if (localTerminal) { + // This hook is outside the builder's error catch. Decline its success shortcut; + // the ordinary buildRequest path then returns the established client-safe 400. + adapter.localTerminal = parsed => untranslatedResponsesInputMedia(parsed._rawBody) + ? undefined + : localTerminal(parsed); + } + return adapter; +} diff --git a/src/adapters/registry.ts b/src/adapters/registry.ts index f9398b4fa8..8d6e8bc63d 100644 --- a/src/adapters/registry.ts +++ b/src/adapters/registry.ts @@ -15,6 +15,7 @@ import { createOllamaNativeAdapter } from "./ollama-native"; import { createResponsesPassthroughAdapter } from "./openai-responses"; import type { OcxProviderConfig } from "../types"; import { createAdapterTierMetadata } from "../providers/fastwire"; +import { withInputMediaGuard } from "./input-media-guard"; export type AdapterCacheRetention = "none" | "short" | "long"; @@ -180,6 +181,9 @@ export function createRegisteredAdapter( const definition = getAdapterDefinition(provider.adapter); if (!definition) throw new Error(`Unknown adapter: ${provider.adapter}`); const adapter = definition.create(provider, context); + if (effectiveAdapterContract(provider.adapter).wire !== "openai-responses") { + withInputMediaGuard(adapter); + } const buildRequest = adapter.buildRequest.bind(adapter); adapter.buildRequest = (parsed, incoming) => { const attachTierMetadata = (request: Awaited>) => { diff --git a/src/chat/inbound.ts b/src/chat/inbound.ts index 6e5538fbe3..bdf7b30551 100644 --- a/src/chat/inbound.ts +++ b/src/chat/inbound.ts @@ -6,6 +6,7 @@ * responsesRequestSchema so routing/OAuth/pool/sidecars are inherited unchanged. */ import { chatImageUrlFromPart } from "./image-parts"; +import { untranslatedChatInputMedia, untranslatedInputMediaMessage } from "../responses/input-media"; export class ChatCompletionsRequestError extends Error {} @@ -277,6 +278,13 @@ function resolveReasoningSummary(raw: Rec): string | undefined { */ export function chatCompletionsToResponsesBody(raw: unknown): Rec { assertChatCompletionsRoutingBody(raw); + // Only the translated path reaches this function. Native Chat can retain its + // provider-specific file/audio blocks; projecting them here would discard them. + const unsupportedMedia = untranslatedChatInputMedia(raw); + if (unsupportedMedia) { + throw new ChatCompletionsRequestError(untranslatedInputMediaMessage(unsupportedMedia)); + } + const systemParts: string[] = []; const input: Rec[] = []; @@ -312,6 +320,16 @@ export function chatCompletionsToResponsesBody(raw: unknown): Rec { if (msg.tool_calls !== undefined) toolCallsToItems(msg.tool_calls, input, knownNameByCallId); break; } + case "function": { + // Native eligibility diverts legacy image results too, but this translator + // has no legacy function_call/name pairing. Never silently discard them. + if (Array.isArray(msg.content) && msg.content.some(part => isRec(part) && imageUrlFromPart(part))) { + throw new ChatCompletionsRequestError( + "Legacy function-result image translation is not implemented. Use tool_calls and role:tool with tool_call_id.", + ); + } + break; + } case "tool": { const callId = typeof msg.tool_call_id === "string" ? msg.tool_call_id : typeof msg.tool_use_id === "string" ? msg.tool_use_id diff --git a/src/responses/input-media.ts b/src/responses/input-media.ts new file mode 100644 index 0000000000..43f8db583e --- /dev/null +++ b/src/responses/input-media.ts @@ -0,0 +1,65 @@ +/** Input kinds for which the normalized request has no lossless content carrier. */ +export type UntranslatedInputMedia = "audio" | "file"; + +type RecordValue = Record; + +function isRecord(value: unknown): value is RecordValue { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function mediaKind(value: unknown): UntranslatedInputMedia | undefined { + if (!isRecord(value)) return undefined; + if (value.type === "input_audio" || value.type === "audio") return "audio"; + if (value.type === "input_file" || value.type === "file" || value.type === "document") return "file"; + // A file-id-only image is not pixels: translated adapters cannot dereference it. + if (value.type === "input_image" && typeof value.file_id === "string" && value.file_id.length > 0 + && !(typeof value.image_url === "string" && value.image_url.length > 0)) return "file"; + return undefined; +} + +function contentMedia(content: unknown): UntranslatedInputMedia | undefined { + if (!Array.isArray(content)) return undefined; + for (const part of content) { + const kind = mediaKind(part); + if (kind) return kind; + } + return undefined; +} + +/** + * Inspect only typed input items and their content arrays, never strings, tool + * arguments, schema properties, or arbitrary nested objects. No payload is copied, + * decoded, fetched or included in the returned value. + */ +export function untranslatedResponsesInputMedia(body: unknown): UntranslatedInputMedia | undefined { + if (!isRecord(body) || !Array.isArray(body.input)) return undefined; + for (const item of body.input) { + if (!isRecord(item)) continue; + const direct = mediaKind(item); + if (direct) return direct; + if (item.type === "function_call_output" || item.type === "custom_tool_call_output") { + const kind = contentMedia(item.output); + if (kind) return kind; + } else if (item.type === "message" || item.type === undefined) { + const kind = contentMedia(item.content); + if (kind) return kind; + } + } + return undefined; +} + +/** Used only when Chat is actually projected, not on the native Chat fast path. */ +export function untranslatedChatInputMedia(body: unknown): UntranslatedInputMedia | undefined { + if (!isRecord(body) || !Array.isArray(body.messages)) return undefined; + for (const message of body.messages) { + if (!isRecord(message)) continue; + const kind = contentMedia(message.content); + if (kind) return kind; + } + return undefined; +} + +/** Fixed vocabulary only: never interpolate filenames, URLs or client metadata. */ +export function untranslatedInputMediaMessage(kind: UntranslatedInputMedia): string { + return `OpenCodex cannot translate ${kind} input on this route. Use a native input wire that supports the attachment, or convert it to text first.`; +} diff --git a/structure/adapters/registry.md b/structure/adapters/registry.md index 78ee66311d..18fe44394f 100644 --- a/structure/adapters/registry.md +++ b/structure/adapters/registry.md @@ -151,3 +151,29 @@ medium/high/max UID before accepting a suffix already present in the model id. The merged `devin` provider uses this resolver for every account, whichever login path minted the credential. Omitted effort preserves an explicit variant; unrelated model families retain their existing suffix precedence. + + +## Untranslated input media + +`src/responses/input-media.ts` inspects actual content blocks and typed tool-output arrays +without parsing text or function arguments, copying attachment payloads, resolving file IDs, +or fetching URLs. Audio, files/documents and file-ID-only images have no lossless normalized +carrier. The scanner returns only an input-kind name, never client content. + +`src/adapters/input-media-guard.ts` guards adapters created by the registry after effective +wire selection. A translated `buildRequest` refuses these inputs through the existing 400 +error path; `runTurn` emits one nonretryable `unsupported_input_modality` error without +starting its underlying transport. `localTerminal` declines a success shortcut for such a +request, letting the guarded builder return the error instead. The original raw body stays +unchanged, including when another final adapter is selected after a failed attempt. + +The effective Responses wire, including both Azure aliases, is excluded: it forwards the +original body and leaves native media acceptance to its upstream. This exception does not +claim that every Responses model supports every attachment. Native Chat also keeps its +existing wire; only an actual Chat-to-Responses projection rejects audio/file blocks before +losing them. Legacy function-result images fail explicitly because that projection does not +implement legacy call/result pairing. Modern tool-image carriers are unchanged. + +`tests/adapters/adapter-input-media-guard.test.ts` covers hook ordering, error events and +raw passthrough; `tests/responses/chat-media-translation.test.ts` reaches the real HTTP +translation boundary and verifies that rejection sends no upstream request. diff --git a/structure/data-planes/inbound-compat.md b/structure/data-planes/inbound-compat.md index 72fdfedfdd..e6e6ecd582 100644 --- a/structure/data-planes/inbound-compat.md +++ b/structure/data-planes/inbound-compat.md @@ -316,3 +316,14 @@ omitting the wire parameter, and the Pi client export maps Pi's `off` thinking l onto it. Dropping it let a provider default re-enable reasoning the caller had explicitly turned off, which is not neutral for the Anthropic families that think by default and require an explicit `thinking:{type:"disabled"}` to stop. + + +## Media at the Chat translation boundary + +The native Chat path retains provider-native file/audio blocks. When a request instead needs +Chat-to-Responses projection, `src/chat/inbound.ts` rejects recognized audio/file content +before it can become empty text, regardless of message role. Legacy `function`-role images +also return an explicit error; their call/result pairing is not implemented by this projection. +Modern `tool` images continue through the existing following-user carrier. These errors state +an OpenCodex conversion limit, not a provider capability claim. Final Responses-to-adapter +admission follows the [registry contract](../adapters/registry.md#untranslated-input-media). diff --git a/structure/providers/chat-compat.md b/structure/providers/chat-compat.md index d3e36717fe..ac37b3e91c 100644 --- a/structure/providers/chat-compat.md +++ b/structure/providers/chat-compat.md @@ -323,13 +323,13 @@ the IR has no audio carrier and no adapter consumes one. The parser stays non-th because the native Responses passthrough also runs through `parseRequest` before the adapter forwards `_rawBody`, so refusing there would regress raw passthrough. -**There is no adapter-level refusal for audio today.** By the time an adapter sees the -turn the part is already a text marker, so it continues rather than rejecting. A typed -unsupported-modality signal that survives to final adapter dispatch — leaving raw -passthrough untouched — is a separate, recorded residual. `input_file` likewise keeps -its existing filename-only marker, and Chat inbound still has no file or audio -translation, so a Chat request can lose media before this parser sees it. No payload -bytes and no media URL ever enter a marker or an error message. +The final registered adapter also checks the original input under the +[untranslated-media contract](../adapters/registry.md#untranslated-input-media). Audio/file +attachments cannot succeed merely because the normalized representation retained a text +marker: translated adapters refuse them, while native Responses retains the original body. +Chat conversion rejects recognized audio/file parts before projection; the native Chat wire +is unchanged. No audio/file transport or automatic URL fetch is added, and no client filename, +payload, URL or metadata is included in the new error messages. The shared coding-agent projection (CodeBuddy, Qoder) carries tool-result images as real image blocks rather than flattening them to the text `[image]`, and orders image diff --git a/structure/providers/cursor.md b/structure/providers/cursor.md index 1dfc6c8553..6d4139b5df 100644 --- a/structure/providers/cursor.md +++ b/structure/providers/cursor.md @@ -127,3 +127,5 @@ Combo child requests normalize effort and thinking controls against the selected `src/adapters/cursor.ts` surfaces the first bare context overflow before attempting conversation remint on later eligible requests. `cursorClientThreadOwner` recognizes both client thread aliases; `src/adapters/cursor/thread-continuity.ts` limits recovery to three remints per retained identity-scoped owner, with a one-hour idle TTL and 2,048-entry bound. Conversation-only requests have no stable owner and do not automatically remint. Quota/rate errors, tool-result resumes, partial output, local side effects, isolated helper/shadow requests and compaction remain fail-closed. Isolated requests neither consume the parent allowance nor invalidate its checkpoint. Eligible overflow checks refresh existing retention timestamps and LRU position even after the cap is exhausted, without allocating absent scopes. Retention expiry, eviction or process restart resets the in-memory allowance; this is not a persistent lifetime cap or semantic-progress policy. Translated Chat request construction uses the [inline-image budget](../transports/streaming-health.md#translated-chat-inline-image-budget); the shared normalizer counts retained bytes even when a wire-specific drop callback keeps the image attached. + +Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/providers/kiro.md b/structure/providers/kiro.md index bf27e3eb4c..80f765adae 100644 --- a/structure/providers/kiro.md +++ b/structure/providers/kiro.md @@ -83,3 +83,5 @@ rebuilds a turn's content from its collected texts and would otherwise discard i No fetch is introduced: resolving the reference server-side would add an outbound request on a request path. The marker carries a count and no URL, because a remote image URL can carry a signed token. + +Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index 6213ce1449..0ca554f4e0 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -136,3 +136,5 @@ Pool quota producers and account commands follow the [bounded raw-observation co Account quota surfaces use [safe probe diagnostics](../transports/inventory.md#account-quota-failure-diagnostics) separately from quota validity, credential health and routing authority. Live sideband admission and its bounded upstream handshake follow the [runtime contract](../runtime.md#live-sideband-handshake); the ordinary Responses WebSocket exchange remains separate. + +Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/runtime.md b/structure/runtime.md index ea80d0a46e..1ef3d78596 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -384,3 +384,5 @@ Exact [model input declarations](config.md#explicit-per-model-capability-declara Provider-scoped approval reviewer settings are projected by the [catalog owner](catalog.md#provider-scoped-approval-reviewer); this surface retains its existing routing, transport and account-selection behavior. Renamed fixed-key providers receive [missing reasoning metadata](catalog.md#renamed-destination-reasoning-metadata) during derivation; explicit per-model entries and provider defaults retain precedence. + +Translated audio/file admission follows the [final-adapter input contract](adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 830d1af7a9..bf30f1740f 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -33,3 +33,5 @@ plus exact separators, without joining a second full JSON array. `src/lib/admiss truncates diagnostic text at UTF-8 code-point boundaries without allocating arrays per character; byte sizing retains TextEncoder's coercion behavior for legacy non-string runtime callers. These optimizations do not add request queues, retry policies, or RSS-based admission gates. + +Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index 10b5be6a6b..381d0abdbe 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -142,3 +142,5 @@ The [explicit model-capability contract](../config.md#explicit-per-model-capabil Provider-scoped approval reviewer settings are projected by the [catalog owner](../catalog.md#provider-scoped-approval-reviewer); this surface retains its existing routing, transport and account-selection behavior. Renamed fixed-key providers receive [missing reasoning metadata](../catalog.md#renamed-destination-reasoning-metadata) during derivation; explicit per-model entries and provider defaults retain precedence. + +Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 1f1bd3cd2f..430598b622 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -597,4 +597,4 @@ Translated Chat request construction uses the [inline-image budget](streaming-he The [explicit model-capability contract](../config.md#explicit-per-model-capability-declarations) preserves operator declarations through provider storage and catalog capture; it does not infer upstream capability or change this surface's routing behavior. -Provider-scoped approval reviewer settings are projected by the [catalog owner](../catalog.md#provider-scoped-approval-reviewer); this surface retains its existing routing, transport and account-selection behavior. +Provider-scoped approval reviewer settings are projected by the [catalog owner](../catalog.md#provider-scoped-approval-reviewer); this surface retains its existing routing, transport and account-selection behavior. Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. diff --git a/tests/adapters/adapter-input-media-guard.test.ts b/tests/adapters/adapter-input-media-guard.test.ts new file mode 100644 index 0000000000..c63c137d52 --- /dev/null +++ b/tests/adapters/adapter-input-media-guard.test.ts @@ -0,0 +1,137 @@ +import { describe, expect, test } from "bun:test"; +import type { ProviderAdapter } from "../../src/adapters/base"; +import { withInputMediaGuard } from "../../src/adapters/input-media-guard"; +import { createRegisteredAdapter } from "../../src/adapters/registry"; +import { parseRequest } from "../../src/responses/parser"; +import { untranslatedResponsesInputMedia } from "../../src/responses/input-media"; +import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig } from "../../src/types"; +import { createTestTranslatorBudget, withTestTranslatorBudget } from "../helpers/translator-budget"; + +const AUDIO = { type: "input_audio", audio_url: "data:audio/wav;base64,YWJj" }; +const FILE = { type: "input_file", filename: "private.pdf", file_data: "data:application/pdf;base64,JVBERi0=" }; + +function request(content: unknown[]): OcxParsedRequest { + return parseRequest({ model: "test-model", input: [{ type: "message", role: "user", content }] }); +} + +function fakeAdapter() { + const seen = { builds: 0, runs: 0, terminals: 0 }; + const adapter: ProviderAdapter = { + name: "stub", + buildRequest() { + seen.builds++; + return { url: "https://example.invalid", method: "POST", headers: {}, body: "{}" }; + }, + async *parseStream() { yield { type: "done", endTurn: true }; }, + async runTurn(_parsed, _incoming, emit) { seen.runs++; emit({ type: "done", endTurn: true }); }, + localTerminal() { seen.terminals++; return { reason: "already answered" }; }, + }; + return { adapter: withInputMediaGuard(adapter), seen }; +} + +describe("typed input media inspection", () => { + test("recognizes user audio and inline file without inspecting payload strings", () => { + expect(untranslatedResponsesInputMedia(request([AUDIO])._rawBody)).toBe("audio"); + expect(untranslatedResponsesInputMedia(request([FILE])._rawBody)).toBe("file"); + }); + + test("recognizes tool and custom-tool attachments", () => { + for (const type of ["function_call_output", "custom_tool_call_output"]) { + expect(untranslatedResponsesInputMedia({ input: [{ type, call_id: "call1", output: [AUDIO] }] })).toBe("audio"); + expect(untranslatedResponsesInputMedia({ input: [{ type, call_id: "call1", output: [FILE] }] })).toBe("file"); + } + }); + + test("recognizes file-id-only images but keeps actual image URLs", () => { + expect(untranslatedResponsesInputMedia({ input: [{ role: "user", content: [{ type: "input_image", file_id: "file-1" }] }] })).toBe("file"); + expect(untranslatedResponsesInputMedia({ input: [{ role: "user", content: [{ type: "input_image", file_id: "file-1", image_url: "https://example.invalid/image.png" }] }] })).toBeUndefined(); + }); + + test("does not parse text, tool arguments or schema properties as attachments", () => { + expect(untranslatedResponsesInputMedia({ + input: [ + { type: "message", role: "user", content: [{ type: "input_text", text: JSON.stringify(AUDIO) }] }, + { type: "function_call", name: "f", call_id: "c", arguments: JSON.stringify(FILE) }, + { type: "function_call_output", call_id: "c", output: JSON.stringify(AUDIO) }, + ], + tools: [{ type: "function", name: "f", parameters: { type: "object", properties: { audio: AUDIO } } }], + })).toBeUndefined(); + }); +}); + +describe("final translated-adapter boundary", () => { + test("build refuses before the adapter can serialize or send; errors contain no payload", () => { + const { adapter, seen } = fakeAdapter(); + const incoming = { headers: new Headers(), translatorBudget: createTestTranslatorBudget() }; + for (const part of [AUDIO, FILE]) { + let failure: unknown; + try { adapter.buildRequest(request([part]), incoming); } catch (error) { failure = error; } + expect(failure).toBeInstanceOf(Error); + const message = (failure as Error).message; + expect(message).toContain("OpenCodex cannot translate"); + expect(message).not.toContain("private.pdf"); + expect(message).not.toContain("base64"); + expect(message).not.toContain("YWJj"); + } + expect(seen.builds).toBe(0); + }); + + test("runTurn emits one terminal nonretryable 400 without invoking the transport", async () => { + const { adapter, seen } = fakeAdapter(); + const events: AdapterEvent[] = []; + await adapter.runTurn!(request([AUDIO]), { + headers: new Headers(), translatorBudget: createTestTranslatorBudget(), + }, event => { events.push(event); }); + expect(seen.runs).toBe(0); + expect(events).toHaveLength(1); + expect(events[0]).toMatchObject({ type: "error", status: 400, code: "unsupported_input_modality", errorType: "invalid_request_error", retryable: false }); + }); + + test("local completion cannot hide a rejected attachment", () => { + const { adapter, seen } = fakeAdapter(); + expect(adapter.localTerminal!(request([FILE]))).toBeUndefined(); + expect(seen.terminals).toBe(0); + expect(adapter.localTerminal!(request([{ type: "input_text", text: "answered" }]))).toEqual({ reason: "already answered" }); + expect(seen.terminals).toBe(1); + }); + + test("ordinary text still reaches build and runTurn", async () => { + const { adapter, seen } = fakeAdapter(); + const parsed = request([{ type: "input_text", text: "hello" }]); + const incoming = { headers: new Headers(), translatorBudget: createTestTranslatorBudget() }; + adapter.buildRequest(parsed, incoming); + const events: AdapterEvent[] = []; + await adapter.runTurn!(parsed, incoming, event => { events.push(event); }); + expect(seen).toMatchObject({ builds: 1, runs: 1 }); + expect(events[0]).toMatchObject({ type: "done" }); + }); +}); + +describe("runtime registry and native passthrough exceptions", () => { + test("registry construction cannot bypass the guard for translated adapters", async () => { + for (const adapter of ["openai-chat", "anthropic", "google", "kiro", "cursor", "devin", "codebuddy", "qoder"]) { + const provider: OcxProviderConfig = { + adapter, baseUrl: "https://example.invalid/v1", authMode: "key", apiKey: "test-placeholder", + }; + const runtime = withTestTranslatorBudget(createRegisteredAdapter(provider)); + await expect(Promise.resolve().then(() => runtime.buildRequest(request([AUDIO])))).rejects.toThrow("OpenCodex cannot translate audio"); + } + }); + + test("Responses and both Azure aliases preserve raw media after a rejected translated attempt", async () => { + for (const adapter of ["openai-responses", "azure", "azure-openai"]) { + const parsed = request([AUDIO, FILE]); + const original = structuredClone(parsed._rawBody); + const rejected = withTestTranslatorBudget(createRegisteredAdapter({ + adapter: "openai-chat", baseUrl: "https://example.invalid/v1", apiKey: "test-placeholder", authMode: "key", + })); + await expect(Promise.resolve().then(() => rejected.buildRequest(parsed))).rejects.toThrow("cannot translate audio"); + const runtime = withTestTranslatorBudget(createRegisteredAdapter({ + adapter, baseUrl: "https://example.invalid/v1", apiKey: "test-placeholder", authMode: "key", + })); + const wire = JSON.parse((await runtime.buildRequest(parsed)).body); + expect(wire.input[0].content).toEqual([AUDIO, FILE]); + expect(parsed._rawBody).toEqual(original); + } + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 99a9318b1f..06a4c14772 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1265,5 +1265,7 @@ "usage-log-ws-stage.test.ts": "usage", "main-device-reauth.test.ts": "codex-integration", "main-device-reauth-api.test.ts": "codex-integration", - "main-device-reauth-ui.test.ts": "gui" + "main-device-reauth-ui.test.ts": "gui", + "adapter-input-media-guard.test.ts": "adapters", + "chat-media-translation.test.ts": "responses" } diff --git a/tests/responses/chat-media-translation.test.ts b/tests/responses/chat-media-translation.test.ts new file mode 100644 index 0000000000..a3042d4f3d --- /dev/null +++ b/tests/responses/chat-media-translation.test.ts @@ -0,0 +1,115 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { ChatCompletionsRequestError, chatCompletionsToResponsesBody } from "../../src/chat/inbound"; +import { buildOpenAIChatPassthroughRequest } from "../../src/adapters/openai-chat"; +import { isNativeChatRouteEligible } from "../../src/server/chat-native"; +import { saveConfig } from "../../src/config"; +import { startServer } from "../../src/server"; +import type { RouteResult } from "../../src/router"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const provider: OcxProviderConfig = { adapter: "openai-chat", baseUrl: "https://example.invalid/v1", apiKey: "test-placeholder", authMode: "key" }; +const media = [ + { type: "input_audio", input_audio: { data: "YWJj", format: "wav" } }, + { type: "input_audio", audio_url: "data:audio/wav;base64,YWJj" }, + { type: "file", file: { filename: "private.pdf", file_data: "data:application/pdf;base64,JVBERi0=" } }, + { type: "input_file", file_id: "file-private" }, +]; + +function chat(part: unknown, role = "user") { + return { model: "model", messages: [{ role, tool_call_id: "call1", content: [{ type: "text", text: "read this" }, part] }] }; +} + +describe("Chat media stays native or fails explicitly at translation", () => { + test("user and tool media never disappear in the converter", () => { + for (const role of ["user", "tool"]) { + for (const part of media) { + expect(() => chatCompletionsToResponsesBody(chat(part, role))).toThrow(ChatCompletionsRequestError); + expect(() => chatCompletionsToResponsesBody(chat(part, role))).toThrow("OpenCodex cannot translate"); + } + } + }); + + test("the native Chat route retains the caller's exact media blocks", () => { + const route = { provider, providerName: "gateway", modelId: "model" } as RouteResult; + for (const part of media) { + const raw = chat(part); + expect(isNativeChatRouteEligible(route, raw)).toBe(true); + const wire = JSON.parse(buildOpenAIChatPassthroughRequest(provider, raw, "model", false).body); + expect(wire.messages).toEqual(raw.messages); + } + }); + + test("diverted legacy function images return an explicit error instead of losing the result", () => { + const raw = { + model: "model", + messages: [ + { role: "user", content: "Inspect the result." }, + { role: "assistant", function_call: { name: "capture", arguments: "{}" }, content: null }, + { role: "function", name: "capture", content: [{ type: "image_url", image_url: { url: "https://example.invalid/secret.png" } }] }, + ], + }; + const route = { provider, providerName: "gateway", modelId: "model" } as RouteResult; + expect(isNativeChatRouteEligible(route, raw)).toBe(false); + expect(() => chatCompletionsToResponsesBody(raw)).toThrow("Legacy function-result image translation is not implemented"); + }); + + test("plain text mentioning an attachment is not treated as one", () => { + const out = chatCompletionsToResponsesBody({ model: "model", messages: [{ role: "user", content: JSON.stringify(media) }] }); + expect(out.input).toEqual([{ type: "message", role: "user", content: [{ type: "input_text", text: JSON.stringify(media) }] }]); + }); +}); + +let home = ""; +let previousHome: string | undefined; +let codexHome: IsolatedCodexHome | undefined; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + codexHome = installIsolatedCodexHome("ocx-media-guard-"); + home = mkdtempSync(join(tmpdir(), "ocx-media-guard-")); + process.env.OPENCODEX_HOME = home; +}); +afterEach(() => { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + codexHome?.restore(); + codexHome = undefined; + try { removeTreeWithRetry(home); } catch { /* Temp cleanup cannot change a passed request assertion. */ } +}); + +test("real HTTP translation refuses media before sending to the selected upstream", async () => { + let sends = 0; + const upstream = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch() { sends++; return new Response("Unexpected upstream request", { status: 500 }); } }); + let server: ReturnType | undefined; + try { + saveConfig({ + port: 0, defaultProvider: "gateway", + providers: { gateway: { ...provider, baseUrl: `${upstream.url.toString().replace(/\/$/, "")}/v1`, allowPrivateNetwork: true } }, + } as OcxConfig); + server = startServer(0); + for (const part of [ + { type: "input_audio", audio_url: "data:audio/wav;base64,YWJj" }, + { type: "input_file", filename: "private.pdf", file_data: "data:application/pdf;base64,JVBERi0=" }, + ]) { + const response = await fetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ model: "gateway/model", stream: false, input: [{ type: "message", role: "user", content: [part] }] }), + signal: AbortSignal.timeout(10000), + }); + expect(response.status).toBe(400); + const body = await response.json() as { error: { type: string; message: string } }; + expect(body.error.type).toBe("invalid_request_error"); + expect(body.error.message).toContain("OpenCodex cannot translate"); + expect(body.error.message).not.toContain("private.pdf"); + expect(body.error.message).not.toContain("YWJj"); + } + expect(sends).toBe(0); + } finally { + await server?.stop(true); + await upstream.stop(true); + } +}, 25000);