Skip to content

Commit cbd37a8

Browse files
author
bcode
committed
fix(provider/openai-compatible): trust user-declared modalities, don't inject fake ERROR text
unsupportedParts (transform.ts) replaces image/file parts with an inline 'ERROR: Cannot read <name> (this model does not support <modality> input)' text part when model.capabilities.input.<modality> is false. The capability is derived in provider.ts:1298-1304 — for @ai-sdk/openai-compatible (the user-configured proxy) there is no models.dev entry, so image/audio/video/pdf all default to false unless the user explicitly declares modalities in opencode.json. This is a silent footgun in the user-configured-proxy case. The screenshot is stripped before the wire, and the model receives the fabricated 'ERROR: Cannot read image' text as if it came from the user — producing nonsensical replies like 'I can't see the image' even though upstream supports vision. Forwarding the part for openai-compatible providers is honest: if upstream truly can't handle it, we get a real provider error from the API call, which is far more debuggable than fabricated capability text. Native providers (@ai-sdk/anthropic, @ai-sdk/openai, @ai-sdk/google, @ai-sdk/amazon-bedrock) keep the existing check because models.dev IS authoritative for them — there the filter prevents a real 4xx upstream and gives the user a clear local error. Reproed against browser-use cloud's V4 LLM gateway (an openai-compatible proxy in front of Anthropic): vision smoketest replied 'I cannot see the image' on every run; gateway-side request audit confirmed zero image_url / file / data:image/ signals in the body even when the agent attached a screenshot via browser_execute. With this change image parts reach the wire and the model produces real vision replies.
1 parent 06293d5 commit cbd37a8

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

packages/opencode/src/provider/transform.ts

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -390,6 +390,21 @@ function applyCaching(msgs: ModelMessage[], model: Provider.Model): ModelMessage
390390
}
391391

392392
function unsupportedParts(msgs: ModelMessage[], model: Provider.Model): ModelMessage[] {
393+
// @ai-sdk/openai-compatible is the user-configured proxy provider — there is
394+
// no models.dev entry to consult, so model.capabilities.input.{image,audio,
395+
// video,pdf} all silently default to false (provider.ts:1298-1304) unless
396+
// the user explicitly declares `modalities` in opencode.json. Replacing
397+
// every image/file part with an inline ERROR text part in that case is a
398+
// silent footgun: bcode strips the screenshot before the wire and the
399+
// model receives "ERROR: Cannot read image..." as if it came from the user,
400+
// producing nonsensical replies ("I can't see the image"). The honest
401+
// behavior is to forward the part — upstream returns a real provider error
402+
// if it truly can't handle it, which is far more debuggable than fabricated
403+
// capability text. Native providers (@ai-sdk/anthropic, @ai-sdk/openai,
404+
// @ai-sdk/google, @ai-sdk/amazon-bedrock) keep the check because models.dev
405+
// is authoritative for them.
406+
if (model.api.npm === "@ai-sdk/openai-compatible") return msgs
407+
393408
return msgs.map((msg) => {
394409
if (msg.role !== "user" || !Array.isArray(msg.content)) return msg
395410

0 commit comments

Comments
 (0)