[Bug]: Vision sidecar never triggers for NVIDIA NIM text-only models (registry `nvidia` entry missing `noVisionModels`)

[brunoflma]

Client or integration

Codex App (also reproducible via Direct HTTP/API client)

Area

Proxy and routing · Catalog / models · Provider adapter

Summary

The built-in nvidia provider entry in src/providers/registry.ts does not declare noVisionModels, so the vision sidecar is never activated for NVIDIA NIM text-only models (e.g. deepseek-ai/deepseek-v4-flash, nvidia/nemotron-3-ultra-550b-a55b, z-ai/glm-5.2, minimaxai/minimax-m3, moonshotai/kimi-k2.6). Users must manually add noVisionModels to the provider entry in ~/.opencodex/config.json as a workaround.

Expected: a text-only NIM model receives an image attachment and the proxy describes it through the vision sidecar before calling the upstream.
Actual: planVisionSidecar returns undefined (the model is not in provider.noVisionModels), so raw image parts are either forwarded to the text-only upstream or the catalog advertises the model as text-only and the app blocks the attachment.

Reproduction

1. Install @bitkyc08/opencodex 2.10.0 and start the proxy (ocx start).
2. Add a provider entry for NVIDIA NIM in ~/.opencodex/config.json with only the documented fields (an empty or absent noVisionModels matches the registry default, which has no noVisionModels for nvidia):

   {
     "providers": {
       "nvidia": {
         "adapter": "openai-chat",
         "baseUrl": "https://integrate.api.nvidia.com/v1",
         "authMode": "key",
         "apiKey": "nvapi-<redacted>"
       }
     }
   }

3. Query the routed model list and confirm the model has no image modality:

   GET /api/models  →  nvidia/deepseek-ai/deepseek-v4-flash  inputModalities: absent
   

4. Send a request that carries an image to the text-only model, e.g. via the Codex App (attach a screenshot) or directly:

   POST /v1/responses
   { "model": "nvidia/deepseek-ai/deepseek-v4-flash", "input": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } } ] } ] }
   

5. Observe: no [vision] sidecar call happens (nothing in the logs); the image either 400s at the upstream or is stripped/blocked.
6. Workaround: set noVisionModels: ["deepseek-ai/deepseek-v4-flash"] on the provider entry and restart. Now /api/models advertises inputModalities: ["text","image"] and the sidecar describes the image successfully (verified end-to-end with describeImage returning a description via the OpenAI forward sidecar).

Version

2.10.0 (npm @bitkyc08/opencodex); Codex runtime 0.146.0

Operating system

Windows 11 (current), America/Fortaleza timezone

Provider and model

nvidia / deepseek-ai/deepseek-v4-flash (also affects nvidia/nvidia/nemotron-3-ultra-550b-a55b, z-ai/glm-5.2, minimaxai/minimax-m3, moonshotai/kimi-k2.6)

Logs or error output

No [vision] log lines are emitted for these models. The model rows in GET /api/models carry no inputModalities. With the workaround applied, the sidecar path works and returns descriptions.

Redacted configuration

{
  "providers": {
    "nvidia": {
      "adapter": "openai-chat",
      "baseUrl": "https://integrate.api.nvidia.com/v1",
      "authMode": "key",
      "apiKey": "nvapi-<redacted>"
    }
  },
  "visionSidecar": {
    "backend": "openai",
    "model": "gpt-5.6-luna"
  }
}

How the vision sidecar is gated

planVisionSidecar in src/vision/index.ts returns a plan only when the routed model is listed in provider.noVisionModels:

if (!modelInList(provider.noVisionModels, modelId)) return undefined;
if (!messagesHaveImage(parsed)) return undefined;

The registry nvidia entry (src/providers/registry.ts, around line 1079) sets noReasoningModels / preserveReasoningContentModels for the Kimi family but no noVisionModels:

{
  id: "nvidia", label: "NVIDIA NIM", baseUrl: "https://integrate.api.nvidia.com/v1", adapter: "openai-chat", authKind: "key", dashboardUrl: "https://build.nvidia.com",
  freeTier: true,
  parallelToolCalls: false,
  noReasoningModels: NVIDIA_NIM_KIMI_MODELS,
  modelReasoningEfforts: Object.fromEntries(NVIDIA_NIM_KIMI_MODELS.map(id => [id, []])),
  preserveReasoningContentModels: NVIDIA_NIM_KIMI_THINKING_MODELS,
  note: "Free tier on NVIDIA NIM — API key still required (get a free key at build.nvidia.com).",
},

Other registry entries covering the same text-only model families do declare it, e.g. ollama-cloud lists deepseek-v4-pro, deepseek-v4-flash, nemotron-3-ultra, glm-5.2, etc., and alibaba-intl lists deepseek-v4-pro/flash, glm-5.2, MiniMax-M2.5.

The NIM /v1/models endpoint does not expose input_modalities, so the registry is the only source of truth for capability classification.

Impact

For every text-only NIM model without noVisionModels:

1. The proxy forwards raw image parts to a text-only upstream (which rejects them with a 400 or silently ignores them), or
2. The Codex catalog advertises the model as text-only (no input_modalities: ["image"]), so the Codex app blocks image attachments client-side before the sidecar ever runs.

Confirmed against 2.10.0: all 101 discovered nvidia/* rows in /api/models carry no inputModalities, and the 4 enabled text-only models (deepseek-ai/deepseek-v4-flash, minimaxai/minimax-m3, nvidia/nemotron-3-ultra-550b-a55b, z-ai/glm-5.2) never produce a vision plan.

Suggested fix

Add noVisionModels to the registry nvidia entry for the known text-only NIM model families (deepseek-v4-flash/pro, glm-5.x, nemotron-3-ultra/super, minimax-m3, kimi-k2.x, qwen, etc.), matching how ollama-cloud / alibaba-intl classify the same families. Vision-capable NIM models (e.g. meta-llama-3.2-11b-vision-instruct, nvidia/nemotron-nano-vl-8b-v1) must stay out of the list.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[github-actions]

Issue reopened

The report now contains the information required by the automated check. Thanks for updating it.