[Bug]: Upload to Provider sends archives, Office files and some text types as file parts the provider rejects (400 on every later turn)

[mihidumh]

What happened?

With "Upload to Provider", some files on the built-in supportedMimeTypes list go to the provider as file parts that the provider always rejects. The provider answers 400. The file stays in the history, so every later turn of the conversation fails the same way.

Measured through a LiteLLM proxy, with an OpenAI file part and data:<mime>;base64,...:

Model behind the gateway	400	200
Azure OpenAI (gpt-5-mini)	application/sql, application/x-sh, application/xml, application/zip, application/octet-stream	text/plain, text/csv, text/markdown, text/html, text/x-python, application/json, application/yaml, application/typescript, docx
Gemini on Vertex AI (gemini-flash-lite-latest)	docx, xlsx, application/sql, application/json, application/yaml, application/xml, application/typescript	text/plain, text/csv, text/markdown, text/html, text/x-python

Azure's error: Invalid file data: 'messages[0].content[1].file.file_data'. Expected a base64-encoded data URL with a supported file MIME type, for example 'data:application/pdf;base64,...'. Vertex's error: Unable to submit request because it has a mimeType parameter with value application/vnd.openxmlformats-officedocument.wordprocessingml.document, which is not supported.

The file picker's accept is images and PDF, but "All files" in the OS dialog bypasses it, and drag-and-drop accepts the file too.

Cause. An endpoint with no supportedMimeTypes of its own inherits the built-in list (mergeWithDefault). That list also serves Code Environment and File Search, so it includes zip, tar, sql, sh, json, yaml, docx/xlsx/pptx and epub. The document branch of processAttachments (api/app/clients/BaseClient.js) and of the child-run encoder (packages/api/src/agents/files/encode.ts) accept any file on that list. #13550 made this deliberate for OpenAI, so CSV and XLSX reach the Responses API (#13547). But some types on the list can never go inline:

• archives (zip, tar, gzip, epub, octet-stream), on every provider;
• Office documents on Gemini, native or behind a gateway (the Anthropic side already has this filter, from 📄 fix: Filter Non-PDF Documents on the Anthropic Encode Path #14535);
• application/sql, x-sh and xml on Azure OpenAI, and every textual application/* type on Gemini.

The encoder already knows "Claude behind a gateway" by the model name (#14650, #16055). Gemini has no such branch.

Suggested fix. Keep the inherited list as the provider opt-in (#13550). Skip archives unless the endpoint lists them itself. Filter Gemini to PDF and text, as #14535 does for Claude. Send the textual types above as a text part (File: "<name>"\n\n<contents>), unless the endpoint lists them. PR to follow.

Version Information

• Image registry.librechat.ai/danny-avila/librechat:v0.8.8-rc3. The code is unchanged in v0.8.8-rc4 and on dev (c8c5478).
• Custom OpenAI-compatible endpoints pointed at a LiteLLM proxy, fileConfig.legacyFileUploadUX: true, no supportedMimeTypes on the endpoint.

Steps to Reproduce

1. Configure a custom endpoint for Azure OpenAI (or Gemini) through an OpenAI-compatible gateway. Do not set supportedMimeTypes on it.
2. Pick "Upload to Provider", choose "All files" in the OS dialog, and attach query.sql (Gemini: report.docx).
3. Send a message. The provider returns 400.
4. Send another message. It fails again, because the file is re-sent from history.

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[mehran-almma]

Thanks for the measurements. I traced this on current main (63363a7) and read #16473 at 5ec8bb3. Three notes for review:

1. Stuck conversations recover once this ships. Files from history come back through addPreviousAttachments → processAttachments in api/app/clients/BaseClient.js. That is the same branch the PR changes, so a conversation already stuck on query.sql works again on its next turn, with no data cleanup. Users hit the repeated 400 hardest, so a regression test is worth having if one isn't there yet: attach the file on an earlier message, not the current turn, and assert the turn succeeds.

2. Gemini behind a gateway alias is not detected. usesGeminiDocumentCapabilities looks for gemini in the model name. A LiteLLM model_name alias (e.g. fast → vertex_ai/gemini-flash-lite-latest) doesn't contain it. On such an endpoint, docx and JSON/YAML still go out as file parts and get the 400. Only sql/sh/xml become text there, through the OpenAI-like set. Claude detection from 📄 fix: Send Textual Documents to Claude Through Gateways as Text #16055 has the same limit. Adding one line to the docs would help: "for aliased Gemini models, set supportedMimeTypes on the endpoint explicitly."

3. Dropped files are invisible to the user. After the fix, Office files sent to Gemini are skipped with a server-side warn. The chip is still in the composer, and the model answers as if nothing was attached. There's already an extractor for exactly this. resolveUploadFallbackText (packages/api/src/files/upload/fallback.ts) reads docx/xlsx through parseDocument. It only runs for an inferred none route with textFallbackWithoutTools, though, and never for a chosen "Upload to Provider". A follow-up could reuse it, so a docx sent to Gemini arrives as a File: "" text part and isn't dropped. That's out of scope for 📎 fix: Keep Provider Uploads the Provider Cannot Read Inline Out of File Parts #16473, but it would close the gap between "no 400" and "the model actually read the file".

[mihidumh]

Thanks for the careful read, @mehran-almma.

1. Recovery from history: added in 📎 fix: Keep Provider Uploads the Provider Cannot Read Inline Out of File Parts #16473 (a3155b430). The test attaches a zip and a PDF on an earlier message and replays them through addPreviousAttachments with the real processAttachments. The zip is skipped with a warning, the PDF is still sent, and message_file_map holds only the PDF. It fails on dev without the fix.
2. Gateway aliases: agreed, the name check cannot see through an alias. I added the limit to the PR description: an alias that hides the family must list its types in supportedMimeTypes explicitly. I can add the same line to the docs once this is merged.
3. Dropped files the user cannot see: agreed, and I would keep it out of this PR. Reusing resolveUploadFallbackText for a chosen "Upload to Provider" would close it, so a docx sent to Gemini arrives as a text part. I can open a follow-up for it after 📎 fix: Keep Provider Uploads the Provider Cannot Read Inline Out of File Parts #16473 lands.

[mihidumh]

@mehran-almma, an update on point 3: it is now in #16473 and is no longer a follow-up. The PR review asked for it before merge (8c83da1).

A file the model would not receive is no longer dropped silently. On the current turn, the send fails with an error that names the file and tells the user to remove it or to upload it as text or to the code environment. On history replay, the turn continues and the message carries a note that names the omitted file. So a docx sent to native Gemini now gets an explicit error, and not an answer as if nothing was attached.

Your resolveUploadFallbackText idea, which would send the docx as a File: "<name>" text part so the model can read it, is still a good follow-up. With this PR it becomes an improvement over a visible error, and no longer the fix for a silent gap.