Skip to content

feat(chat): decode and render file_search and memory attachment payloads - #357

Merged
garfiec merged 1 commit into
developfrom
feat/attachment-structured-payloads
Aug 13, 2026
Merged

feat(chat): decode and render file_search and memory attachment payloads#357
garfiec merged 1 commit into
developfrom
feat/attachment-structured-payloads

Conversation

@garfiec

@garfiec garfiec commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Summary

LibreChat sends four structured "pseudo-attachments" — web_search, file_search, memory, ui_resources — each carrying a payload on its own key rather than being a file. The Attachment DTO modelled only web_search, so the other three were discarded at decode and nothing could render them. This adds the missing fields and their render surfaces, on the SSE path as well as the REST DTO.

Changes

  • Attachment gains file_search, memory and ui_resources, alongside the existing web_search. The new payload types live in AttachmentPayloads.kt.
  • file_search citations render as a sources card. The tool's own output is a flat human-readable digest; the per-source structure — and the fileId tying a citation back to a real uploaded file — exists only in the payload. Sources describing the same file (one per matched chunk) are merged: pages unioned, best relevance kept, passages joined, most-relevant file first. retrieval calls route to the same card, since both carry their citations in the file_search payload.
  • Background memory agent writes now appear in chat. This is the case with no fallback: the agent's sub-run contributes no content parts, so the attachment is the only sign it ran, and mobile previously showed nothing at all. Rendered as a section mirroring upstream's MemoryArtifacts, keyed on writes that no rendered tool call accounts for so an inline write is never double-reported. The section also renders during streaming, not only on a settled message.
  • Inline set_memory/delete_memory cards use the structured payload when present, recovering the remembered key as the title and the value as the body. The existing output-parsing fallback is kept for servers that don't send one, so those are unaffected. The inline card also gains an error state (error-container styling and a "Memory Error" label) and a default body for a delete, which previously rendered as a bare key; a call reporting several writes now renders one card each rather than only the first.
  • Decoded on the SSE path too (SseEventMapper), before the file guard that would otherwise drop a payload-only attachment as empty.
  • ui_resources is carried but deliberately not rendered — see Notes.

Testing

  • ./gradlew test detektMetadataCommonMain detekt :app:lint :app:assembleDebug — green.
  • New unit tests: FileSearchSourcesTest (8) and MemoryArtifactsTest (10), plus SseEventMapperTest coverage for the payloads surviving the SSE decode (61 in that suite).
  • No device verification. This has not been run on a device or emulator. In particular the background-memory-agent path — the one payload that renders nothing at all today — requires memory.agent.enabled server-side and has not been exercised end to end.

Notes

  • ui_resources is carried as raw JSON and deliberately not rendered. Upstream hands it to @mcp-ui/client's UIResourceRenderer, which mounts each resource as a sandboxed auto-resizing iframe — a WebView surface with no counterpart in this client, and far larger than the DTO gap this change closes. Modelling it structurally would also be a guess: the server forwards the payload verbatim and it is not always an array, so a typed decode could reject the whole message's attachments and take the rest of them down with it. Raw JSON preserves it losslessly for whoever builds the renderer.
  • Citation merging is close to upstream's extractFileSources but not identical, in two places worth knowing before comparing them: the dedup key is fileId ?: fileName rather than fileId alone, so a source with no id still merges by name instead of being dropped; and pages are ordered by relevance within each source and then concatenated, where upstream merges pageRelevance across chunks and orders the combined list. A file merged from several chunks therefore lists pages in arrival order rather than by relevance.
  • scripts/mirrors.json gains an entry for the two set_memory storage-limit errorType values, which decide which sentence the user sees and are sent as raw keys only. It is watched file-level rather than by block: both literals sit inside createMemoryTool's body with no smaller exported symbol wrapping them, and a block anchored on the function extracts only its signature — the inert-watch case check-mirrors.py warns about.
  • The 13 new strings are English-only; the sibling locale directories are untouched, consistent with the existing i18n debt.

The Attachment DTO modelled only web_search of the four structured pseudo-attachment payloads, so file_search citations, memory writes and ui_resources were discarded at decode and nothing could render them. Adds the missing payload fields and their render surfaces, decoded on the SSE path as well as the REST DTO.

The background memory agent is the case with no fallback: its sub-run contributes no content parts, so the attachment is the only sign it ran and mobile showed nothing at all. The inline set_memory/delete_memory tools already rendered a card from their own output; they now use the structured payload when it is present, keeping the output fallback for servers that do not send one.

ui_resources is carried as raw JSON and deliberately not rendered: upstream hands it to @mcp-ui/client's UIResourceRenderer, a sandboxed-iframe surface with no counterpart in this client, and the server forwards the payload verbatim in a shape that is not always an array — so a structural decode would risk rejecting the whole attachments array.

Payload parsing lives in its own file: ToolCallParsing.kt sat at 26 top-level functions against a TooManyFunctions threshold of 30, so adding these seven would have crossed it.
@github-actions

Copy link
Copy Markdown
Contributor

Android debug APK

Artifact: switchboard-android-debug-357
Download: switchboard-android-debug-357.zip
Retention: 90 days
Commit: b77e708b0ea96cb9022a440ada7cfcb493c1b30d

Download requires a GitHub login. Installs over previous debug builds without uninstalling (stable signing key).

@garfiec
garfiec merged commit fc86305 into develop Aug 13, 2026
5 checks passed
@garfiec
garfiec deleted the feat/attachment-structured-payloads branch August 13, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant