feat(chat): decode and render file_search and memory attachment payloads - #357
Merged
Merged
Conversation
The Attachment DTO modelled only web_search of the four structured pseudo-attachment payloads, so file_search citations, memory writes and ui_resources were discarded at decode and nothing could render them. Adds the missing payload fields and their render surfaces, decoded on the SSE path as well as the REST DTO. The background memory agent is the case with no fallback: its sub-run contributes no content parts, so the attachment is the only sign it ran and mobile showed nothing at all. The inline set_memory/delete_memory tools already rendered a card from their own output; they now use the structured payload when it is present, keeping the output fallback for servers that do not send one. ui_resources is carried as raw JSON and deliberately not rendered: upstream hands it to @mcp-ui/client's UIResourceRenderer, a sandboxed-iframe surface with no counterpart in this client, and the server forwards the payload verbatim in a shape that is not always an array — so a structural decode would risk rejecting the whole attachments array. Payload parsing lives in its own file: ToolCallParsing.kt sat at 26 top-level functions against a TooManyFunctions threshold of 30, so adding these seven would have crossed it.
Contributor
Android debug APKArtifact:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LibreChat sends four structured "pseudo-attachments" —
web_search,file_search,memory,ui_resources— each carrying a payload on its own key rather than being a file. TheAttachmentDTO modelled onlyweb_search, so the other three were discarded at decode and nothing could render them. This adds the missing fields and their render surfaces, on the SSE path as well as the REST DTO.Changes
Attachmentgainsfile_search,memoryandui_resources, alongside the existingweb_search. The new payload types live inAttachmentPayloads.kt.file_searchcitations render as a sources card. The tool's own output is a flat human-readable digest; the per-source structure — and thefileIdtying a citation back to a real uploaded file — exists only in the payload. Sources describing the same file (one per matched chunk) are merged: pages unioned, best relevance kept, passages joined, most-relevant file first.retrievalcalls route to the same card, since both carry their citations in thefile_searchpayload.MemoryArtifacts, keyed on writes that no rendered tool call accounts for so an inline write is never double-reported. The section also renders during streaming, not only on a settled message.set_memory/delete_memorycards use the structured payload when present, recovering the remembered key as the title and the value as the body. The existing output-parsing fallback is kept for servers that don't send one, so those are unaffected. The inline card also gains an error state (error-container styling and a "Memory Error" label) and a default body for a delete, which previously rendered as a bare key; a call reporting several writes now renders one card each rather than only the first.SseEventMapper), before the file guard that would otherwise drop a payload-only attachment as empty.ui_resourcesis carried but deliberately not rendered — see Notes.Testing
./gradlew test detektMetadataCommonMain detekt :app:lint :app:assembleDebug— green.FileSearchSourcesTest(8) andMemoryArtifactsTest(10), plusSseEventMapperTestcoverage for the payloads surviving the SSE decode (61 in that suite).memory.agent.enabledserver-side and has not been exercised end to end.Notes
ui_resourcesis carried as raw JSON and deliberately not rendered. Upstream hands it to@mcp-ui/client'sUIResourceRenderer, which mounts each resource as a sandboxed auto-resizing iframe — a WebView surface with no counterpart in this client, and far larger than the DTO gap this change closes. Modelling it structurally would also be a guess: the server forwards the payload verbatim and it is not always an array, so a typed decode could reject the whole message'sattachmentsand take the rest of them down with it. Raw JSON preserves it losslessly for whoever builds the renderer.extractFileSourcesbut not identical, in two places worth knowing before comparing them: the dedup key isfileId ?: fileNamerather thanfileIdalone, so a source with no id still merges by name instead of being dropped; and pages are ordered by relevance within each source and then concatenated, where upstream mergespageRelevanceacross chunks and orders the combined list. A file merged from several chunks therefore lists pages in arrival order rather than by relevance.scripts/mirrors.jsongains an entry for the twoset_memorystorage-limiterrorTypevalues, which decide which sentence the user sees and are sent as raw keys only. It is watched file-level rather than by block: both literals sit insidecreateMemoryTool's body with no smaller exported symbol wrapping them, and a block anchored on the function extracts only its signature — the inert-watch casecheck-mirrors.pywarns about.