feat: support image content in tool responses - #101
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR standardizes image handling and tool execution outputs across @llm-space/core, the runtime, and the UI by replacing the legacy "image_data" message content type with the unified "image" shape and by switching built-in tool call results from contentText to structured content (text + images). This aligns persisted thread data, RPC contracts, and UI rendering so tool responses can carry image payloads end-to-end.
Changes:
- Replaced
"image_data"with"image"across message/tool schemas, converters/parsers, UI rendering, storage, and docs. - Introduced
BuiltinToolCallResponse(structuredcontent) and updated built-in + MCP tool execution plumbing to return structured outputs. - Updated/added tests to validate structured tool results (including image bytes) and the new message content type.
Reviewed changes
Copilot reviewed 36 out of 38 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/ui/src/host/types.ts | Updates UI host types to use BuiltinToolCallResponse for tool results. |
| packages/ui/src/components/thread-playground/stores/thread-store.ts | Stores structured tool outputs and preserves images when editing tool-result text. |
| packages/ui/src/components/thread-playground/stores/thread-store-tool-results.test.ts | Adds coverage ensuring auto-run tool results preserve image content. |
| packages/ui/src/components/thread-playground/stores/thread-history.ts | Counts assistant tool-result images for history memory accounting. |
| packages/ui/src/components/thread-playground/message/use-tool-call-runner.ts | Writes structured tool results to the store; uses text extraction for error classification. |
| packages/ui/src/components/thread-playground/message/tool-call-list-item.tsx | Renders image thumbnails from tool call outputs. |
| packages/ui/src/components/thread-playground/message/message-list-view.tsx | Updates image numbering logic for "image". |
| packages/ui/src/components/thread-playground/message/message-list-item.tsx | Updates user-message image rendering types to "image". |
| packages/ui/src/components/thread-playground/message/image-content-view.tsx | Updates image component types to ImageContent. |
| packages/ui/src/components/thread-playground/examples/tools.ts | Updates read tool description to describe structured image output. |
| packages/runtime/src/tools/tool-registry.ts | Adds structured tool response contract + safe opt-in marker via createToolCallResponse. |
| packages/runtime/src/tools/tool-registry.test.ts | Updates expectations to structured content and adds regression for JSON {content:…} payloads. |
| packages/runtime/src/tools/built-in/fs.ts | Enhances read to return model-facing image content for supported formats (size-limited). |
| packages/runtime/src/tools/built-in/built-in-tools-module.test.ts | Adds tests for read returning image bytes / placeholders / oversize rejection. |
| packages/runtime/src/runtime/types.ts | Updates runtime client built-in tool RPC to return BuiltinToolCallResponse. |
| packages/runtime/src/mcp/mcp-manager.ts | Converts MCP tool results into structured content with truncation. |
| packages/core/src/types/tools/index.ts | Defines BuiltinToolCallResponse in core types. |
| packages/core/src/types/threads/thread.test.ts | Adds schema test asserting "image" accepted and "image_data" rejected. |
| packages/core/src/types/messages/tools.ts | Allows tool outputs to contain image content. |
| packages/core/src/types/messages/messages.ts | Updates user message content union to use ImageContent. |
| packages/core/src/types/messages/contents.ts | Renames image content schema to "image" and aligns types with pi. |
| packages/core/src/types/mcp.ts | Updates MCP call-tool response to structured content. |
| packages/core/src/thread/tool-call-status.ts | Introduces getToolResultText and updates output text extraction to skip images. |
| packages/core/src/thread/run-history-utils.ts | Updates run summaries and tool-result text extraction for "image" + structured tool output. |
| packages/core/src/server/storage/local/file-system.test.ts | Updates stored thread fixture to use "image". |
| packages/core/src/server/storage/blob/image-blobs.ts | Updates blob pack/unpack to match "image" content type. |
| packages/core/src/parsers/normalize-thread.ts | Updates normalization to produce "image" content blocks. |
| packages/core/src/client/converters.ts | Simplifies conversion now that user image content is already pi-compatible. |
| packages/core/src/client/converters.test.ts | Adds coverage ensuring tool-result image content passes through to pi context. |
| docs/core-concepts.zh-CN.md | Updates docs examples to "image". |
| docs/core-concepts.md | Updates docs examples to "image". |
| apps/server/src/rpc.test.ts | Updates RPC tests to use structured content tool responses. |
| apps/desktop/src/shared/rpc.ts | Updates desktop RPC types to return BuiltinToolCallResponse. |
| apps/desktop/src/client/tool-execution.ts | Updates desktop tool execution wrapper to return structured content. |
| apps/desktop/src/client/built-in-tools.ts | Updates built-in tool RPC client to return BuiltinToolCallResponse. |
| apps/desktop/src/bun/remote/remote-runtime-client.ts | Updates remote runtime client typing for the new built-in call response. |
Comments suppressed due to low confidence (1)
packages/core/src/server/storage/blob/image-blobs.ts:79
- Even after accepting legacy
image_data, the current mapping returns the original object unchanged whendatais unchanged, sotype: "image_data"won’t be migrated totype: "image". That keeps older threads incompatible with the rest of the code that now checks fortype === "image".
if (_isImageContent(value)) {
const nextData = fn(value.data);
return nextData === value.data ? value : { ...value, data: nextData };
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+39
to
+43
| function _isImageContent(value: unknown): value is StoredImageContent { | ||
| return ( | ||
| !!value && | ||
| typeof value === "object" && | ||
| (value as { type?: unknown }).type === "image_data" && | ||
| (value as { type?: unknown }).type === "image" && |
Comment on lines
522
to
+526
| const match = /^data:(image\/\w+);base64,(.*)$/s.exec(url); | ||
| if (match?.[1] === undefined || match[2] === undefined) { | ||
| return undefined; | ||
| } | ||
| return _imageData(match[1], match[2]); | ||
| return _imageContent(match[1], match[2]); |
Comment on lines
31
to
35
| * `image/png`, `image/jpeg`, `image/gif`, `image/webp`, etc. | ||
| */ | ||
| mimeType: Type.String({ pattern: /^image\/\w+$/ }), | ||
| mimeType: Type.String(), | ||
|
|
||
| /** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request standardizes the handling of image content throughout the codebase by replacing the legacy
"image_data"type with the unified"image"type. It updates type definitions, parsing logic, storage, and tests to ensure consistency and compatibility with the new format. Additionally, it improves the handling of built-in tool call responses by updating their type signatures and usage across client and server code.Image Content Standardization
"image_data"type with"image"in message content, including type definitions, parsing logic, and documentation, ensuring a consistent image representation throughout the system. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]"image"type, including example messages and storage handling. [1] [2] [3] [4]Built-in Tool Call Response Improvements
BuiltinToolCallResponsetype for built-in tool call responses, updating function signatures, type imports, and RPC interfaces to use the new structure. [1] [2] [3] [4] [5] [6] [7] [8]contentfield is used instead of the previouscontentText. [1] [2] [3]Utility and Summary Function Updates
These changes together ensure a more robust, consistent, and future-proof handling of image content and tool call responses across the codebase.