feat(relay): support GIF image prompts#64
Open
zikolach wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds inbound image/gif support across PiRelay’s messenger relays by treating GIF as a convertible inbound image format (converted to first-frame PNG) while keeping the existing direct/static image MIME allow-list semantics for model-ready and outbound image handling.
Changes:
- Introduces shared inbound-image helpers to classify direct vs convertible image MIME types and convert GIF first frames to PNG with size/pixel bounds.
- Wires GIF prompt handling through Telegram (runtime + broker parity) and through Discord/Slack attachment download → prompt injection flows.
- Updates tests, OpenSpec requirements, and user-facing docs to describe GIF first-frame conversion and limits.
Reviewed changes
Copilot reviewed 35 out of 36 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/telegram-api.test.ts | Adds Telegram API tests for GIF document normalization and GIF→PNG download conversion. |
| tests/slack-runtime.test.ts | Adds Slack runtime test ensuring downloaded GIF attachments become PNG prompt images. |
| tests/slack-adapter.test.ts | Extends Slack adapter tests for GIF normalization + capability reporting of convertible formats. |
| tests/runtime.test.ts | Adds end-to-end-ish Telegram runtime tests for caption/fallback behavior and safe conversion failures. |
| tests/relay/media.test.ts | Adds unit tests for direct vs convertible MIME classification and GIF→PNG conversion bounds/errors. |
| tests/discord-runtime.test.ts | Adds Discord runtime test ensuring downloaded GIF attachments become PNG prompt images. |
| tests/discord-adapter.test.ts | Extends Discord adapter tests for GIF normalization + capability reporting of convertible formats. |
| README.md | Updates feature/docs text to mention inbound GIF first-frame conversion. |
| package.json | Adds runtime deps gifuct-js, pngjs (+ @types/pngjs). |
| package-lock.json | Locks new GIF/PNG conversion dependencies. |
| openspec/specs/relay-configuration/spec.md | Adds requirement describing safe GIF conversion defaults and limit enforcement. |
| openspec/specs/relay-channel-adapters/spec.md | Adds requirement that adapters distinguish direct vs convertible image MIME support. |
| openspec/specs/messenger-relay-sessions/spec.md | Adds requirement for GIF first-frame conversion prompt semantics and failure safety. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/tasks.md | Archives spec-driven change task list for GIF first-frame input. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/specs/relay-configuration/spec.md | Archived spec delta for relay configuration requirement. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/specs/relay-channel-adapters/spec.md | Archived spec delta for adapter capability requirement. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/specs/messenger-relay-sessions/spec.md | Archived spec delta for GIF prompt requirement. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/proposal.md | Archived proposal describing rationale and scope. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/design.md | Archived design describing conversion approach + safety constraints. |
| openspec/changes/archive/2026-05-27-add-gif-first-frame-image-input/.openspec.yaml | Adds OpenSpec metadata for the archived change. |
| extensions/relay/media/index.ts | Exports the new inbound image helper module. |
| extensions/relay/media/inbound-image.ts | Implements MIME classification + GIF first-frame decode → PNG encode with bounds checks. |
| extensions/relay/core/utils.ts | Adds wrappers that expose inbound image acceptance/format text and prompt-image preparation. |
| extensions/relay/core/channel-adapter.ts | Extends ChannelCapabilities with optional convertibleInboundImageMimeTypes. |
| extensions/relay/core/adapter-contracts.ts | Extends MessengerCapabilities with optional convertibleInboundImageMimeTypes. |
| extensions/relay/broker/process.js | Updates broker Telegram image handling to accept/convert GIFs and improve format messaging. |
| extensions/relay/adapters/telegram/runtime.ts | Updates Telegram runtime accepted formats messaging + consistent fallback text for image-only prompts. |
| extensions/relay/adapters/telegram/middleware.ts | Declares Telegram convertible inbound GIF capability. |
| extensions/relay/adapters/telegram/api.ts | Accepts inbound GIF documents and converts downloads to PNG prompt images. |
| extensions/relay/adapters/telegram/adapter.ts | Declares Telegram capabilities including convertible inbound GIF formats. |
| extensions/relay/adapters/slack/runtime.ts | Downloads Slack image attachments and injects converted PNG prompt image blocks. |
| extensions/relay/adapters/slack/adapter.ts | Treats GIF as convertible inbound image; updates supported formats + error messaging. |
| extensions/relay/adapters/discord/runtime.ts | Downloads Discord image attachments and injects converted PNG prompt image blocks. |
| extensions/relay/adapters/discord/adapter.ts | Treats GIF as convertible inbound image; updates supported formats + error messaging. |
| docs/testing.md | Adds Telegram GIF smoke-test steps and negative-case coverage. |
| docs/config.md | Documents allowedImageMimeTypes semantics vs inbound GIF conversion behavior. |
Comment on lines
4
to
8
| import type { AgentMessage } from "@mariozechner/pi-agent-core"; | ||
| import type { AssistantMessage, ImageContent, Model, TextContent } from "@mariozechner/pi-ai"; | ||
| import type { ImageFileLoadResult, LatestTurnImage, LatestTurnImageFileCandidate } from "./types.js"; | ||
| import { acceptedInboundImageFormatsText as formatAcceptedInboundImageFormats, acceptedInboundImageMimeTypes as listAcceptedInboundImageMimeTypes, isAcceptedInboundImageMimeType as mediaIsAcceptedInboundImageMimeType, prepareInboundImageForPrompt as mediaPrepareInboundImageForPrompt } from "../media/inbound-image.js"; | ||
| import type { DeliveryMode, ParsedTelegramCommand, TelegramBindingMetadata, TelegramOutboundChunk, TelegramUserSummary } from "./types.js"; |
Comment on lines
+117
to
+123
| const png = new PNG({ width, height }); | ||
| for (let y = 0; y < firstFrame.dims.height; y++) { | ||
| const sourceStart = y * firstFrame.dims.width * 4; | ||
| const targetStart = ((firstFrame.dims.top + y) * width + firstFrame.dims.left) * 4; | ||
| png.data.set(firstFrame.patch.subarray(sourceStart, sourceStart + firstFrame.dims.width * 4), targetStart); | ||
| } | ||
|
|
zikolach
force-pushed
the
feat/gif-first-frame-image-input
branch
from
7b33817 to
8a33b01
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation