Skip to content

fix(images): preserve cross-client attachments and image understanding - #2946

Merged
bobleer merged 3 commits into
GCWing:mainfrom
bobleer:bob/fix-cross-client-image-understanding
Sep 10, 2026
Merged

bobleer merged 3 commits into
GCWing:mainfrom
bobleer:bob/fix-cross-client-image-understanding

Conversation

@bobleer

@bobleer bobleer commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and behavior

Inline images from mobile and CLI could reach a text-only primary model without a path that analyze_image could read. Context persistence also stripped the only copy of those pixels, while some native send/retry paths discarded attachments before the host acknowledged them. CLI Peer submissions silently omitted imageContexts.

The receiving Runtime now saves inline attachments atomically under a content-addressed runtime artifact path before turn admission or steering. Native multimodal models receive pixels on every round; text-only models receive the exact tool-readable path and require the configured image-understanding model and enabled image tool. Canonical image messages remain intact for retries and restoration. SSH workspace reads use workspace services, with no fallback to same-named controller files.

Mobile web, Android, iOS and HarmonyOS prepare supported image payloads within their transport budgets, preserve attachments until acknowledgement, and retain newer drafts on success or failure. Native retries include the original images. CLI Peer accepts image inputs; a negotiated inline-attachment capability preserves the old upload path for older hosts. Web payloads retain pixels for Detached Dispatch.

Validation

  • Focused Rust image preparation, provider projection, persistence, remote filesystem and analyze_image tests: 26 passed on the final revision. Provider projection covers OpenAI, Anthropic, Responses and Gemini, including steering.
  • CLI image-related tests: 19 passed, including Peer payload parsing and legacy requests.
  • Shared native session store: 62 passed, including image-only send, failed send, retry, acknowledgement and concurrent draft edits.
  • Android debug build, iOS arm64 simulator build, HarmonyOS assemble and local tests passed.
  • Web/Peer/mobile submission focused tests: 37 passed; Detached Dispatch contract/API tests: 13 passed. Mobile image preparation: 4 passed; mobile component checks, type check and build passed.
  • Chrome manual image preparation check: small PNG retained, 3000×1800 image converted to 1920×1152 JPEG, corrupt image rejected.
  • iOS pure Swift tests and Web lint passed.
  • Web type/theme checks, i18n audit, capability generation/contract tests, core boundaries, repository hygiene and changed Rust formatting passed.

Remote evidence uses simulated SSH filesystem ports and Peer/Dispatch/RemoteCommand contracts. No live provider, actual SSH connection, or physical phone end-to-end run was performed. Shared TUI keeps its explicit image-unsupported response because its existing bounded IPC transport does not carry these attachments.

Two unchanged account declarations fail the local native architecture default-argument audit; the minimal Core image filter also encounters an unchanged catalog test that assumes a product tool plan. Focused owner tests pass; upstream CI results are tracked on this PR.

@bobleer
bobleer merged commit e0ddcf9 into GCWing:main Sep 10, 2026
13 checks passed
@bobleer
bobleer deleted the bob/fix-cross-client-image-understanding branch September 24, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant