Skip to content

fix: Linux/WebKitGTK image intake — file picker, clipboard paste, and drop-zone claim - #3124

Merged
kev1n77 merged 4 commits into
GCWing:mainfrom
EthanCheung-gh:fix/webkitgtk-image-intake
Sep 20, 2026
Merged

kev1n77 merged 4 commits into
GCWing:mainfrom
EthanCheung-gh:fix/webkitgtk-image-intake

Conversation

@EthanCheung-gh

Copy link
Copy Markdown
Contributor

Summary

On Linux (WebKitGTK 2.52.x), images could not be inserted into the chat
composer at all: the "Add image" picker silently dropped every selection,
and pasting a screenshot did nothing. Both paths are now fixed and verified
on real hardware; a third, partial improvement un-claims the composer drop
zone for unidentified drags.

All three failures are WebKitGTK-specific engine gaps, not regressions in
this repo — WebView2 and WKWebView never hit them, which is why they went
unnoticed upstream.

Root causes (probed on WebKitGTK 2.52.6 / GNOME 50.1 / Wayland)

  1. File picker — detached <input type=file> never receives change.
    The picker was created via createElement and clicked without being
    mounted. WebKitGTK opens the native chooser, the user picks a file, and
    the engine never fires change on the detached element — the selection
    is silently dropped (no thumbnail, no log, no error). Verified with a
    minimal GTK+WebKit probe: attached inputs fire change; detached ones
    do not.

  2. Paste — DataTransfer is permanently empty.
    WebKitGTK delivers paste events with zero items and zero types
    (even for text), while getData() keeps working — so text paste works
    but the in-page file branch can never see an image. The Async Clipboard
    API (navigator.clipboard.read()) is not a way out either: its promise
    never settles on this engine. Pasted images were simply unreachable
    from the page.

  3. Drag & drop — the composer drop zone claims unidentified drags.
    ContextDropZone called preventDefault on every dragenter/dragover.
    On WebKitGTK an OS file drag reports no types at all, so the claim
    swallowed the drag while the in-page handler found nothing to handle.
    (Partial fix — see "Known limitations".)

The commits

fix(web-ui): mount the image picker so WebKitGTK delivers change

Mounts the picker offscreen (position: fixed; left: -9999px —
display: none is deliberately avoided because some WebKit builds refuse
to open a chooser for it) and reclaims the node on both the change path
and the cancel path (one-shot window focus listener).

fix(desktop): read pasted clipboard images through the host on WebKitGTK

Adds a get_clipboard_image desktop command that reads the clipboard via
the same wl-paste/xclip tools already used by get_clipboard_files,
sniffs PNG/JPEG magic bytes, and returns base64 pixels. The composer now
listens for paste directly and — only when the webview reported zero
types (the WebKitGTK shape; WebView2/WKWebView deliver images in-band and
keep their existing path) — asks the host for the image and reuses the
existing clipboard-image intake, so limits and error reporting are
identical. Registers the command in the remote-surface registry
(LocalOnly) and in the attachments capability evidence; capability
artifacts regenerated.

fix(web-ui): stop claiming empty-typed drags in the composer drop zone

The zone now claims only drags it can positively identify and handle:
internal context payloads, typed non-file drags, and Files drags when a
DOM file handler is actually provided (the Windows desktop runtime).
Empty-typed drags stay unclaimed and keep flowing to the pane-level
native drop path.

Verification

Real device: Ubuntu 26.04, GNOME 50.1, Wayland, WebKitGTK 2.52.6, deb
install shape.

Path Before After
+ → Add image → picker silently dropped ✅ attaches (select and cancel paths)
Ctrl+V screenshot paste nothing ✅ attaches
Drop file on conversation pane worked still works
Drop file directly on composer silently dropped ⚠️ still not attaching (see below)
  • Automated end-to-end through the embedded WebDriver of an isolated
    desktop instance: synthetic empty-typed paste on the composer →
    host clipboard read → attachment thumbnail rendered.
  • Focused suites: web-ui 24/24 (new regression tests for the mounted
    picker, the paste fallback gating, and the unclaimed drag), desktop
    clipboard API 14/14, product-domains registry 40/40,
    check:core-boundaries green, capabilities:generate regenerated.

Known limitations

  • Dropping an OS file directly on the composer still does not attach
    on WebKitGTK: a document-level dragover preventDefault keeps the
    subtree claimed and the native window drop handler never sees the
    release over the composer. The third commit removes the zone's own
    silent swallow and is a step toward the fix, but full on-composer
    native drops need follow-up at the WebKitGTK/wry interaction level.
  • jsdom cannot reproduce any of the three engine gaps (detached-input
    change fires fine there; DataTransfer shape is synthetic), so the
    new tests lock in the wiring and the gating decisions, and the real
    verification is the on-device matrix above.

Platform impact

No behavior change on Windows or macOS: WebView2/WKWebView populate
DataTransfer and deliver change for detached inputs, so the fallback
paths never trigger and the in-band paths are untouched.

@GCWing
GCWing requested a review from kev1n77 September 19, 2026 07:55
@kev1n77

kev1n77 commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the fix.

Before merging, please address the remaining integration issues:

  • Missing wl-paste/xclip currently results in a silent empty response. Please return an explicit error/unsupported state or ensure these dependencies are packaged.
  • Use the repository’s process-manager abstraction instead of calling std::process::Command directly.
  • Please rebase/merge the latest main and resolve the current conflicts.
  • Please attach screenshots or a short recording showing the file-picker and clipboard-image flows on Linux/WebKitGTK.

After these are addressed, this should be ready for another review.

The composer's Add-image flow created a detached <input type=file> and
clicked it. On WebKitGTK (Linux) the native chooser opens, the user
picks a file, and the engine never fires change on the detached input,
so every selection was silently dropped - no thumbnail, no log, no
error. WebView2 and WKWebView deliver change either way, which is why
only Linux users hit this.

Mount the picker offscreen before clicking it (display:none is avoided
on purpose: some WebKit builds refuse to open a chooser for it), and
reclaim the node on both the change path and the cancel path via a
one-shot window focus listener.

Verified on Linux/WebKitGTK 2.52 with an isolated desktop instance:
picker selection now lands in the composer.
WebKitGTK delivers paste events with empty DataTransfer items and types,
and its Async Clipboard API promise never settles, so a pasted image was
unreachable from the page: the composer's file branch never ran and the
paste landed as nothing. Add a get_clipboard_image desktop command that
reads the clipboard through the same wl-paste/xclip tools as
get_clipboard_files, sniffs magic bytes, and returns base64 pixels. The
composer now listens for paste directly and, only when the webview
reported zero types (the WebKitGTK shape; WebView2 and WKWebView
deliver images in-band), asks the host for the image and reuses the
existing clipboard-image intake with identical limits and errors.

Registers the command in the remote-surface registry (LocalOnly) and in
the attachments capability evidence; regenerates the capability
artifacts.
ContextDropZone called preventDefault on every dragenter/dragover,
which turned the composer into a DOM drop target for OS file drags. On
WebKitGTK an OS file drag reports no dataTransfer types, so once the
zone was claimed, the in-page handler saw nothing to handle while the
claim itself interfered with the drag pass-through.

Claim only drags the zone can positively identify and handle: internal
context payloads, typed non-file drags, and Files drags when a DOM file
handler is actually provided (the Windows desktop runtime). Empty-typed
drags stay unclaimed and keep flowing to the pane-level native drop
path.

Known limitation: with this change the zone no longer swallows
empty-typed drags, but dropping an OS file directly on the composer
still does not attach on WebKitGTK 2.52 - a document-level dragover
preventDefault keeps the subtree claimed and the native window drop
handler still never sees the release over the composer. Drops elsewhere
in the conversation pane work. Full on-composer native drops need
follow-up at the WebKitGTK/wry interaction level.
Review follow-up for the WebKitGTK clipboard-image fallback:

- get_clipboard_image now distinguishes "clipboard holds no image"
  from "neither wl-paste nor xclip could be spawned". The latter
  returns an explicit clipboard_image_unsupported error that the
  composer surfaces as a localized warning instead of silently doing
  nothing.
- Linux clipboard tool spawns go through the repository process-manager
  facade (openbitfun_core::util::process_manager::create_command)
  instead of bare std::process::Command, matching the repo rule for
  GUI-hosted child processes.
@EthanCheung-gh
EthanCheung-gh force-pushed the fix/webkitgtk-image-intake branch from 2f9b9f8 to 610801c Compare September 20, 2026 10:34
@EthanCheung-gh

Copy link
Copy Markdown
Contributor Author

1.mark-shot-20260920-191435
2.
mark-shot-20260920-191510
All four points addressed — force-pushed as 09710bd9e → 610801ce4 (4 commits, rebased onto current main).

1. Missing wl-paste/xclip is now an explicit unsupported state. get_clipboard_image distinguishes "clipboard holds no image" from "neither reader tool could be spawned". The latter returns a clipboard_image_unsupported: error, which the composer surfaces as a localized warning (input.clipboardImageToolsUnavailable, en-US/zh-CN/zh-TW; i18n:audit passes) instead of silently doing nothing. We chose the explicit error over packaging the tools: wl-clipboard/xclip pull in display-stack dependencies that don't belong in the deb Depends for an optional fallback path.

2. Process-manager abstraction. All Linux clipboard tool spawns — the new image readers and the pre-existing get_clipboard_files uri-list helpers in the same module — now go through openbitfun_core::util::process_manager::create_command instead of bare std::process::Command. The untouched macos_clipboard osascript call predates this PR; happy to convert it in a follow-up.

3. Rebased onto latest main (8471a65cd). Conflicts were limited to generated capability artifacts; resolved by taking main's versions and regenerating with capabilities:generate from the merged sources.

4. Screenshots on Ubuntu 26.04 / GNOME Wayland / WebKitGTK 2.52.6 (attached):

  • Screenshot 1 — "+" → Add image → file-picker selection renders as an attachment thumbnail.
  • Screenshot 2 — screenshot into the clipboard → Ctrl+V on the composer → pasted image attaches through the host clipboard read.

Focused suites: desktop clipboard API 14/14, web-ui intake/drop-zone tests 9/9. As with #3123, fork-PR checks need a maintainer workflow approval to run.

@EthanCheung-gh EthanCheung-gh changed the title fix: Linux/WebKitGTK image intake — file picker and clipboard pasteintake fix: Linux/WebKitGTK image intake — file picker, clipboard paste, and drop-zone claim Sep 20, 2026
@kev1n77
kev1n77 merged commit 757c4c7 into GCWing:main Sep 20, 2026
13 checks passed
@EthanCheung-gh
EthanCheung-gh deleted the fix/webkitgtk-image-intake branch September 20, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants