Skip to content

mobile: Support IME composition, file picking and native touch gestures - #53

Open
huacnlee wants to merge 5 commits into
itsbalamurali:mainfrom
longbridge:upstream/06-fix-support-mobile-ime-file-picking-and
Open

huacnlee wants to merge 5 commits into
itsbalamurali:mainfrom
longbridge:upstream/06-fix-support-mobile-ime-file-picking-and

Conversation

@huacnlee

@huacnlee huacnlee commented Sep 27, 2026 •

Copy link
Copy Markdown

Summary

iOS and Android Chinese keyboards could not reliably compose text, Add files returned an error or no result, and converting every drag to scroll prevented controls from claiming touch gestures.

  • Use a native UITextView composition buffer to provide UITextInput support and forward plain and attributed marked text, UTF-16 selection, commits, and deletion to GPUI.
  • Present an asynchronous UIDocumentPicker, returning sandbox copies of selected files (including images in Files), with multi-selection and cancellation support.
  • Forward UIKit contacts as GPUI TouchEvents with stable IDs. GPUI's existing recognizer handles taps, long presses, control drags, scrolling, and inertia, replacing the duplicate iOS mouse/scroll synthesis.
  • Add an Android InputConnection bridge with native-thread composition delivery, UTF-16 selection, Unicode-safe deletion, and session checks for stale callbacks. Keep legacy TextInput commit/backspace callbacks working. Custom hosts use GpuiInputActivity; the example inherits it.
  • Connect Android file prompts to SAF and copy content URIs into app cache on a worker before returning readable paths. Register the picker Activity in the example.
  • Forward Android contacts as TouchEvents, preserve multiple pointer identities, and distinguish cancellation from release. Remove duplicate mouse/momentum synthesis, desktop Fontconfig loading on Android, and unconditional frame presentation (matching the existing ai-chat host workarounds).
  • Fix two simulator Debug crashes: UIKit dynamically forwards public keyboard trait setters, and NSString byte initializers require const void * rather than byte-pointer encoding. Correct the matching URL/maps initializer calls too.

This uses existing gpui-pre APIs; no GPUI core changes. TextView selection and scrollbar thumb dragging also require the companion gpui-kit handlers. The file pickers accept images accessible through Files/Documents; they do not add a separate Photos library entry point.

Touch-driven focus changes finish the current composition before dispatch. On iOS, programmatic focus switches without a touch or keyboard hide still need a platform focus identity signal; clearing the previous GPUI field's marked state on programmatic blur remains a follow-up.

Test Plan

  • Passed: cargo check --target aarch64-apple-ios --lib --offline --no-default-features --features camera,video_player.

  • Passed: cargo check --target aarch64-apple-ios-sim --lib --offline --no-default-features --features camera,video_player.

  • Passed: Android backend library/test cross-compilation with NDK 27, and the complete ai-chat Android host's cargo ndk -t arm64-v8a --platform 26 check --locked --offline using these local changes.

  • Passed: Android API 34 Java compilation, including the host's :app:compileDebugJavaWithJavac task.

  • Passed: 2 host-run regression tests for the actual Unicode deletion and legacy callback helpers; Android touch regression tests compile but were not run on a device.

  • Passed: Rust formatting and git diff --check.

  • Passed on iOS 26.5 simulator Debug: input focus and English input; software Pinyin ni followed by selecting 你 inserts Chinese into the GPUI input; explicit keyboard dismissal; Files picker opens and cancels twice without crashing or reopening the keyboard.

  • Passed: native objc2 Debug probe reproduces the old NSString signature rejection and verifies corrected English, Chinese/emoji, Chinese URL and empty-string roundtrips. Keyboard-trait forwarding was also checked against this UIKit runtime.

  • Passed: attributed-marked-text runtime probe reaches the override once, preserves ni composition and UTF-16 selection, then commits Chinese. Requester confirmed Chinese input and subsequent iOS/Android device testing complete.

  • Passed: full ai-chat Android Debug and optimized Release builds, installation and startup on Samsung Android 12 ARM64; Release package verified non-debuggable.

  • Not addressed by a dedicated fix: the reported held-key repeat case (discrete 256 insert/delete callbacks pass), or the separately identified insertText:alternatives:style: entry point. No system autorepeat validation is claimed by those discrete-callback tests.

🤖 Generated with Claude Code

huacnlee and others added 5 commits September 27, 2026 20:05
Switch the mobile platform and the example from Zed git dependencies to the
matching `gpui-pre` / `gpui-pre-wgpu` 0.3.4 crates.io packages (imported as
`gpui` / `gpui_wgpu`, so no `use` changes are needed). This removes the
`[patch.crates-io]` async-task fork and every git dependency, which makes the
crate publishable to crates.io and lets consumers depend on it without
reproducing Zed's patches.

Adapt to the platform and styling API differences, fix the "wpgu" typo in the
crate description, and update the publish workflow to check the iOS backends
before publishing and to treat the `dry-run` input as the boolean it is.

Test plan:
- cargo check --all-targets
- cargo check --target aarch64-apple-ios / aarch64-apple-ios-sim
- cargo test --lib
Embed GPUI Kit's Markdown TextView in a Swift-owned UIKit window instead
of letting GPUI create the iOS example's window. A child controller
renders into `GPUITextView: UIView`; native Compact / Expanded controls
exercise layout changes. The light English sample puts emphasis, quotes,
nested lists, code, tables, and tasks on the first screen.

Fix two startup failures exposed by the simulator: supply the UIKit
display handle to wgpu 29, and retain the `run_embedded` application
handle so native callbacks cannot outlive the GPUI app. Synchronize
layout and redraw without implicit animations during size changes.
Correct the Xcode static-library paths and build directory.

Pins GPUI Kit to commit `7d9efcd2069f9eaa6eb3ba6345aac4aa7d87c9f7` from
longbridge/gpui-kit#3045, so a sibling local
checkout is not required. No crates are published.

The embedding example supports one GPUI view for the application
lifetime; independent destruction and multiple instances are not
implemented.

- Passed `cargo check --manifest-path example/Cargo.toml --target
aarch64-apple-ios-sim` with the pinned Git dependency.
- Built the Swift app with Xcode for the iOS simulator and ran it on
iPhone 17 Pro / iOS 26.5 using the same GPUI Kit source revision
locally.
- Verified light Markdown rendering, native controls, and Compact /
Expanded resizing.
- Passed `git diff --check` and `bash -n example/build.sh`.
- Embedded scrolling needs manual verification: the automation tool
emitted touch begin/end without move events. Transient resize jitter
also needs visual confirmation.
- No Android, physical-device, or performance testing performed.

---------

Co-authored-by: Codex <codex@openai.com>
A released iOS touch currently leaves GPUI's synthetic mouse over the
last control, keeping hover feedback active. Clear the pointer hit test
and emit MouseExited after touch end/cancellation and momentum frames,
while preserving the original position for click and scroll dispatch.

Handle UIKit cancellation separately so it never commits a deferred tap
or starts a momentum fling. Cancel an active scroll with
TouchPhase::Cancelled.

Update the example's GPUI Kit revision to include mobile tooltip
suppression, tap-triggered HoverCard behavior, and the inline is_mobile
helper from longbridge/gpui-kit#3045.

- Built the Swift-hosted iOS example for aarch64-apple-ios-sim.
- Used a temporary simulator smoke view to check HoverCard opens on tap,
remains open when its content is tapped, and closes on outside tap.
- Checked a normal ghost button with a configured tooltip: no lingering
hover background or tooltip after release.
- Restored the original chat example and rebuilt it with the final Kit
revision.
- GPUI Kit regression tests cover desktop hover and mobile tap
interaction.
- UIKit cancellation was reviewed in code; no system-cancellation
gesture was injected. Android was not changed or tested. No crates
published.

Co-authored-by: Codex <codex@openai.com>
## Summary

Apply the iOS touch cleanup from #3 to Android. After touch
release/cancellation and every momentum frame, move the synthetic
pointer outside the window and emit MouseExited so hover does not remain
on the last control. Preserve original coordinates for click and scroll
dispatch.

Handle ACTION_CANCEL independently: discard queued coalesced scroll,
reset velocity, stop inertia, and emit ScrollWheel(Cancelled) for an
active scroll. Never turn cancellation into a tap or fling, and ignore
stray moves with no active touch.

Add regression tests and update an existing test to use the renamed
gpu_specs API.

## Test Plan

- Android arm64/API 31 cargo-ndk library check passed.
- Built the Android unit-test executable and ran touch tests on an
Android API 34 arm64 emulator: 5 passed.
- Tests cover tap coordinates, pending-tap cancellation,
coalesced-scroll cancellation, and post-momentum hover cleanup.
- This verification covers platform event dispatch; Android app UI
testing is ongoing separately.

Co-authored-by: Codex <codex@openai.com>
iOS and Android Chinese keyboards could not reliably compose text, Add
files returned an error or no result, and converting every drag to
scroll prevented controls from claiming touch gestures.

- Use a native UITextView composition buffer to provide UITextInput
support and forward plain and attributed marked text, UTF-16 selection,
commits, and deletion to GPUI.
- Present an asynchronous UIDocumentPicker, returning sandbox copies of
selected files (including images in Files), with multi-selection and
cancellation support.
- Forward UIKit contacts as GPUI TouchEvents with stable IDs. GPUI's
existing recognizer handles taps, long presses, control drags,
scrolling, and inertia, replacing the duplicate iOS mouse/scroll
synthesis.
- Add an Android InputConnection bridge with native-thread composition
delivery, UTF-16 selection, Unicode-safe deletion, and session checks
for stale callbacks. Keep legacy TextInput commit/backspace callbacks
working. Custom hosts use GpuiInputActivity; the example inherits it.
- Connect Android file prompts to SAF and copy content URIs into app
cache on a worker before returning readable paths. Register the picker
Activity in the example.
- Forward Android contacts as TouchEvents, preserve multiple pointer
identities, and distinguish cancellation from release. Remove duplicate
mouse/momentum synthesis, desktop Fontconfig loading on Android, and
unconditional frame presentation (matching the existing ai-chat host
workarounds).
- Fix two simulator Debug crashes: UIKit dynamically forwards public
keyboard trait setters, and NSString byte initializers require `const
void *` rather than byte-pointer encoding. Correct the matching URL/maps
initializer calls too.

This uses existing GPUI 0.3.4 APIs; no GPUI core changes. TextView
selection and scrollbar thumb dragging also require the companion
gpui-kit handlers. The file pickers accept images accessible through
Files/Documents; they do not add a separate Photos library entry point.

Touch-driven focus changes finish the current composition before
dispatch. On iOS, programmatic focus switches without a touch or
keyboard hide still need a platform focus identity signal; clearing the
previous GPUI field's marked state on programmatic blur remains a
follow-up.

- Passed: `cargo check --target aarch64-apple-ios --lib --offline
--no-default-features --features camera,video_player`.
- Passed: `cargo check --target aarch64-apple-ios-sim --lib --offline
--no-default-features --features camera,video_player`.
- Passed: Android backend library/test cross-compilation with NDK 27,
and the complete ai-chat Android host's `cargo ndk -t arm64-v8a
--platform 26 check --locked --offline` using these local changes.
- Passed: Android API 34 Java compilation, including the host's
`:app:compileDebugJavaWithJavac` task.
- Passed: 2 host-run regression tests for the actual Unicode deletion
and legacy callback helpers; Android touch regression tests compile but
were not run on a device.
- Passed: Rust formatting and `git diff --check`.
- Passed on iOS 26.5 simulator Debug: input focus and English input;
software Pinyin `ni` followed by selecting `你` inserts Chinese into the
GPUI input; explicit keyboard dismissal; Files picker opens and cancels
twice without crashing or reopening the keyboard.
- Passed: native objc2 Debug probe reproduces the old NSString signature
rejection and verifies corrected English, Chinese/emoji, Chinese URL and
empty-string roundtrips. Keyboard-trait forwarding was also checked
against this UIKit runtime.

- Passed: attributed-marked-text runtime probe reaches the override
once, preserves ni composition and UTF-16 selection, then commits
Chinese. Requester confirmed Chinese input and subsequent iOS/Android
device testing complete.
- Passed: full ai-chat Android Debug and optimized Release builds,
installation and startup on Samsung Android 12 ARM64; Release package
verified non-debuggable.
- Not addressed by a dedicated fix: the reported held-key repeat case
(discrete 256 insert/delete callbacks pass), or the separately
identified insertText:alternatives:style: entry point. No system
autorepeat validation is claimed by those discrete-callback tests.

---------

Co-authored-by: Codex <codex@openai.com>
@huacnlee
huacnlee force-pushed the upstream/06-fix-support-mobile-ime-file-picking-and branch from 7591973 to b5fed30 Compare September 27, 2026 12:06
@huacnlee huacnlee changed the title Fix support mobile IME, file picking and native touch gestures mobile: Support IME composition, file picking and native touch gestures Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant