android: expose GPUI's accessibility tree to TalkBack

[Bombatomica64]

GPUI 0.3.7 already builds an AccessKit tree and hands it to PlatformWindow::a11y_init / a11y_tree_update, but the Android window leaves both as no-ops, so TalkBack only sees one opaque surface. This PR connects them to accesskit_android's InjectingAdapter. That adapter installs an AccessibilityDelegate on the existing host View, so hosts don't need any Java changes.

It's one new module (src/android/accessibility.rs) plus two hooks in window.rs and two small changes in jni.rs, about 250 lines in total.

How it works

• a11y_init stores GPUI's callbacks and attaches the adapter to the host view. That's the content view on the host-driven path, where the SurfaceView lives, and the decor view on the android-activity path.
• a11y_tree_update forwards each tree to the adapter. The adapter posts its work to the UI thread itself.
• set_host_activity moves the adapter onto the new Activity's views when the Activity is recreated.
• Each window keeps its own callbacks. With one window per host Activity (android: one window per host, GL fallback, backend pinning and IME fixes #26), the window that becomes active takes the adapter over.
• On the android-activity path, hover events are now reported as unhandled. GPUI doesn't use hover on mobile, and TalkBack's touch exploration arrives as hover events, which the delegate on the decor view needs to see.

Two things worth flagging

• Version. accesskit_android 0.7.5 is the newest release built on accesskit 0.24, which is the version gpui-pre 0.3.7 uses. It can move to 0.9 when GPUI moves to accesskit 0.25.
• Workaround. 0.7.5 raises accessibility events without checking AccessibilityManager.isEnabled(). Android throws on the UI thread when no service is listening, so the app crashed as soon as the screen reader disconnected. This PR checks first. Once accessibility is off, it calls GPUI's deactivation callback and attaches a fresh adapter, so turning the screen reader back on activates GPUI again instead of reading a stale tree. 0.9 does the isEnabled() check internally, so most of this can go away with the upgrade.

Testing

Tested with the host-driven path in a small lab app on GPUI Kit 0.7.0.

OnePlus CPH2581 (Android 16), TalkBack. Touch exploration lands on the right controls, double-tap toggles a checkbox, and turning TalkBack off with the volume keys no longer crashes the app:

Redroid (Android 14), uiautomator dump. Every node's bounds drawn over the rendering. Kit's checkboxes, radios, switches and sliders come through as their native Android classes with labels and checked state:

android-activity (NativeActivity) path, same phone. I tested the same bridge ported to the NativeActivity example in itsbalamurali/gpui-mobile#69, because the lab app uses the host-driven path. That's where the hover fix above came from: before it, touch exploration never reached the delegate. With it, TalkBack reads the controls and double-tap toggles them. One limitation remains: Android doesn't draw its green focus ring, because NativeActivity takes over the window surface that the ring would be drawn on.

Happy to change the approach if you'd prefer something different, for example a Java-side AccessibilityNodeProvider in the host instead of the injected delegate.

🤖 Generated with Claude Code

[Bombatomica64]

I added two fixes after more testing, each as its own commit:

• d4ae7c7. TalkBack turned off and back on read a stale screen. The adapter stayed active, so GPUI was never activated again, and it kept building trees nobody read. Now, once AccessibilityManager.isEnabled() is false, the bridge calls GPUI's deactivation callback and swaps in a fresh adapter. The next screen reader then activates GPUI cleanly.
• a37bff0. Touch exploration on the android-activity (NativeActivity) path. Hover events went to the native input queue and were reported as handled even though GPUI skips them, so they never reached the decor view's accessibility delegate. They're now reported as unhandled, the same way touches on platform views already are.

I found both while porting this to the current Zed GPUI for itsbalamurali/gpui-mobile, whose example only runs as a NativeActivity. Verified with TalkBack on a OnePlus CPH2581 (Android 16), and with repeated activate/deactivate cycles on redroid.

[Bombatomica64]

Rebased onto main after #24, #26 and #27. It applied cleanly, but #26 now gives each host Activity its own window. The bridge assumed a single window, so I added one commit:

• d7e7a73. The screen reader follows the active window. Each window keeps its own GPUI callbacks. When a window becomes active (set_active(true), e.g. going back to an earlier Activity), it takes the adapter over, and the window it replaces gets GPUI's deactivation callback so it stops building trees. Trees from a window that isn't the active one are ignored.

One thing I noticed while doing this, not changed here: HOST_ACTIVITY only follows set_host_activity, which hosts call from onCreate. When the user goes back from B to A, activity_as_ptr() still names B, so the bridge (like the IME's restore_keyboard) would attach to B's views. A host that also calls set_host_activity from onResume avoids it. Happy to send that as a doc change or a separate fix if it's useful.

Checked with cargo ndk -t arm64-v8a clippy (no new warnings). I haven't re-run TalkBack on a device with two stacked Activities yet.

[huacnlee]

Thanks, getting GPUI's tree in front of TalkBack is a big step. A few issues need fixing before this can merge. These come from reading the diff together with accesskit_android 0.7.5; I have not reproduced them on a device.

1. Wrong Activity after going back (accessibility.rs, init)
init (also run from set_active(true)) attaches through activity_as_ptr(), which is the Activity that most recently called set_host_activity. With host A, then host B, then Back: A resumes, but HOST_ACTIVITY.current is still B, so A's adapter is installed on B's finishing content view and A goes back to being one opaque surface for TalkBack. The adapter should be tied to the window's own host Activity/view.

2. The isEnabled() check still races with raising events (accessibility.rs, update)
update_if_active only posts events.raise to the UI thread, and that runnable doesn't check again. If TalkBack is turned off between the GPUI-thread isEnabled() call and the runnable running, AccessibilityManager throws, the unwrap inside the crate's extern "system" callback panics across FFI, and the process aborts. That is the same crash this workaround is meant to prevent. The check has to happen on the UI thread, right before sending (or the exception has to be caught there).

3. BRIDGE is never cleared when the window closes (window.rs)
After HostDestroyed closes the active window, BRIDGE still holds that window's callbacks and an adapter installed on its view. A later TalkBack activation calls the dead window's activation and gets a stale tree, and the next init calls deactivation on the closed window. Please clear the bridge (dropping the adapter) when the window that owns it is dropped.

4. Abort if the view already has an AccessibilityDelegate (accessibility.rs, attach)
InjectingAdapter::new posts a runnable that panics with "host already has an accessibility delegate". Because it runs inside an extern "system" callback, this aborts the process. A host app, or an androidx/Compose layer, can easily have set its own delegate on android.R.id.content. Please check for an existing delegate first and fall back (log and skip) rather than crash.

5. attach is never retried after a failure (accessibility.rs:85)
init returns early when the same callbacks are already registered, even if adapter is None because the earlier attach failed (VM/Activity not ready, or a JNI error). Accessibility then stays off for that window. It should retry when adapter.is_none().

6. Attaching during onCreate happens before setContentView (jni.rs, activity_changed)
With AppCompat, the android.R.id.content id moves to the subDecor's ContentFrameLayout after setContentView, so the delegate can end up on an outer frame, which offsets node bounds. Consider attaching lazily, or when the surface is created.

Non-blocking:

• update makes a JNI isEnabled() call every frame while holding the global BRIDGE lock, which also blocks activity_changed on the UI thread. An AccessibilityStateChangeListener, or a cached AtomicBool, would avoid this.
• with_host_view uses the jni 0.21 that accesskit_android re-exports and rebuilds the JavaVM on every attach, while the crate already uses newer jni and caches the VM. That means two jni major versions in the build and two JNI styles to maintain.
• In jni.rs, the hover early return comes after the platform-view hit_test, so every hover event during touch exploration pays for a registry lookup that is thrown away. Move the check above it.

[Bombatomica64]

Thanks for the careful review! I rebased onto main (it conflicted with #29)
and added one commit per point:

1. Wrong Activity after Back (2b761c1). New host::set_activity(env, host, activity) does what set_host_activity does and also remembers the Activity
   per HostId. Each window knows its host, so the adapter goes on that host's
   own Activity. Hosts that only call set_host_activity keep the old behaviour,
   and the "What the host has to do" docs now point to set_activity.

2. isEnabled() race (not fully fixed, see below). I don't think
   gpui-mobile can close it on its own. The runnable that raises the events is
   posted by InjectingAdapter::update_if_active inside accesskit_android,
   and nothing runs between it and requestSendAccessibilityEvent. A
   Looper-side check could still lose to a queued MSG_SET_STATE.
   accesskit_android 0.9 fixes this the way you describe: QueuedEvents::raise
   checks isEnabled() on the UI thread. But 0.9 needs accesskit 0.25, and
   gpui-pre 0.3.8 is on 0.24. What changed here:

   • 7006f50 clears the Java exceptions that accesskit_android's jni 0.21
     leaves pending.
   • 31d79d8 makes the GPUI-thread check cheaper and documents the remaining
     window in the module docs.

   The real fix is either a backported accesskit_android 0.7.x, which I can
   ask AccessKit for, or moving to 0.9 once gpui-pre is on accesskit 0.25.
   Would you merge with this documented, or wait for one of those?

3. BRIDGE not cleared (d0bea7c). HostDestroyed and Drop for AndroidWindow now drop the bridge, and with it the adapter, when it belongs
   to that window. The closed window's callbacks are not called.

4. Existing AccessibilityDelegate (34f8c96). Before creating the adapter,
   attach reads getAccessibilityDelegate(). If the delegate is not an earlier
   AccessKit one (whose removal is already queued ahead), it logs and skips the
   view.

5. No retry (ea97c6d). init now attaches again when there is no adapter.
   Since 9418278 it also does so when the window's Activity has changed.

6. Attach before setContentView (9418278). set_host_activity no longer
   touches the adapter. It is attached whenever the window becomes active: on
   resume, and again (as a cheap no-op if nothing changed) when its surface
   arrives.

Non-blocking:

• isEnabled() under the lock (31d79d8): the AccessibilityManager and
  the JavaVM are cached once, and the check runs before BRIDGE is taken.
  Nothing takes BRIDGE on the UI thread any more. An
  AccessibilityStateChangeListener would need a Java class, which this crate
  doesn't ship, so I kept the per-frame call. It is a synchronized field read.
• Two jni versions: the second one stays as long as we use
  accesskit_android, because InjectingAdapter::new takes its jni 0.21
  JNIEnv. The VM is no longer rebuilt per attach (31d79d8).
• Hover before hit_test (6d443bd): moved.

Checked: cargo check and cargo clippy --target aarch64-linux-android (no
warnings in the touched files), cargo fmt --check, and cargo test --lib
(46 passed). I haven't re-tested these fixes on a device yet; I'll follow up with TalkBack results (turning it off mid-draw, Back from a second host) once my phone is reconnected.

[Bombatomica64]

Device results for 31d79d8, on a OnePlus CPH2581 (Android 16), with the lab app (one host Activity):

• TalkBack on from a cold start: every node is read, and focus rectangles sit on their items. The adapter now attaches on resume and surface creation instead of onCreate.
• Going Back from an Activity on top, Home and reopen, and turning TalkBack off and on again: the tree is still read, with no restart needed.
• Turning TalkBack off while the app was drawing: no abort. The race window in point 2 is narrow, so this doesn't prove it's closed.
• The crash log buffer is empty, and the only process exit is the force stop I did for the cold start.

The lab has a single host Activity, so I couldn't stack two GPUI hosts on the device. The A → B → Back case is covered by reading the code, not by a device run.

[huacnlee]

Reviewed 31d79d8. The per-host Activity registration, window callback ownership, bridge cleanup, and attach retry changes address the earlier lifecycle concerns. Two correctness issues still need fixing before merge: the TalkBack shutdown race can still abort the process, and per-frame adapter updates leak JNI local references. Details are inline. CI is green, but these findings come from source inspection of this PR and accesskit_android 0.7.5; I have not reproduced them on an Android device.

[huacnlee]

[P1] Check accessibility state where events are actually raised on the UI thread

The GPUI-thread isEnabled() check still races with the runnable posted by InjectingAdapter::update_if_active. If TalkBack turns off after this check but before that runnable runs, AccessibilityManager can throw; accesskit_android 0.7.5 unwraps the JNI error inside its extern "system" callback, aborting the process. clear_exception below runs on the GPUI thread, so it cannot clear an exception raised later on the UI thread and cannot recover from that panic. The new module comment acknowledges the race, but it remains a crash path. Please backport the UI-thread guard into a compatible adapter release (or use a patched/low-level adapter that checks immediately before raising events) before merging. Device runs that happen not to hit the race do not establish that it is fixed.

[huacnlee]

[P2] Release JNI local references around every adapter update

InjectingAdapter::update_if_active in accesskit_android 0.7.5 creates a local host reference via upgrade_local and, when posting events, a local Runnable reference. Neither is explicitly deleted by that implementation. This call runs on the permanently attached render thread outside a local-reference frame, so those references accumulate across frames and retain Java objects, including old Activity views. with_raw_activity only establishes a frame for attach/is_current; it does not cover this update call. Please obtain the current JNIEnv and wrap update_if_active in with_local_frame so all local references created by each update are released when the update returns. The adapter drop path also creates local references, so cover that path when replacing or clearing the bridge as well. Android documents that references on an attached native thread are not automatically freed until it detaches: https://developer.android.com/ndk/guides/jni-tips#local-and-global-references

[Bombatomica64]

Thanks for the second pass!

P2, local references (fafa6f0): update_if_active now runs in its own local reference frame, and so does the adapter's drop. That covers replacing the adapter, closing the window and switching windows.

P1, the isEnabled() race: agreed, it has to be fixed in the adapter. I see two ways to do that and would like your preference:

1. Backport to accesskit_android 0.7.x. I'd open a PR on AccessKit from the 0.7.5 tag that adds the UI-thread isEnabled() check from QueuedEvents::raise (and #796's local-reference fix), and ask for a 0.7.6 release. Here it would be a version bump, and it works with the current gpui-pre. It depends on AccessKit being willing to cut a release from an old line.
2. Bump gpui-pre to accesskit 0.25 and move this PR to accesskit_android 0.9, which already has both fixes. I tried it locally: I patched gpui-pre 0.3.8's accesskit requirement to 0.25.1 and bumped accesskit_android to 0.9. cargo check --target aarch64-linux-android passes with no changes to this PR's code. The only new output is two deprecation warnings in gpui (accesskit::Tree is now TreeInfo). For desktop, compatible adapters already exist: accesskit_macos 0.27, accesskit_unix 0.23 and accesskit_windows 0.35. But Zed main is still on 0.24, so this means publishing gpui-pre ahead of Zed. I haven't run this setup on a device yet.

Which would you prefer? With 2, part of this PR's workaround could probably go as well, since 0.9 checks isEnabled() itself.

[huacnlee]

审查了最新提交 fafa6f0。新增的 JNI 局部引用清理已覆盖更新和适配器销毁路径，但仍有两项需要在合并前修复：关闭 TalkBack 时的异步事件发送仍可能导致进程 abort，以及 API 26–28 上无条件调用 API 29 方法导致适配器挂载失败。详见行内评论。CI 全部通过；这些结论来自源码审查，未在 Android 真机复现。

[huacnlee]

[P1] 在实际发送事件的 UI 线程修复 TalkBack 关闭竞态

这里的 isEnabled() 检查发生在 GPUI 线程，而 accesskit_android 0.7.5 的 update_if_active() 只将 events.raise 排到 UI 线程。如果 TalkBack 在检查后、runnable 执行前关闭，requestSendAccessibilityEvent 的 JNI 调用仍可能抛异常，依赖中的 unwrap() 会在 extern "system" 回调里 panic，导致整个进程 abort。这里的 clear_exception() 在 GPUI 线程执行，无法清理 UI 线程的异常。因此目前的检查和异常清理仍没有解决此前的崩溃问题。请回移适配器发送事件前的 isEnabled() 检查，或升级到兼容的已修复版本后再合并。

[huacnlee]

[P2] 为 API 26–28 提供兼容的 delegate 检查和挂载路径

项目 README 和示例 minSdk 均声明支持 API 26+，但 View.getAccessibilityDelegate() 从 API 29 才提供。在 API 26–28 上，这个无条件 JNI 调用会发生方法查找错误，attach() 随后记录警告并退出；每次重试都会失败，TalkBack 因而仍只能看到不透明表面。accesskit_android 0.7.5 的 InjectingAdapter 在安装和移除 delegate 时也调用该方法，所以只跳过这里的检查还不够。请同时修复依赖内部的兼容路径。Android API 文档：https://developer.android.com/reference/android/view/View#getAccessibilityDelegate()