Real-time browser streaming and remote input for openpilot UI.
This is a port of the original Qt-based remote UI streamer by ecpp. The openpilot UI has moved from Qt (C++) to raylib/pyray (Python), so the original implementation is no longer functional in newer releases of openpilot. This fixes that.
Current primary path is Python + raylib/pyray:
- Frame capture:
frame_streamer.py - Input injection:
touch_injector.py - UI integration helper:
remote_ui.py - Browser server:
stream_server.py
The C++/Qt files are kept in legacy/ as an implementation reference:
legacy/frame_streamer.cc/.hlegacy/touch_injector.cc/.hlegacy/main.cc
This project moves openpilot UI frames from the render loop to a browser with no disk writes:
- Once a browser connects, the UI process grabs the framebuffer each frame (only the GPU readback runs on the render thread) and a worker thread JPEG-encodes it.
- The encoded frame is written into POSIX shared memory at
/dev/shm/openpilot_ui_frames. stream_server.pyreads shared memory and broadcasts frames over one WebSocket per browser.- The browser draws frames on a
<canvas>and sends touch, mouse, scroll and keyboard input back over the same WebSocket (POST /inputis a fallback). - Input is forwarded to
/tmp/ui_touch_socket, wheretouch_injector.pyapplies it to pyray input hooks. - The server publishes its client count, the browsers' network delay and their requested frame width in
/dev/shm/openpilot_ui_clients, which the UI uses to adapt.
The browser page can be installed as a PWA ("Add to Home Screen"), which runs it full screen in landscape.
-
frame_streamer.py- Captures with
pyray.load_image_from_screen()and hands the image to an encoder thread, so encoding doesn't add to UI frame time. Frames are skipped, not queued, while the encoder is busy. - Skips re-encoding when the screen hasn't changed, re-announcing the previous frame once a second so the stream doesn't look stalled.
- Adapts JPEG quality (55 to 90) and frame rate (4 to 10 FPS) to encode time, frame size and network delay; drops to 1 FPS with no browser connected.
- Encodes at half resolution when every connected browser is small enough not to need full size.
- Publishes to shared memory using a packed 31-byte header + payload.
- Captures with
-
touch_injector.py- Listens on Unix socket
/tmp/ui_touch_socketfor newline-delimited JSON input events. - Monkeypatches pyray polled input functions (mouse, touch, wheel and keyboard) to expose remote state, falling back to local input.
- Keyboard events and pasted text are queued and released one frame at a time.
- Listens on Unix socket
-
remote_ui_protocol.py- Shared IPC layout: shared memory name and header format, and the touch socket path.
-
client_state.py- Tiny shared memory segment the server writes and the UI maps read-only: client count, worst network delay and requested frame width.
-
remote_ui.py- Starts
TouchInjectorandFrameStreamer. Remote input is ignored while openpilot is engaged (local touch keeps working).
- Starts
-
remote_ui_auto.py- Zero-integration hook (see Installation). Starts the server, creates
RemoteUIwhen the first browser connects and streams only while one is connected.
- Zero-integration hook (see Installation). Starts the server, creates
-
stream_server.py- Flask + flask-sock server on port
8081. - Reads shared memory via
mmapand broadcasts the latest frame to connected WebSocket clients. - Forwards browser input to the UI and refuses it while openpilot is engaged (the browser shows a banner).
- Serves the browser client:
remote_ui_client.js,remote_ui.cssand the PWA manifestremote_ui.webmanifest.
- Flask + flask-sock server on port
-
screenshot_server.py(legacy)- Older polling server that serves PNG screenshots from
/tmp/ui_frame_*.png. - Kept for compatibility/testing, not the recommended runtime path.
- Older polling server that serves PNG screenshots from
The C++ files in legacy/ implement the same shared-memory + socket concepts for the older Qt UI. They are useful as protocol references, but are not the primary integration path for current raylib/pyray UI.
Both writer and reader use this packed layout:
struct SharedFrame {
uint64_t timestamp; // epoch ms
uint32_t width;
uint32_t height;
uint32_t size; // JPEG byte length
uint32_t format; // 1 = JPEG
uint8_t ready;
uint16_t source_width; // UI size before scaling (input coordinates use it)
uint16_t source_height;
uint8_t padding[2];
uint8_t data[4 * 1920 * 1080];
};- Metadata size: 31 bytes
- Data capacity: 8,294,400 bytes
- Total shared memory size: 8,294,431 bytes
On the device (or any machine with an openpilot checkout):
curl -sL https://raw.githubusercontent.com/CHaucke89/op-remote-ui-raylib/main/install.sh | bashOr from a clone: ./install.sh [OPENPILOT_DIR] (default /data/openpilot).
The installer:
- Copies the files to
/data/op-remote-ui(override withINSTALL_DIR). - Installs Flask, flask-sock and Pillow into
/data/op-remote-ui/vendorusing openpilot's Python (.venvif present), so openpilot's own environment is untouched. - Adds one marked line to
selfdrive/ui/ui.pythat importsremote_ui_auto. It is safe to re-run.
Restart the UI (or reboot) and open http://DEVICE_IP:8081.
remote_ui_auto wraps pyray.end_drawing to capture each frame before the buffer swap, launches stream_server.py itself, and starts RemoteUI when the first browser connects. No render-loop edits or launch-script changes are needed. Set OP_REMOTE_UI=0 to disable it without uninstalling.
To remove: ./uninstall.sh [OPENPILOT_DIR]. openpilot updates may reset ui.py; just re-run install.sh.
If you prefer to wire it in yourself, put this directory on sys.path and call RemoteUI from your render loop:
from remote_ui import RemoteUI
remote = RemoteUI() # after the window/GL context exists
...
remote.update() # each frame
remote.stream_frame() # each frame while a browser is connected, before end_drawing
...
remote.close()Then run python3 stream_server.py yourself.
GET /: Built-in HTML viewer (installable as a PWA)GET /remote_ui.js,/remote_ui.css,/remote_ui.webmanifest: Browser client assetsWS /stream: Real-time frame stream, plus input, feedback and view-size messages from the browserPOST /input: Remote input events (JSON), fallback when the WebSocket is downGET /stats: Cache/client statsGET /stream-sse: SSE fallback stream
Input events are JSON, sent over the WebSocket as {"type":"input","event":{...}} or to POST /input, and forwarded to /tmp/ui_touch_socket.
Examples:
{"type":"click","x":1080,"y":540}{"type":"scroll","x":1080,"y":540,"deltaY":120}{"type":"drag","x":1200,"y":600,"startX":1080,"startY":540}{"type":"keydown","key":"Enter"}{"type":"text","text":"pasted text"}remote_ui.py includes a standalone test window in its __main__ block.
Run it directly from this directory (python3 remote_ui.py); imports are flat, so no openpilot package layout is required.
- Confirm shared memory exists:
ls -lh /dev/shm/openpilot_ui_frames - Confirm server is running on port 8081.
- Check server logs for shared-memory connect/read errors.
- Confirm
remote.stream_frame()is being called from the render loop.
- Confirm socket exists:
ls -lh /tmp/ui_touch_socket - Confirm
TouchInjectorwas started (RemoteUI()created successfully). - Remote input is ignored while openpilot is engaged; the browser shows a banner when it is.
- Check network quality between browser and host.
- Quality and frame rate adapt automatically; the limits are the
*_JPEG_QUALITYand*_FRAME_RATE_LIMITconstants inframe_streamer.py. - Reduce competing load on the UI host.
- This repository was initially meant to be a straight port of the Qt-based streamer to Python, but has evolved into an ongoing project that has now been fully integrated into my openpilot/sunnypilot fork, cloudypilot, requiring no installation after-the-fact.
legacy/main.ccand the Qt classes document the older integration style and protocol history.