Native visuals: agents show interactive Luau visuals rendered by GPUI

[wingleeio]

Agents can show interactive visuals (charts, tables, dashboards, stepping explainers, simulations, real 3D) inline in their replies. A visual is a sandboxed Luau script that Zeron renders with its own GPUI elements, not a web page. This replaces the WebKit approach in #898.

Hosting a browser engine inline kept producing workarounds: offscreen compositing on Linux, snapshotting an invisible WKWebView on macOS, pointer arming and disarming to route clicks, focus handoffs, a loopback server, height reporting, injected font and zoom fixes, and a card fallback on Windows. All of that is gone. A visual is now ordinary transcript content, so it scrolls, clips, blurs under the glass composer, takes clicks and drags, and follows theme and text-size changes the same way on Linux, macOS, and Windows.

Why Luau instead of a JSON component tree

A JSON tree of preset components can't express a visual we didn't predict, and its state handling turns into a small expression language. A script can draw anything and owns its state, so I went with Luau, but kept the output as a tree of Zeron nodes so the style rules still hold:

• The script builds data and state once, then returns a view function. The view returns ui.* nodes: column, row, grid, text/span, stat, table, chart (bar, line, area, scatter, with axes, legend and hover readout built in), tabs, toggle, slider, button, link, canvas (2D vector drawing), and scene (3D). Controls call handlers, the handlers assign to state, and Zeron renders the view again.
• The style rules are the API, not advice. Colors are theme tokens (text, muted, accent, chart1…chart6, and so on), text sizes are steps on the transcript's type scale, the font is always the interface font, and layout nodes have no fill, border, shadow, padding, or font props. Using a hex color, background, or font fails with an error that explains the rule, so agents can't build cards and every visual reads on glass or opaque, light or dark.
• Real 3D without a GPU pipeline. ui.scene takes boxes, spheres, cylinders, height-field surfaces, meshes, lines, points and labels. The runtime frames an orbit camera on the geometry (with a fit that doesn't change size while it spins), lights faces with one soft light, culls and depth-sorts them, and hands the desktop 2D polygons to paint as GPUI paths. The reader drags to orbit, and spin adds a turntable.
• Sandboxed and bounded. Only the string, table, math, utf8 and bit32 libraries are loaded (no I/O, network or require), with a 64 MB memory cap and a time budget per call: 250 ms for setup, renders and handlers, 60 ms per animated frame. An endless loop fails with a message instead of freezing the app. math.random is reseeded from the visual's id before every call, so generated data is identical on every render and device.
• Checked before anyone sees it. PublishVisual runs the script the way the desktop will: setup, the first render, every canvas and scene draw, then each tab, toggle, slider end and button once. The first error goes back to the agent with its line number. In testing, an agent used a pixel size for canvas text, got visual script failed: in a canvas draw: visual:133: unknown text size '13': use xs, sm, body, lg, xl, or 2xl (the transcript's type scale), fixed it and resent.

Agent-authored examples

These were written by Claude from the tool description alone (orbit speeds and inclinations are Claude's figures):

3D orbits, spinning, with a speed slider (dark glass)	Quicksort stepper: buttons, scrub slider, live stats (dark opaque)

A height-field surface whose frequency slider and wireframe toggle re-run the scene (light glass):

What changed

• crates/visual (zeron-visual, new, GPUI-free): the Luau runtime (mlua with Luau), the ui prelude, node parsing with strict props, canvas drawing commands, 3D scene building and projection, and check(). 17 tests cover the sandbox, time and memory limits, style enforcement, canvas, 3D culling and depth order, and the publish-time check.
• crates/ui/src/visuals.rs (rewritten): renders the tree with existing idioms: markdown's table treatment, the settings tab, switch and button widgets, the transcript's rule, and canvases and scenes painted as GPUI paths and shaped text in theme colors. Canvas draws are cached per render unless animated, and only visible rows paint, so offscreen animation costs nothing. When the tree changes shape (a tab switch, for example), the visual emits an event and the transcript remeasures that row.
• Plumbing carried over from Visual replies: agents show HTML pages inline, composited by GPUI #898, reshaped: show_visual {title, script} on the Zeron MCP server (only inside a chat), the PublishVisual RPC (now runs check on a blocking thread), uploads store {id}-visual.luau, and an AgentEvent::Visual / visual doc part carrying only the path and title (the height field is gone, since native layout measures itself).
• Agent guidance: the tool description is a compact API reference with a worked example. A test runs that example through check, and another asserts the description names every color the runtime accepts.
• Removed from Visual replies: agents show HTML pages inline, composited by GPUI #898: the loopback server, theme CSS and bootstrap injection, web-font serving, the Linux helper transparent/zoom commands, the inline BrowserSurface mode, and inline_macos.rs.

Since the first version: GPU rendering, assets, iOS, hardening

• GPU rendering (crates/visual-gpu, new). ui.scene renders through wgpu, with physically based materials, soft shadows, glass that refracts with dispersion, bloom and 4× multisampling. The software rasterizer is now the fallback where there's no GPU. A new ui.shader runs Shadertoy-style GLSL with iMouse, an orbit camera, named inputs bound to state, up to four feedback buffers, iNoise and blend modes, and naga validates it at publish time. Visuals fade out toward their edges instead of stopping at a hard line.

• Assets. show_visual takes files from the agent's working directory:

  • glTF models (s:model);
  • PNG, JPEG, WebP and HDR images, used as scene textures, environment maps (s:environment) and sprites (s:sprite), or as shader images (iTexture0-3).

  The engine checks the files and stores them beside the script, and they travel to other devices like attachments. The limits are 16 files, 8 MB each and 10 MB together, sized so the request fits in one WebSocket message to the engine.

• iOS (crates/mobile/src/visual, VisualView.swift). Each visual gets a Rust worker thread that:

  • runs the script;
  • lays the tree out with the desktop's rules, measuring text with the same font files iOS draws with;
  • draws canvases and charts as vector paint;
  • renders scenes and shaders with wgpu (Metal).

  Swift only paints what arrives and forwards touches, so the main thread stays nearly idle: 1.7% busy while a canvas, a scene and a shader all animate. Transcript rows take each visual's measured height. Visuals pause when they're offscreen or the app is in the background. The demo has an "Agent visuals" chat (-demo -route chat:chat-visuals).

• Hardening. A security audit and code review of the whole change led to these fixes:

  • glTF: embedded images stay encoded and are size-checked when drawn; accessor shapes are validated; node walks are bounded (a looping file used to hang the UI thread); parsing can't panic.
  • Images: a pixel budget on top of the side limit.
  • GPU errors: wgpu errors are caught or logged; before, a 508-byte model could crash the app through a validation panic.
  • Texture cache: keyed by content and bounded.
  • Engine: checks for a running turn first and removes partial writes.
  • Assets: names may not differ only by case, and the MCP server reads only regular files, with a bounded read.

Testing

• cargo test -p zeron-visual -p zeron-mcp -p zeron-doc, the engine e2e publish_visual_lands_in_the_live_turn (now also checks that a broken script is refused and leaves no part), and the UI visuals tests.
• Live on Linux in an isolated rig (own data dir, engine port and X display) with Claude Code. In the app I clicked tabs (the whole view re-rendered), hovered charts (inline readout), dragged sliders, toggled switches, pressed buttons, orbit-dragged scenes, and watched an animated canvas.
• Light and dark, opaque and glass. Linux has no window glass, so the glass screenshots come from a local-only build with glass turned on, over a wallpaper with picom; that build is not part of this PR.
• macOS: the GPU path in the desktop dev app (scenes, shaders, assets, the Prism examples), and the iOS simulator with the desktop's own visuals loaded through ZERON_DEMO_VISUALS. A UI test (VisualFlowTests) drives a dashboard's toggle, slider and tabs.
• cargo test for zeron-visual, zeron-visual-gpu, zeron-mobile, zeron-client, zeron-mcp and zeron-proto. New tests cover the glTF attacks (a node cycle, short attributes, a mis-shaped accessor, an oversized embedded image), shader image orientation, translucency, the edge fade, the iOS layout and worker, and load retries.
• Not run on Windows or on a physical iPhone.

Follow-ups

• Text inside a visual isn't selectable yet. Prose text could join the transcript's selection.
• Visual state resets when the app restarts. Handlers could persist a small state table in the doc.
• An agent-side preview (render a visual to a PNG the agent can look at) would close the loop on layout, not just errors.

Supersedes #898.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}