Skip to content

Native visuals: agents show interactive Luau visuals rendered by GPUI #900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wingleeio merged 4 commits into from
Oct 15, 2026
Merged

Native visuals: agents show interactive Luau visuals rendered by GPUI #900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1f30c49
Render agent visuals natively from sandboxed Luau scripts
wing-anara Oct 14, 2026
771b402
Render visual scenes and shaders on the GPU
wingleeio Oct 14, 2026
d9db3e4
Add visual assets and iOS visuals, and harden untrusted visual input
wingleeio Oct 14, 2026
5ae7223
Render desktop visual frames off the main thread and reuse GPU targets
wingleeio Oct 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 139 additions & 10 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ members = [
"crates/client",
"crates/mcp",
"crates/text",
"crates/visual",
"crates/visual-gpu",
"apps/zeron",
]

Expand Down Expand Up @@ -54,6 +56,8 @@ zeron-client = { path = "crates/client" }
zeron-theme = { path = "crates/theme" }
zeron-mcp = { path = "crates/mcp" }
zeron-text = { path = "crates/text" }
zeron-visual = { path = "crates/visual" }
zeron-visual-gpu = { path = "crates/visual-gpu" }

# CRDT
loro = "1.13"
Expand Down Expand Up @@ -170,6 +174,13 @@ opt-level = 0
[profile.dev.package."*"]
opt-level = 2

# Scenes rasterize in software every frame the camera moves.
[profile.dev.package.zeron-visual]
opt-level = 3

[profile.dev.package.zeron-visual-gpu]
opt-level = 3

# Mobile core (scripts/ios/build-core.sh). Always optimized — layout and sync
# run on the phone's hot paths, so even Debug app builds link an -O core.
[profile.mobile]
Expand Down
6 changes: 6 additions & 0 deletions apps/ios/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@ ZeronShared/ Compiled into the app and the extension: the activity's
paints runs with CoreText at the Rust coordinates — measurement and
rendering can't disagree. Display models for rows beyond the viewport are
prefetched off the main thread.
5. Agent visuals (`show_visual` Luau scripts) get a worker thread each
(`crates/mobile/src/visual`): it runs the script, lays the node tree out,
draws canvases and charts as vector paint, renders scenes and shaders with
wgpu, and reports its height back to the row. `VisualView` paints what
arrives (rasterized on the worker thread) and forwards touches; the demo's
"Agent visuals" chat (`-demo -route chat:chat-visuals`) has one of each.

## Launch arguments

Expand Down
Loading
Loading