Windows: roam, multi-model chat, island questions, hats and lighter memory

[arnavaggarwal-dev]

Windows-side features I've been running daily. Everything is under windows/; macOS is untouched.

What's in it

• Roam: drag Mochi out of the island to look at the screen. It scans, screenshots, and opens the chat with the image attached. Cancel with Esc, a right click, or by dropping it back on the island. A "quick screenshot" setting skips the scan animation. On Linux the overlay is a transparent layer-shell surface that takes the keyboard (Esc) but no clicks; the island forwards the held pointer, and the screenshot uses the desktop's tool (spectacle, gnome-screenshot, grim, or import/scrot on X11).
• Area scan: hold Shift while carrying Mochi to pick an area instead of the whole screen. The rest of the screen dims, the area shows its size, and on drop Mochi hops onto its edge, scans only inside it, and only that area is captured. Windows captures the area directly; on Linux it is cropped out of the full screenshot (shot.rs, with a unit test).
• Chat providers: any OpenAI-compatible endpoint (OpenRouter, NVIDIA NIM, Groq...), a saved model list, and a picker next to Send. Keys live in Credential Manager, one per host. Replies render GitHub-style Markdown (below). Sending an image to a model that can't take images asks first.
• Questions on the island: AskUserQuestion multiple choice is answerable from the card, labelled yes/no, pick one or pick any. The card closes live when answered in the terminal. Claude Code's questions open even when another Mochi is focused.
• Mochi names, shapes and hats: each Mochi can be renamed and given a body shape and a hat. The vector art is generated by windows/scripts/wardrobe. Hats ride the head on a fixed seat (no drifting); only tassels swing.
• GitHub card: latest Actions runs with a bar of finished steps while one runs, latest pushes, stars and repos in the header; refreshes every 20 s during a run. A git push that Claude Code runs pops the card open with the push in progress (from the hooks, since GitHub can't see a push in flight), keeps it open until the push ends, then shows Pushed or Push failed.
• Chat extras: replies end with a hidden {"mood": ...} that Mochi acts out (new sad, shy and scared emotes); code blocks have a copy button; a dropped code file is sent as a fenced block and shown as a code card; "Clear" under Mochi, which also drops the attached file so it isn't sent again.
• Upload: the bar never moved (nothing updated State.uploadProgress); it now follows Mochi's progress curve and says the file is read locally, not uploaded. The Mochi that eats the file is the focused one, in its colour, shape and hat, and the bar and glow take its colour.
• Closed island: the middle shows the focused Mochi's current step, or the time when nothing is going on.
• Image, video, speech and 3D models: a saved model says what it makes (guessed from its id: flux → images, sora → video, orpheus → speech, trellis → 3D, whisper → transcribes speech).
  • Images use /images/generations, or the chat route with modalities for OpenRouter. While one is made, the chat shows a glowing tile with a sparkle and the seconds.
  • Video uses OpenAI's Videos API (create, poll with progress, download), with a progress bar; it gives up after 20 minutes.
  • Speech uses /audio/speech; Groq's Orpheus gets its six voices, and its 200-character limit is checked before sending.
  • 3D uses Pollinations' /3d route (TRELLIS 2, low/medium/high detail). The dropped picture is uploaded to Pollinations' media store first, and the chat says so. The reply shows a rendered still; "View in 3D" opens a turntable. three.js is loaded only when a model is shown.
  • Results show inline with Download (to Downloads, never overwriting) and full screen on the roam overlay (Esc, click outside or close). The webview can only read a generated file by its bare name in the media folder, and the API key only goes to the provider's own host.
• Mic: mic and Send are one button, a mic while the field is empty and Send once it holds text.
  • Words appear while you talk. Speech is split into phrases at short pauses and sent to your saved speech-to-text model (e.g. Groq Whisper), with a long phrase re-read about every 1.5 s, under a budget of about 18 requests a minute.
  • It stops and sends after 1.6 s of quiet. Silence is never uploaded, and Whisper's stock inventions ("Thanks for watching", a lone "Thank you." from noise) are dropped. Esc cancels; leaving the chat or hiding the island releases the mic.
  • The mic turns off after each prompt; Settings → General → "Keep the mic on" listens again after the reply to a spoken prompt.
  • On Windows the island grants its own page the microphone. Linux isn't wired yet (WebKitGTK needs a permission handler).
• Markdown like GitHub: replies go through marked + marked-footnote.
  • Tables with alignment wrap to fit the chat instead of scrolling sideways. Also task lists, strikethrough, autolinks, nested lists, footnotes, details/summary, sub/sup/kbd, and GitHub's alerts (> [!NOTE] and friends).
  • Math ($…$, $$…$$, ```math, \(…\); "$5 and $10" stays text) is typeset by KaTeX, loaded the first time a reply has math. Its fonts are emitted as files, since the CSP has no font-src data:.
  • Model output is untrusted and the page holds Tauri IPC. So the HTML goes through DOMPurify with an allowlist: no script, on*, style or class, and ids are prefixed user-content-. Images never get a live src, and no link keeps an href: web links open in the browser through Rust.
• Fixes:
  • File drops reach the island at last. WebView2's own child window (owned by its browser process) is what Windows asks during a drag, so Tauri's native drop events never fired, and no app can revoke another process's drop target. The island now takes a plain HTML5 drop and sends the bytes to a new ingest_bytes command; the same code serves Linux.
  • An integration's key button no longer saves an empty key over a stored one: it becomes "Clear key" and asks first.
  • The island is a tool window, so taskbar replacements don't list it as an app.
  • Tool step labels are in English (they were the macOS French, e.g. "Exécute").
  • Sweat drops run down instead of floating up like the other particles.
  • Settings no longer showed a newly added model twice (two overlapping redraws each added their rows).
  • Linux build: roam_linux.rs imports tauri::Manager (thanks @arreina).
• Music (Windows): a Music pill (Melody) that reads the Windows media controls, the same feed as the volume flyout's player, so Spotify, Apple Music, YouTube Music and any player reporting to it show the track with play/pause and skip. Event-driven (GlobalSystemMediaTransportControlsSessionManager events), so nothing polls. Mochi dances only to music: browser sessions (YouTube videos) don't count, installed web apps like YouTube Music do. While dancing, the eyes groove to the beat instead of following the cursor, music notes fly off, and the head opens like a real AirPods case (lid hinged at the back, white inside, two wells, front light): the buds rise out and shoot off to the left and right edges of the monitor on the screen overlay (borrowed like the media preview, click-through, skipped when busy), then the lid snaps shut. 15 s after the music stops the lid opens, the buds swoop back in and drop into their wells. The buds are drawn in code (airpod.ts). Linux reads nothing yet (MPRIS would be the route).
• Branch updated with main (0.1.3). One merge commit, no conflicts.
• Music, steadier: skipping no longer flashes "Nothing playing" (the last track holds for 3 s while the player reports nothing between songs), and Melody takes the colour of the song's cover (pill, card and its Mochi; red without art).
• Ticker fix: Claude's tool activity froze after a session's 20th step, because the capped step list stopped its index moving. It follows a running step count now. With several sessions, one session's Stop no longer idles Mochi while another works. (The macOS ticker has the same cap issue; untouched.)
• Loaders: speech plays a music-note Lottie recoloured with a turning rainbow (lottie-web light canvas build, loaded only while speech generates); image and video show an "aurora" placeholder instead of the sparkle tile.
• Text: no em dashes in on-screen strings.
• Smaller: "keep island visible", auto-close down to 3 s, single-line ticker.
• Memory: the three webviews share one renderer process, and hidden ones ask WebView2 to trim. Working set dropped from about 790 MB to 650 MB on my machine.

Heads-up

• Some of these change existing views (ticker, question card height, settings sections, the closed island), which CLAUDE.md asks to avoid unless requested. Happy to split those out or drop them.
• The first commit is large. I can break it into smaller PRs per feature if that's easier to review.
• New dependencies, against CLAUDE.md's "no third-party dependencies unless truly unavoidable": three.js (showing 3D models), marked + marked-footnote (GitHub-flavoured Markdown), DOMPurify (sanitizing it) and KaTeX (math), and lottie-web for the speech loader. The heavy ones (three.js, KaTeX) load only when a reply needs them. Writing a safe GitHub-compatible parser, sanitizer or 3D renderer by hand didn't seem wise; happy to discuss.

Tested

• Windows 11: cargo test (29 pass), tsc --noEmit, release build, daily use. The area scan's animation and capture call were also checked in a headless harness.
• Markdown: src/dev/markdown-check.ts (33 checks in headless Edge, including hostile HTML: <script>, <img onerror>, javascript: links, style, id clobbering, KaTeX \href). Dictation: src/dev/dictation-check.ts with a fake microphone and stubbed transcription. The 3D viewer was checked by rendering Khronos's sample Duck.glb.
• Music: checked live with YouTube Music (app) and Chrome tabs; music.rs unit tests cover browser vs. player detection and names. The lid and buds were checked rendered in headless Edge.
• Not yet tried against the real providers (image, video, speech, 3D and transcription calls), or with a real microphone.
• Linux: not built locally. The roam overlay, area crop, screenshot tools and HTML5 drop are written for it but untested; the Linux CI run on this PR should cover the build.

[arreina]

Heads-up from a Linux build check (cargo check --workspace --all-targets on Linux Mint 22.3, with this PR merged onto current main, head 75db4c7): it doesn't compile on Linux.

roam_linux.rs:81 and :131 call app.get_webview_window(…), which needs the Manager trait in scope (E0599). Adding use tauri::Manager; at the top of roam_linux.rs fixes it: I checked, and with that one line the whole workspace builds on Linux with no errors.

(Linux CI now runs on pull requests that touch windows/, so this will show up there too once it runs.)

[arnavaggarwal-dev]

Thanks for checking on Linux! Fixed in ba196cf: roam_linux.rs now imports tauri::Manager. The PR has also moved on since 75db4c7 (Shift-drag to scan an area, which crops the screenshot on Linux), so the Linux CI run on this push covers that too.