OpenLive 0.2.0: coding agents over ACP, voice cloning, and a hardened live pipeline - #8
Merged
Conversation
- packages/shared/src/agent-registry.ts: single source of agent identity (labels, adapters w/ pinned versions, install/auth recipes, session parsers, credential probes) replacing ~6 duplicated per-file registries - @openlive/shared/node: expandHome, widenedPath (path.delimiter + win npm), evalCredProbe (file/json/keychain), terminalCommand (mac/win/linux) - fix: opencode/hermes external sessions were discovered then dropped from History (AGENT_ORDER omitted them); order now registry-driven - fix: POSIX-only path splits in agent session discovery (Windows) - cross-platform dev scripts (cross-env, node free-ports/dev launchers) - vitest wired at root; 4 existing assert-script tests converted; new registry tests guard the claude/hermes adapter pins; CI runs tests + a windows-latest job building an unsigned installer - scoped loggers (agent service + web) replace ad-hoc console.*; toast store surfaces user-actionable failures (delete/download/preview/picker) - dedupe: isDesktop/basename/bridge (lib/platform), settings Section - README + docs/ARCHITECTURE rewritten for the v2 ACP agent pipeline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…logos - /api/agents now returns three-state status per agent: installed (which/where), credState from read-only credential probes (file/JSON/keychain — never reads secrets, only probed when the CLI exists), authDetail (cursor account email), canLogout, hidden - Sign in shows only when login is actually needed; Sign out appears when signed in and the agent has a logout command (hermes has none); both open the agent's own flow in a real terminal (mac/win); status re-probes on window focus - per-row visibility toggle (agentHidden:<id> setting) hides an agent from the Talk-to pickers and History discovery; sessions stay on disk, bound chats keep working, un-hiding restores everything - real brand marks for OpenCode (official block-O) and Hermes (its ACP registry caduceus icon), themed via currentColor; letter badge stays as the fallback for future agents Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- /api/history reshaped: HistoryWorkspace[] grouped by realpath'd cwd only — all agents' chats for the same project live together, each chat carrying its agentId for the row's brand mark; most-recent workspace first, "No folder" bucket always last; dedup/empty-chat/hidden-agent filtering preserved - sidebar rewritten to 2 layers: workspace node (folder, chat count, persisted collapse, bulk delete) → chat rows with the agent's mark + relative time; New Chat pinned at the top as the primary action; search box filters chat titles (incl. local rename overrides), workspace basenames and full paths, rendering flat results with a workspace subtitle - rename/delete/resume flows carried over unchanged; query key bumped to ["history","v2"] so stale cache shapes never render Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… presets - new lib/live/supertonic.ts: Supertone's supertonic-3 on onnxruntime-web (4 ONNX sessions, unicode indexer, 10 preset voices, 44.1 kHz), adapted from the MIT reference with flat Float32Array latents; shares the model worker's ort instance; assets stream through the Cache API with byte progress - pipeline config gains tts.engine (kokoro default) with per-engine voice catalogs + clamping (voice snaps to the engine's default on switch); engine picker cards in Pipeline settings; cache tag folds the engine in so a switch re-prompts the download; worker lazy-loads the other engine mid-call (tts calls get a 120s leash for exactly that first download) - measured on this machine (WebGPU): supertonic ~310ms vs kokoro ~610ms for the same sentence — 2× faster first-audio - warmup-on-launch (lib/live/warmup.ts): cached weights + shaders + mic driver warm quietly after first paint (never prompts, never downloads, skips the mini panel) — first call starts instantly - barge-in: playback-aware gate — while agent audio plays, mic RMS must clear the noise gate scaled with playback level, so speaker echo can't self-trigger - devicechange handling: unplugging the active mic falls back to the system default with a toast; mic switch mid-call already existed - turn-taking presets (Relaxed/Balanced/Snappy) as a derived one-tap control over trailing silence + threshold + hold; editing raw sliders = custom - fix: TTS Preview stayed disabled after a finished download (stale modelsCached() read) — Preview now self-loads with its spinner Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- de-boxing (HIG-style): new elevation scale (--shadow-xs/card/pop, --surface-raised); Settings header/nav, History sidebar, section dividers, agent rows, confirm modal, in-call top bar, transcript panel, lobby aside, engine cards and About links now separate by whitespace + surface shifts + soft shadows — hairlines only inside lists/menus; focus rings untouched - Settings grows a General tab (now 5): theme (moved out of About), Space hold-vs-toggle voice input, editable global mini-mode talk hotkey (capture field → openlive:set-mini-hotkey IPC, live re-register with taken-shortcut fallback + persisted, re-sent on each mini entry), and launch-at-login (openlive:login-item → app.setLoginItemSettings) - hints: ACP available_commands_update now flows acp-agent → AgentMeta → agent_meta wire schema → liveStore; lib/hints.ts is a pure ranked selector (first-calls PTT coaching, the agent's own slash commands, error recovery with a one-tap "Open Agents settings" fix) rendered as at most two quiet chips above the in-call control bar (HintChips) - onboarding: four-card skippable first-run walkthrough (agent → folder → one-time download → how to talk), Escape/skip closes forever - cleanup: the last hardcoded agent-id list (useLiveSession) now uses the shared registry's isAgentId Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…der in main view
- hermes: a pooled credential is NOT a configured provider — the probe now
requires auth.json `providers` non-empty (verified: hermes-acp exits with
"No LLM provider configured" when providers:{}), so Settings honestly shows
"Sign in needed" and its Sign in button runs `hermes setup`; startHint
rewritten to name the fix. (Codex "error" was the user's ChatGPT usage
limit, relayed verbatim — verified codex-acp works end-to-end.)
- push-to-talk is now OPT-IN: a keyboard toggle sits left of the mic in the
call bar (persisted); Space does nothing until armed; the PTT/Enter coaching
chips are gone (mini-pill global hotkey unchanged)
- lobby: project folder + recents moved into the MAIN pre-call view (it gates
Start); the right panel is now a floating elevated card (rounded, raised
surface, shadow) — same treatment + slide-in for the in-call Activity panel
- agents settings: Advanced ACP command UI removed (the acpCommand:<id>
setting is still honored server-side); the duplicated section/description
collapsed into one honest blurb
- window chrome: Windows/Linux now get native-idiom min/max/close top-RIGHT
(mac keeps traffic lights top-left); headers clear the correct edge per
platform (pl-84 mac / pr-140 win) and the stray pl-[80px] is unified
- verified live end-to-end: real Claude Code call (STT → agent → TTS →
captions), prewarmed model/mode/effort pickers, light+dark lobby
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- agent failures are now SPOKEN (first sentence, capped) through the same TTS chain as replies — voice-first users hear "Codex hit its usage limit" the moment it happens; the full text stays in the banner + hint chip - onboarding rebuilt as a real spotlight tour: each step dims the page with a cutout + accent ring around the ACTUAL control (Talk to, New, Resume, Settings) and points an arrowed tooltip at it; skippable via Skip/Escape/×, follows resize, shown once (openlive-onboarded-v2) - home hero gains a round Settings button beside New/Resume (kept the top-right icon too) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olish, mini-mode fix
- SpotlightTour: one reusable, CSS-animated coach-mark component (cutout +
accent ring + arrowed tooltip on the real control). The old Onboarding ran
GSAP hooks while unmounted — the "Invalid scope"/"target not found" console
spam — replaced wholesale; hooks now only exist while the tour is live, and
a missing anchor closes the tour instead of stranding a dim overlay
- tours run once per surface on first visit: home (4 steps), lobby (folder +
setup panel), settings (nav), history sidebar (actions row), in-call
(controls incl. the PTT toggle); the mini pill gets a one-time inline hint
- hermes lifecycle made honest: installed = uvx on PATH AND ~/.hermes exists
(new registry installedProbe — uv alone proves nothing); Install opens the
interactive setup wizard in a terminal (new install.terminal recipe);
Uninstall/Sign-out hidden where no command exists (was streaming a 400);
terminal-run actions get one truthful "continues in the terminal" message
- history sidebar is a floating elevated panel; New chat + search share one
row — tapping the magnifier width-animates the field over the button,
Escape/empty-blur collapses
- lobby workspace picker compacted: label row with Browse… on the right,
one-line recent chips with ~-shortened paths, verbose description gone
- hints reduced to error-recovery only — the slash-command chip surfaced
skill ids nobody would say ("/gsap-core") and is removed; picker footer
blurb removed
- mini mode: preview relay 1 → ~10 fps, previews render in fixed 16:9 boxes
and size reports are delta-gated — per-frame pixel jitter was firing
ResizeObserver → miniSize IPC → an ANIMATED window resize per frame (the
glitch); comments synced
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-write Both the web and agent processes write settings.json/conversations.json; plain read-modify-write could drop the other side's update. All mutations now run through updateJson(), which serializes same-process callers on a per-file promise chain and holds a proper-lockfile lock across the whole read→modify→write cycle for cross-process safety. Reads stay lock-free (atomic rename gives consistent snapshots). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The server has emitted todos (ACP plan / update_todos) and usage events since day one, but the client silently dropped both. Plans now render as a pinned checklist card above the transcript (Step m/n, strike-through), and a compact context/cost chip sits in the top bar. Slash commands were tried and dropped as voice noise long ago — the transmit-only path is now removed end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hema, symlink fence - headLines now reads a single bounded 1 MB head instead of slurping the whole session log (Codex rollouts reach hundreds of MB). - Hermes session discovery rewritten against the REAL hermes-agent 0.18.2 schema (verified from hermes_state.py + a synthetic DB): table sessions, PK id, started_at/ended_at REAL seconds, sub-sessions and archived rows excluded. The old query targeted columns that never existed. - confine() now realpath-checks the deepest existing component, so a symlink inside the workspace pointing outside is refused for both reads and writes; covered by a real-symlink test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unanswered agent permission silently auto-denied after 120s. The server now sends expiresAt with each ask; the prompt shows a ticking 'Auto-deny in m:ss' (red under 30s) and the engine speaks a reminder 30s before the deadline so voice-first users aren't blindsided. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… verified paths - Agents settings now shows each installed CLI's version (probed with --version on the widened PATH; skipped for hermes whose runner is uvx) and an Update button that reruns the install recipe pinned @latest. - A failed global npm install with EACCES appends actionable guidance (user-writable prefix / nvm) instead of a raw error dump. - OpenCode's Windows data path verified against its docs (same ~/.local/share/opencode everywhere); discovery now honors XDG_DATA_HOME. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Silero VAD worklet/onnx/ort-wasm vendored from node_modules into public/vad at predev/prebuild — the voice loop no longer fetches from jsdelivr at runtime (CSP keeps the host only for transformers.js's own wasm fallback). Versions track package.json; binaries stay gitignored. - Desktop: removed the OPENLIVE_AGENT_SECRET that was passed to the web server but never enforced anywhere on the desktop path; documented the real trust model. Fixed the misleading WS/CSP comment in next.config.ts. - Version 0.2.0 across the workspace + CHANGELOG entry (package.json had drifted to 0.1.0 while CHANGELOG was at 0.1.9). - Deleted 130MB of untracked demo videos, the unused CameraCapture.recent(), and the stale better-sqlite3 build allowance (store is JSON files). - Crash screen: OS-theme aware, brand accent instead of off-palette indigo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uts, lobby readiness - Light mode rebuilt as a 4-step oat-to-cream ladder (page < raised < card < popover), no pure white, borders at visible alpha — cards no longer fuse into one white field. Control-bar shadows use the token. - Transcript renders agent replies as markdown (GFM) with copy buttons on code blocks and messages, plus a one-tap Markdown export. - In-call keyboard shortcuts (M/C/S/T/H, Cmd-E end) with a '?' cheat sheet listing every binding including push-to-talk. - Lobby now surfaces 'agent isn't installed / needs sign-in' with a one-tap jump to Settings → Agents, instead of letting Start fail into a spoken error on first run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README now leads with what OpenLive is — the open voice and vision layer for AI agents (bring your own model; open alternative to ElevenLabs Agents / Gemini Live / OpenAI Realtime) — with coding agents over ACP as the flagship integration, matching the in-app tagline. It documents Supertonic, the real first-run download sizes, and states plainly that the pipeline is cascaded rather than full-duplex speech-to-speech and why that trade buys local + BYOK. Also fixed: harness registry comment claiming 'three supported providers' (there are 15), hints engine comment promising 2 hints (max is 1), desktop README's per-arch DMG names (builds are universal), ARCHITECTURE's hermes pin string, and the app metadata description. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Settings → General gains 'Your assistant's style' — free text (2000 char cap, debounced save) describing how the assistant should behave and speak. It's appended to the built-in assistant's system prompt and to every ACP agent's session preamble (Claude via its system-prompt append, others via the first-turn preamble), read fresh per session. Replay stripping recognizes the new block so resumed transcripts stay clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the loop The desktop shell gains a tray icon (Open / Mini mode / Quit) and an openlive:notify IPC that shows an OS notification only when the app isn't focused; clicking it brings OpenLive forward (also out of mini mode). The renderer notifies on two events: an agent permission ask (which would otherwise silently auto-deny in 2 minutes) and a turn finishing after more than 5 seconds of work. Mini enter/restore logic is factored into shared functions used by IPC, tray, and notification clicks. Web build no-ops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Long tool runs are dead air in a voice call. A new narrator wraps the
ACP turn's emit stream and voices a short line ('Step 2 of 4 — fix the
parser.') when a tool has run 1.5s with no reply text — plan-aware when
the agent published one, tool-name + file basename otherwise. Throttled
to 4 lines a turn, 8s apart, silenced by agent speech or barge-in, and
delivered as standalone sentences through the normal TTS chunker so it
can never splice into a real reply. Toggle lives in Settings → Pipeline
(off by default); the built-in brain keeps its existing worker
narration. Covered by throttle/gating tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zero-shot voice cloning as the third TTS engine. Record 5-30s in Settings → Pipeline → Voice Studio, the on-device Whisper transcribes the reference (editable), and from the next reply the assistant speaks in that voice — synthesized by the agent service on CPU with ZipVoice (k2-fsa, Apache-2.0, 123M distilled int8) through the sherpa-onnx Node addon. Verified on this machine: ~0.22x realtime, first sentence in ~0.7s, engine cold-start hidden by a warm-up ping at call start. Design points: - The reference rides every generate() call → one engine serves all profiles; generateAsync runs on sherpa's native thread pool so live sessions never block; engine unloads after 5 idle minutes (~700MB). - Optional install: honest 156 MB download / 208 MB on disk, streamed progress, atomic (.part + full cleanup on failure), deletable. - Profiles = wav + transcript (0600) with preview / export / delete and a required consent checkbox; responsible-use note in the README. - One transport for every deployment: a same-origin /api/voice Next proxy forwards to the agent (secret injected server-side in the container). Cloned PCM plays through the existing AudioPlayer with the same epoch/barge-in semantics; any failure falls back to Kokoro with a single toast. - Packaging: sherpa-onnx-node ships as real node_modules beside agent.mjs (native addon, resolved lazily via createRequire); the app still boots fine without it. Verified end-to-end through the proxy: model detect, profile create, synthesis (2.95s of speech, RMS 0.157), export, delete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… flow Voice cloning was buried inside Pipeline → TTS. It's now its own tab: - Guided flow: pick a reading script (Everyday / Expressive / Calm — or just talk), record with a live level meter and an elapsed bar that marks the 5s minimum, LISTEN BACK before anything is saved, re-record, fix the Whisper auto-transcript, name it, consent, save. - Profile manager: preview any profile with any text you type, set the speaking voice, play the original recording, inline rename, export to a file and import on another machine, delete (active voice falls back to Kokoro cleanly). Durations shown per profile. - New server surface: PATCH rename, GET /profiles/:id/audio, POST /profiles/import — all verified live (rename 200, audio 200, export→import round-trip). - Pipeline → TTS now just picks among cloned voices with a 'Manage in Voices' jump; the settings tour copy is updated. Cross-platform packaging: pack-agent ships BOTH darwin archs for the universal mac DMG — the build machine's copy plus an npm-pack fetch of the missing arch (win/linux ship their own). Verified: dist bundle boots, serves /voice, and resolves the addon from its shipped node_modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A crashed or force-killed run can leave the spawned web/agent servers alive (Windows especially — children aren't tied to the parent's lifetime), still listening on 47823/47824. Every later launch then dies in an EADDRINUSE respawn loop ending at 'The web service keeps crashing', and relaunching never helps. We hold the single-instance lock, so at startup any OTHER process of OUR binary listening on our ports is a zombie: kill it (netstat/taskkill on Windows, lsof/kill elsewhere). Foreign processes are left alone. The keeps-crashing dialog now also says which ports to check and asks for console output.
fetch_url's DNS-rebinding defense imports undici's Agent dispatcher directly, but the package was never declared — it typechecked locally through a transitive copy and failed CI's frozen-lockfile install.
The security fence itself is platform-correct (path.resolve + path.sep throughout); the test hardcoded a POSIX root, so every startsWith comparison failed on the new Windows CI job.
electron-builder's publish auto-detection demands a GH_TOKEN in pull_request runs and failed the job after the installer had already built. --publish never makes the intent explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The 0.2.0 release. The core is unchanged — OpenLive is the open voice and vision layer for AI agents — and this release adds the integrations that serve it, plus a round of correctness and polish work. Full details in CHANGELOG.md.
Added
Fixed (highlights)
CI
Docs