Skip to content

feat(voice): add OpenAI Realtime Expert and Spokesperson mode - #283

Merged
johnmatthewtennant merged 41 commits into
mainfrom
jt/realtime-master-emissary
Sep 3, 2026
Merged

feat(voice): add OpenAI Realtime Expert and Spokesperson mode#283
johnmatthewtennant merged 41 commits into
mainfrom
jt/realtime-master-emissary

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Category: feature
User Impact: Users can choose OpenAI Realtime voice for a responsive spoken interface backed by the full Berd coding agent.

Problem: Chained speech-to-text and text-to-speech cannot provide the same low-latency, interruptible conversation as the OpenAI Realtime API, while a Realtime voice model alone cannot perform the full work available to a Berd agent.

Solution: Add an experimental Realtime voice mode that presents one seamless assistant using a fast Spokesperson for live conversation and the normal Berd Expert for tools and longer-running work. The two roles exchange ordered handoffs and results behind the scenes, with explicit debug rendering available for development. Chained voice remains the fresh-install default.

Key changes
  • Add the Realtime WebRTC voice lifecycle, server VAD, barge-in, transcript correction, typed-message forwarding, pause/resume replay, and cross-window ownership.
  • Add ordered Expert/Spokesperson handoffs with cursors, bounded reminders, SAY/context delivery, dismissal, and debug coordination bubbles.
  • Preserve normal Expert thinking and tool output while rendering spoken turns consistently with Berd voice conversations.
  • Add Voice settings for Realtime model, transcription, voice, speed, turn detection, VAD, noise reduction, reasoning, and subtle/debug presentation.
  • Reuse the Berd-owned OpenAI voice credential and keep the long-lived key in Tauri while exposing only an ephemeral Realtime client secret to the renderer.
  • Keep ordinary text sends and text-only transcript projection on their existing fast paths; bound cross-window Realtime status checks so voice coordination cannot hold prompt ownership indefinitely.
  • Add berdctl routing commands plus deterministic text-driven evaluation scenarios and lifecycle regression coverage.

Validation

  • Independent Opus risk review cleared the final isolation changes after checking shared chat, transcript, release-gate, and cross-window paths.
  • Focused Realtime/chat/projection tests: 26 passed. Release-script tests: 95 passed.
  • Full pre-push gate passed: formatting, frontend checks, Tauri checks, and Clippy.

@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review September 3, 2026 12:40
@johnmatthewtennant
johnmatthewtennant requested a review from a team September 3, 2026 12:40

@morgmart morgmart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated code review

REQUEST_CHANGES. Corrected the completed static Engineering review against the exact three-dot comparison 5c31189...09f6b1b and the supplied current changed-line anchors. One blocking lifecycle finding remains: failed transport-only Realtime coordination can exhaust its retry budget while remaining an invisible, persistent queue head that blocks every later message. The earlier startup-cancellation finding was removed after re-checking the complete flow: stop invalidates the run, cleans resources, and late microphone completion is rejected by run identity, so the claimed stuck user state was not supported. Final self-check covered every changed user flow; accessibility; i18n/localization completeness; navigation and consent guards; async, error, never-completes, lifecycle, and race behavior; test honesty; design-system/project rules; duplicate/overlapping findings; and concrete evidence plus user effect for blocking severity. No findings fall outside the exact PR comparison or supplied anchors. The supplied GitHub evidence was inspected, records no local execution, and is structurally valid for the exact base and head; all ten supplied check runs completed successfully. Required checks still independently govern merge readiness.

Deterministic publication result: 1 blocking and 0 non-blocking inline finding(s) publishable; 0 duplicate(s) suppressed; 0 blocking screenshot-evidence requirement(s) in this review body.

Comment thread src/features/chat/hooks/useMessageQueue.ts

@morgmart morgmart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated code review

APPROVE. Completed a fresh static Engineering review of the exact three-dot comparison 5c31189...9fd6643 after explicitly applying the project code-review skill and the user wes-review skill. No publishable findings remain. The previously reported hidden-coordination queue failure is fixed at this head: after the bounded retry budget, the implementation emits localized generic feedback without exposing private coordination text, removes and persists removal of the blocking queue head, stops only the matching Realtime session, and allows the next queued message to drain; the new regression test covers exhaustion, subsequent user-message delivery, payload privacy, and restart persistence. The resolved authoritative thread has a substantive human reply and was not re-raised. The review traced Realtime startup and teardown, microphone and credential boundaries, typed and spoken transcript delivery, Expert and Spokesperson handoffs and reminders, cross-window bridge routing, queue persistence and recovery, session promotion, navigation and consent guards, CLI contracts, release gates, accessibility, localization, and changed-test honesty. Final self-check covered every changed user flow; accessibility; i18n/localization completeness; navigation and consent guards; async, error, never-completes, lifecycle, and race behavior; test honesty; design-system/project rules; duplicate and overlapping findings; and concrete evidence plus user effect for blocking severity. Unsupported claims were removed. The supplied GitHub evidence was inspected, records no local execution, is structurally valid for the exact base and head, and reports ten successful check runs. Required checks still independently govern merge readiness.

Deterministic publication result: 0 blocking and 0 non-blocking inline finding(s) publishable; 0 duplicate(s) suppressed; 0 blocking screenshot-evidence requirement(s) in this review body.

Pending checks: 1 check(s) are not complete.

This approval reflects the completed code review only; merge readiness remains governed by the repository's required checks.

@johnmatthewtennant
johnmatthewtennant merged commit 1b4e511 into main Sep 3, 2026
10 checks passed
@johnmatthewtennant
johnmatthewtennant deleted the jt/realtime-master-emissary branch September 3, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants