Live Reply Coach — MVP

[lulu0119]

Live Reply Coach — MVP

Problem Statement

A foreign-language learner (e.g., a Chinese student working a convenience-store part-time job in Japan) can often understand what the other person says but doesn't know how to reply. Existing AI voice tools translate (help you understand) or speak on your behalf (AI代说), but neither helps the learner open their own mouth in the few seconds after the other person finishes speaking. The learner needs several reply options — native-language meaning + target-language text + reading annotation — to pick from and say themselves, in real time during a live conversation.

Solution

A real-time reply coach that runs as a PWA in the browser. It listens via the microphone, VAD segments speech, local speaker verification distinguishes the user from the other person, STT transcribes each segment, and only on the other person's turns does an LLM stream 3 reply candidates (Chinese meaning / target-language text / reading). The user's own speech is also transcribed and enters the conversation context (affecting the next round's suggestions) but produces no candidates. When the other person speaks again while candidates are streaming, the in-flight LLM is aborted and partial candidates are discarded. A session ends with a text summary the user can copy.

Orchestration lives entirely in the browser (VAD, speaker gate, conversation store). A thin Hono proxy on Railway forwards LLM and STT calls, hides API keys, and streams SSE. MVP has no accounts; voiceprint and conversation persist in IndexedDB per device.

User Stories

Onboarding

1. As a learner, I want to enroll my voiceprint by reading a fixed passphrase, so that the app can distinguish my speech from the other person's.
2. As a learner, I want to skip re-enrolling on the same device, so that I don't read the passphrase every time I open the app.
3. As a learner, I want to re-record my voiceprint when my environment changes (e.g., I have a cold), so that speaker detection stays accurate.
4. As a learner, I want to set my target-language level (N5–N1), so that reply candidates match my proficiency.
5. As a learner, I want to pick a scene (convenience store / general), so that reply candidates fit the context.

Session — listening & candidates

6. As a learner, I want to start a session, so that the app begins listening.
7. As a learner, I want the app to detect when the other person finishes a sentence (pause threshold), so that candidates appear at the right moment.
8. As a learner, I want the app to distinguish the other person's speech from mine, so that candidates only appear when the other person speaks.
9. As a learner, I want 3 reply candidates (Chinese meaning / target text / reading) when the other person finishes, so that I can pick one to say.
10. As a learner, I want candidates to stream in as they're generated, so that I can start reading the first one before all three are done.
11. As a learner, I want my own speech to be transcribed and remembered, so that the next round's candidates account for what I actually said.
12. As a learner, I want my own speech to NOT produce candidates, so that I'm not shown replies to myself.
13. As a learner, I want the conversation timeline to show each turn with speaker label and transcription, so that I can follow what's been said.
14. As a learner, I want the timeline transcription to be read-only, so that I see what the ASR heard (even if wrong) without editing complexity.
15. As a learner, I want candidates to be replaced when the other person speaks again (interrupting), so that I always see suggestions relevant to the latest context.
16. As a learner, I want a "换一批" (regenerate) option, so that I can get fresh candidates if the current ones aren't good.
17. As a learner, when the other person speaks several times in a row without me speaking, I want candidates to refresh each time, so that I always see suggestions for the latest turn.
18. As a learner, when I start speaking during the other person's pause, I want the app to listen to me instead of showing candidates, so that my speech is captured.

Session — resilience & end

19. As a learner, I want the session to survive an accidental refresh / iOS background kill, so that I don't lose the conversation.
20. As a learner, I want to end the session and get a text summary, so that I can review what I learned.
21. As a learner, I want the summary to be copyable, so that I can save it elsewhere.
22. As a learner, I want the summary to include the transcribed conversation, so that I can review what was actually said.
23. As a learner, I want to start a new session after ending one, so that the previous conversation doesn't clutter the new one.
24. As a learner, I want a failed transcription to not kill the session, so that one network hiccup doesn't end my conversation.
25. As a learner, I want a failed candidate generation to show a retry button, so that I can try again if the LLM call fails.
26. As a learner, I want the app to keep listening even after an error, so that the conversation continues.

Platform

27. As a learner on iPhone, I want to add the app to my home screen (PWA), so that it feels like an app without an install.
28. As a learner on iPhone, I want full-screen PWA mode, so that the browser chrome doesn't get in the way.
29. As a learner on Mac, I want to open the same URL in a browser, so that I can use it on desktop too.
30. As a learner, I want a responsive UI, so that it works in portrait (mobile) and landscape (desktop).

Operator / developer

31. As the operator, I want API keys to live only in server env, so that users never see provider keys.
32. As the operator, I want users to go through our proxy, so that I control which providers/models are used.
33. As the operator, I want to switch LLM/STT provider via env, so that I can change models without code changes.
34. As the operator, I want to configure the pause threshold via env, so that I can tune it per scene without code changes.
35. As a developer, I want a playground to test pipeline modules in isolation, so that I can develop without the full UI.
36. As a developer, I want to inject mock speaker labels in the playground, so that I can test downstream pipeline without trusting the speaker model.
37. As a developer, I want vieval to evaluate prompt quality, so that I can iterate on reply candidate generation.

Implementation Decisions

Full detail lives in docs/2026-07-20-live-reply-coach-mvp-tech-spec.md and docs/adr/0001-client-orchestration-thin-proxy.md. Summary:

• Monorepo: pnpm workspace + Turborepo. apps/ are thin entry points; packages/ hold nearly all implementation (AIRI pattern).
• Apps: apps/web (Vite + React + PWA, thin mount), apps/api (Hono proxy), apps/playground (dev tool for pipeline modules), apps/desktop (P1, Electron thin shell for Mac system audio).
• Packages: pipeline (conversation loop state machine), speaker (enroll + verify, WASM), conversation (turn store, IndexedDB), llm (provider-agnostic client factory), stt (provider-agnostic client factory), audio (AudioSource + encodeWav), prompts (Velin TSX), ui, pages, shared.
• Orchestration in browser: VAD, speaker gate, STT upload, conversation store all run client-side. The server only hides keys and forwards.
• Server: Hono thin proxy, two routes. POST /llm is SSE streaming (raw tokens, browser incrementally parses structured 3-candidate output). POST /stt is batch POST returning JSON. The proxy also serves apps/web's static build via serveStatic (same origin, no CORS). Deployed on Railway as a long-running process (no serverless timeout for SSE).
• Provider config: all via env, never DB. Naming: prefix + LLM_ACTIVE / STT_ACTIVE selector (e.g., LLM_OPENROUTER_BASE_URL / _API_KEY / _MODEL). MVP defaults both LLM and STT to OpenRouter (one key, one bill). Code is provider-agnostic — createLlmClient({ provider, baseUrl, apiKey, model }) / createSttClient(...); OpenRouter is one provider value, not hardcoded. Adding a provider = add adapter + env block, no refactor.
• STT: cloud, batch per VAD-cut segment (not streaming). Default openai/gpt-4o-transcribe (best multilingual/Japanese accuracy), groq/whisper-large-v3-turbo as cost-fallback. Audio sent as WAV 16kHz mono PCM (packages/audio encodeWav).
• Conversation state machine (§2.4 of spec): states IDLE / OTHER_SPEAKING / USER_SPEAKING / LLM_STREAMING. Pause threshold (default 1s, env VAD_OTHER_PAUSE_MS / VAD_USER_PAUSE_MS) triggers LLM after other stops. Interruption: new other speech during LLM_STREAMING aborts the in-flight request, discards partial candidates, starts capturing new other turn; next LLM receives all completed ConversationTurns (partial candidates never enter context). User抢说 during other's pause cancels the pending LLM trigger and captures the user turn instead. Multi-other-turn loops with no user speech keep refreshing candidates.
• Speaker: local verification (1-user enroll + binary classify), not open diarization. No LLM speaker correction (costs 2× and is itself unreliable). No post-hoc correction. Manual labeling is a playground-only test-harness capability (inject mock labels), never a production env flag.
• Enrollment persistence: voiceprint embedding cached in IndexedDB; re-record button; each device enrolls once (different device = different mic = re-enroll). Interface: enroll / loadEmbedding / saveEmbedding / verify.
• Conversation persistence: MVP persists active session to IndexedDB (append-only log), no history list — crash/refresh recovery + F09 regeneratable. Later: history list + cloud sync to Supabase per userId.
• Accounts: MVP has none. ConversationTurn reserves optional userId. Later: hosted Supabase auth + JWT middleware on proxy.
• Transcription visibility: shown in timeline (read-only), not editable (F03 changed from original "可编辑"). Candidates are not editable either — use "换一批" to regenerate.
• Error handling: retry once (1s backoff) for both STT and LLM; STT failure → appendTurn({ text: '', sttFailed: true }), UI marks red, not user-editable, loop continues; LLM failure → candidate area shows retry button, other turn already stored, loop continues. Neither kills the session. No fallback provider / circuit breaker / multi-round backoff in MVP. Retry lives in packages/pipeline; clients stay simple.
• Prompts: Velin TSX in packages/prompts, scene-split (reply-suggestions.tsx, session-summary.tsx). Evaluated via vieval (root vieval.config.ts + evals/), matrix over level / scene / model / historyDepth.

Testing Decisions

A good test asserts external behavior, not implementation details. The repo is greenfield, so all seams are new; they are placed at the highest point possible.

Primary seam (the one) — integration test: real packages/pipeline + real apps/api Hono proxy + mocked upstream LLM/STT providers (no real OpenRouter calls) + injected audio segments / speaker results (no microphone, no WASM). Asserts: the ConversationTurn[] sequence, candidate emissions, interruption (abort propagation), pause-threshold triggering, retry behavior, multi-turn-no-user, user-抢说 cancellation, SSE chunk framing, key non-leakage (env keys never appear in responses), and proxy abort propagation. This single high-level seam covers orchestration + proxy. Audio and WASM are mocked at the input; OpenRouter is mocked at the upstream; everything between is real.

Optional fast variant — pure state machine: same injected inputs but provider clients mocked at the packages/llm / packages/stt level (no HTTP, no proxy spawn), for fast deterministic iteration on state-machine logic. Add only if the integration seam is too slow to run frequently.

Existing seam — vieval (§2.6 of spec): evaluates reply-candidate prompt quality over a level / scene / model / historyDepth matrix. Already specified; not new.

Out of Scope

• Candidate editing / self-authoring (team decision; use "换一批" instead).
• TTS / AI speaking on the user's behalf (product is the user opening their own mouth).
• iPhone call monitoring (impossible on web/PWA).
• Login accounts (MVP; later).
• Offline ASR + LLM (online APIs only).
• LLM-based speaker correction (hot path or post-hoc).
• Editable transcription (read-only display).
• Fallback provider routing / circuit breakers / multi-round exponential backoff.
• Cross-device voiceprint sync (each device enrolls once).
• History session list (MVP; later with cloud sync).
• Mac system audio (P1 Electron, not MVP).

Further Notes

• Full technical detail: docs/2026-07-20-live-reply-coach-mvp-tech-spec.md (§1.3 feature list, §2.1 stack, §2.2 monorepo structure, §2.4 pipeline + state machine + enrollment/conversation persistence + error transitions, §2.7 playground, §2.8 server & deployment, §2.9 config & env).
• Architecture rationale: docs/adr/0001-client-orchestration-thin-proxy.md (why client-side orchestration + thin Hono proxy + Railway; why not Next.js+Vercel / serverless / VPS).
• Product origin: docs/2026-07-16-live-reply-coach-language-assist.md.
• Reference repos: airi, webai-example-realtime-voice-chat, velin, vieval, deepchat.