Skip to content

feat(voice): add bounded local dictation and guarded reply playback - #467

Open
chayapats wants to merge 12 commits into
mainfrom
jit/422-bounded-voice
Open

chayapats wants to merge 12 commits into
mainfrom
jit/422-bounded-voice

Conversation

@chayapats

@chayapats chayapats commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Closes #422
Refs #392

Add explicit local dictation and reply playback to the shared client and /s16 composer. Mic reserves finite audio quota before permission/capture; Stop produces an editable private transcript. Add to message and ordinary Send remain separate. Speech never decides an approval. Listen binds the exact currently guarded conversation/request/revision/text; interruption, route loss and stale views/sessions stop client audio independently of Bot work.

The optional protected-host route uses pinned whisper.cpp plus eSpeak NG (explicit macOS say for development/evidence), fixed supervised CPU commands, bounded WAV/text/output, private scratch outside backups and a metadata-only durable quota journal. Actual controller loss kills and joins the CLI; unknown processing retains charges and never replays. Final synchronous HTTP publishers recheck owner/source permission after processing. No remote speech fallback, credentials, automatic downloads or new package dependency is introduced.

ADR 0096, s16-voice/v1, the runbook and native proof record contracts, limits, artifact/license provenance and recovery. Runtime source is 60679513fdf70d0f9b32e409387023e7068dafae; final head is 8ac7df173c862d4bb6e1151507ff6b5d1c06ee13 (its final commit only refreshes evidence).

Review fixes separate the recording deadline from a submission deadline fifteen seconds later, refuse expired recording receipts before microphone permission, and preserve a completed draft when a recording-health read finishes late. Authentication rejection still clears private state. Crash-test and supervisor JSON receipts now publish atomically after complete writes; the initially failing ARM64 marker race is corrected without weaker assertions or longer timeouts.

Validation:

  • Frozen install, typecheck, upstream provenance and committed diff checks passed.
  • Affected voice/client/composition/runtime/package suites: 254 pass, 0 fail, 1,479 assertions, thirteen files on the verified runtime source.
  • Real Chrome bundle/AudioWorklet: 7 pass, 71 assertions, English 1440 px / Thai 390 px, fake native media, keyboard playback, cancel, route loss, denied/unusable capture, delayed permission/navigation and rejected-session cleanup; screenshots inspected.
  • Actual local speech through the composed client: 1 pass / 10 assertions each on macOS ARM64 and Linux ARM64; actual known-sentence STT and guarded-reply WAV, with no work before ordinary Send. Linux uses pinned base.en; the unchanged tiny.en quality failure is documented. Linux controlled core also passed 30 tests / 164 assertions, typecheck and provenance.
  • Exact failing shard reproduction: 641 pass, 17 opt-in skips, 0 fail, 66,627 assertions across 52 files. A fresh CI run on the new head remains required.
  • Native subprocess fixtures exercise real cancellation/deadline/close/SIGKILL settlement and quota recovery. A held Google action stays pending after transcribed approval words; source withdrawal after synthesis resolution still withholds HTTP audio.

S14/default/legacy selection, S15 opt-in, installed policy and historical private scope are unchanged. No owner-host activation or production deploy. English speech proof does not establish Thai recognition quality; #423 continuous calls and #431/#432 physical-device/account/owner-host acceptance remain open. Parent #392 is not closed by this slice.

@chayapats chayapats left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 285c1a933ea7488bb751798fb226395197dca725 against main 7431400a7d1d462c70d9a457d476d270f86f8d59.

The local route is optional and remains outside Bot execution. Checked exact owner/conversation/generation bindings, PCM and output bounds, quota persistence/clock rewind, private scratch/actual subprocess settlement, no automatic replay, raw-body reauthentication, final synchronous transcript/audio publication, current reply/source withdrawal, and client media cleanup across cancel/navigation/session/route loss. Speech cannot send or decide an action; ordinary Send and exact decision cards remain the owning ports. Review fixes include generation-bound Cancel and playback health interruption.

Validation is recorded in the PR: 253 affected/package tests, four actual Chrome flow cases, actual macOS/Linux ARM64 speech and Linux controlled-process recovery/typecheck/provenance. Native runtime source is immutable bd6e317669a0f40d105636977f0379399989fa34; final head additionally records evidence/docs and the worklet authentication test. No remaining blocking finding from this review. Merge remains contingent on successful latest-head checks and resolution of any new review findings. Continuous calls and physical owner-host/device acceptance stay separately open.

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach the opt-in S16 voice path and its protected local speech service.

Fix the recording and submission deadlines before merging so completed dictation is not lost.

Findings

  1. P1 Late permission loses dictation ▶
Fix with agent prompt
### Issue 1
src/s16/client/voice-recording.ts:63
**Late permission loses dictation.** When microphone permission takes more than the 15-second allowance, the recorder uses the remaining reservation time and schedules Stop at the reservation deadline. It still needs to finish the worklet, close the audio context and upload the WAV afterward. The server rejects transcription once `expiresAt` is reached, so a completed recording can be discarded with `voice_capture_expired`. Leave time for cleanup and upload, or use separate recording and submission deadlines.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds optional local dictation and guarded reply playback to the shared client and /s16 composer.

  • The S16 client lets owners dictate drafts and listen to current replies.
  • Configured local speech uses bounded processes and keeps quota across restarts.

One recording deadline issue needs fixing: slow microphone permission can leave no time to upload the completed clip.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Mic[Owner presses Dictate] --> Reserve[Reserve audio quota]
  Reserve --> Capture[Capture bounded local audio]
  Capture --> STT[Local transcription]
  STT --> Draft[Editable private draft]
  Draft --> Add[Add to composer]
  Add --> Send[Ordinary Send]
  Reply[Currently guarded reply] --> Check[Check exact reply and owner]
  Check --> TTS[Local synthesis]
  TTS --> Recheck[Recheck before release]
  Recheck --> Audio[Client playback]
  Interrupt[Interrupt] --> Stop[Stop client audio]
Loading

Reviews (1) · Last reviewed commit: "Document local voice operation and nativ..." · Reviewed by Greptile

Comment thread src/s16/client/voice-recording.ts Resolved

@chayapats chayapats left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 8ac7df173c862d4bb6e1151507ff6b5d1c06ee13 after the deadline and CI corrections. Runtime source is 60679513fdf70d0f9b32e409387023e7068dafae; the final commit only refreshes evidence.

Recording now ends before a separate finite submission expiry, already expired receipts cannot open microphone permission, and an old health response cannot discard a completed draft. Authentication/source/generation checks still withhold private results. The crash-test marker and supervisor settlement receipt publish only complete JSON via atomic rename, after the actual CLI joins; no assertion or timeout was weakened.

Verified 254 affected/client/composition/runtime/package tests (1,479 assertions), seven actual Chrome flow cases (71 assertions), the exact formerly failing shard command (641 pass, 17 opt-in skips, 66,627 assertions), and thirty controlled core tests plus actual native speech on both macOS/Linux ARM64. Typecheck, provenance and committed diff checks passed. Updated evidence records the new Linux executable hash and exact proof revision. No remaining blocking finding from primary review; merge is contingent on successful new-head checks and resolution of any new findings. #423/#431/#432 and parents retain their separate acceptance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Frontier] [VOICE16-01] Dictate messages and speak replies with bounded usage

1 participant