Skip to content

feat(desktop): hold-to-dictate in the message composer#2709

Open
chillerno1 wants to merge 10 commits into
block:mainfrom
chillerno1:feat/composer-dictation
Open

feat(desktop): hold-to-dictate in the message composer#2709
chillerno1 wants to merge 10 commits into
block:mainfrom
chillerno1:feat/composer-dictation

Conversation

@chillerno1

Copy link
Copy Markdown

Closes #2708

Adds dictation to the message composer, inspired by the Claude Code implementation. The user can click the mic button in the composer toolbar, hold Space (~800 ms; a quick tap still types a space), or hold ⌃Space for an instant start. Words stream into the composer live from the on-device STT stack, releasing the held key commits the trailing phrase, and Enter while dictating sends the message once every in-flight word has arrived. Ships with a Voice & dictation settings section (Standard parakeet-en / Enhanced parakeet-v3 picker with download-on-select). Everything runs locally, so no audio leaves the machine.

How it works

  • Composer (useDictation.ts): one hook owns a dictation session across all three triggers (mic button toggle, Space hold, ⌃Space). It anchors a text stream in the editor, replaces the live partial in place as re-decodes refine it, and commits on finals. Enter is intercepted at window capture during a session so the editor's own submit-on-Enter can't fire early.
  • Rust (dictation.rs + huddle/stt.rs): reuses the huddle STT pipeline in a live-transcript mode. A self-pacing re-decode loop emits partials for the growing phrase buffer and a final at phrase end, over a strictly ordered channel.
  • Enter-to-send is deterministic: stop_dictation pushes a zero-length sentinel after the flush silence, and the STT worker echoes it back as a Flushed event. The marker guarantees every transcript of the stopped session has been delivered, so the send fires on the marker rather than on heuristics, and rapid dictate/Enter/dictate cycles can't lose or splice messages.
  • Model picker (huddle/models.rs): per-model download slots, resolve_dictation_model (preferred model if ready, else the startup model), and download-on-select with progress from the settings card.

Decode guards

While testing this I kept hitting cases where Parakeet's transducer lied on a re-decode of audio it previously got right: sentence fronts vanished, finals came back empty, and random noise turned into "Yeah." The three guards below came out of those sessions. Each one encodes a preference I settled on while dictating day to day rather than a rule from the model's docs, so they're the judgment calls that deserve the closest review (huddle/stt.rs, tests in huddle/stt_tests.rs; assertions use strings from my captured traces).

  1. Collapse guard: every re-decode of a phrase hears a superset of the audio, so it can revise words but never honestly lose them. A decode with fewer words than the best partial is discarded, and a collapsed final commits the best partial instead.
  2. Prefix stitch: after a mid-phrase pause the model sometimes drops everything before the pause while gaining tail words, so the word count still grows and the collapse guard passes it. A settled front of 4+ words never legitimately vanishes wholesale. When a re-decode disagrees on both of the phrase's first two words, its opening is anchored inside the sentence-so-far and the preserved front is spliced back on.
  3. Lone-filler noise gate: a breath or hum that outlasts the VAD onset debounce decodes as a lone filler ("Yeah.", "Mm."). A phrase whose decodes never grow past a single filler is treated as noise and dropped end to end.

Two tradeoffs I made deliberately: I'd rather keep a phantom word the model hallucinated from noise than let an empty final erase a sentence I actually said, and I accept that dictating a bare standalone "Yeah." and nothing else gets dropped by the noise gate. Both felt right in practice; happy to revisit either if reviewers disagree.

Also in here

  • Live-mode pipeline fixes: a wall-clock decode hold (prevents decode storms from overflowing the audio queue and splicing phrases), the audio queue deepened to 30 s, a ~96 ms VAD onset debounce, and empty finals propagating so stranded partials clear.
  • The Space hold shortened to 800 ms; at 1.5 s people start talking before the mic opens.

Testing

  • Unit tests for all three guards plus sticky punctuation in huddle/stt_tests.rs, using strings from captured live traces. #[ignore]d decode experiments document the measured model behavior that motivated the guards.
  • Full suites green at 74683cac: cargo check/clippy clean, Rust 1634 passed, tsc + biome clean, frontend 3461 passed.
  • I exercised it end to end while developing: long rambles, rapid dictate/Enter/dictate cycles, noise-only input (breath, hum, and clicks produce nothing), and model switching mid-session.

The 10 commits build up in review order: composer feature, settings and model picker, Enter-to-send, then the guards.


🤖 Generated with Claude Code

@chillerno1
chillerno1 requested a review from a team as a code owner July 24, 2026 11:38
chillerno1 and others added 10 commits July 24, 2026 21:41
Hold-key dictation using the local huddle STT stack: live streaming
decode with self-pacing re-decode loop, stable text commitment at
pauses, sticky punctuation, Parakeet v3 (0.6B) model support via the
model registry, and caret hidden while words stream in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Three fixes for live (dictation) mode, found by measuring a 35 s ramble
through the stt test harness:

- Wall-clock decode hold: after every decode (partials and flushes) the
  worker waits 2x that decode's duration before the next partial. The
  previous samples-based back-off collapsed into decode storms after a
  long flush stall, overflowing the audio queue and silently dropping
  mic audio - which spliced the phrase and made later decodes delete
  words earlier partials had shown (pause-then-resume text loss).
- Audio queue deepened 50 -> 300 slots (5 s -> 30 s) so the worst-case
  flush stall of a 30 s phrase cannot overflow it.
- Onset debounce (~96 ms of sustained VAD speech) before opening a
  phrase, so keyboard clicks and background blips no longer decode to
  hallucinated filler ('yeah', 'uh'). Onset audio is buffered while it
  proves itself; huddle mode untouched.
- Empty finals now propagate to the frontend so a partial shown for a
  phrase that decoded to nothing is cleared instead of stranded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Adds a "Voice & dictation" section under Personal settings with a
Standard (parakeet-en) / Enhanced (parakeet-v3) model picker:

- models.rs: per-model download slots (startup model shares its slot),
  SttModelInfo for the frontend, resolve_dictation_model (preferred if
  ready, else startup model), start_stt_download_for.
- dictation.rs: start_dictation takes an optional model id and rebuilds
  the pipeline when the model changes; new get_dictation_models and
  download_dictation_model commands.
- Frontend: localStorage-backed preference (dictationModelPreference),
  passed to start_dictation per session; VoiceSettingsCard with
  download-on-select, progress polling, and fallback note while the
  chosen model is still downloading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Enter during a live dictation session is intercepted at window capture
(ahead of the editor's submitOnEnter, which would send before the
trailing phrase decoded): dictation stops, and the submit fires once the
trailing final lands in the doc (3 s cap if it never arrives; immediate
when no partial is showing). Repeat Enters while the send is queued are
swallowed so it can't double-send.

Also de-slops the Voice & dictation settings description (em dash out,
Enter-to-send mentioned).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Rapid dictate -> Enter -> dictate cycles lost messages two ways: a
restart cancelled a send still waiting on its trailing phrase (the
message never went out), and the "nothing on screen, send now" fast
path fired before the trailing phrase had decoded (the last words were
cut off). Stragglers from the stopped session could also splice into
the next session's text through the 15 s ownership window.

Replace the heuristics with a deterministic marker: stop_dictation
pushes a zero-length sentinel batch after the flush silence, the STT
worker echoes it back on the (strictly ordered) live channel as
LiveEvent::Flushed, and it reaches the webview as a dictation-flushed
event -- a hard guarantee that every transcript of the stopped session
has been delivered. Enter-to-send now submits on the marker; restarting
with a send pending fires that send immediately instead of dropping it,
and discards the old session's stragglers until its marker arrives.
Ownership also releases on the marker instead of decaying for 15 s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
At 1.5 s people start talking well before the mic opens, so the first
words never reach the pipeline and the surviving fragment decodes as
filler ("Yeah").

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
The Enhanced (parakeet-v3) transducer occasionally collapses a re-decode
of a phrase to empty or one junk word ("Yeah") even though earlier
decodes of the same growing buffer were fine. The frontend trusted
finals unconditionally, so a collapsed final replaced the sentence on
screen with nothing.

Every decode of a phrase sees a superset of the audio any earlier one
saw, so word count can honestly grow or hold but not shrink. Track the
best partial per phrase; suppress partials that lose words against it,
and when the final loses words, commit the best partial instead.

Also includes temporary DEBUG(dictation-yeah) tracing on both sides of
the event stream — to be stripped before the PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
The word-count guard (dff5f8d) catches decodes that collapse outright,
but chl's next trace showed a second shape: once a phrase contains an
internal pause, a re-decode sometimes returns only the words after the
pause — dropping the front while gaining new tail words, so the word
count still grows and the guard passes it ("Big questions are coming
out, I need to start asking." -> "I need to start asking whether
it's...").

A settled front (>= 4 words) never legitimately vanishes wholesale —
early decodes only rewrite a 1-3 word start. When a re-decode disagrees
on both of the phrase's first two words, anchor its opening words inside
the best partial and splice the preserved front back on; with no anchor,
append. Applied to partials and finals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Parakeet decodes breaths/hums/clicks that outlast the VAD onset debounce
as a lone filler word ("Yeah.", "Mm."), so every random noise inserted
"Yeah." into the composer. A phrase whose decodes never grow past a lone
filler never contained speech: its partials are suppressed (and never
recorded as best/punct partial), and its final is dropped — the empty
final still flows to the frontend to clear phrase state.

Trace strings from the 2026-07-24 ramble session are unit-tested.
Deliberately dictating a bare "Yeah." is dropped too — known tradeoff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Removes the temporary DEBUG(dictation-yeah) event traces from the Rust
dictation commands and the composer hook now that the decode-collapse,
prefix-stitch, and lone-filler guards are verified against live repros.
Moves the stt.rs unit tests and #[ignore]d decode experiments into a
sibling stt_tests.rs, dropping the file-size override for stt.rs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
@chillerno1
chillerno1 force-pushed the feat/composer-dictation branch from 74683ca to 29e8355 Compare July 24, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(desktop): dictation in the message composer with live on-device transcription

1 participant