feat(desktop): hold-to-dictate in the message composer#2709
Open
chillerno1 wants to merge 10 commits into
Open
feat(desktop): hold-to-dictate in the message composer#2709chillerno1 wants to merge 10 commits into
chillerno1 wants to merge 10 commits into
Conversation
Hold-key dictation using the local huddle STT stack: live streaming decode with self-pacing re-decode loop, stable text commitment at pauses, sticky punctuation, Parakeet v3 (0.6B) model support via the model registry, and caret hidden while words stream in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
Three fixes for live (dictation) mode, found by measuring a 35 s ramble
through the stt test harness:
- Wall-clock decode hold: after every decode (partials and flushes) the
worker waits 2x that decode's duration before the next partial. The
previous samples-based back-off collapsed into decode storms after a
long flush stall, overflowing the audio queue and silently dropping
mic audio - which spliced the phrase and made later decodes delete
words earlier partials had shown (pause-then-resume text loss).
- Audio queue deepened 50 -> 300 slots (5 s -> 30 s) so the worst-case
flush stall of a 30 s phrase cannot overflow it.
- Onset debounce (~96 ms of sustained VAD speech) before opening a
phrase, so keyboard clicks and background blips no longer decode to
hallucinated filler ('yeah', 'uh'). Onset audio is buffered while it
proves itself; huddle mode untouched.
- Empty finals now propagate to the frontend so a partial shown for a
phrase that decoded to nothing is cleared instead of stranded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Adds a "Voice & dictation" section under Personal settings with a Standard (parakeet-en) / Enhanced (parakeet-v3) model picker: - models.rs: per-model download slots (startup model shares its slot), SttModelInfo for the frontend, resolve_dictation_model (preferred if ready, else startup model), start_stt_download_for. - dictation.rs: start_dictation takes an optional model id and rebuilds the pipeline when the model changes; new get_dictation_models and download_dictation_model commands. - Frontend: localStorage-backed preference (dictationModelPreference), passed to start_dictation per session; VoiceSettingsCard with download-on-select, progress polling, and fallback note while the chosen model is still downloading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
Enter during a live dictation session is intercepted at window capture (ahead of the editor's submitOnEnter, which would send before the trailing phrase decoded): dictation stops, and the submit fires once the trailing final lands in the doc (3 s cap if it never arrives; immediate when no partial is showing). Repeat Enters while the send is queued are swallowed so it can't double-send. Also de-slops the Voice & dictation settings description (em dash out, Enter-to-send mentioned). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
Rapid dictate -> Enter -> dictate cycles lost messages two ways: a restart cancelled a send still waiting on its trailing phrase (the message never went out), and the "nothing on screen, send now" fast path fired before the trailing phrase had decoded (the last words were cut off). Stragglers from the stopped session could also splice into the next session's text through the 15 s ownership window. Replace the heuristics with a deterministic marker: stop_dictation pushes a zero-length sentinel batch after the flush silence, the STT worker echoes it back on the (strictly ordered) live channel as LiveEvent::Flushed, and it reaches the webview as a dictation-flushed event -- a hard guarantee that every transcript of the stopped session has been delivered. Enter-to-send now submits on the marker; restarting with a send pending fires that send immediately instead of dropping it, and discards the old session's stragglers until its marker arrives. Ownership also releases on the marker instead of decaying for 15 s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
At 1.5 s people start talking well before the mic opens, so the first
words never reach the pipeline and the surviving fragment decodes as
filler ("Yeah").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
The Enhanced (parakeet-v3) transducer occasionally collapses a re-decode
of a phrase to empty or one junk word ("Yeah") even though earlier
decodes of the same growing buffer were fine. The frontend trusted
finals unconditionally, so a collapsed final replaced the sentence on
screen with nothing.
Every decode of a phrase sees a superset of the audio any earlier one
saw, so word count can honestly grow or hold but not shrink. Track the
best partial per phrase; suppress partials that lose words against it,
and when the final loses words, commit the best partial instead.
Also includes temporary DEBUG(dictation-yeah) tracing on both sides of
the event stream — to be stripped before the PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
The word-count guard (dff5f8d) catches decodes that collapse outright, but chl's next trace showed a second shape: once a phrase contains an internal pause, a re-decode sometimes returns only the words after the pause — dropping the front while gaining new tail words, so the word count still grows and the guard passes it ("Big questions are coming out, I need to start asking." -> "I need to start asking whether it's..."). A settled front (>= 4 words) never legitimately vanishes wholesale — early decodes only rewrite a 1-3 word start. When a re-decode disagrees on both of the phrase's first two words, anchor its opening words inside the best partial and splice the preserved front back on; with no anchor, append. Applied to partials and finals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
Parakeet decodes breaths/hums/clicks that outlast the VAD onset debounce
as a lone filler word ("Yeah.", "Mm."), so every random noise inserted
"Yeah." into the composer. A phrase whose decodes never grow past a lone
filler never contained speech: its partials are suppressed (and never
recorded as best/punct partial), and its final is dropped — the empty
final still flows to the frontend to clear phrase state.
Trace strings from the 2026-07-24 ramble session are unit-tested.
Deliberately dictating a bare "Yeah." is dropped too — known tradeoff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: chillerno1 <gh.chiller@pm.me>
Removes the temporary DEBUG(dictation-yeah) event traces from the Rust dictation commands and the composer hook now that the decode-collapse, prefix-stitch, and lone-filler guards are verified against live repros. Moves the stt.rs unit tests and #[ignore]d decode experiments into a sibling stt_tests.rs, dropping the file-size override for stt.rs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: chillerno1 <gh.chiller@pm.me>
chillerno1
force-pushed
the
feat/composer-dictation
branch
from
July 24, 2026 11:41
74683ca to
29e8355
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2708
Adds dictation to the message composer, inspired by the Claude Code implementation. The user can click the mic button in the composer toolbar, hold Space (~800 ms; a quick tap still types a space), or hold ⌃Space for an instant start. Words stream into the composer live from the on-device STT stack, releasing the held key commits the trailing phrase, and Enter while dictating sends the message once every in-flight word has arrived. Ships with a Voice & dictation settings section (Standard
parakeet-en/ Enhancedparakeet-v3picker with download-on-select). Everything runs locally, so no audio leaves the machine.How it works
useDictation.ts): one hook owns a dictation session across all three triggers (mic button toggle, Space hold, ⌃Space). It anchors a text stream in the editor, replaces the live partial in place as re-decodes refine it, and commits on finals. Enter is intercepted at window capture during a session so the editor's own submit-on-Enter can't fire early.dictation.rs+huddle/stt.rs): reuses the huddle STT pipeline in a live-transcript mode. A self-pacing re-decode loop emits partials for the growing phrase buffer and a final at phrase end, over a strictly ordered channel.stop_dictationpushes a zero-length sentinel after the flush silence, and the STT worker echoes it back as aFlushedevent. The marker guarantees every transcript of the stopped session has been delivered, so the send fires on the marker rather than on heuristics, and rapid dictate/Enter/dictate cycles can't lose or splice messages.huddle/models.rs): per-model download slots,resolve_dictation_model(preferred model if ready, else the startup model), and download-on-select with progress from the settings card.Decode guards
While testing this I kept hitting cases where Parakeet's transducer lied on a re-decode of audio it previously got right: sentence fronts vanished, finals came back empty, and random noise turned into "Yeah." The three guards below came out of those sessions. Each one encodes a preference I settled on while dictating day to day rather than a rule from the model's docs, so they're the judgment calls that deserve the closest review (
huddle/stt.rs, tests inhuddle/stt_tests.rs; assertions use strings from my captured traces).Two tradeoffs I made deliberately: I'd rather keep a phantom word the model hallucinated from noise than let an empty final erase a sentence I actually said, and I accept that dictating a bare standalone "Yeah." and nothing else gets dropped by the noise gate. Both felt right in practice; happy to revisit either if reviewers disagree.
Also in here
Testing
huddle/stt_tests.rs, using strings from captured live traces.#[ignore]d decode experiments document the measured model behavior that motivated the guards.74683cac: cargo check/clippy clean, Rust 1634 passed, tsc + biome clean, frontend 3461 passed.The 10 commits build up in review order: composer feature, settings and model picker, Enter-to-send, then the guards.
🤖 Generated with Claude Code