Two suggestions from someone who works on transcription, in the r/macapps thread for 2.0. Word timing precision itself is tracked in #948; this issue covers what that one does not.
1. A custom vocabulary field, passed as Whisper's initial prompt
Names, product names, libraries and acronyms are the words Whisper mishears most, and they are the ones a demo is full of. Correcting them after the fact works (double-click a word), but the corrected word keeps the timings of what the model thought it heard.
whisper.cpp already conditions decoding on an initial prompt (whisper_full_params.initial_prompt). Our helper (electron/native/whisper-stt/src/main.cpp) does not pass one. A plain text field where the user lists the terms they are about to say would bias the decode before it runs, and should also give cleaner boundaries on those words.
- Where: a "Names and terms" field next to the transcription language, saved with the project (and maybe a global default).
- How: pass it as the initial prompt on every chunk the helper decodes (
electron/stt/chunking.ts), not only the first one.
- Limit: Whisper only reads the last 224 tokens of a prompt, so cap the field and say so.
- Check: run
tools/stt-eval on a clip full of jargon, with and without the prompt, and compare word error rate and boundary error on those words.
2. Snap cuts made inside continuous speech to the quietest nearby point
Word times come from DTW, then the CTC aligner. On clear speech they land within about 30 ms (#948). On fast speech, run-together words and plosive onsets they drift, and a cut taken exactly at the reported boundary can clip the start of the next word or leave a scrap of the deleted one. Removing short fillers such as "um" is the worst case.
Suggestion: when a transcript cut lands inside continuous speech, place the actual cut at the lowest-energy point within a short window around the boundary (a few tens of milliseconds), without moving the word times shown in the transcript.
Caution from #948: the old RMS lookback snap dragged correct boundaries about 120 ms early, which is why it was dropped. Whatever replaces it must be symmetric, bounded, applied to the cut point only, and measured with the word-timing harness (tools/stt-eval/word-timing) before it ships.
3. Fades at every join
The export already applies a 240-sample (5 ms) fade at internal segment boundaries (crates/compositor/src/audio.rs). Whether transcript cuts inside a clip go through that path, and whether preview playback does the same, is still an open check in #948. Listed here only so the two issues stay linked.
Two suggestions from someone who works on transcription, in the r/macapps thread for 2.0. Word timing precision itself is tracked in #948; this issue covers what that one does not.
1. A custom vocabulary field, passed as Whisper's initial prompt
Names, product names, libraries and acronyms are the words Whisper mishears most, and they are the ones a demo is full of. Correcting them after the fact works (double-click a word), but the corrected word keeps the timings of what the model thought it heard.
whisper.cpp already conditions decoding on an initial prompt (
whisper_full_params.initial_prompt). Our helper (electron/native/whisper-stt/src/main.cpp) does not pass one. A plain text field where the user lists the terms they are about to say would bias the decode before it runs, and should also give cleaner boundaries on those words.electron/stt/chunking.ts), not only the first one.tools/stt-evalon a clip full of jargon, with and without the prompt, and compare word error rate and boundary error on those words.2. Snap cuts made inside continuous speech to the quietest nearby point
Word times come from DTW, then the CTC aligner. On clear speech they land within about 30 ms (#948). On fast speech, run-together words and plosive onsets they drift, and a cut taken exactly at the reported boundary can clip the start of the next word or leave a scrap of the deleted one. Removing short fillers such as "um" is the worst case.
Suggestion: when a transcript cut lands inside continuous speech, place the actual cut at the lowest-energy point within a short window around the boundary (a few tens of milliseconds), without moving the word times shown in the transcript.
Caution from #948: the old RMS lookback snap dragged correct boundaries about 120 ms early, which is why it was dropped. Whatever replaces it must be symmetric, bounded, applied to the cut point only, and measured with the word-timing harness (
tools/stt-eval/word-timing) before it ships.3. Fades at every join
The export already applies a 240-sample (5 ms) fade at internal segment boundaries (
crates/compositor/src/audio.rs). Whether transcript cuts inside a clip go through that path, and whether preview playback does the same, is still an open check in #948. Listed here only so the two issues stay linked.