CTC forced alignment for word timings (supersedes #61, #65)

[wassgha]

CTC forced alignment for word timings

Replaces Whisper's DTW-derived word boundaries (and Parakeet's TDT ones) with boundaries measured against the audio: a language-specific CTC acoustic model, Viterbi over the standard CTC lattice, and an envelope pass to widen the peaky spans that CTC returns.

Supersedes #61 and #65. This is #65's branch merged onto current main, plus a fix for two bugs in its multilingual layer. #61 should be closed unmerged — its commits are all contained here, and on its own it aligns every language against an English-only wav2vec2 (stripping accents to nothing and Han characters entirely) and skips Parakeet.

What lands

lib/forcedAlign.ts — the algorithm, pure and model-free so it's testable without loading weights. Viterbi over blank t0 blank t1 … blank, word spans read off the path, unspellable words (digits, bare punctuation) interpolated from their neighbours, batching at the widest available pause so the frames × tokens lattice stays bounded.

lib/alignModels.ts — language → aligner map. English keeps the small Librispeech wav2vec2 (~86 MB q4); es/fr/de share the MMS forced-aligner; zh uses an XLS-R CTC model with Han in its vocab (~240 MB q4 each). A language with no entry skips CTC and keeps the heuristic path.

lib/align.ts — new work that needs no model at all and stands on its own:

• speechEnvelope — loudness sampled every 5 ms, as the max of a broadband and a >2 kHz envelope each normalised against its own 95th percentile. The high band is what lets an unvoiced fricative register: broadband alone puts the onset of "shot" on the vowel, ~0.2 s late.
• estimateLagFromEnvelope — estimates the global shift from loudness rises rather than the VAD mask, which goes flat (and silently returns ~0) on speech that never pauses.
• refineOnsets — pulls each VAD onset back off the vowel onto the sound that actually starts the word.
• repairCollapsedWords — re-splits words Whisper's DTW collapsed to a sliver and handed to a neighbour. On the test clip "evening" spanned 0.06 s while the preceding "and" took 0.40 s.

ALIGN_LEAD_S, the fixed 80 ms perceptual nudge, is removed — see the open question below.

workers/transcription.worker.ts — the aligner gets its own ModelManager rather than joining the ASR registry. The deciding factor is lifecycle: the ASR registry runs under a WebGPU→WASM fallback policy whose recovery step is unloadAll(), and the aligner never asks for a device, so a GPU loss would otherwise throw away perfectly good WASM weights. Whisper and Parakeet share one post-ASR timing pass; disfluency placeholders run after alignment, since ... has no CTC spelling.

The two bugs fixed on top (b207e41)

Both derivations in alignerModel() were correct on two of the three shipped aligners and silently wrong on the third. Neither raises; both just cost timing accuracy on es/fr/de.

Blank. blankId came from pad_token_id. For wav2vec2-base-960h and XLS-R-zh, <pad> is the CTC blank and sits at 0, so that was accidentally right. The MMS forced-aligner declares them separately — <blank> at 0, <pad> at 1 — so MMS scored the lattice's blank states against a column the model never activates. Viterbi doesn't fail there; it avoids blank states, which smears every boundary.

Delimiter. delimiterId came from encode("|"), which cannot detect absence: a vocab without | maps it to <unk>, one id, indistinguishable from a real hit. MMS has no |, so <unk> was threaded between every pair of words. The old comment ("MMS has none") described the intent correctly; the check didn't implement it.

Both now go through convert_tokens_to_ids, which yields undefined rather than <unk> for an absent token. Moved out of the worker into ctcVocabFromTokenizer() so it's unit-testable.

Verification

tsc --noEmit, eslint, and all 17 test files pass.

Checked against the live tokenizers, all five languages:

en  blank=0  delim=4          "HELLOTHERE"      unk-hits=0/10
es  blank=0  delim=undefined  "cancioncafe"     unk-hits=0/11
fr  blank=0  delim=undefined  "ouetesvous"      unk-hits=0/10
de  blank=0  delim=undefined  "gruesseschoen"   unk-hits=0/13
zh  blank=0  delim=4          "你好世界"           unk-hits=0/4

Before the fix, es/fr/de resolved blank=1 and delim=3. Eight new assertions in tests/forced-align-test.ts pin this using fakes built from the ids the real tokenizers report.

Not verified: vad-regression-test.ts skips its audio-coverage check when ffmpeg isn't on PATH, which was the case in my environment. Worth a run somewhere with ffmpeg before merge.

Open questions

• ALIGN_LEAD_S removal. The 80 ms nudge existed because a highlight landing early reads as in-sync while one landing late reads as lagging. CTC measures properly so the nudge shouldn't be needed — but forceAlign falls back to the heuristic path on any failure (cold-cache error, failed batch, unsupported language), and that fallback now has no lead. Worth deciding whether to keep the nudge on the fallback path only.
• Download and memory cost lands hardest on WebKit. +86 MB (en) or +240 MB (multilingual) on top of Parakeet's 1.3 GB, plus a full-audio CTC forward pass. Transcription accuracy improvements #69 just forced Safari and all iOS browsers onto WASM specifically because of memory-driven tab reloads; this pushes against that ceiling. The aligner is already WASM-only so there's no direct conflict, but it's more pressure on the platform we just hardened.
• ROADMAP.md:33 already marks "Better word↔audio alignment (speech-onset anchoring / lag correction)" as done. This is the tier beyond that — the entry probably wants splitting or updating.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
app.rescript	Ready	Preview	Aug 6, 2026 2:44am