[codex] Improve macOS dictation latency

[murmer-blip]

Summary

• keep macOS stop status correct: REC while capture is active/stopping, then TXT... once transcription starts
• make MLX Whisper decode deterministic for short dictation by disabling the temperature fallback ladder
• reduce macOS stop-wrapper polling latency while preserving the TXT... transition
• use non-silent ASR warmup audio so service startup avoids the slow pure-silence path
• trim quiet leading/trailing edges before MLX transcription, preserving padding around speech

Benchmark Results

Area	Benchmark	Before	After	Change
TXT warm generated cases	median wall time	1524 ms	1256 ms	-17.6%
TXT warm generated cases	mean wall time	1903 ms	1246 ms	-34.5%
TXT warm generated cases	worst wall time	2809 ms	1480 ms	-47.3%
macOS stop wrapper	median runtime	331 ms	115 ms	-65.4%
padded generated speech	median wall time	~1393 ms	~1322 ms	-5.1%
cold ASR warmup probe	one-shot warmup	~10.0 s	~8.8 s	-12.1%

Notes:

• Rejected timestamp-free Whisper decode after benchmarking because it worsened to about 2980 ms median.
• Rejected direct model.generate() and manual sample_len caps because they were unstable or slower on this MLX build.
• Final warm TXT benchmark used 3 reps across generated silence, tone, and chirp cases.

Validation

• uv run python -m tests (360 tests OK)
• uv run ruff check transclip/asr.py transclip/service/engine.py transclip/desktop/hotkey/macos.py tests/test_asr.py tests/test_service.py tests/test_macos_hotkey.py
• uv run scripts/bench_mlx_txt_latency.py --reps 3 --label final-speedups
• uv run scripts/bench_macos_hotkey_latency.py --repetitions 50 --warmups 5 --scenario start
• uv run scripts/bench_macos_hotkey_latency.py --repetitions 50 --warmups 5 --scenario stop

Local Install Smoke Test

Regenerated ~/bin/transclip-toggle, restarted the TransClip service, kicked the macOS hotkey LaunchAgent, and confirmed local transclip status reports ready.

[murmer-blip]

Update from the latest latency pass:

• Added MLX background warmup for 28s, 32s, ..., 60s audio buckets after service readiness.
• Kept 4s-24s MLX warmup synchronous before readiness.
• Fixed MLX sample_len to use the padded bucket duration; this was the reason a 27.3s clip padded to 28s still compiled a fresh decode shape.
• Switched the first MLX startup warmup to prefer a short macOS spoken warmup clip (say + afconvert) when available, with chirp fallback.
• Added a final short spoken 4s rewarm after the long background buckets, because the long bucket pass can cool/evict the common short dictation path.
• Background warmup now yields while dictation status is recording, stopping, or transcribing between buckets.

Checks run locally:

• uv run ruff check transclip/asr.py transclip/service/engine.py tests/test_asr.py tests/test_service.py
• uv run python -m tests -> 369 tests OK
• Restarted local LaunchAgent from this checkout and verified /health ready.

Live macOS benchmark notes from this machine:

• Final restart readiness: 13.11s.
• Post-background final probe, Testing 1, 2, 3. TTS WAV (1.603s): first run 4.05s wall / 3.94s ASR, repeat 1.80s wall / 1.71s ASR.
• Post-background long synthetic probes: 27.3s -> 1.51s wall / 1.50s ASR; 59.3s -> 3.20s wall / 3.19s ASR.

One observed tradeoff: the background warmup is cooperative between buckets, but an individual MLX generation cannot be interrupted mid-bucket. It runs after readiness and restores the short speech bucket last to protect the common short-dictation path.

[murmer-blip]

Follow-up update for the common short-dictation path:

• Changed MLX audio padding to use 1s buckets for short clips up through 12s.
  • Example: 2.5s now pads to 3s instead of 4s.
  • Example: 11.3s pads to 12s.
  • Above 12s, MLX keeps 4s long buckets, so 12.3s pads to 16s.
• Startup warmup now covers the 1s, 2s, ..., 12s MLX buckets before readiness.
• The long background warmup now starts at 16s and continues 20s, 24s, ..., 60s.
• Startup and post-background warmup both run the short spoken Testing 1, 2, 3. warmup after the numeric buckets, so it does not replace the true 1s bucket.

Checks run locally:

• uv run ruff check transclip/asr.py transclip/service/engine.py tests/test_asr.py tests/test_service.py
• uv run python -m tests -> 371 tests OK
• Restarted local LaunchAgent from this checkout and verified /health ready.

Live macOS benchmark from this machine after background warmup completed:

• Restart readiness with 1-12s startup warmup: 25.21s on this run.
• Exact Testing 1, 2, 3. TTS WAV (1.603s):
  • run 1: 2.00s wall / 1.89s ASR
  • run 2: 1.70s wall / 1.69s ASR
  • run 3: 1.52s wall / 1.51s ASR
• Synthetic chirp probes after warmup:
  • 2.5s -> 1.79s wall
  • 5.3s -> 1.49s wall
  • 11.3s -> 1.80s wall
  • 12.3s -> 1.48s wall
  • 15.3s -> 1.73s wall

Tradeoff: startup readiness got heavier than the previous 4s-bucket-only warmup, but the common short path no longer pays the 4s minimum bucket or the first-use spoken-shape spike.

[murmer-blip]

Updated with a real-audio latency fix after Matthew's latest hotkey trace disproved the synthetic benchmark.

Finding:

• Latest real toggle at 2026-06-13 15:23 local recorded ~2.2s but spent 27.6s in MLX ASR (generate_write 27.4s).
• Replaying captured microphone WAVs through the same HTTP /transcribe path reproduced the issue: a 2.5s mic clip took 42.2s ASR and a 3.86s mic clip took 15.4s ASR.
• The earlier generated/synthetic benchmarks were therefore not representative.

Fix in 6659e8e:

• For MLX Whisper interactive dictation, pass return_timestamps=false and condition_on_previous_text=false.
• TransClip only needs plain transcript text for paste, and each toggle recording is independent, so the timestamp/previous-context decode path was unnecessary.

Benchmarks on the same Mac, same daemon/HTTP route, same captured mic files:

• Before: 2.5s mic clip -> 42.2s ASR; after: 2.5s mic clip -> 2.5s ASR.
• Before: 3.86s mic clip -> 15.4s ASR; after: 3.86s mic clip -> 2.9s ASR.
• Additional replayed mic clips after patch: ~3.0s and ~3.6s first-pass; repeat runs varied ~1.6-4.7s.

Verification:

• uv run -m unittest tests.test_asr -v passes.
• uv run python -m tests passes: 371 tests.

Caveat:

• Latency is now in the intended range for the failing real-audio cases, but still has some variance. If another real hotkey attempt is slow, debug capture should now give us the exact WAV/timing to compare.

[murmer-blip]

Follow-up fix pushed in 006b76d after another real captured-audio probe exposed a second issue.

Second finding:

• After the first MLX decode-options fix, short real-audio probes were ~2.5-3s immediately after readiness.
• But after the long 16-60s background bucket warmup completed, the same 2.5s captured mic clip got slow again (~40s ASR). That means long-bucket warmup can evict or disturb the short real-audio compiled path.

Follow-up fix:

• When debug_capture=true, the MLX background warmup now finishes by selecting recent local debug-captures/*/audio.wav clips, one per short bucket, and rewarming those real microphone-like buckets.
• This is gated on debug capture so normal installs do not unexpectedly replay old captured audio; Matthew's local machine has debug enabled while we tune this.

Verification:

• Added regression coverage: test_background_mlx_warmup_rewarms_recent_debug_capture_buckets.
• uv run -m unittest tests.test_service tests.test_asr -v passes.
• uv run python -m tests passes: 372 tests.

[murmer-blip]

Correction after checking Matthew's latest real shortcut use: the previous fix was not sufficient.

Latest live use:

• At 2026-06-13 15:50:52 +0700, a 3.1s recording still took 23.9s in ASR (generate_write 23.8s), with debug capture at debug-captures/20260613T085116Z.

What reproduced the failure:

• Replaying that exact captured WAV through the daemon was slow on first pass: 19.0s, then 3.9s, then 2.0s.
• Running an in-process engine with startup warmup only, and no post-ready MLX background warmup, transcribed the same WAV in 1.4-1.7s across three runs.

Root cause update:

• The aggressive MLX post-ready background warmup for 16-60s buckets was counterproductive for the real use case. It churned/poisoned MLX compiled state so the next short microphone dictation could still pay a 20s+ first-pass cost.

Fix in 2ca2f12:

• MLX now keeps startup warmup for 1-12s short dictation buckets.
• MLX skips post-ready background bucket warming entirely.
• Granite/NAR background shape warmup is unchanged.

Verification:

• uv run -m unittest tests.test_service tests.test_asr -v passes.
• uv run python -m tests passes: 370 tests.
• After restarting the real macOS daemon with this commit, replaying the latest slow capture through HTTP was 2.2s, 1.5s, 2.0s instead of 19-24s.