Skip to content

Choose the speech model's language, with Automatic the default - #66

Merged
purinda merged 2 commits into
mainfrom
speech-language
Oct 17, 2026
Merged

purinda merged 2 commits into
mainfrom
speech-language

Conversation

@purinda

@purinda purinda commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

This lets you set the language the speech model transcribes in, e.g. English only, so a multilingual model doesn't drift into another language now and then. It's built on #65, which has merged.

Where

  • The Language menu in the menu bar (Mac) and the tray (Linux, Windows). It sits after Cleanup and lists the languages for the model you're using. A model that can't be told a language shows Automatic and nothing else, greyed out.
  • The model's row in Settings › Models, in the same spot as Cohere's language picker.

Both menus write the same sttLanguage setting. Automatic (no setting) is the default for every model. The setting is shared across models, so if you switch to a model that doesn't have the chosen language, it shows as Automatic, and the choice comes back when you switch to a model that has it.

What each model offers

Model Languages Automatic
Qwen3-ASR 0.6B, 1.7B 30 (from the model's support_languages) the model finds the language
Qwen3-ASR 0.6B Sinhala 31 (the 30 plus Sinhala) the model finds the language
Whisper large-v3 turbo 100 the model finds the language
Cohere Transcribe 14 English (Cohere can't find the language itself, so the menu shows "Automatic (English)")
Parakeet TDT v2, v3 none (can't be told one) the model finds the language

English comes first in each list, then the rest by name.

Qwen3-ASR is kept to the language's letters

Testing showed telling Qwen3-ASR a language only replaces its own guess at it: with Sinhala chosen, English speech came out in English, and with English chosen, Sinhala came out in Sinhala. So when a language is chosen, Qwen3-ASR may now write only that language's letters, plus Latin letters for English words (as the Sinhala model writes them), and the digits, punctuation and spaces every language shares.

  • Each Qwen language in speech-models.json names its script (ISO 15924: Latn, Sinh, Hani…).
  • At each step the model picks the likeliest token whose bytes keep to those letters. A character split across tokens is tracked, and the reply can only end after a whole character.
  • Mac: Qwen3-ASR is decoded in Qwen3ASR+ScriptLock.swift, on mlx-audio-swift's public model API, so there's no fork. Other models still use mlx-audio-swift's generate.
  • Linux/Windows: the greedy decoder takes the same lock (script_lock.rs, qwen3_asr/locked_tokens.rs).
  • Automatic never locks.

What it does (Mac Bench, default Sinhala Qwen model):

  • On the 65 English eval clips, English locked gives exactly the same output as Automatic (WER vs said 12.8% both; p50 140 vs 158 ms).
  • With English locked, a Chinese clip comes out as an English translation ("We'll meet you at 10:00 in the office.").
  • It can't tell English from German, since both use Latin letters.
  • With Sinhala chosen, other languages come out as Sinhala-script gibberish that can run to the output limit. This happened before the lock too; the loop guard misses multi-token Sinhala words.

How the language is told

  • Qwen3-ASR is told the language by name, by adding language English<asr_text> after the assistant turn starts, as mlx-audio-swift's buildPromptText does.
  • Whisper and Cohere are told the code, as Cohere was before.
  • --language automatic clears the setting for a single run on Linux and Windows.

Not tested yet

  • Whisper's 100 codes haven't been checked against a real Whisper tokenizer.
  • The Linux/Windows tray, settings window and dictation code is behind cfg(linux|windows), so CI is the first place it compiles.
  • The Linux OpenVINO lock is covered by unit tests only; I haven't run it on a real model.

Checks

  • make test: everything passes. The two wall-clock cleanup timing tests failed once under load and passed on a rerun. The Mac app builds.
  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass on macOS.

🤖 Generated with Claude Code

purinda and others added 2 commits October 14, 2026 23:32
A speech model that can be told which language to write now offers its languages in two places:
the menu bar's or tray's Language menu, and the Language menu on its row in Settings › Models.
Both list Automatic first, then only the languages that model takes: Qwen3-ASR's 30 (31 with
Sinhala in the default model), Whisper's 100 on the Mac, and Cohere Transcribe's 14. Parakeet
can't be told one, so its menu is greyed out with Automatic alone.

Automatic, the default for every model, tells the model nothing and it works the language out,
as before. Cohere Transcribe can't, so the catalog's automatic_language gives it English, as it
had. Choosing English stops a short or noisy recording coming out in another language. The one
Language setting holds the choice for every model; a model without that language uses Automatic.

Qwen3-ASR is told the language as mlx-audio-swift's prompt does ("language English<asr_text>"
after the assistant's turn), by the name its config lists, and Whisper and Cohere by the code.
The Linux and Windows Qwen3-ASR runtime now builds that prompt too, so it reads the model's merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Telling Qwen3-ASR a language only stands in for its own guess: clear speech
in another language still came out in that language. Now, with a language
chosen, Qwen3-ASR may write only that language's letters, plus Latin letters
for English words, and the digits, punctuation and spaces every language
shares. Told English, a Chinese clip comes out translated into English, and
the 65 English eval clips come out exactly as before.

Each Qwen language in the catalog names its script (ISO 15924). On the Mac,
Qwen3-ASR is decoded here, on mlx-audio-swift's public model API, with a mask
on the logits before each greedy choice. On Linux and Windows, the greedy
decoder takes the same lock. Automatic never locks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@purinda
purinda merged commit 322ef21 into main Oct 17, 2026
2 checks passed
@purinda
purinda deleted the speech-language branch October 17, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant