Add MOSS-VoiceGenerator (voice design from a text instruction)

[jrohde]

What

MOSS-VoiceGenerator creates a speaker from a written description instead of a reference WAV — "a warm male radio voice in his fifties, calm, never shrill" — and speaks the supplied text in it. Apache-2.0 weights, ~2B params, Chinese and English.

Would you welcome a port? And is anyone already working on it?

Why it fits here

• It slots into the existing vdes task with --instruct, the same shape Qwen3-TTS VoiceDesign already uses. No new task type needed.
• It uses the MOSS audio tokenizer that src/models/moss/shared/ already implements — same tokenizer as moss_tts_local, the model just takes the first 16 of its 32 RVQ layers.
• It fills a gap: voice design currently means Qwen3-TTS or OmniVoice, and neither is a MOSS-family option.

What is actually new

VoiceGenerator is the moss_tts_delay architecture, not the local-transformer path — so instead of moss_tts_local's depth transformer it needs a delay-pattern frame decoder plus that family's prompt template. That is the bulk of the work; the codec, sampling helpers and token-row utilities under moss/shared/ should carry over.

What I can bring

I implemented this family in pwilkin/openmoss (another Apache-2.0 C++/ggml port), where I contributed multi-speaker cloning, the continuation prefix, a voice registry and converter fixes. Some findings that cost me time and would otherwise be rediscovered:

• It is genuinely sensitive to decoding hyperparameters. The model card documents 1.5 / 0.6 / 50 / 1.1 and warns about it; at the flagship's defaults it degenerates into an immediate end-of-speech.
• It needs explicit full language names ("English", not "en").
• It has no reference audio to anchor length, so left unbounded it either collapses on the first frame or rambles for minutes. Deriving min/max frame bounds from the text length fixes it.
• bf16 for the backbone — f16 overflows its attention sinks.

Happy to start under community_models and follow the reproducibility bar in docs/community_models/models.md (build/run commands, WAVs, backend coverage, timings, VRAM). I test on Intel Arc / Vulkan, so I can cover that backend; I would need help confirming CUDA numbers.

If this lands well, MOSS-TTSD (multi-speaker dialogue, same delay family) and MOSS-SoundEffect-v2.0 (DiT + flow matching + DAC VAE, close to what ace_step/ already does) are the natural follow-ups.

[0xShug0]

@jrohde Thanks for the proposal! That sounds exciting! I'd really appreciate the port because it would add new reusable primitives to audio.cpp. No one is currently working on other MOSS models as far as I know.

Starting under community_models sounds great. It probably makes sense to implement the model(s) locally first, and then we can discuss how to promote reusable components into framework modules/templates (or extending existing ones) once the shape is stable.

[jrohde]

Great — thanks. We're going to get to work on it.

I'll start MOSS-VoiceGenerator under community_models, local-first the way you described, and open a draft PR once the first end-to-end WAV is out so you can steer the shape early rather than review a finished wall of code. MOSS-TTSD is the natural follow-up.

On backends: I can test Vulkan, ROCm/HIP on Radeon, and CPU — basically everything except CUDA. For CUDA numbers I'd need someone with the hardware.

One practical question: release packages resolve to audio-cpp/audio.cpp-gguf. Should I hand converted GGUFs over to you for upload, or is there a contributor path into that repo?

No timeline promises — this is evenings-and-weekends work — but it's underway.

[0xShug0]

@jrohde Sounds great, and thank you so much! I’ll test CUDA. For GGUF, you can host the files on your own HF repo. Some community models already use their own HF repos.