Add MiniMax-Music3 community model

CMakeLists.txt

@@ -540,6 +540,23 @@ audiocpp_add_model(inflect_v2
         engine::models::inflect_v2::make_inflect_v2_loader
 )
 
+audiocpp_add_model(minimax_music3
+    SOURCES
+        src/community_models/minimax_music3/assets.cpp
+        src/community_models/minimax_music3/condition_encoder.cpp
+        src/community_models/minimax_music3/depth_decoder.cpp
+        src/community_models/minimax_music3/dit.cpp
+        src/community_models/minimax_music3/lm.cpp
+        src/community_models/minimax_music3/pipeline.cpp
+        src/community_models/minimax_music3/session.cpp
+        src/community_models/minimax_music3/tokenizer_text.cpp
+        src/community_models/minimax_music3/vocoder.cpp
+    INCLUDES
+        engine/community_models/minimax_music3/session.h
+    LOADERS
+        engine::models::minimax_music3::make_minimax_music3_loader
+)
+
 audiocpp_add_model(minimax_h3
     SOURCES
         src/community_models/minimax_h3/assets.cpp
@@ -1435,6 +1452,11 @@ if (ENGINE_ENABLE_OPENMP)
     target_link_libraries(audiocpp_server PRIVATE OpenMP::OpenMP_CXX)
 endif()
 
+add_executable(ggml-quantize-raw
+    tools/ggml_quantize_raw.c
+)
+target_link_libraries(ggml-quantize-raw PRIVATE ggml)
+
 add_executable(audiocpp_gguf
     app/gguf/main.cpp
 )
@@ -1532,6 +1554,7 @@ if (ENGINE_BUILD_WARMBENCH)
     endfunction()
 
     add_engine_warmbench(campplus_shared_default_probe tests/glm_tts/campplus_shared_default_probe.cpp)
+    add_engine_warmbench(minimax_music3_component_probe tests/minimax_music3/minimax_music3_component_probe.cpp)
     add_engine_warmbench(chatterbox_warm_bench tests/chatterbox/chatterbox_warm_bench.cpp)
     add_engine_warmbench(citrinet_asr_warm_bench tests/citrinet_asr/citrinet_asr_warm_bench.cpp)
     add_engine_warmbench(confucius4_tts_warm_bench tests/confucius4_tts/confucius4_tts_warm_bench.cpp)

docs/community_models/minimax_music3.md

@@ -0,0 +1,152 @@
+# MiniMax-Music3
+
+MiniMax-Music3 generates full songs (vocals plus arrangement, 44.1 kHz stereo, up to six
+minutes) from a music description caption and lyrics. The port follows the diffusers
+reference pipeline: a Qwen3-8B autoregressive stage emits one semantic code per 40 ms
+frame with classifier-free guidance, a 4-layer RVQ depth decoder fills seven residual
+codebooks per frame, the fused per-frame hidden states condition a 36-layer flow-matching
+transformer over 200-frame windows, and a DAC-style Flow-VAE decoder renders the latents
+to stereo audio.
+
+Upstream weights: [MiniMaxAI/MiniMax-Music3](https://huggingface.co/MiniMaxAI/MiniMax-Music3).
+Reference implementation: `MiniMaxMusic3ModularPipeline` in diffusers (0.40.0.dev0 or newer).
+
+## Package Layout
+
+The package directory must contain the files referenced by `model_specs/minimax_music3.json`:
+
+```text
+MiniMax-Music3-GGUF/
+  lm_q8_0.gguf                  global Qwen3-8B, Q8_0, lm_head sliced to the sampleable rows
+  lm_q4_k.gguf                  optional Q4_K variant of the global LM
+  depth_decoder_f16.gguf
+  dit_f16.gguf                  flow-matching transformer
+  condition_encoder_f32.gguf
+  vocoder_f16.gguf              Flow-VAE decoder, torch weight norm folded
+  tokenizer/tokenizer.json
+  tokenizer/tokenizer_config.json
+```
+
+The DiT and depth decoder are stored F16 rather than the checkpoint's BF16: the CUDA
+BF16 matmul path is several times slower than F16 on Ampere, the weights fit F16's range
+with a wide margin (max magnitude about 3.2), and component parity is equal or better.
+
+`--model` takes the `lm_*.gguf` entry file; the runtime resolves the other component
+files from its parent directory.
+
+## Conversion
+
+`scripts/minimax_music3/convert_gguf.py` converts the HF snapshot per component:
+
+```bash
+hf download MiniMaxAI/MiniMax-Music3 --local-dir models/MiniMax-Music3-hf \
+  --exclude "qwen_7B/qwen_7B/*.safetensors"
+
+python scripts/minimax_music3/convert_gguf.py --component lm \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/lm_q8_0.gguf --type q8_0 \
+  --override "lm_head_sliced.weight=bf16"
+python scripts/minimax_music3/convert_gguf.py --component depth_decoder \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/depth_decoder_f16.gguf --type f16 \
+  --override "*norm*=f32" --override "pos_embedding*=f32"
+python scripts/minimax_music3/convert_gguf.py --component dit \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/dit_f16.gguf --type f16 \
+  --override "*norm*=f32" --override "*bias*=f32" --override "time_proj.weight=f32"
+python scripts/minimax_music3/convert_gguf.py --component condition_encoder \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/condition_encoder_f32.gguf --type f32
+python scripts/minimax_music3/convert_gguf.py --component vocoder \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/vocoder_f16.gguf --type f16 \
+  --override "*.alpha=f32" --override "*.bias=f32"
+cp models/MiniMax-Music3-hf/tokenizer/tokenizer.json \
+   models/MiniMax-Music3-hf/tokenizer/tokenizer_config.json \
+   models/MiniMax-Music3-GGUF/tokenizer/
+```
+
+The LM conversion slices the 200k-row lm_head to the 16385 rows the sampler can ever
+pick (the audio end token plus the 16384 semantic codes); the vocoder conversion folds
+torch `weight_g`/`weight_v` weight-norm pairs into plain conv weights. The `qwen_7B/`
+safetensors in the upstream repo are an alternative packaging of the same LM and are
+not needed. The optional Q4_K LM variant needs the `ggml-quantize-raw` build target
+(gguf-py cannot produce K-quants):
+
+```bash
+cmake --build build/linux-cuda-release --target ggml-quantize-raw
+python scripts/minimax_music3/convert_gguf.py --component lm \
+  --snapshot models/MiniMax-Music3-hf \
+  --output models/MiniMax-Music3-GGUF/lm_q4_k.gguf --type q4_k \
+  --override "lm_head_sliced.weight=bf16" --override "model.embed_tokens.weight=q8_0"
+```
+
+Pass `--model .../lm_q4_k.gguf` to select it; the other components resolve from the
+package directory either way.
+
+## Run
+
+```bash
+build/linux-cuda-release/bin/audiocpp_cli \
+  --task gen \
+  --family minimax_music3 \
+  --model models/MiniMax-Music3-GGUF/lm_q8_0.gguf \
+  --backend cuda \
+  --threads 8 \
+  --text "$CAPTION" \
+  --request-option lyrics="$LYRICS" \
+  --request-option duration_seconds=60 \
+  --seed 42 \
+  --metrics \
+  --out song.wav
+```
+
+`--text` carries the music description caption (genre, mood, vocals, instrumentation,
+arrangement). `lyrics` carries the lyrics; structure tags such as `[verse]` or `[chorus]`
+must each be on their own line. Options: `duration_seconds` (upper bound in seconds, the
+model may stop earlier, maximum 360), `num_inference_steps` (flow Euler steps per window,
+default 30), `guidance_scale` (flow CFG, default 1.7), `seed`. The autoregressive stage's
+sampling recipe (CFG 1.5, top-50) is fixed by the checkpoint contract.
+
+## Validation
+
+Component parity against the diffusers reference (`tests/minimax_music3/reference_dump.py`
+generates fixtures, `tests/minimax_music3/minimax_music3_component_probe.cpp` runs the
+same component in isolation; build the probe with `-DENGINE_BUILD_WARMBENCH=ON`):
+
+| Component | Result |
+|---|---|
+| Tokenizer (prompt template, caption cleaning, lyrics normalization) | exact id match |
+| LM prefill (both CFG branches) | corr 0.9998 Q8_0 / 0.993 Q4_K, argmax match |
+| RVQ depth decoder (argmax rollout) | all 8 codes exact, hidden max diff 2.1e-3 (f16) |
+| Condition encoder | max diff 5.7e-5 |
+| Flow transformer forward | corr 0.99998, max diff 5.2e-2 (f16 flash attention) |
+| Vocoder | about 48 dB SNR (f16) |
+
+## Performance
+
+RTX 3090, CUDA, 32 s of audio at 30 flow steps:
+
+| Configuration | AR | Flow | Vocode | Wall | RTF |
+|---|---|---|---|---|---|
+| Q8_0 LM, BF16 DiT (first pass) | 39.1 s | 84.5 s | 2.0 s | 130.6 s | 4.08 |
+| Q8_0 LM, F16 DiT, flash attention | 39.1 s | 36.9 s | 1.9 s | 82.1 s | 2.57 |
+| + batched CFG decode, on-device depth sampling | 23.9 s | 37.0 s | 2.0 s | 66.6 s | 2.08 |
+| same with Q4_K LM | 22.0 s | 37.3 s | 2.0 s | 64.5 s | 2.02 |
+
+The flow transformer runs flash attention with F16 weights and activations (norms and
+residuals stay F32); the BF16-to-F16 storage switch alone is a 3.9x DiT forward speedup
+on Ampere. The autoregressive stage batches the conditional and unconditional CFG
+branches into one decode graph (weights stream once per frame; the two sequences always
+share positions, so one KV write slot and mask serve both), and the seven depth-decoder
+codebook steps run as a single unrolled graph with on-device sampling: classifier-free
+guidance, a top-k mask built from `ggml_top_k`, host-supplied Gumbel noise, and an
+argmax, which draws exactly from the reference's renormalized top-k distribution. Both
+stages now sit close to their weight-bandwidth floors (`minimax_music3.ar_lm_decode_ms`
+and `minimax_music3.ar_depth_ms` timing logs give the split). Remaining headroom is
+architectural: pipelining flow-matching windows onto a second GPU while the
+autoregressive stage streams frames, and bucketed KV-cache views for very long songs.
+
+VRAM peaks around 14 GB during the autoregressive phase (Q8_0 LM, two KV states, depth
+decoder) and around 8 GB during the flow phase; `mem_saver` (default on) loads each
+phase's weights on demand and frees them afterwards.

docs/community_models/models.md

@@ -20,6 +20,7 @@ Practical expectations:
 | **inflect_v2** | TTS | en | Community | [Inflect Micro v2 and Nano v2](inflect_v2.md) native FP32 offline synthesis |
 | **kroko_asr** | ASR | de, en, es, fr, it, he, nl, pt, sv, tr | Mirek [@mirek190](https://github.com/mirek190) | [Kroko Community ASR](kroko_asr.md) native offline/streaming Zipformer2/RNN-T transcription with word timestamps |
 | **minimax_h3** | Video, Music, TTS/Dialogue | auto | [@0xShug0](https://github.com/0xShug0) | [MiniMax-H3](minimax_h3.md) text-to-audio/video generation with Q4_K and optional INT8 ConvRot DiT |
+| **minimax_music3** | Music (vocals, lyrics) | auto | Joe Mattie | [MiniMax-Music3](minimax_music3.md) lyrics- and caption-conditioned song generation: Qwen3-8B AR codes, RVQ depth decoding, flow matching, and Flow-VAE decode to 44.1 kHz stereo |
 | **moss_tts_local** | TTS, voice cloning | auto, optional language hint | [@justinjohn0306](https://github.com/justinjohn0306) | [MOSS-TTS-Local Transformer v1.5](../models/moss_tts.md) support in the core model tree |
 | **outetts** | TTS, voice cloning | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | Mirek [@mirek190](https://github.com/mirek190) | [Llama-OuteTTS-1.0-1B](outetts.md) TTS and voice cloning support |
 | **parakeet_tdt** | ASR | auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk | [@dleiferives](https://github.com/dleiferives) | [Parakeet-TDT 0.6B v3](parakeet_tdt.md) offline, long-form, and buffered-streaming ASR support |

docs/proposals/minimax_music3.md

@@ -0,0 +1,142 @@
+# MiniMax-Music3 port design
+
+Status: implemented; see docs/community_models/minimax_music3.md for the user-facing
+documentation and validation results.
+
+MiniMax-Music3 ([MiniMaxAI/MiniMax-Music3](https://huggingface.co/MiniMaxAI/MiniMax-Music3))
+generates full songs (vocals plus arrangement, up to six minutes, 44.1 kHz stereo) from a
+music description caption and lyrics. The reference implementation is the diffusers
+modular pipeline `MiniMaxMusic3ModularPipeline` (diffusers >= 0.40.0.dev0).
+
+## Architecture summary
+
+Five checkpoint components run in four stages:
+
+1. **Tokenize.** Qwen2 BPE tokenizer (`tokenizer/` subfolder). The prompt is a fixed
+   special-token template over the cleaned caption and normalized lyrics:
+   `<|im_start|><|caption_start|>C<|caption_end|><|lyrics_start|>[start]\nL<|lyrics_end|><|im_end|><|audio_start|>`.
+   Maximum 5000 prompt tokens. Classifier-free guidance uses a token-level pair: the
+   unconditional prompt is the conditional one with every id except the first and the
+   trailing two replaced by the audio-CFG token (id 151654).
+2. **Autoregressive stage, 25 frames/s.** The conditional and unconditional sequences run
+   as a batch of two through a Qwen3-8B causal LM (36 layers, hidden 4096, 32 query and
+   8 KV heads, head dim 128, ffn 12288, vocab 200000, untied lm_head). Per frame the LM
+   samples one semantic code out of 16384 (logits masked to the code range at offset
+   151675 plus the end token 151670, CFG scale 1.5 restricted to the conditional top-50,
+   then top-50 sampling). A 4-layer depth decoder (hidden 4096, 16 heads, ffn 6144,
+   SwiGLU, RMSNorm, learned positions, causal, no RoPE) then autoregressively samples the
+   seven residual codebooks (1024 entries each) with the same CFG and top-50 recipe. The
+   frame feedback embedding is `embed(semantic) + sum(residual embeds)` scaled by
+   `8^-0.5`. The stage's real output is not the codes but the per-frame hidden states:
+   `concat(LM last hidden, 7 depth-step hiddens)`, shape `[frames, 8 * 4096]`.
+3. **Flow matching over 200-frame windows** (100-frame hop). A small condition encoder
+   (softmax-weighted mix of the 8 hidden slots, 3-tap Conv1d 4096 to 2048, nearest
+   resample by 3.4453125) puts the window's hiddens on the Flow-VAE latent timeline
+   (44100 / 512 latents per second, 689 latents per full window). A 36-layer, 2048-wide
+   DiT (32 heads by 64, partial RoPE over the first 32 of 64 dims, LayerNorm, gated ff of
+   inner size 8192, Fourier time embedding prepended as one token; the input is
+   `concat(latent 128, zeros 128, condition 2048)` through a residual 1x1 conv) predicts
+   flow velocity. The scheduler reduces to plain uniform Euler: `t_k = k / N`,
+   `x += (1 / N) * v`, default N = 30, CFG scale 1.7 with all-zero conditioning as the
+   unconditional branch. Window overlap is handled by re-injecting
+   `(1 - (1 - 1e-6) t) * noise_prompt + t * previous_latent` over the first 172 latent
+   frames before every step, and by carrying latent frames `[L-344, L-172)` to the next
+   window.
+4. **Vocoder.** DAC-style decoder: the 128-channel latent folds to two 64-channel streams
+   (stereo), each runs in_proj, conv_in, four upsample blocks (strides 8, 8, 4, 2, snake
+   activations, weight-norm convs, dilated residual units), and a tanh output conv.
+   Total upsampling 512, so 44.1 kHz stereo. Waveform windows are stitched by dropping
+   86 leading latent frames (times 512 samples) on every window after the first and 258
+   trailing latent frames on every window before the last.
+
+All AR-stage constants (special token ids, code offset, CFG scales, top-k, frame rate,
+chunk sizes, overlap lengths) are checkpoint contract, fixed in the reference code rather
+than configs. We keep them as named constants in the family code.
+
+## Port plan
+
+Family `minimax_music3` under `src/community_models/minimax_music3/` and
+`include/engine/community_models/minimax_music3/`, registered with
+`audiocpp_add_model`, spec-backed loader, task `gen` (`tasks: ["music"]`), offline mode,
+CUDA-first (`runtime.tags: ["cuda", "gguf"]`).
+
+### Package layout (multi-file, minimax_h3 idiom)
+
+```text
+MiniMax-Music3-<precision>-GGUF/
+  lm_<p>.gguf                  global Qwen3-8B (Q4_K default, Q8_0 variant)
+  depth_decoder_<p>.gguf       RVQ depth decoder
+  dit_<p>.gguf                 flow-matching transformer
+  condition_encoder_f16.gguf   condition encoder
+  vocoder_f16.gguf             Flow-VAE decoder, weight norm folded
+  tokenizer/tokenizer.json
+  tokenizer/tokenizer_config.json
+```
+
+`model_specs/minimax_music3.json` (schema v1) maps these as `sources[].tensors` /
+`files` entries with `roots.model = "."`; `--model` takes the `lm_*.gguf` entry file and
+the parent directory is the package root, like minimax_h3's `dit.gguf` convention.
+Multi-file packages do not embed a spec, so `default_contract_spec_path` in
+`src/framework/model_spec/package.cpp` needs `minimax_music3` added to the same
+workspace/builtin fallback as `minimax_h3`.
+
+Converter: `scripts/minimax_music3/convert_gguf.py` (one script, `--component` selector),
+reading the diffusers-format safetensors. Component notes:
+
+- **lm**: lm_head is sliced to the 16385 rows that can ever be sampled (row 0 = end
+  token 151670, rows 1..16384 = semantic codes at offset 151675) and stored bf16; the
+  full embedding table stays (prompt tokens and code feedback need it). Norms f32.
+- **depth_decoder**: fused q/k/v kept separate as in the checkpoint; bf16 or Q8_0.
+- **dit**: bf16 default; `ff_in` is stored fused (gate and value in one matrix) and kept
+  that way. Attention out projections and time embedding stay bf16 in quantized variants.
+- **vocoder**: fold `weight_g`/`weight_v` pairs into plain conv weights at conversion
+  (torch `weight_norm` dim 0 convention; ConvTranspose1d normalizes over dims 1, 2),
+  store f16, following `scripts/minimax_h3/convert_fold_audio_vae_gguf.py`.
+
+Shapes are derived from GGUF tensor metadata at load (minimax_h3 house style); the only
+sidecars are the two tokenizer files.
+
+### Runtime components and reuse
+
+| Component | Implementation |
+|---|---|
+| Text tokenizer and prompt build | `tokenizers::LlamaBpeTokenizer` (`Qwen2` pre-type, `tokenizer.json` path), caption cleaning and lyrics normalization ported from the reference, special ids resolved with `find_token_id` |
+| Global LM | shared `modules::QwenCausalDecodeRuntime` (`use_qk_norm`, untied head, `output_mode` logits plus hidden), two KV states for the conditional and unconditional sequences; `decode_embedding` carries the frame feedback |
+| AR sampler | host-side: CFG on the 16385 sliced logits, conditional top-50 restriction, top-50 softmax multinomial with a seeded RNG (`engine::sampling` helpers) |
+| Depth decoder | hand-rolled small graph modeled on `qwen3_tts` `CodePredictorGraph` (4 layers, seq <= 9, batch 2, learned positions, no RoPE, no qk-norm) |
+| Condition encoder | tiny graph (weighted mix plus Conv1d k3) with host-side nearest resample, modeled on `ace_step`'s condition encoder runtime |
+| Flow DiT | dedicated graph (LayerNorm blocks, partial RoPE including the time token at position 0, fused gated ff); driven per window by `modules::FlowSamplerRuntime` (cond/uncond branches, CFG 1.7, Euler) or, if the per-step overlap injection does not fit its hooks, a host loop over `engine::sampling::diffusion_math` (`cfg_guidance`, `euler_step_in_place`) |
+| Vocoder | dedicated DAC-decoder graph (snake, ConvTranspose1d, dilated residual units); stereo via the two folded channel groups, `interleave_planar_channels`, output 44.1 kHz stereo `AudioBuffer` |
+| Session | `RuntimeSessionBase` + `IOfflineVoiceTaskSession`, spec-backed loader, `mem_saver` default on |
+
+### Memory plan (single 24 GB GPU)
+
+Sequential phases with `mem_saver`: the AR phase holds the LM (Q4_K about 5 GB) plus the
+depth decoder and two KV states (about 4 GB at 60 s); frame hiddens accumulate on the
+host (32 KB times frames times 4 bytes, about 200 MB per minute). The flow phase frees
+the LM and holds the DiT (bf16 9.7 GB) plus per-window activations. The vocoder phase is
+negligible. Peak stays under 16 GB, so bf16 DiT plus Q4_K LM fits one RTX 3090.
+
+### Request surface
+
+- `--text`: the music description caption (required).
+- `--request-option lyrics=...` or `lyrics_file=...`: the lyrics (required).
+- `audio_duration` (default 60 s, max 360 s), `num_inference_steps` (default 30),
+  `seed`, and the standard `gen` options.
+
+### Validation
+
+Parity seams against the diffusers reference (bf16, `readback_round_type` BF16):
+
+1. token ids of the assembled conditional and unconditional prompts,
+2. prefill last hidden state,
+3. first-frame guided logits with a pinned code sequence,
+4. `frame_hiddens` for a short forced-code rollout,
+5. condition encoder output for a fixed window,
+6. one DiT forward at fixed t and latents,
+7. vocoder waveform for a fixed latent,
+8. end-to-end generation listening check plus RTF and VRAM numbers for the
+   community-models table.
+
+Python dump scripts live in `tests/minimax_music3/` next to a warm bench, following the
+house pattern.