X2-Turn

Frame-synchronous streaming ASR and turn-state prediction

English | Chinese

Overview

X2 Turn transcribes speech and, every 80 ms, predicts a turn-taking state: idle, noidle, speaking, turn_end, backchannel, or uncertain.

The fastest way to try it is the hosted online demo (no local GPU).

The in-repo browser Turn Demo needs a GPU and the 4B weights. It does not need an LLM, TTS, or vLLM.

turn_demo_v2.mp4

🔥 News

• [2026-09-17] We add a hosted online demo on Hugging Face Spaces. Thanks to @multimodalart.
• [2026-09-05] We update the full-duplex dialogue demo for faster, smoother interactions, with pause handling support.
• [2026-08-28] We release X2-ASR-4B-0812, the Stage 1 bilingual streaming ASR backbone (no turn head).
• [2026-08-20] We release the paper, code, X2-Turn-4B-0812, and the Turn Demo.

Quick start

To try the model without installing anything, open the online demo.

To run the same Turn Demo on your machine, follow this path only. Save vLLM, the full-duplex stack, and the Python API until the demo has produced the expected result below.

You need

• Linux with an NVIDIA GPU and at least 24 GB of VRAM
• Miniforge or Conda, and Git
• Network access to Hugging Face

The first run is slow. Creating the environment, downloading the 4B weights, and the first Run scenario click each take several minutes. The web page can appear before the model is on the GPU. That is expected.

1. Create the environment

git clone https://github.com/X-Square-Robot/X2-Turn.git
cd X2-Turn

conda env create -f environments/environment-transformers.yml
conda activate x2-turn

These packages are not on PyPI. The environment file installs them from this checkout.

If you already have a CUDA PyTorch environment and prefer pip:

python -m pip install -e "./voxtral-realtime[transformers]"
python -m pip install -e "./turn-demo"

2. Download the model

The demo can fetch x-square-robot/X2-Turn-4B-0812 on first use. If Hugging Face hangs, times out, or cannot reach the Hub, download the weights once:

# from the X2-Turn repository root, with x2-turn activated
huggingface-cli download x-square-robot/X2-Turn-4B-0812 \
  --local-dir ./models/X2-Turn-4B-0812

Then point MODEL at that folder in the next step.

3. Start the demo

cd turn-demo
MODEL=x-square-robot/X2-Turn-4B-0812 bash run.sh

If you used the local download:

cd turn-demo
MODEL="$PWD/../models/X2-Turn-4B-0812" bash run.sh

Wait until the log shows Uvicorn running on http://127.0.0.1:7860. The server listens on localhost and does not load the 4B weights yet.

4. Open the page and run the built-in sample

Open http://localhost:7860.

1. Choose [built-in] English question.
2. Click Run scenario.

You do not need a microphone. The first click loads the model onto the GPU and can take several minutes. Transformers may print attention_mask or pad_token warnings. Those are harmless.

5. Expected result

The bundled clip is about 3.4 seconds of synthetic English. A successful run looks like this:

Field	Typical value
ASR text	hello can you tell me what the weather is like today
Frames	about 53 frames of 80 ms
Histogram	idle 32, noidle 4, speaking 15, turn_end 2
Timeline	speech, then turn_end, then idle

The prompt text on the page is Hello, could you tell me what the weather is like today? The ASR line above is the model output, not a copy of that prompt. Counts can shift by a frame or two across GPUs and library versions. If you see a transcript close to that sentence and a turn_end near the end of the utterance, the install worked.

After the demo works

Python API (local Transformers)

No server and no vLLM. From the repository root, with x2-turn activated:

import torch
from transformers import AutoProcessor

from voxtral_realtime.transformers import (
    infer_asr_turn,
    load_mtp_checkpoint,
)

model_id = "x-square-robot/X2-Turn-4B-0812"  # or ./models/X2-Turn-4B-0812
processor = AutoProcessor.from_pretrained(model_id)
model = load_mtp_checkpoint(
    model_id,
    device="cuda",
    dtype=torch.bfloat16,
).eval()

result = infer_asr_turn(model, processor, "turn-demo/assets/sample_en.wav")

print("ASR:", result.transcript)
for frame in result.turn_frames:
    print(frame.start_ms, frame.end_ms, frame.label, frame.confidence)

Write the same result to JSON:

python voxtral-realtime/integrations/transformers/examples/offline_inference.py \
  --model x-square-robot/X2-Turn-4B-0812 \
  --audio turn-demo/assets/sample_en.wav \
  --output offline_frames.json

The loader does not patch Transformers and does not need trust_remote_code. Details: voxtral-realtime/integrations/transformers/README.md. The bundled sample's text, license, and FFmpeg command are in turn-demo/assets/README.md.

Full-duplex dialogue demo

This stack adds a reply LLM and local Qwen3TTS-Streaming, and shows barge-in during playback. It is a separate setup: patched vLLM, the dialogue app, and a local Qwen3TTS-Streaming engine. Start from full-duplex-demo/README.md.

dialogue_demo_v4.mp4

Realtime serving with vLLM

Stock vLLM does not emit the custom turn.delta events. Follow the vLLM integration guide from the voxtral-realtime/ directory. To replay a WAV through the production turn controller, use voxtral-realtime/examples/offline_inference.py after that runtime is up.

Local services bind to 127.0.0.1 by default. Set BIND_HOST=0.0.0.0 only when another machine must connect.

Repository layout

• voxtral-realtime/ — model wrapper, local ASR + turn inference, the realtime controller, and the patched vLLM integration.
• turn-demo/ — browser demo for raw ASR, 80 ms Turn states, and the frame-level token / class / probability table.
• full-duplex-demo/ — full conversational stack with a reply LLM and local Qwen3TTS-Streaming.
• environments/ — separate Miniforge environments so Transformers, patched vLLM, and the dialogue app do not share one CUDA/Torch tree.

Release boundary

Each component keeps its own license and notice. Model weights live on Hugging Face, not in this source tree.

Do not publish local logs, certificates, datasets, external source checkouts, or credentials.

Citation

If you find X2-Turn useful in your research, please cite:

@article{fu2026x2turn,
  title = {X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction},
  author = {Fu, Kaiqi and Wen, Rime and Lin, Altman and Qin, Shawn and Gan, Roy and Wang, Hao and Wang, Qian},
  journal = {arXiv preprint arXiv:2608.10878},
  year = {2026},
}

Acknowledgments

X2 Turn builds on ideas, models, and infrastructure from the open-source speech and machine-learning community. We thank:

• Mistral AI for Voxtral Mini 4B Realtime, which provides the realtime speech backbone.
• SoulX-Duplug for its semantic turn-taking work and the dialogue-system foundation adapted by the full-duplex demo.
• vLLM for the high-throughput serving runtime extended by the X2 Turn realtime overlay.
• Hugging Face Transformers for model loading, processing, and the local inference ecosystem.
• Qwen3TTS-Streaming for the local streaming TTS engine used by the full-duplex demo.

See the component NOTICE files and the available THIRD_PARTY_NOTICES.md documents for detailed attribution and license information.