EMNLP 2026 (Main Conference)
Jingyu Lu1,2, Yuhan Wang1,2, Jianming Luo1, Yifu Chen1, Tianle Liang1, Shengpeng Ji3, Ziyue Jiang1, Xiaoda Yang1,
Yu Zhang4, Xize Cheng1, Chenyuhao Wen1, Changhao Pan1, Haoxiao Wang1, Chen Ye2, Jian Wu2,
Xiaoxi Jiang2, Guanjun Jiang2, Zhou Zhao1
1Zhejiang University · 2Qwen Business Unit, Alibaba · 3HunYuan Team, Tencent · 4ByteDance
📄 Paper | 🎧 Demo | 📌 Citation
This repository is the official repository for Speaking While Listening: A Survey and Empirical Audit of Full-Duplex Spoken Dialogue Systems, accepted to the EMNLP 2026 Main Conference. It is maintained by the MM-Speech group; the anonymous review-period URL (DuplexLM/DuplexSurvey) now redirects here.
- We organize full-duplex spoken dialogue systems with three lenses: the L0-L3 Architectural Hierarchy, the T x I x R Interaction Ontology, and the Full-Duplex Decision State Machine.
- We provide visual summaries for the system timeline, architectural hierarchy, interaction scenarios, state machine, and cross-system audit.
- We collect public full-duplex models, datasets, and benchmarks for follow-up research.
- 📦 [2026/08] This repository has moved to
MM-Speech/DuplexSurvey. Links to the old anonymous URL still redirect here. - 🏆 [2026/08] Our paper has been accepted to the EMNLP 2026 Main Conference!
- 📄 [2026/06] The preprint is available on arXiv:2606.19453.
- 🎉 [2026/06] We have officially released this survey repository for Full-Duplex Spoken Dialogue Systems!
More than a dozen recent spoken dialogue systems have claimed full-duplex capability, but the term is used for systems with substantially different behaviors. This project organizes the space with three complementary frameworks:
- L0-L3 Architectural Hierarchy: where the duplex decision is made, from external controllers to hidden states, token streams, and future shared latent representations.
- T x I x R Interaction Ontology: what interaction the system must handle, decomposed into temporal relation, user intent, and system response.
- Decision State Machine: what the system is doing moment by moment across Idle, Listen, Speak, Wait, and Dual states.
Together, these frameworks turn the vague question "is this system full-duplex?" into a concrete profile: which interaction cells are covered, which states are reachable, and where the decision lives in the model stack.
- Quick Start
- What's new
- Overview
- A Brief History of Full-Duplex Spoken Dialogue Systems
- Architectural Hierarchy and Cross-System Audit
- The T x I x R Interaction Ontology
- The Full-Duplex Decision State Machine
- Frontiers and Conclusion
- Appendix: Public Resources
- Citation
- Contributing
Full-duplex spoken dialogue systems have moved from modular pre-LLM pipelines, to speech-token LLMs, to GPT-4o-style live interaction. After LSLM and Moshi, the field quickly split across L0 modular control, L1 hidden-state prediction, L2 token-level synchronization, and the still-unrealized L3 representation-level design.
Figure 1: Timeline of published full-duplex spoken dialogue systems, 2021-2026, grouped by the L0-L3 architectural layer defined below.
This section reframes architecture around one diagnostic question: where does the system decide whether to listen, speak, wait, or handle overlap? Cascaded/end-to-end and engineered/learned labels are too coarse for this purpose. The L0-L3 hierarchy instead locates duplex decisions in an external controller, a hidden-state predictor, the token sequence, or a shared latent representation.
The hierarchy is not a progress ladder. Our audit shows that L0 remains competitive, L1 has become an industrial attractor, L2 is the largest and most Dual-capable family but remains internally heterogeneous, and L3 is still open.
Figure 2: The L0-L3 architectural hierarchy. The decision migrates from an external scheduler (L0), to a sidecar predictor reading LLM hidden states (L1), to the token stream itself (L2), and finally to a hypothetical shared latent representation (L3).
Main lesson: architecture gives reachability; data and training determine realization. Two L2 systems may both reach Dual, yet still diverge on T3-I2-R1 backchanneling.
L0-L3 asks where the duplex decision lives. T x I x R asks what interaction that decision must handle. "Overlap" is not one capability: user speech during system output may be a backchannel, a floor claim, a repair, or third-party audio, and each case demands a different response. The ontology splits every moment along three axes: temporal relation (T), user intent (I), and system response (R).
We highlight six acid-test cells: a compact stress suite that exposes the main ways current systems fail.
Figure 3: Six canonical full-duplex interaction scenarios. Each panel concretizes one (T, I, R) ontology cell on User (U) / Assistant (A) audio timelines.
These cells can be used directly as data slices, benchmark criteria, and ablation targets. "Fails on T3-I2-R1" is more precise than "not always full-duplex": it says the system mistakes a backchannel for an interruption.
The FSM closes the three-part framework. L0-L3 says where the decision lives, TIR says what interaction must be handled, and the FSM says what the system is doing right now and when it should switch. TIR names the interaction cell; the FSM turns that cell into a policy trace over time.
Five states extend the simple Speak/Listen view with two key additions: Wait, where silence does not mean the user has finished; and Dual, where both speakers are active and the system must still interpret the user. Eleven transitions cover onset, turn-handoff, and overlap. Overlap is the hard case: backchannel, floor claim, and third-party speech each lead to different destinations.
Figure 4: The full-duplex decision state machine: five states and eleven transitions.
The audit leaves two open frontiers. Data coverage: architecture can make a state reachable, but public corpora still underrepresent T4 concurrent speech and I7 third-party speech, so many systems cannot learn the hardest cells. L3 architecture: no published system yet makes Dual native to a shared latent representation.
Together, T x I x R, the five-state FSM, and L0-L3 replace "is this full-duplex?" with a structured profile: cells covered, states reached, and implementation layer. The same profile explains past systems and points toward future data construction, benchmarks, and model design.
| Model | Code Repository |
|---|---|
| dGSLM | 🔗 fairseq/dgslm |
| Duplex-Model | 🔗 thunlp/duplex-model |
| FireRedChat | 🔗 FireRedTeam/FireRedChat |
| X-Talk | 🔗 xcc-zach/xtalk |
| SoulX-Duplug | 🔗 Soul-AILab/SoulX-Duplug |
| Freeze-Omni | 🔗 VITA-MLLM/Freeze-Omni |
| Fun-Audio-Chat | 🔗 FunAudioLLM/Fun-Audio-Chat |
| Covo-Audio | 🔗 Tencent/Covo-Audio |
| Step-Audio R1.1 | 🔗 stepfun-ai/Step-Audio-R1 |
| Moshi | 🔗 kyutai-labs/moshi |
| Mini-Omni2 | 🔗 gpt-omni/mini-omni2 |
| DuplexMamba | 🔗 khfs/DuplexMamba |
| SALMONN-omni | 🔗 bytedance/SALMONN |
Note: For TIR abbreviations, T = Temporal Relation, I = User Intent, R = System Response. Refer to The T x I x R Interaction Ontology for detailed definitions.
| Name | TIR Coverage | Scale | URL |
|---|---|---|---|
| Fisher English Training | T1, T3 I2, I4, I6 R1, R2, R4 |
~1,960 h | 🔗 Part 1 🔗 Part 2 |
| Switchboard-1 Release 2 | T1, T3, T5 I2, I3, I4, I6 R3, R4 |
~260 h | 🔗 LDC97S62 |
| CALLHOME Am. English | T1, T3 I4, I6, I7 |
120 calls (≤30 min/ea) |
🔗 LDC97S42 |
| HCRC Map Task Corpus | T1, T2, T3, T5 I3, I5, I6 R4 |
128 dialogues (~18 h) |
🔗 Official |
| Chiba Univ. JP Map Task | T1, T2, T3, T5 I2, I3, I6 |
128 dialogues (~23 h) |
🔗 NII-SRC |
| IFADV Corpus | T3 I2 |
20 dialogues (~5 h) |
🔗 Official |
| CANDOR Corpus | T1, T3 I2, I4 |
1,656 convs (850+ h) |
🔗 TalkBank |
| Multi-stream Spon. (ZH) | T1, T3 I2, I4 |
27 convs (~10 h) |
🔗 MagicHub |
| Multi-stream Spon. (EN) | T1, T3 I2, I4 |
8 convs (5 h) |
🔗 MagicHub |
| otoSpeech-FD-280h | T1, T3 I4 |
~280 h | 🔗 HuggingFace |
| otoSpeech-FD-141h | T1, T3 I4 |
141 h | 🔗 HuggingFace |
| Japanese CALLHOME | T1, T3, T5 I2, I6 R4 |
~49 h | 🔗 LDC96S37 |
| AliMeeting | T1, T3, T5 | 118.75 h | 🔗 OpenSLR |
| MMedFD | T1, T3 I4 |
39.04 h | 🔗 HuggingFace |
| HumDial-FDBench (Tr.2) | T3, T5 I2, I4, I6, I7 R2, R3, R5 |
Train 107+ h | 🔗 HuggingFace |
| Name | TIR Coverage | Scale | URL |
|---|---|---|---|
| Full-Duplex-Bench | T3 I2, I4 R2, R4 |
727 samples | 🔗 GitHub |
| FD-Bench v1.5 | T3 I2, I4, I7 R1, R2 |
499 samples | 🔗 GitHub |
| FD-Bench v2 | T3 I4 R2, R4 |
200 tasks | 🔗 GitHub |
| FD-Bench v3 | T3 I4 R2 |
100 examples | 🔗 GitHub |
| FD-Bench (Audio) | T3 I2, I4 R2, R4 |
~40 h | 🔗 HuggingFace |
| MTR-DuplexBench | T3 I2, I4, I7 R2, R4 |
1,220 items | 🔗 HuggingFace |
| MMedFD | T1, T3 I4 |
39.04 h | 🔗 HuggingFace |
| HumDial Track 2 | T3, T5 I2, I4, I6, I7 R2, R3, R5 |
Test 5,000 inst. | 🔗 HuggingFace |
| τ-Voice | T3 I2, I4, I7 R2, R4, R5 |
278 tasks | 🔗 GitHub |
| SID-Bench | T3 I2, I4 R2 |
~10 h | 🔗 HuggingFace |
| DualTurn Turn-Taking | T3 I2 R4 |
Oto: 287 h SW: 256 h |
🔗 Oto HF 🔗 SW HF |
If you find this survey or repository useful, please cite our paper:
@article{lu2026fullduplex,
title = {Speaking While Listening: A Survey and Empirical Audit of Full-Duplex Spoken Dialogue Systems},
author = {Lu, Jingyu and Wang, Yuhan and Luo, Jianming and Chen, Yifu and Liang, Tianle and
Ji, Shengpeng and Jiang, Ziyue and Yang, Xiaoda and Zhang, Yu and Cheng, Xize and
Wen, Chenyuhao and Pan, Changhao and Wang, Haoxiao and Ye, Chen and Wu, Jian and
Jiang, Xiaoxi and Jiang, Guanjun and Zhao, Zhou},
journal = {arXiv preprint arXiv:2606.19453},
year = {2026}
}This list is meant to keep growing. If a full-duplex model, dataset, or benchmark is missing -- or if you spot an error in the L0--L3 / T x I x R / FSM audit -- please open an issue or a pull request.