backchannel

An open, full-duplex realtime voice-agent framework, built Apple-Silicon-first, with first-class barge-in and turn detection — the parts that make a voice agent feel like a conversation instead of a walkie-talkie.

mic ─► VAD ─► streaming ASR ─► LLM (streamed) ─► streaming TTS ─► speaker
          │                                                          ▲
          └──────────── barge-in: user talks over the agent ─────────┘

Why

Most voice demos are half-duplex: you talk, it talks, you wait. Real conversation is full-duplex — you interrupt, you back-channel ("mm-hmm"), you finish each other's sentences. backchannel treats interruption and turn-taking as core, not an afterthought:

• Turn detection decides when you are actually done, using VAD silence plus optional semantic endpointing, so it does not cut you off at the first pause.
• Barge-in lets you talk over the agent; it cancels generation, flushes speech, and yields the floor in under ~200ms, with echo gating so it does not interrupt itself.
• Latency is measured, not guessed: every turn reports time-to-first-audio and a full stage-by-stage breakdown.

Design

• Async all the way down. The pipeline is asyncio; stages are wired by queues with backpressure. Barge-in is task cancellation, done cleanly.
• Swappable backends. ASR, LLM, and TTS sit behind small ABCs. Defaults are Apple-Silicon-native (mlx-whisper, mlx-lm, Kokoro); an HF Inference backend is planned.
• One event stream, many transports. The agent emits a single ordered event stream. A local mic/speaker loop, a WebSocket server, and a FastRTC/Gradio demo all consume the same stream, so behaviour is identical everywhere.

See docs/architecture.md for the design, docs/benchmarks.md for measured results, PROTOCOL.md for the wire protocol, and PLAN.md for the roadmap.

Status

Working end to end: full-duplex local loop, barge-in + turn detection, streaming ASR, and a WebSocket server with a browser demo — all verified with the real models. Highlights:

• Barge-in interrupts the agent mid-sentence in ~200ms, verified end to end.
• Streaming ASR + whisper-small keeps recognition off the critical path (ttft_asr ≈ 60ms).
• 25 model-free tests (uv run pytest -q, ~2s); real-model behaviour covered by benchmarks/.

Run it

Local mic loop (talk + interrupt in your terminal):

uv run backchannel

Server + browser demo (open http://127.0.0.1:8000, use headphones):

uv run backchannel-serve          # FastAPI + WebSocket voice server, serves the demo at /
uv run python benchmarks/ws_smoke.py   # headless client that drives it with synth audio

The wire protocol is documented in PROTOCOL.md. To run the demo publicly on a Hugging Face Space (Linux/CPU backend), see deploy/DEPLOY.md.

Backends

• mlx (default on Apple Silicon): mlx-whisper + mlx-lm, local and fast.
• linux (servers / HF Spaces): faster-whisper on CPU/CUDA + the HF Inference API for the LLM. Selected automatically off Apple Silicon, or with BACKCHANNEL_BACKEND=linux.

Develop

uv venv --python 3.12
uv pip install -e ".[dev,speech,serve]"
uv run pytest -q

Benchmarks: benchmarks/warm_latency.py, benchmarks/streaming_ab.py (ASR trade-offs), benchmarks/bargein_demo.py (barge-in with real models).

License

MIT.