Multi-Agent Voice Systems with LiveKit

Build a medical office triage system where specialized voice agents (triage, support, billing) hand off conversations seamlessly while preserving full context across transfers.

Start learning at learnwithparam.com. Regional pricing available with discounts of up to 60%.

What You'll Learn

• Build multi-agent voice systems with LiveKit
• Implement seamless agent-to-agent transfers
• Preserve conversation context across handoffs
• Create specialized agents with distinct roles and responsibilities
• Manage shared state across agent transfers

Tech Stack

• LiveKit Agents - Real-time voice AI agent framework
• Deepgram - Speech-to-text and text-to-speech
• Silero VAD - Voice activity detection
• FastAPI - High-performance async Python web framework
• LLM Provider Pattern - Supports Fireworks, OpenRouter, OpenAI
• Docker - Containerized development

Getting Started

Prerequisites

• Python 3.11+
• uv (installed automatically by make setup)
• An API key from any supported LLM provider
• LiveKit Cloud account (or self-hosted LiveKit server)
• Deepgram API key

Quick Start

# One command to set up and run
make dev

# Or step by step:
make setup          # Create .env and install dependencies
# Edit .env with your API keys
make run            # Start the FastAPI server

With Docker

make build          # Build the Docker image
make up             # Start the container
make logs           # View logs
make down           # Stop the container

API Documentation

Once running, open http://localhost:8000/docs for the interactive Swagger UI.

Challenges

Work through these incrementally to build the full application:

1. The "Hello, Voice!" Agent - Basic single voice agent with LiveKit
2. The Specialist Agent - Create agents with specialized prompts and roles
3. The Multi-Agent Scaffold - Set up shared state and agent registry
4. The Transfer Signal - Implement function tools for agent transfer triggers
5. The Seamless Handoff - Transfer between agents with context preservation
6. The Efficient Handoff - Chat history truncation and context management
7. The Full Application - Complete medical office triage system
8. Production-Ready Features - Room attributes, error handling, and monitoring

Makefile Targets

make help           Show all available commands
make setup          Initial setup (create .env, install deps)
make dev            Setup and run (one command!)
make run            Start FastAPI server
make build          Build Docker image
make up             Start container
make down           Stop container
make clean          Remove venv and cache

Learn more

• Start the course: learnwithparam.com/courses/multi-agent-voice-systems
• AI Bootcamp for Software Engineers: learnwithparam.com/ai-bootcamp
• All courses: learnwithparam.com/courses