ai-voice-agent

A real-time conversational AI agent that auto-answers calls on FreeSWITCH — speech-to-text → LLM → text-to-speech, streamed back into the live call with low enough latency to function as an actual phone conversation rather than a turn-and-wait IVR menu.

Architecture

 Caller dials 1199 ──► FreeSWITCH answers immediately (ai_auto_answer.xml)
                                │
                        mod_audio_fork streams audio
                         both directions over WebSocket
                                │
                                ▼
                          ai_agent.py
                    ┌───────────┼───────────┐
                    ▼           ▼           ▼
                  STT  ──►   LLM    ──►   TTS
              (transcribe) (respond)  (synthesize)
                                │
                        audio streamed back
                          into the live call

Why voice-activity-based turn-taking, not push-to-talk

A phone call has no "press a key when you're done talking" affordance — the agent has to infer when the caller finished their thought from audio alone. ai_agent.py tracks trailing silence via RMS energy per audio chunk (CallSession.feed_audio) and treats a configurable silence threshold as the end of the caller's turn. This is a cheap, dependency-free approach — accurate enough for clear turn-taking, though a dedicated VAD model (e.g. WebRTC VAD or Silero VAD) would handle background noise and false triggers better in a noisy environment.

Files

• freeswitch/ai_auto_answer.xml — the dialplan extension that answers and bridges the call into mod_audio_fork, FreeSWITCH's real-time audio streaming module.
• ai_agent.py — the WebSocket server handling the STT → LLM → TTS pipeline per call session. The provider functions (speech_to_text, query_llm, text_to_speech) are intentionally left as integration points — which STT/LLM/TTS vendor you plug in is a swappable implementation detail, not the core design.

Setup

cp freeswitch/ai_auto_answer.xml /etc/freeswitch/dialplan/acme_ai.xml
fs_cli -x "reloadxml"

pip install -r requirements.txt
# Wire up speech_to_text(), query_llm(), text_to_speech() in ai_agent.py
# to your providers of choice, then:
python ai_agent.py

Where the real engineering challenge is

Anyone can call an LLM API. The actual hard problem in a telecom context is latency and turn-taking feeling natural over a live RTP stream — every step (STT transcription, LLM response generation, TTS synthesis) adds delay, and a caller will hang up on a 3-second dead-air pause far faster than they'd tolerate the same delay in a text chat. This is why the pipeline streams audio in both directions rather than waiting for a full recording before processing, and why voice-activity detection (not a fixed timer) decides when a "turn" is actually over.

Pairs with

freeswitch-cloud-pbx — the same multi-tenant dialplan structure this extension drops into, and the CDR pipeline that would log AI-handled calls alongside human-handled ones for reporting.