Agent Voice Kit

Local-first voice capabilities for AI agents: language/speech recognition, text-to-speech, and speaker voiceprint recognition.

What This Repo Is For

This project provides a small integration layer for adding voice I/O to agents without locking the agent runtime to one vendor.

Core capabilities:

• Language and speech recognition: detect spoken language and transcribe audio.
• Text-to-speech: turn agent responses into spoken audio.
• Speaker recognition: enroll and match speaker voiceprints.
• Provider abstraction: use local models by default, with cloud providers as fallback.

Recommended Stack

The first supported provider targets are:

Capability	Local-first option	Cloud fallback
Speech recognition	FunASR / SenseVoice, faster-whisper, whisper.cpp	OpenAI transcribe, Deepgram
Text-to-speech	Kokoro, Piper	OpenAI TTS, ElevenLabs
Speaker recognition	SpeechBrain, pyannote.audio	pyannoteAI, provider diarization APIs
Emotion signal	SenseVoice	Provider-specific audio intelligence

Install

pip install "git+https://github.com/zimoos/agent-voice-kit.git"

For local development:

git clone https://github.com/zimoos/agent-voice-kit.git
cd agent-voice-kit
uv sync --all-extras
uv run agent-voice doctor

Quick Start

from agent_voice_kit import VoiceGateway, UnavailableSpeechRecognizer

gateway = VoiceGateway(
    recognizer=UnavailableSpeechRecognizer("wire FunASR, Whisper, or OpenAI here"),
)

result = gateway.recognize("sample.wav")
print(result.text)

Design

Agents should not call ASR, TTS, or speaker models directly. They should call a single voice gateway:

microphone/file -> VAD -> recognizer -> agent text loop -> synthesizer -> speaker
                              |
                              +-> speaker identifier / emotion signal

This keeps model choice, cost control, privacy policy, and fallback behavior outside the agent business logic.

Roadmap

• FunASR / SenseVoice recognizer adapter
• faster-whisper recognizer adapter
• Piper synthesizer adapter
• Kokoro synthesizer adapter
• SpeechBrain speaker embedding adapter
• OpenAI-compatible ASR/TTS adapters
• FastAPI gateway service
• WebSocket streaming mode

License

MIT