xAI Voice Agent

Real-time voice conversations with Grok — a local, open-source voice assistant powered by the xAI Realtime API (grok-voice-latest).

Talk naturally with Grok using your microphone. Interrupt mid-sentence, get instant responses, and keep full conversation history — all running locally on your machine.

Beta software — Rapidly built in ~2 hours during rest time using Claude Code and OpenAI Codex. Functional and exciting, but needs testing, polish, and community contributions.

Features

Core Voice Experience

• True real-time voice powered by grok-voice-latest via the xAI Realtime API
• Full barge-in support — interrupt Grok mid-response by speaking
• Server-side Voice Activity Detection (VAD) with configurable threshold and silence timeout
• Streaming audio + text — hear responses while they are being generated
• Live transcription of your voice using grok-2-audio

Conversation Intelligence

• Automatic dialogue titles generated by grok-3-mini-fast after your first message
• Full conversation history with SQLite persistence
• FTS5 full-text search across all messages
• Per-message and per-conversation token accounting (input/output/total)
• Deduplicated usage tracking — prevents double counting on retries

Advanced Capabilities

• Settings versioning system — create multiple agent personalities (different system prompts, voices, VAD settings, temperature) and switch between them
• Message-level settings inspection — see exactly which configuration was used for any response
• Built-in tools — Grok has access to web_search and x_search (X/Twitter) during voice conversations
• Live settings panel — change instructions, voice, temperature, and VAD parameters without restarting

Developer Experience

• Detailed debug logging (JSONL) with optional full audio capture
• Clean async architecture (FastAPI + WebSocket)

Quick Start

git clone https://github.com/arthurkatcher/xai-voice-agent.git
cd xai-voice-agent

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Edit .env and add your xAI API key

Add Your xAI API Key

1. Copy the example file:

   cp .env.example .env

2. Open .env and set your key:

   XAI_API_KEY=xai-your-key-here

Get your API key at: https://console.x.ai

Run the App

python server.py

The interface will automatically open at http://127.0.0.1:8765.

Configuration

All settings are managed via .env. Key options:

Variable	Default	Description
XAI_API_KEY	(required)	Your xAI API key
VOICE	eve	Voice: eve, ara, leo, rex, sal
INSTRUCTIONS	"You are..."	System prompt for Grok
TEMPERATURE	0.8	Creativity level
MAX_RESPONSE_OUTPUT_TOKENS	4096	Maximum response length
VAD_THRESHOLD	0.85	Voice detection sensitivity (0.0–1.0)
VAD_SILENCE_MS	800	Silence duration before ending turn (ms)

Debug Logging Options

XAI_VOICE_LOG_IO=1              # Log all WebSocket events
XAI_VOICE_LOG_FULL_AUDIO=1      # Include raw audio frames (very verbose)

How It Works

Browser
        ↕ WebSocket (audio + events)
FastAPI server (server.py)
        ↕
xAI Realtime API (grok-voice-latest)
        ↕ (tools)
Web Search + X Search

Key Components

• app/realtime.py — WebSocket bridge, barge-in logic, streaming audio/text handling
• app/db.py — Robust SQLite layer with WAL, FTS5, token tracking, and settings versioning
• app/titles.py — Async title generation using grok-3-mini-fast
• app/api.py — REST endpoints for conversations, search, and settings management
• static/app.js — Real-time frontend with streaming transcript and settings UI

Technical Highlights

• Sophisticated interruption handling with state machine and 900ms grace period
• Token deduplication via counted_response_ids table
• Settings versioning — every message is linked to the exact configuration used
• Tools enabled — Grok can search the web and X during voice conversations
• Production-grade database — WAL mode, 256MB mmap, incremental vacuum, FTS5 triggers
• Detailed observability — every event can be logged with optional audio capture

Project Status

This is early beta software.

It was built extremely fast as a proof-of-concept. While the core voice loop is surprisingly robust, the project needs:

• Broader testing across different environments and microphones
• UI/UX refinements and accessibility improvements
• Better error handling and reconnection logic
• Export/import functionality
• Documentation and usage examples
• Feature requests and real-world feedback

Contributions are very welcome! Open an issue or pull request.

Tech Stack

• Python 3.10+
• FastAPI + WebSocket
• xAI Realtime API (grok-voice-latest)
• aiosqlite + FTS5
• Vanilla JavaScript frontend

License

MIT License

---

Built with Claude Code & OpenAI Codex • Beta • Seeking contributors

https://github.com/arthurkatcher/xai-voice-agent