Callaude 📞

⚠️ Experimental: This project is in early development and needs more testing with real phone calls. Use at your own risk and expect rough edges.

Talk to LLMs (Claude, Gemini, GPT) over the phone via Twilio.

Uses simonw's llm CLI for model access, supporting Claude, Gemini, GPT, and local models.

📋 Twilio Setup Note: Setting up a Twilio account requires full KYC (Know Your Customer) verification depending on your country/state. Be aware that this process may take time and require additional documentation.

Architecture

Caller → Twilio → ngrok → Python Server
                              ↓
                    Whisper (STT) ← audio
                    llm CLI ← text → response
                    Piper (TTS) → audio
                              ↓
                    audio → Twilio → Caller

Requirements

• Python 3.10+
• llm CLI with model plugins
• ngrok free account (for Twilio webhooks)

Quick Start

1. Install dependencies

uv pip install -r requirements.txt

Or with traditional pip:

pip install -r requirements.txt

2. Run the setup wizard

python scripts/setup.py

This will:

• Install LLM provider plugins (Claude, Gemini, or GPT)
• Configure your API keys
• Set your security PIN
• Set up ngrok tunnel
• Download the TTS model

3. Start Callaude

python scripts/start.py

What start.py does:

• Starts the Python HTTP server that handles incoming Twilio calls
• Initiates an ngrok tunnel (secure public URL) that connects your local server to Twilio
• The ngrok tunnel creates a public endpoint (https://YOUR-NGROK-URL/incoming-call) that Twilio webhooks can reach
• Displays your PIN and configured model
• All call audio, transcription (Whisper), LLM processing, and synthesis (Piper) happens through this local server

4. Configure Twilio

• Go to your Twilio phone number settings
• Set the webhook URL shown in the console: https://YOUR-NGROK-URL/incoming-call

5. Call your Twilio number and enter your PIN!

Configuration

Edit config.yaml:

security:
  pin: "123456"              # 6-digit PIN to access
  pin_timeout: 10            # Seconds to enter PIN
  whitelist:                 # Allowed phone numbers (E.164 format)
    - "+1234567890"
  working_hours:
    enabled: true
    start: "09:00"
    end: "18:00"
    timezone: "America/New_York"

llm:
  default_model: "claude-3-5-sonnet-latest"  # or gemini-1.5-flash, gpt-4o, etc.
  system_prompt_file: "prompts/system.md"    # Path to system prompt file (see below)

audio:
  stt_model: "base.en"       # Whisper model
  tts_model: "en_US-lessac-medium"  # Piper voice

System Prompt Configuration

Instead of embedding the system prompt in config.yaml, Callaude reads it from a separate Markdown file. This keeps your configuration clean and makes it easy to edit prompts without touching the main config.

To customize the system prompt:

1. Create or edit prompts/system.md:

You are an AI assistant speaking on a phone call.
Keep responses concise and conversational.
Avoid markdown, code blocks, or visual formatting.
Use natural spoken language.

2. Update config.yaml to point to your prompt file:

llm:
  system_prompt_file: "prompts/system.md"

The path is relative to your project root. The setup wizard will show you the file path, making it easy to find and edit later.

Security Features

• PIN Protection: Caller must enter correct PIN within timeout
• Whitelist: Only specified numbers can call (optional)
• Working Hours: Calls rejected outside configured hours
• Call Logging: All calls logged with timestamps

Session Readers (Amp & Claude Code)

Callaude exposes tools to the LLM for reading past sessions from AI coding assistants. During a call, you can ask things like:

• "What Amp threads do I have?"
• "Read my last Amp session"
• "Search my sessions for authentication bug"

The LLM will use list_sessions, search_sessions, and read_session tools automatically.

Supported Sources

Source	Requirements	Storage
Amp	amp CLI installed	Cloud (ampcode.com)
Claude Code	~/.claude/ exists	Local JSONL files

Note: Creating new sessions is not supported—Amp and Claude Code are TUI apps that don't accept piped input cleanly.

Supported Models

Any model supported by llm works:

# List available models
llm models

# Test a model
llm -m claude-3-5-sonnet-latest "Hello!"
llm -m gemini-2.5-flash "Hello!"
llm -m gpt-4o "Hello!"

Project Structure

callaude/
├── callaude/
│   ├── config.py          # YAML config loader
│   ├── security.py        # PIN, whitelist, working hours
│   ├── call_handler.py    # Call state machine
│   ├── llm_bridge.py      # llm CLI wrapper
│   ├── tools.py           # LLM tools (session readers)
│   ├── session_readers.py # Amp & Claude Code readers
│   ├── server.py          # FastAPI server (Twilio webhooks)
│   └── audio/
│       ├── codec.py       # μ-law encode/decode
│       ├── stt.py         # Whisper (STT)
│       └── tts.py         # Piper (TTS)
├── scripts/
│   ├── setup.py           # Setup wizard
│   ├── start.py           # Start server + ngrok
│   └── download_models.py
├── docs/
│   └── NETGSM.md          # Alternative provider (Turkey)
├── config.example.yaml
└── requirements.txt

License

MIT