Built by implicator.ai with Claude Code
Claude Code runs in a terminal. Terminals take text. But sometimes you are walking, cooking, driving, or just thinking faster than you can type. This project connects Telegram voice messages to Claude Code through a Whisper transcription server, so you can talk to your AI coding assistant from your phone without touching a keyboard.
The reference implementation runs entirely on local hardware using a GPU-accelerated Whisper server. Cloud adapters for OpenAI and Groq are included for users who prefer not to self-host. A typical 15-second voice message returns a transcript in under two seconds on local GPU, or 1-3 seconds via cloud API.
Claude Code's Telegram plugin already handles text and photo messages. You type on your phone, Claude sees it in the terminal. But Telegram's voice message feature, the hold-to-record button that sends .ogg audio, was ignored entirely. The bot received voice messages, had no handler for them, and dropped them silently.
That created a gap in the workflow. Voice input is faster than thumb-typing for anything longer than a sentence. It is also the natural input method when you are away from a desk. Without voice support, the Telegram channel was limited to short text exchanges. Anything that required context, explanation, or nuance meant either switching to a laptop or laboriously typing paragraphs on a phone keyboard.
Two code changes. One adds a transcription endpoint to an existing Whisper server. The other teaches the Telegram plugin to download voice messages and send them to that endpoint.
Total lines of code added: roughly 120 (50 Python, 70 TypeScript). No new dependencies. No new services. Both changes plug into existing infrastructure that was already running for other purposes.
┌─────────────┐ .ogg ┌───────────────────┐
│ Telegram │──────────────►│ server.ts │
│ (phone) │ Bot API │ (Telegram plugin) │
└─────────────┘ └────────┬───────────┘
│
download .ogg via getFile API
save to inbox/
│
POST multipart/form-data
│
▼
┌───────────────────┐
│ ytwhisper │
│ Flask + GPU │
│ │
│ faster-whisper │
│ large-v3-turbo │
│ RTX 3080 Ti │
│ CUDA / float16 │
└────────┬───────────┘
│
{"text":"...","duration":12.3}
│
▼
┌───────────────────┐
│ Claude Code │
│ (MCP notification)│
│ │
│ [voice] Hello, │
│ can you refactor │
│ the auth module...│
└───────────────────┘
Three components, all on a local network. The Telegram plugin (server.ts) is an MCP server that runs alongside Claude Code. It receives messages from Telegram's Bot API via long polling, applies access control, and forwards approved messages into the Claude Code session as MCP notifications. The ytwhisper server is a Flask app backed by faster-whisper on a CUDA GPU, originally built for transcribing YouTube videos. The new /transcribe-file endpoint reuses the same model and post-processing pipeline.
1. The user records a voice message. Hold the mic button in Telegram, speak, release. Telegram encodes the audio as Opus in an .ogg container and sends it to the bot via the Bot API.
2. The plugin receives a message:voice event. grammY, the Telegram bot framework, dispatches this to the voice handler registered in server.ts.
3. Access control runs first. The same gate() function that protects text messages checks whether the sender is allowlisted, has a valid pairing code, or is posting in an approved group. Unauthorized voice messages are dropped before any download occurs. This is deliberate. Downloading files from unapproved senders would let anyone with the bot's username consume bandwidth and fill the inbox directory.
4. The .ogg file downloads. On approval, the handler calls Telegram's getFile API to get a temporary URL, fetches the binary content, and writes it to ~/.claude/channels/telegram/inbox/ with a timestamped filename. The file stays on disk as an archival copy.
5. Transcription request fires. The handler constructs a FormData object with the .ogg buffer and POSTs it to the ytwhisper server's /transcribe-file endpoint. A 30-second timeout (AbortSignal.timeout(30000)) prevents the handler from blocking indefinitely if the GPU server is unreachable.
6. faster-whisper transcribes. The endpoint saves the upload to a temp file, calls model.transcribe() with VAD (Voice Activity Detection) filtering enabled, and collects the resulting segments. The model is a lazy singleton. It loads into VRAM on the first request (~3 seconds) and stays resident for all subsequent calls.
7. Post-processing cleans the transcript. The post_process_transcript() function handles the rough edges that Whisper leaves behind: duplicate words from stutters, missing spaces after punctuation, spelled-out numbers that should be digits, and long walls of text that need paragraph breaks. For voice messages, this step is lighter than for YouTube videos because the audio is typically cleaner, shorter, and single-speaker.
8. The transcript returns as JSON. {"text": "Hello, can you refactor the auth module to use JWT tokens instead of session cookies?", "duration": 8.2}. Duration comes from ffprobe, not from Whisper, so it is the actual audio length regardless of silence trimming.
9. Claude Code receives an MCP notification. The content field is [voice] Hello, can you refactor the auth module to use JWT tokens instead of session cookies?. The [voice] prefix tells Claude this was spoken, not typed. The meta object includes voice_path (the saved .ogg on disk), voice_duration (seconds), chat_id, message_id, user, and a timestamp.
10. Claude responds normally. From Claude's perspective, this is a text message with a [voice] prefix. It can reply via the reply MCP tool, which sends a Telegram message back to the user's phone. The full conversational loop works: voice in, text out.
11. Failure is graceful. If ytwhisper is down, the fetch times out, or the transcription errors, the notification content becomes [voice message — transcription failed]. Claude sees this and can ask the user to type their message instead. No crash, no silent failure.
The transcription server in this project is not new. It was built months earlier to solve a different problem: transcribing YouTube videos for a journalism research workflow. When a source publishes a 45-minute conference talk or a podcast interview, the newsroom needs a text version to search, quote, and fact-check against. Manually transcribing that much audio is impractical. Paying per-minute cloud transcription rates at the volume a daily publication requires gets expensive.
The solution was ytwhisper: a Flask web application backed by faster-whisper, an optimized C++ reimplementation of OpenAI's Whisper model that runs 4x faster than the original PyTorch version. It accepts a YouTube URL, downloads the audio with yt-dlp, transcribes it on the GPU, runs post-processing to fix common Whisper artifacts (stutters, missing punctuation, number formatting), and optionally sends the result through an LLM for a final polish pass to catch misheard proper nouns.
The server runs on a Proxmox LXC container with PCIe GPU passthrough. The container gets direct access to the RTX 3080 Ti as if it were bare metal, but remains isolated from other workloads on the same Proxmox host. This is a common homelab pattern: one physical machine running multiple containers, each with access to specific hardware resources.
Server environment:
| Component | Detail |
|---|---|
| OS | Debian 12 (Bookworm) on Proxmox LXC |
| CPU | 12 cores allocated |
| RAM | 12GB |
| GPU | NVIDIA RTX 3080 Ti, 12GB VRAM |
| CUDA | 13.0, Driver 580.142 |
| Python | 3.11.2 |
| Flask | 3.1.2 |
| faster-whisper | 1.2.1 |
| Whisper model | large-v3-turbo (float16, beam size 8) |
What already existed before this project:
get_model()— Lazy singleton that loads the Whisper model into VRAM on first call. Subsequent calls return the cached instance. This is the function that makes warm transcriptions fast: the 3-second model load only happens once.get_audio_duration()— Calls ffprobe to get the exact audio duration in seconds. Used for progress tracking in the YouTube workflow, reused here for thedurationfield in the API response.post_process_transcript()— A pipeline of regex transformations that fixes Whisper's most common formatting mistakes. It normalizes spelled-out numbers ("twenty" to "20") in technical contexts (e.g., "M20" meaning "Mac mini M2"), removes word-level stutters ("the the server" to "the server"), adds spaces after punctuation that Whisper sometimes omits, and inserts paragraph breaks at natural transition points ("However,", "Moving on,", "First,").llm_polish()— An optional pass that sends the transcript through a language model (Claude Haiku via LiteLLM) to fix misheard proper nouns. "Clock code" becomes "Claude Code," "enthropy" becomes "Anthropic." This is off by default for voice messages (speed matters more than polish for short dictation), but available via?polish=true.DEFAULT_VOCAB— A vocabulary hint list of ~200 technical terms that Whisper uses as a conditioning signal. When the audio sounds like "Claude Code," this list makes the model more likely to produce the correct spelling rather than a phonetic approximation.
The existing architecture meant adding voice message support required exactly one new function: a Flask route that accepts a file upload instead of a YouTube URL. Every other piece, the model management, transcription, post-processing, and cleanup, was already written and tested against months of daily YouTube transcriptions.
The new endpoint (/transcribe-file) is a synchronous Flask route. Unlike the YouTube transcription workflow, which spawns a background thread for downloads that can take minutes, voice messages are short enough to handle in the request cycle. A 15-second voice message transcribes in roughly one second. Making it asynchronous would add complexity (job polling, status endpoints) for no practical benefit.
@app.route("/transcribe-file", methods=["POST"])
def transcribe_file():
"""Synchronous transcription of an uploaded audio file."""
if "file" not in request.files:
return jsonify({"error": "No file provided"}), 400
uploaded = request.files["file"]
# Check file size before processing (25MB limit)
uploaded.seek(0, 2)
size = uploaded.tell()
uploaded.seek(0)
if size > 25 * 1024 * 1024:
return jsonify({"error": "File too large (25MB max)"}), 413
ext = Path(uploaded.filename).suffix or ".ogg"
tmp_path = DOWNLOADS_DIR / f"voice_{uuid.uuid4().hex[:8]}{ext}"
try:
uploaded.save(str(tmp_path))
duration = get_audio_duration(tmp_path)
model = get_model()
segments, info = model.transcribe(
str(tmp_path),
beam_size=BEAM_SIZE,
language="en",
initial_prompt=DEFAULT_VOCAB,
temperature=[0.0, 0.2, 0.4],
vad_filter=True,
vad_parameters=dict(min_silence_duration_ms=500),
)
text = " ".join(seg.text.strip() for seg in segments)
text = post_process_transcript(text)
# Optional LLM cleanup pass (off by default for speed)
use_polish = request.args.get("polish", "false").lower() == "true"
if use_polish:
text = llm_polish(text, "voice message")
return jsonify({"text": text, "duration": round(duration, 1)})
except Exception as e:
app.logger.error(f"transcribe-file error: {e}")
return jsonify({"error": str(e)}), 500
finally:
tmp_path.unlink(missing_ok=True)Design notes:
The temperature parameter is set to [0.0, 0.2, 0.4], narrower than the [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] range used for YouTube videos. Whisper uses temperature fallback: it tries the lowest temperature first, and only escalates if the output quality metrics (log probability, compression ratio) are poor. Voice messages from a phone's microphone in a quiet room produce clean audio that Whisper handles well at temperature 0. The narrower range means less time spent on fallback attempts that are unlikely to trigger.
The initial_prompt contains a vocabulary hint list (DEFAULT_VOCAB) of roughly 200 technical terms: product names (Claude, GPT, Anthropic), programming terms (TypeScript, REST API, GraphQL), and infrastructure vocabulary (Proxmox, TrueNAS, Docker). Whisper uses this as a conditioning signal. When the audio contains "Claude Code," the model is more likely to output that exact string rather than "clock code" or "cloud code."
The vad_filter enables Silero VAD (Voice Activity Detection) to skip silence at the beginning and end of the recording. Phone voice messages often have a half-second of silence before the speaker starts and after they stop. Without VAD, Whisper sometimes hallucinates filler text in these silent sections.
Cleanup happens in a finally block. The temp file is deleted whether transcription succeeds or fails. The original .ogg remains in the Telegram plugin's inbox directory as the archival copy.
The Telegram plugin is implemented as an MCP (Model Context Protocol) server. MCP is the standard that Claude Code uses to communicate with external tools and data sources. The plugin registers three tools (reply, react, edit_message) that Claude can call to send messages back to Telegram, and uses MCP notifications to push inbound messages into the session.
The plugin runs as a child process of Claude Code, communicating over stdio. When launched with --channels plugin:telegram@claude-plugins-official, Claude Code starts the MCP server, which initializes a grammY bot that long-polls the Telegram Bot API.
The voice handler follows the same pattern as the existing photo handler. Both defer expensive operations (file downloads) until after the access gate approves the sender. The voice handler adds a second deferred operation: transcription.
bot.on('message:voice', async ctx => {
await handleInboundVoice(ctx)
})The handleInboundVoice function is intentionally separate from the existing handleInbound function rather than being integrated into it. This keeps the diff minimal. The existing text and photo handling code is unchanged. If a future plugin update ships a new server.ts, the voice handler can be re-added without resolving merge conflicts in existing functions.
async function handleInboundVoice(ctx: Context): Promise<void> {
const result = gate(ctx)
if (result.action === 'drop') return
if (result.action === 'pair') {
// Same pairing flow as text messages
await ctx.reply(`Pairing required — run in Claude Code:\n\n/telegram:access pair ${result.code}`)
return
}
// Approved — show typing indicator
void bot.api.sendChatAction(chat_id, 'typing').catch(() => {})
// Download the .ogg
const voice = ctx.message!.voice!
const file = await ctx.api.getFile(voice.file_id)
const url = `https://api.telegram.org/file/bot${TOKEN}/${file.file_path}`
const res = await fetch(url)
const buf = Buffer.from(await res.arrayBuffer())
writeFileSync(path, buf)
// Transcribe via ytwhisper
const form = new FormData()
form.append('file', new Blob([buf]), 'voice.ogg')
const transcribeRes = await fetch(`${YTWHISPER_URL}/transcribe-file`, {
method: 'POST',
body: form,
signal: AbortSignal.timeout(30000),
})
const content = transcribeRes.ok
? `[voice] ${(await transcribeRes.json()).text}`
: '[voice message — transcription failed]'
// Fire MCP notification — Claude sees this as a message
void mcp.notification({
method: 'notifications/claude/channel',
params: {
content,
meta: { chat_id, voice_path, voice_duration, user, ts },
},
})
}The YTWHISPER_URL is configurable via environment variable, loaded from ~/.claude/channels/telegram/.env at boot. Default: http://localhost:5000.
| Metric | Value |
|---|---|
| Model load (first request only) | ~3 seconds |
| Telegram file download | 300-800ms |
| Transcription (10s audio) | ~500ms |
| Transcription (30s audio) | ~1.2s |
| Transcription (60s audio) | ~2.5s |
| Post-processing | <10ms |
| Total end-to-end (warm model) | 1-2s typical |
| VRAM usage (model loaded) | ~2.7GB of 12GB |
The model stays loaded in GPU memory as a Python global. After the first voice message warms it up, subsequent transcriptions skip the load entirely. The RTX 3080 Ti has 12GB of VRAM; the large-v3-turbo model in float16 uses roughly 2.7GB, leaving plenty of headroom for other GPU workloads.
The bottleneck is usually the network hop to Telegram's file servers, not the transcription itself. Voice messages under 30 seconds transcribe faster than they play.
Dictating while mobile. Walk to the coffee shop, record a voice message describing the bug you just thought of, and Claude starts working on it before you sit down.
Longer instructions without typing. Explaining a refactoring strategy in 30 seconds of speech is faster and more detailed than thumb-typing the same thing.
Accessibility. For anyone with repetitive strain injuries or other conditions that make typing painful, voice input turns the phone into a full interface for an AI coding assistant.
Multi-tasking. Review a pull request description by listening to Claude's reply (Telegram reads messages aloud), then dictate your response.
Cloud speech-to-text APIs (Google Cloud Speech, AWS Transcribe, OpenAI Whisper API) would work, but local transcription has three advantages for this use case:
Privacy. Voice recordings contain biometric data. They may also contain proprietary code discussions, unreleased product details, or personal information. Keeping audio on the local network means no third-party retention policies apply.
Latency. A local GPU transcribes a 15-second clip in under a second. Cloud APIs add network round-trip time, queue time, and cold-start latency. For a real-time conversational flow, every extra 500ms matters.
Cost. After the GPU hardware cost (amortized across many uses), local transcription is free per request. Cloud APIs charge per minute of audio. At high volumes this adds up; at any volume it is an ongoing operational cost rather than a one-time capital expense.
The tradeoff is maintenance. You need a machine with an NVIDIA GPU, a working CUDA installation, and a running Flask server. For anyone already running a homelab, this is trivial. For someone starting from scratch, a cloud API is simpler to set up.
Not everyone has an RTX card sitting in a server rack. The architecture is designed so that the transcription backend is a black box: server.ts POSTs a file and expects {"text": "...", "duration": N} back. Any service that speaks this contract works. Here are the practical options for users who want voice messages without running local hardware.
The most direct replacement. OpenAI hosts the same Whisper model family as a cloud API. Send audio, get text.
Endpoint: POST https://api.openai.com/v1/audio/transcriptions
Format: Multipart form data with file and model fields
Pricing: $0.006 per minute of audio (as of early 2026)
Latency: 1-3 seconds for short clips, depending on load
To use it as a drop-in backend, you would write a small adapter that receives the POST /transcribe-file request, forwards the file to OpenAI's API, and reformats the response. A minimal Flask proxy:
@app.route("/transcribe-file", methods=["POST"])
def transcribe_file():
uploaded = request.files["file"]
import openai
client = openai.OpenAI()
result = client.audio.transcriptions.create(
model="whisper-1",
file=uploaded,
response_format="json",
)
return jsonify({"text": result.text, "duration": 0})The duration field won't be accurate without running ffprobe locally, but server.ts uses it only for metadata, not for any logic.
Pros: No GPU needed. No model management. Works from any machine with internet access. Cons: Audio leaves your network. Per-minute cost adds up. Adds 1-2 seconds of network latency versus a local GPU.
Google's offering with broad language support and speaker diarization. The v2 API supports file uploads and streaming.
Pricing: $0.016 per minute (standard), $0.006 per minute (data logging enabled, Google may use your audio to improve models) Latency: 1-4 seconds for short clips Languages: 125+ languages, auto-detection available
More setup overhead than OpenAI (GCP project, service account, client library), but stronger if you need multilingual transcription. The data logging pricing tier matches OpenAI's rate but comes with the tradeoff of Google retaining your audio.
Groq runs Whisper on their LPU (Language Processing Unit) hardware, producing the fastest cloud transcription available. Particularly interesting for the voice message use case where latency matters most.
Endpoint: POST https://api.groq.com/openai/v1/audio/transcriptions
Format: OpenAI-compatible (same multipart form data)
Pricing: Free tier available, paid plans for higher throughput
Latency: Often sub-second, even for 30-second clips
Because Groq's API is OpenAI-compatible, the same adapter code works. Change the base URL and API key.
client = openai.OpenAI(
base_url="https://api.groq.com/openai/v1",
api_key=os.environ["GROQ_API_KEY"],
)Pros: Fastest cloud option. Free tier for experimentation. OpenAI-compatible, so existing code works. Cons: Availability depends on Groq's capacity. Free tier has rate limits. Less battle-tested than Google or OpenAI at scale.
Real-time and batch transcription with a developer-focused API. Strong at noisy audio and accented speech.
Pricing: Pay-as-you-go starting at $0.0043 per minute (Nova-2 model) Latency: Sub-second for short clips with their Nova model Languages: 36 languages
The cheapest per-minute option among the commercial APIs. Their Nova-2 model is competitive with Whisper large-v3 on English benchmarks and faster on their hardware.
If you want to avoid cloud APIs but don't have a dedicated GPU, faster-whisper can run on CPU. It is significantly slower, but for voice messages under 30 seconds, it may still be acceptable.
Change the ytwhisper environment variables:
WHISPER_DEVICE=cpu
WHISPER_COMPUTE=int8
WHISPER_MODEL=baseThe base model on CPU with int8 quantization transcribes a 15-second clip in roughly 5-8 seconds on a modern Intel/AMD processor. Not instant, but usable. The quality is lower than large-v3-turbo, but adequate for clear voice recordings from a phone microphone.
You could also use the small model as a middle ground: better quality than base, roughly 2x slower.
| Option | Latency | Cost/min | Privacy | Setup |
|---|---|---|---|---|
| Local GPU (this project) | <1s | $0 | Full | Medium |
| CPU fallback | 5-8s | $0 | Full | Easy |
| Groq Whisper | <1s | Free tier | Cloud | Easy |
| OpenAI Whisper API | 1-3s | $0.006 | Cloud | Easy |
| Deepgram Nova-2 | <1s | $0.0043 | Cloud | Easy |
| Google Cloud STT | 1-4s | $0.006-0.016 | Cloud | Medium |
For the best experience, local GPU wins on every metric except setup complexity. For users without GPU hardware, Groq's free tier is the fastest way to get started. OpenAI's Whisper API is the safest bet for production reliability. The CPU fallback works in a pinch but the 5-8 second delay makes the conversational flow feel sluggish.
- English only. The transcription call hardcodes
language="en". Whisper supports 100+ languages; changing this to auto-detect or accept a language parameter is straightforward but not yet implemented. - Plugin cache volatility. The Telegram plugin's
server.tslives in Claude Code's plugin cache (~/.claude/plugins/cache/). Plugin updates will overwrite it. The patches inpatches/can be reapplied. - Single-worker GPU. faster-whisper holds a CUDA context lock during transcription. Concurrent voice messages queue behind each other on the Flask server. For single-user use this is invisible. For a group chat with many voice senders, it could introduce latency.
- No video notes. Telegram distinguishes between "voice messages" (audio-only, hold to record) and "video notes" (round video circles). Only voice messages are handled.
- 30-second timeout. Very long voice messages (several minutes) could exceed the fetch timeout in
server.ts. Adjustable in the code.
See setup-guide.md for step-by-step replication instructions.
README.md — This document
setup-guide.md — Step-by-step replication guide
LICENSE — MIT
patches/
app.py.patch — ytwhisper /transcribe-file endpoint
server.ts.patch — Telegram plugin voice handler
adapters/
openai-whisper-proxy.py — Drop-in adapter using OpenAI Whisper API
groq-whisper-proxy.py — Drop-in adapter using Groq Whisper API
Voice Mode for Claude Code Channels is an implicator.ai project. Built with Claude Code. Licensed under MIT.