From 5c8269d8522b176cbdbeafcf3dd286c889976b4a Mon Sep 17 00:00:00 2001 From: Spaceben123 Date: Sat, 5 Sep 2026 20:34:46 +0200 Subject: [PATCH] Add guide: Transcribing Audio with Whisper via OpenAI and Groq Covers SAPAT (nkkko/sapat), a multi-provider Whisper transcription CLI, for issue #13. Grounded in a direct read of the source (v0.3.0, provider-plugin-architecture): explains how Whisper's encoder-decoder architecture actually works, walks through Azure OpenAI and Groq end-to-end, adds a new native api.openai.com provider using the existing OpenAICompatProvider mixin (the current release only ships azure/groq for the OpenAI family), surveys the other 25+ registered providers, and documents the automatic large-file chunking pipeline. Also flags a real, source-verified gotcha: the azure provider calls Whisper's /audio/translations endpoint rather than /audio/transcriptions, so non-English audio comes back translated into English rather than transcribed in its original language. Adds a first-time author profile and a new speech-to-text definition per CONTRIBUTING.md. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01MQvd9avZenwbcumQHWuusX Signed-off-by: Spaceben123 --- authors/benjamin_eikrem.md | 8 + .../20260905_definition_speech-to-text.md | 47 ++ ...scribing_audio_with_whisper_openai_groq.md | 699 ++++++++++++++++++ 3 files changed, 754 insertions(+) create mode 100644 authors/benjamin_eikrem.md create mode 100644 definitions/20260905_definition_speech-to-text.md create mode 100644 guides/20260905_guide_transcribing_audio_with_whisper_openai_groq.md diff --git a/authors/benjamin_eikrem.md b/authors/benjamin_eikrem.md new file mode 100644 index 00000000..a456b14e --- /dev/null +++ b/authors/benjamin_eikrem.md @@ -0,0 +1,8 @@ +Author: Benjamin Eikrem +Title: Software Developer +Description: Benjamin Eikrem is a developer who writes technical guides and tooling documentation, with an interest in AI engineering workflows and developer tooling. This guide was researched and drafted with AI assistance (Claude/Fable), grounded in a direct read of the source repository it documents. +Company Name: +Company Description: +Author Image: +Company Logo Dark: +Company Logo White: diff --git a/definitions/20260905_definition_speech-to-text.md b/definitions/20260905_definition_speech-to-text.md new file mode 100644 index 00000000..d3573c2d --- /dev/null +++ b/definitions/20260905_definition_speech-to-text.md @@ -0,0 +1,47 @@ +--- +title: 'Speech-to-Text (STT)' +description: + 'The process of converting spoken audio into written text, typically using + a trained acoustic/language model such as OpenAI Whisper.' +date: 2026-09-05 +author: 'Benjamin Eikrem' +--- + +# Speech-to-Text (STT) + +## Definition + +Speech-to-text (STT) is the process of automatically converting spoken audio +into written text. Modern STT systems are neural networks trained on large +paired datasets of audio and matching transcripts. Rather than using +hand-built acoustic rules, they learn to map raw or preprocessed audio (most +commonly a log-mel spectrogram — a time/frequency representation of the +sound) directly to sequences of text tokens. + +Not all STT models share the same architecture. Encoder-decoder transformer +models such as OpenAI's Whisper generate text autoregressively, one token at +a time, conditioned on the audio. CTC-based models (Connectionist Temporal +Classification), such as NVIDIA's Parakeet, instead predict a character or +token for each audio frame directly and collapse repeated/blank predictions +into the final transcript — a different, generally faster but historically +less flexible approach for handling variable-length alignment between audio +and text. + +## Context and Usage + +STT underpins most transcription tooling: meeting-recording transcripts, +podcast show notes, voice assistants, subtitle generation, and call-center +analytics all depend on it as a first step before any text-based processing +(search, summarization, sentiment analysis) can happen. + +In practice, "STT" and "the model behind it" are often conflated. A CLI or +SDK might expose one uniform `transcribe()`-style interface across many +providers, but the acoustic model actually doing the work can differ +significantly between them — some are literally OpenAI's open Whisper +weights hosted on different infrastructure (OpenAI's own API, Azure OpenAI, +Groq's LPU hardware, or run entirely offline via `whisper.cpp`), while +others are unrelated proprietary models that simply expose a +Whisper-compatible request/response shape for drop-in compatibility. When +comparing STT providers on accuracy or latency, it's worth checking which +underlying model each one actually runs rather than assuming "Whisper API" +always means the same acoustic model. diff --git a/guides/20260905_guide_transcribing_audio_with_whisper_openai_groq.md b/guides/20260905_guide_transcribing_audio_with_whisper_openai_groq.md new file mode 100644 index 00000000..350050cb --- /dev/null +++ b/guides/20260905_guide_transcribing_audio_with_whisper_openai_groq.md @@ -0,0 +1,699 @@ +--- +title: 'Transcribing Audio with Whisper via OpenAI and Groq' +description: + 'A deep, step-by-step guide to SAPAT: how Whisper transcription actually + works under the hood, and how to run it with Azure OpenAI, Groq, and 25+ + other speech-to-text providers.' +date: 2026-09-05 +author: 'Benjamin Eikrem' +tags: ['whisper', 'transcription', 'openai', 'groq', 'ai engineering'] +--- + +# Transcribing Audio with Whisper via OpenAI and Groq + +## Introduction + +[Speech-to-text (STT)](/definitions/20260905_definition_speech-to-text.md) has +quietly become one of the most useful primitives in an AI engineer's toolkit. +Meeting recordings, podcasts, lecture videos, and user-research calls all +arrive as audio or video, and almost every downstream workflow — search, +summarization, translation, feeding a transcript into an +[LLM](/definitions/20241219_definition_llm.md) — needs text first. OpenAI's +Whisper model became the de-facto standard for this because it is open, +accurate across dozens of languages, and available both as a hosted +[API](/definitions/20241212_definition_api.md) and as weights you can run +yourself. + +This guide walks through [SAPAT](https://github.com/nkkko/sapat) (Speech +Audio Processing And Transcription), a command-line tool that wraps Whisper +and Whisper-compatible transcription services behind one consistent +interface. Rather than treating it as a black box, we're going to open it up: +you'll learn exactly how the CLI is structured, how audio actually flows +through Whisper's encoder-decoder architecture, and how to run real +transcriptions against Azure OpenAI's Whisper deployment and Groq's +Whisper endpoint — the two APIs this repo's maintainer specifically asked to +be covered. Along the way we'll also extend the tool to talk to OpenAI's +`api.openai.com` endpoint directly (which, as you'll see, the current release +doesn't ship out of the box), and survey the other 25+ providers SAPAT +already supports. + +Everything below is grounded in a direct read of the `sapat` source at commit +`ba900b7` (the `feat/provider-plugin-architecture` merge, package version +`0.3.0`) — not the project's README, which describes an older CLI shape. +Where the two disagree, this guide follows the code. + +Prerequisites: + +- A [Daytona](https://github.com/daytonaio/daytona) account, or Daytona + installed locally, to spin up a disposable + [development environment](). +- `ffmpeg` (SAPAT shells out to it for every audio conversion and split). +- At least one of: an Azure OpenAI resource with a Whisper deployment, or a + free [Groq Cloud](https://console.groq.com) API key. Groq's free tier is + the fastest way to follow along without billing setup. +- Basic comfort with the command line and Python virtual environments. + +## TL;DR + +- **SAPAT** is a Python CLI that converts video/audio to MP3 (or another + format), sends it to a speech-to-text provider, and writes a `.txt` + transcript — with automatic chunking for files over a provider's size + limit. +- **Whisper** is a transformer encoder-decoder trained on 680,000 hours of + audio. It works on 30-second log-mel spectrogram windows, not raw audio, + and was trained on transcription, translation, language ID, and timestamp + prediction simultaneously — which explains both its strengths and its + well-known hallucination quirks. +- SAPAT's current provider registry includes `azure` (Azure OpenAI's hosted + Whisper) and `groq` (Groq's LPU-accelerated Whisper) — the two "big" APIs + the source asks about — plus 25+ others, some of which are Whisper under + the hood and some of which are not. +- There is **no plain `openai` provider registered today** despite the + README implying one. We'll add one in Step 5 using the same + `OpenAICompatProvider` mixin the codebase already uses for Together, + Venice, and Lemonfox. +- Run it with: `sapat meeting.mp4 --provider groq --language en --quality H`. + +## How Whisper Actually Works + +Before running any commands, it's worth understanding what's actually +happening when you call a "Whisper" endpoint, because SAPAT talks to at least +four meaningfully different things and calls all of them "transcription." + +### The model itself + +Whisper is a sequence-to-sequence transformer, not a specialized speech +model with hand-built acoustic features. Audio is first resampled to 16 kHz +and converted into an 80-channel log-mel spectrogram — a 2D representation +of frequency energy over time that compresses raw waveform samples into +something a transformer can attend over efficiently. That spectrogram is +sliced into fixed 30-second windows (padded with silence if the input is +shorter), because the model was trained on fixed-length windows and has no +native mechanism for variable-length audio. + +An **encoder** stack of transformer blocks turns each 30-second spectrogram +window into a sequence of hidden states. A **decoder** stack then generates +text tokens autoregressively, conditioned on those hidden states — the same +way GPT-style models generate text conditioned on a prompt, except the +"prompt" here is audio, not text. This is why the `--transcription-prompt` +flag in SAPAT (and the `prompt` field every provider sends) actually works: +Whisper was trained to optionally condition on preceding text context, so +feeding it prior vocabulary (names, jargon, spellings) measurably improves +accuracy on that vocabulary. + +Whisper was also trained as a **multitask** model. The same weights handle +transcription, translation-to-English, language identification, and +timestamp prediction, all controlled by special tokens the decoder emits at +the start of generation. This is significant for two practical reasons: + +1. It's why passing the wrong `--language` code doesn't necessarily produce + an error — the model will often silently transcribe in the language it + *thinks* it detects, which can look like garbage output when your audio + has heavy accents, code-switching, or background noise the model + mistakes for a different language's phonetics. +2. It's the root cause of Whisper's most infamous failure mode: + **hallucination during silence**. Because the model always emits *some* + text for every 30-second window (it was never trained to emit "nothing"), + long stretches of silence or non-speech audio can produce fabricated, + repetitive, or copyright-notice-like text. SAPAT doesn't do voice-activity + detection before transcribing, so if your source video has long silent + gaps, expect to see this in your output. + +### Same weights, different hosts + +This is the part most walkthroughs skip, and it's exactly what the issue +behind this guide asked to have explained: **"Whisper" is not one API, it's +one set of open model weights that multiple companies host differently.** + +- **OpenAI's hosted API** (`api.openai.com/v1/audio/transcriptions`) runs + Whisper (specifically `whisper-1`, alongside newer `gpt-4o-transcribe` + models) on OpenAI's own infrastructure. +- **Azure OpenAI** runs the *same* Whisper weights, but on Microsoft's Azure + infrastructure, behind a per-resource deployment you create and name + yourself (`AZURE_OPENAI_STT_MODEL_NAME`), with a different URL shape + (`/openai/deployments/{model}/audio/translations`) and Azure-native + authentication (an `api-key` header instead of a bearer token). +- **Groq** also runs literal Whisper weights (`whisper-large-v3` and a + distilled `distil-whisper-large-v3-en`), but on Groq's custom LPU + (Language Processing Unit) hardware instead of GPUs. The model's outputs + are essentially the same as OpenAI's hosted Whisper for the same weights + and audio — what Groq sells is raw throughput: transcription that would + take real-time-or-slower on a GPU often comes back in a fraction of the + audio's actual duration. +- **whisper.cpp and WhisperX**, two of SAPAT's local providers, run Whisper + *weights you download yourself* entirely offline, using optimized C++ + (ggml/GGUF quantized) or a Python pipeline with forced alignment, + respectively. No API key, no network call, no per-minute cost — at the + expense of needing local compute and, for whisper.cpp, correctly + installing a compiled binary. + +Meanwhile, several of SAPAT's other providers expose the *same* +`/audio/transcriptions`-shaped API but are **not** Whisper at all under the +hood: Mistral's provider calls their `voxtral-mini-latest` model (a +different architecture Mistral trained specifically for audio), ElevenLabs +transcribes with their own `scribe_v2` model, Google's provider goes through +Gemini's multimodal understanding rather than a dedicated STT model, and +NVIDIA's provider defaults to `parakeet-ctc-0.6b-asr`, a CTC-based model +architecture that predates and works differently from Whisper's +encoder-decoder design. Vosk (offline) is built on Kaldi, a much older +statistical/neural hybrid ASR toolkit unrelated to Whisper. + +SAPAT flattens all of this behind one `transcribe()` interface, which is +convenient — but it means the provider name in your `.env` file tells you +who's billing you, not which acoustic model actually produced your +transcript. Keep that distinction in mind when you're debugging accuracy +differences between providers. + +### Why there's a file-size limit at all + +Whisper-family hosted APIs cap uploads at 25 MB. That's not an arbitrary +platform limit — it's a practical ceiling on how much audio a single +inference request should carry, driven by upload time, request timeout +budgets, and the fact that a 30-second-windowed model gets no real accuracy +benefit from being handed hours of audio in one call. This is precisely why +SAPAT ships its own chunking logic, which we'll cover in Step 6. + +## Step 1: Set Up Your Development Environment + +The cleanest way to follow along is inside a disposable Daytona workspace, so +none of this touches your host machine's Python environment. + +Install Daytona if you haven't already: + +```bash +curl -L https://download.daytona.io/daytona/install.sh | sudo bash +``` + +Create a workspace directly from the SAPAT repository: + +```bash +daytona create https://github.com/nkkko/sapat --code +``` + +This clones the repo and opens it in your IDE inside an isolated +[development environment](). +The repo also ships a `.devcontainer/devcontainer.json` that installs +`ffmpeg` and the Python requirements automatically via `postCreateCommand` if +your editor supports Dev Containers — Daytona will pick this up. + +If you'd rather set things up manually (inside the workspace or on your own +machine): + +```bash +git clone https://github.com/nkkko/sapat.git +cd sapat + +python -m venv .venv +source .venv/bin/activate # .venv\Scripts\activate on Windows + +pip install -r requirements.txt +pip install -e . +``` + +`pip install -e .` registers the `sapat` console script (defined in +`pyproject.toml` as `sapat = "sapat.cli:main"`) so you can run `sapat` from +anywhere in the environment instead of `python -m sapat`. + +Confirm `ffmpeg` and `ffprobe` are both on your `PATH` — SAPAT calls both of +them directly via `subprocess`, and it will fail with a clear error if +they're missing: + +```bash +sudo apt update && sudo apt install -y ffmpeg # Debian/Ubuntu-based images +ffmpeg -version +ffprobe -version +``` + +Finally, copy the example environment file and fill in only the providers +you plan to use — SAPAT only registers a provider if its required +environment variables are present, so unused providers simply don't show up: + +```bash +cp .env.example .env +``` + +## Step 2: How SAPAT Is Put Together + +It's worth spending two minutes on the architecture before you run anything, +because it explains every flag you'll use later. + +``` +sapat/ +├── cli.py # Click entry point; discovers providers, parses flags +├── process.py # Orchestrates: convert -> split (if needed) -> transcribe -> correct -> write +├── convert.py # ffmpeg wrappers: MP3 (3 quality tiers), WAV, FLAC, Ogg Opus +├── audio_splitter.py # ffprobe-based chunking for files over a provider's size limit +└── providers/ + ├── base.py # TranscriptionProvider ABC + ProviderConfig dataclass + ├── __init__.py # Registry: auto-discovers every module in this package + ├── openai_compat.py # Shared mixin for OpenAI-shaped /audio/transcriptions APIs + ├── azure.py, groq.py, mistral.py, gemini.py, ... (30 provider modules) +``` + +The registry in `sapat/providers/__init__.py` walks every module in the +`providers/` package with `pkgutil.iter_modules`, imports it, and lets each +module's own `@register` decorator add itself to a dict — but only if +`TranscriptionProvider.is_available()` returns `True` for it, which checks +that every environment variable in that provider's `ProviderConfig` is set. +This is why setting `GROQ_API_KEY` alone is enough to make `--provider groq` +appear as a valid option: nothing needs to be hardcoded in `cli.py` itself. + +`process.py` is the actual pipeline: + +``` +input.mp4 + │ convert_audio() (ffmpeg: video/audio -> provider's preferred format) + ▼ +input.mp3 + │ should_split_file()? (ffprobe: is it over the provider's max_file_size_mb?) + ├── no -> provider.transcribe() + └── yes -> split_audio_file() -> provider.transcribe() per chunk -> join text + ▼ +result.text + │ --correct flag AND provider.config.supports_correction? + ├── yes -> provider.correct_transcript() (LLM cleanup pass) + └── no -> (warns and skips if --correct was requested but unsupported) + ▼ +input.txt (written next to the source file, temp audio file deleted) +``` + +Every provider subclasses `TranscriptionProvider` (in `providers/base.py`) +and implements one required method, `transcribe()`, plus an optional +`correct_transcript()` and `resolve_model()`. `ProviderConfig` is a small +dataclass each provider fills in with its required env vars, its Whisper +file-size ceiling, its preferred audio format, and its default model string. +This plugin shape is exactly why "add support for another API" (which we'll +do in Step 5) is a same-day task, not a rearchitecture. + +## Step 3: Transcribing with Azure OpenAI's Whisper + +The README calls this simply "OpenAI," but reading `sapat/providers/azure.py` +shows it specifically targets **Azure OpenAI**, not `api.openai.com`. If your +API key starts with `sk-` and you're calling `api.openai.com` directly, +skip ahead to Step 5 — this section is for an Azure OpenAI resource with a +Whisper model deployed to it. + +In the Azure portal, deploy a `whisper` model to your Azure OpenAI resource +and note the deployment name, your resource endpoint, and an API version +(the code defaults expect `2024-06-01` for the STT deployment). Add these to +your `.env`: + +```env +AZURE_OPENAI_API_KEY=your_azure_api_key +AZURE_OPENAI_ENDPOINT=https://YOUR-RESOURCE.openai.azure.com +AZURE_OPENAI_STT_MODEL_NAME=whisper +AZURE_OPENAI_STT_API_VERSION=2024-06-01 + +# Only needed if you'll use --correct with this provider +AZURE_OPENAI_DEPLOYMENT_NAME_CHAT=gpt-4o +AZURE_OPENAI_API_VERSION_CHAT=2023-03-15-preview +``` + +Run a transcription: + +```bash +sapat meeting.mp4 --provider azure --language en --quality M +``` + +Reading `AzureProvider.transcribe()`, here's what actually happens on the +wire: SAPAT builds a URL of the form +`{endpoint}/openai/deployments/{model}/audio/translations?api-version={version}`, +sends your API key in an `api-key` header (Azure's convention, not a +`Bearer` token like every other provider in this codebase), and multipart- +uploads the converted MP3 along with `response_format`, `temperature`, and +optionally `language` and `prompt` fields. The response is Whisper's plain +`{"text": "..."}` JSON, same shape you'd get from `api.openai.com`. + +Look closely at that URL: it hits Whisper's **`/audio/translations`** +endpoint, not `/audio/transcriptions`. That's not a typo in this guide — it's +in the shipped code. The practical difference matters: `/audio/translations` +always returns **English** text, translating from whatever language the +source audio is in, whereas `/audio/transcriptions` returns text in the +audio's original spoken language. If you point `--provider azure` at +non-English audio expecting a same-language transcript, you'll get an +English translation instead — see the Troubleshooting section below. + +Azure is also the only provider besides Mistral with +`supports_correction=True`. Add `--correct` and, after transcription +finishes, SAPAT makes a second call — this time to your `gpt-4o` chat +deployment — with a system prompt instructing it to fix spelling and +punctuation without changing wording. This is a genuinely useful pass for +proper nouns and product names Whisper tends to mangle, but it costs an +extra LLM call and can occasionally over-correct correctly-transcribed +slang or code terms, so treat it as a suggestion, not a source of truth. + +## Step 4: Transcribing with Groq + +Groq is the fastest way to see SAPAT work end-to-end, since Groq Cloud's +free tier needs nothing but an email signup. Grab a key from +[console.groq.com](https://console.groq.com/keys) and add it to `.env`: + +```env +GROQ_API_KEY=your_groq_api_key +``` + +Run: + +```bash +sapat meeting.mp4 --provider groq --model w --language en +``` + +`GroqProvider.resolve_model()` (in `sapat/providers/groq.py`) maps short +aliases to real model IDs: `w` and `whisper` both resolve to +`whisper-large-v3` (Groq's full-accuracy model), and `dw` resolves to +`distil-whisper-large-v3-en` — a distilled, English-only variant that trades +a small amount of accuracy for noticeably lower latency and cost. If you +skip `--model` entirely, `cli.py` falls back to the provider's +`config.default_model`, which for Groq is `whisper-large-v3`. + +Under the hood, `GroqProvider.transcribe()` posts directly to +`https://api.groq.com/openai/v1/audio/transcriptions` with a standard +`Authorization: Bearer` header — notice the URL literally contains +`/openai/`, because Groq deliberately mirrors OpenAI's request/response +schema so existing OpenAI SDK code (and tools like SAPAT) can point at Groq +with almost no changes. `max_file_size_mb` is set to 25.0, same ceiling as +OpenAI's own API, and `supports_correction=False` — Groq's provider in this +codebase does not currently implement a correction pass, so `--correct` with +`--provider groq` will print a warning and transcribe without correction. + +For a rough sense of the speed difference: transcribing a 10-minute audio +file that takes 60-90 seconds against a typical GPU-hosted Whisper endpoint +frequently completes in single-digit seconds on Groq's LPUs, for the exact +same model weights and (in practice) near-identical output text. This is +the main reason to reach for `--provider groq` over Azure when you're +iterating quickly on transcripts. + +## Step 5: Adding Direct OpenAI (`api.openai.com`) Support + +This is the extension the underlying issue specifically asked for. As of +this codebase's `0.3.0` release, `sapat/providers/` has no module for +`api.openai.com` — only `azure.py` for Azure-hosted Whisper. (There *is* a +`src/sapat/transcription/openai.py` file in the repo, but it's leftover code +from an earlier, pre-plugin architecture — it's not under the `sapat/` +package that `pyproject.toml` actually builds and installs, so it's dead +code the CLI never imports.) + +Adding a real, registered `openai` provider is a good exercise because it's +almost entirely boilerplate, thanks to the `OpenAICompatProvider` mixin +already used by `together.py`, `venice.py`, `xai.py`, and `lemonfox.py`. +Here's `lemonfox.py` in full, as a template: + +```python +# sapat/providers/lemonfox.py +from sapat.providers import register +from sapat.providers.base import AudioFormat, ProviderConfig +from sapat.providers.openai_compat import OpenAICompatProvider + + +@register +class LemonfoxProvider(OpenAICompatProvider): + name = "lemonfox" + config = ProviderConfig( + required_env_vars=["LEMONFOX_API_KEY"], + default_model="whisper-1", + ) + base_url = "https://api.lemonfox.ai/v1/audio/transcriptions" + _env_key_for_auth = "LEMONFOX_API_KEY" +``` + +`OpenAICompatProvider.transcribe()` already does the multipart upload, +`Bearer` auth header construction, and JSON response parsing for you — a +subclass only needs to set `base_url` and `_env_key_for_auth`. Create +`sapat/providers/openai.py` with the direct OpenAI equivalent: + +```python +# sapat/providers/openai.py +# ABOUTME: Direct OpenAI transcription provider (api.openai.com, not Azure) + +from sapat.providers import register +from sapat.providers.base import AudioFormat, ProviderConfig +from sapat.providers.openai_compat import OpenAICompatProvider + + +@register +class OpenAIProvider(OpenAICompatProvider): + name = "openai" + config = ProviderConfig( + required_env_vars=["OPENAI_API_KEY"], + max_file_size_mb=25.0, + preferred_format=AudioFormat.MP3, + supports_correction=False, + default_model="whisper-1", + ) + base_url = "https://api.openai.com/v1/audio/transcriptions" + _env_key_for_auth = "OPENAI_API_KEY" + + def resolve_model(self, model_alias: str) -> str: + aliases = { + "w": "whisper-1", + "whisper": "whisper-1", + "gpt4o": "gpt-4o-transcribe", + "gpt4o-mini": "gpt-4o-mini-transcribe", + } + return aliases.get(model_alias, model_alias) +``` + +Because the registry in `providers/__init__.py` discovers every module in +the package automatically via `pkgutil.iter_modules`, you don't need to +touch `cli.py`, `process.py`, or any registration list — dropping this file +in and setting `OPENAI_API_KEY` in `.env` is sufficient for `--provider +openai` to appear: + +```env +OPENAI_API_KEY=sk-your-real-openai-key +``` + +```bash +sapat meeting.mp4 --provider openai --model whisper --language en +``` + +Two things worth flagging if you build on this: first, OpenAI's newer +`gpt-4o-transcribe` and `gpt-4o-mini-transcribe` models (mapped above via +the `gpt4o`/`gpt4o-mini` aliases) are not classic Whisper — they're +GPT-4o-family audio-input models repurposed for transcription, generally +more accurate on noisy audio but with different pricing and rate limits +than `whisper-1`. Second, `OpenAICompatProvider._build_data()` always sends +a `language` field; OpenAI's API accepts this, but if you extend the mixin +for a provider that rejects unexpected fields, you'll need to override +`_build_data()` rather than assuming every "OpenAI-compatible" API is +byte-for-byte compatible. + +## Step 6: The Other 25+ Providers + +SAPAT's provider list is much broader than "OpenAI and Groq." Reading every +file in `sapat/providers/`, here's what's actually registered, split by +whether the provider is genuinely running Whisper weights or something else +entirely: + +| Provider (`--provider`) | Underlying model | Whisper? | Notes | +| --- | --- | --- | --- | +| `azure` | Whisper (Azure-hosted) | Yes | See Step 3 | +| `groq` | `whisper-large-v3` | Yes | See Step 4 | +| `deepinfra` | `openai/whisper-large-v3` | Yes | OpenAI-compatible endpoint | +| `together` | `openai/whisper-large-v3` | Yes | OpenAI-compatible endpoint | +| `venice` | `whisper-large-v3` | Yes | Privacy-focused hosting | +| `xai` | `whisper-large-v3` | Yes | OpenAI-compatible endpoint | +| `lemonfox` | `whisper-1` | Yes | OpenAI-compatible endpoint | +| `whisper_cpp` | Local ggml/GGUF Whisper | Yes | Fully offline, needs a compiled binary | +| `whisperx` | Local Whisper + alignment | Yes | Offline, adds word-level timestamps | +| `mistral` | `voxtral-mini-latest` | No | Mistral's own audio model; supports correction | +| `elevenlabs` | `scribe_v2` | No | ElevenLabs' own STT model | +| `gemini` | `gemini-2.0-flash` | No | Multimodal LLM, not a dedicated STT model | +| `nvidia` | `parakeet-ctc-0.6b-asr` | No | CTC architecture, not encoder-decoder | +| `baidu` | Baidu's short-form ASR | No | Requires `BAIDU_API_KEY` + `BAIDU_SECRET_KEY` | +| `cloudflare` | `@cf/openai/whisper` | Yes | Whisper via Cloudflare Workers AI | +| `sarvam` | `saaras:v3` | No | Focused on Indian languages | +| `falai` | `fal-ai/whisper` | Yes | Whisper via fal.ai's inference platform | +| `soniox` | `stt-async-v4` | No | Soniox's own real-time-capable model | +| `symbl` | Symbl's ASR | No | Async job-based API | +| `gladia` | Gladia's ASR | No | Async job-based API | +| `speechmatics` | Speechmatics' ASR | No | Async job-based API | +| `yandex` | Yandex SpeechKit | No | Configurable max file size via env var | +| `oracle` | OCI AI Speech | No | Uploads audio to an OCI bucket first | +| `replicate` | `openai/whisper` | Yes | Whisper hosted on Replicate | +| `vosk` | Vosk (Kaldi-based) | No | Fully offline, no API key needed | +| `moonshine` | Moonshine (Useful Sensors) | No | Fully offline, tiny footprint | +| `picovoice` | Leopard | No | Fully offline, needs `PICOVOICE_ACCESS_KEY` | + +A few of these — `oracle`, `gladia`, `speechmatics`, and `symbl` — subclass a +different base, `AsyncPollProvider` (`sapat/providers/async_poll.py`), +because their APIs are job-based: you submit audio, get back a job ID, and +poll until the transcript is ready, rather than getting text back +synchronously in one HTTP response. If you're extending SAPAT for a new +async provider, that base class — not `OpenAICompatProvider` — is the one to +subclass. + +## Step 7: Handling Large Files + +Whisper-family APIs cap uploads around 25 MB, but a one-hour meeting +recording at reasonable quality is easily 3-4x that. Rather than making you +split files by hand, `process_file()` in `process.py` checks +`should_split_file()` (in `audio_splitter.py`) against the active provider's +`max_file_size_mb`, and automatically kicks off chunking when needed — you +don't pass any extra flag for this, it's automatic: + +```bash +sapat long_lecture.mp4 --provider groq --quality H +``` + +Here's exactly what `split_audio_file()` does, since it's a genuinely +useful technique beyond just this tool: + +1. Runs `ffprobe` to get the file's duration and bitrate. +2. Computes a target segment duration: + `segment_duration = (max_size_mb * 1024 * 1024 * 8) / bitrate_bps` + — i.e., "how many seconds of audio at this bitrate fit under the size + ceiling" — with a 30-second floor so you don't end up with hundreds of + tiny chunks on very high-bitrate audio. +3. Runs a single `ffmpeg -f segment -segment_time -c copy` command, + which uses **stream copy** (`-c copy`) rather than re-encoding — this is + fast (no re-compression) and lossless, since it just cuts the compressed + audio stream at the nearest keyframe/packet boundary. +4. Transcribes each chunk independently and joins the resulting text with + spaces. If any individual chunk's request fails, that chunk is replaced + with a `[Chunk N transcription failed]` marker rather than aborting the + whole job — worth knowing so a failed chunk in the middle of your + transcript doesn't silently disappear. +5. Deletes the temporary chunk directory afterward, whether or not + transcription succeeded. + +One consequence worth knowing: because chunking happens on ffmpeg keyframe +boundaries and each chunk is transcribed independently with no +overlap, a sentence that happens to span exactly across a chunk boundary can +occasionally be split awkwardly in the output text. For most long-form +content (meetings, lectures, podcasts) this is a minor, rare cosmetic issue, +not a correctness bug — but if you need perfectly seamless chunk joins for a +downstream use case, consider adding a small overlap window and de-duplicating +the overlapping text after transcription. + +## Full CLI Reference + +Read directly from `sapat/cli.py`: + +| Flag | Short | Description | +| --- | --- | --- | +| `INPUT_PATH` | (positional) | A single video/audio file, or a directory (only `.mp4` files inside it are processed) | +| `--provider` | `-p` | Provider name; if omitted, defaults to `azure` if available, else the first available provider alphabetically | +| `--model` | `-m` | Model ID or provider-specific alias (e.g. `w`, `dw` for Groq); defaults to the provider's `default_model` | +| `--language` | `-l` | Language code, default `en` | +| `--transcription-prompt` | `-t` | Optional prompt text passed to the model for vocabulary/context conditioning | +| `--temperature` | `-temp` | Sampling temperature, `0`-`1`, default `0` | +| `--quality` | `-q` | MP3 conversion quality: `L` (22.05kHz mono, 96kbps), `M` (44.1kHz mono, 96kbps), or `H` (44.1kHz stereo, 160kbps); default `L` | +| `--correct` | — | Run an LLM correction pass after transcription, if the provider supports it | +| `--version` | — | Print the installed SAPAT version | + +Note the default quality is `L` in the current code — the README's older +example (`--quality M` default) is stale, another spot where the shipped +CLI and the README have drifted apart. For anything you plan to actually +read closely (as opposed to feeding straight into an LLM for summarization), +`--quality M` or `H` is worth the extra bandwidth. + +## Common Issues and Troubleshooting + +**Problem:** `click.ClickException: No providers available.` + +**Solution:** No provider's required environment variables are set. Check +that your `.env` file is in the directory you're running `sapat` from (it's +loaded via `python-dotenv`, relative to the current working directory, not +the repo root), and that variable names match exactly — e.g. `GROQ_API_KEY`, +not `GROQCLOUD_API_KEY` (the older README-documented name that the current +`groq.py` does not read). + +**Problem:** `RuntimeError: ffmpeg and ffprobe are required for splitting +large audio files.` + +**Solution:** `ffmpeg` isn't on `PATH`, or only one of `ffmpeg`/`ffprobe` is +installed. Both ship together in the standard `ffmpeg` package on every +major OS, so `sudo apt install ffmpeg` (or `brew install ffmpeg`) covers +both. + +**Problem:** Azure requests fail with a 404 or "deployment not found." + +**Solution:** `AZURE_OPENAI_STT_MODEL_NAME` must exactly match the +**deployment name** you chose in the Azure portal, not the underlying model +name — Azure lets you name a Whisper deployment anything, and the URL SAPAT +builds (`/openai/deployments/{model}/...`) uses whatever you resolve to +here. + +**Problem:** `--provider azure` on non-English audio comes back translated +into English instead of transcribed in the original language. + +**Solution:** This isn't a misconfiguration — `azure.py` calls Whisper's +`/audio/translations` endpoint rather than `/audio/transcriptions`, and +translation endpoints always output English by design, regardless of the +`--language` value you pass. If you need same-language transcripts for +non-English audio, use `--provider groq` (which correctly calls +`/audio/transcriptions`) or the `openai` provider from Step 5 instead. + +**Problem:** `--correct` silently does nothing. + +**Solution:** Check the console output for a yellow warning — +`process_file()` only runs correction when +`provider.config.supports_correction` is `True`. Today that's only `azure` +and `mistral`. Passing `--correct` with `--provider groq` (or most other +providers) prints a warning and continues without correcting, by design, +rather than failing the whole run. + +**Problem:** Transcript has repeated phrases or text during quiet sections. + +**Solution:** This is Whisper hallucinating during silence, as covered +above — it's a known model behavior, not a SAPAT bug. Trimming long silent +gaps before transcription (e.g., with `ffmpeg`'s `silenceremove` filter) or +post-processing to collapse exact-duplicate repeated sentences are the +usual mitigations. + +**Problem:** Large-file chunking finishes, but a sentence near a chunk +boundary looks cut off or duplicated. + +**Solution:** Expected, per the chunking discussion above — chunks are +transcribed independently with no overlap. For most use cases this is +cosmetic. + +## Real-World Use Cases + +**Meeting and interview archives.** Point SAPAT at a folder of recorded +calls (`sapat ./recordings/ --provider groq --quality M`) to batch-produce +searchable text transcripts you can grep, diff, or feed into a +retrieval pipeline, without manually transcribing anything. + +**Podcast show notes.** Run a higher-quality conversion +(`--quality H`) and pair it with `--correct` on the `azure` provider to get +a cleaned-up transcript that's closer to publish-ready than raw Whisper +output, then hand it to an LLM to summarize into show notes. + +**Multilingual content review.** Because every provider accepts a +`--language` flag, you can transcribe the same interview across multiple +target languages (where the provider supports translation) or run the same +audio through two providers to sanity-check dialect/accent handling before +committing to one vendor for a larger batch job. + +**Local-first / offline pipelines.** For sensitive audio you don't want +leaving your machine, `whisper_cpp` or `whisperx` give you the exact same +Whisper transcription quality as the hosted APIs, with zero network calls — +useful for compliance-sensitive interview or legal-recording workflows. + +## Conclusion + +SAPAT is a small tool, but its provider-plugin architecture is a genuinely +good pattern to learn from: one abstract base class, one dataclass for +config, and an auto-discovering registry mean adding a new speech-to-text +backend is a 15-line file, not a fork. More importantly, "Whisper" turned +out to be several different things wearing the same name — the same open +weights hosted three different ways (OpenAI, Azure, Groq), plus local +offline variants, plus a long tail of providers that mimic Whisper's API +shape while running entirely different models underneath. Understanding +that distinction is what actually helps you debug accuracy or latency +differences between providers, rather than just swapping `--provider` +values and hoping. + +From here, a natural next step is contributing the `openai.py` provider from +Step 5 back upstream, or wiring up SAPAT inside a CI job that transcribes +recorded demo videos automatically as part of a documentation pipeline. + +## References + +- [SAPAT source repository](https://github.com/nkkko/sapat) +- [Groq Cloud API keys](https://console.groq.com/keys) +- [Azure OpenAI Whisper documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/whisper-quickstart) +- [OpenAI Whisper paper: "Robust Speech Recognition via Large-Scale Weak Supervision"](https://arxiv.org/abs/2212.04356) +- [Daytona Installation Guide](https://daytona.io/docs/installation/installation/)