🗣️ feat: First-Class gpt-transcribe and Live gpt-live-transcribe Speech-to-Text

[Nk-rodrigues]

Summary

Related to #14503 (claimed in #14503 (comment)).

Speech-to-text could only send a finished recording with a singular language. OpenAI's recommended transcription models don't fit that: gpt-transcribe replaces language with languages[] and adds keywords[], and gpt-live-transcribe only works over a Realtime session. Setting either name as stt.openai.model therefore neither uses the new hints nor gives live dictation.

Recorded dictation now builds the request from the model. gpt-transcribe and gpt-live-transcribe (including snapshots) receive the admin's optional prompt, keywords, and languages and never the singular language; when the user has picked a speech language, languages narrows to it. Older models on the openai provider (Whisper, gpt-4o(-mini)-transcribe) keep the singular language and also receive the optional prompt; the azureOpenAI provider is unchanged. A response that carries detected-language metadata still resolves to its text.

Live dictation is opt-in through a new speech.stt.openai.realtime block. When it's configured, Settings > Speech > Engine offers a third option, "Live", and speechTab.speechToText.engineSTT: 'realtime' can make it the default. A Live take streams transcript deltas into the composer while the user speaks. Stop, or the existing silence auto-stop, commits the audio, and the final transcript stays editable. Auto Send, Cancel, the elapsed timer, and the draft that was in the composer before recording behave as they do for External. The composer shows "Connecting" until audio can reach the provider and "Reconnecting" while WebRTC recovers. Permission, connection, and provider failures each get a toast and keep whatever text was already heard.

Out of scope: STT usage accounting (#13308 adds it for all non-chat models) and stream=true for recorded clips (optional in the issue; Live already covers progressive text).

How it works

The browser never holds a provider credential, not even an ephemeral one. It posts its WebRTC offer to LibreChat. The server uses the configured key to mint a 60-second client secret, spends that secret on the SDP exchange, and returns only the SDP answer. Media and transcript events then flow between the browser and the provider over WebRTC. Azure OpenAI's /realtime/calls accepts only ephemeral secrets, so this flow works there too.

sequenceDiagram
    participant Browser
    participant API as LibreChat API
    participant Provider as OpenAI / Azure OpenAI
    Browser->>API: POST /api/files/speech/stt/realtime { sdp, language? }
    API->>Provider: POST /realtime/client_secrets (API key, transcription session)
    Provider->>API: ek_... (60 s)
    API->>Provider: POST /realtime/calls (offer, ephemeral secret)
    Provider->>API: SDP answer
    API->>Browser: { sdp }
    Browser->>Provider: microphone audio (WebRTC)
    Provider->>Browser: transcription delta / completed events (data channel "oai-events")
    Browser->>Provider: input_audio_buffer.commit on Stop or silence

Loading

The session is transcription-only with turn_detection: null. gpt-live-transcribe has no server VAD, and for older models this stops a pause from splitting one dictation into several items. The route sits behind JWT auth and the existing STT IP and user limiters. Both upstream calls go through the proxy config and the SSRF guard, which honors stt.allowedAddresses. Failures come back as coded errors (stt_realtime_not_configured, stt_realtime_invalid_offer, stt_realtime_provider_error), and provider errors are logged with getSafeErrorMetadata.

speech:
  stt:
    openai:
      apiKey: '${STT_API_KEY}'
      model: 'gpt-transcribe'
      keywords: ['LibreChat', 'OpenShift']
      languages: ['de', 'en']
      realtime:
        model: 'gpt-live-transcribe'
        delay: 'low' # minimal | low | medium | high | xhigh, gpt-live-transcribe only
        # url defaults to https://api.openai.com/v1

packages/api/src/files/
├── transcription.ts   # model-aware hints shared by both paths
└── realtime.ts        # live target resolution, session config, secret + SDP exchange, route handler
client/src/hooks/Input/
├── useSpeechToTextRealtime.ts  # one LiveTake per recording: WebRTC, data-channel events, commit/timeout
└── silence.ts                  # silence auto-stop shared with the External engine

Type of change

• Feature

Testing

Tested environments/configuration:

• App built from this branch (npm run frontend, npm run backend) with MongoDB 8.0.20 in Docker, Node 24.16.0, macOS
• Browser: Chrome (Playwright, headless) with a fake microphone playing a recorded English sentence
• Provider: Azure OpenAI (East US 2), deployments gpt-4o-mini-transcribe and gpt-4o-transcribe

What I ran against the real API:

1. Live, configured with stt.openai.realtime: { url: 'https://<resource>.openai.azure.com/openai/v1', model: 'gpt-4o-mini-transcribe' } and engineSTT: 'realtime'. Settings listed Browser / External / Live with Live preselected. Each take went Connecting, then Listening once the data channel opened. Stop sent the commit, the transcript landed in the composer, and the mic re-enabled. 6 of 7 takes completed, each within 0.8 s of the commit. In the other one, Azure acknowledged the commit (committed, item.added, item.done) but sent no transcript. The take ended at the 15 s completion timeout with the "stopped unexpectedly" toast, and the composer stayed usable. A standalone WebRTC script outside LibreChat hit the same provider flakiness: 1 of 4 client_secrets calls timed out with a 408.
2. External through the openai provider, with url pointing at the Azure deployment's /audio/transcriptions and model: 'gpt-4o-mini-transcribe': 200, and the transcript landed in the composer.
3. Fallback plus Azure regression: the azureOpenAI provider with engineSTT: 'realtime' in the yaml. The speech config reports sttRealtime: false, Live isn't offered, the engine falls back to External, and the take transcribes through the unchanged Azure provider.
4. POST /api/files/speech/stt/realtime returns 404 stt_realtime_not_configured when live isn't configured, and 401 without a token.

What I could not verify against the real API: gpt-transcribe and gpt-live-transcribe weren't deployed on the Azure resource I had access to. Their request shapes (languages, keywords, prompt, delay) are covered by unit tests only. With gpt-4o-mini-transcribe and turn_detection: null, Azure sent all deltas in a burst right after the commit. So the real-API runs exercise the transport, the commit, and finalization, but not deltas arriving while the user is still speaking; that rendering is covered by the mocked hook tests. Separately, this Azure resource returned DeploymentNotFound for /openai/v1/audio/transcriptions, which is why step 2 used the deployments URL.

Automated tests:

• New: packages/api/src/files/transcription.spec.ts, packages/api/src/files/realtime.spec.ts, api/server/routes/files/speech/index.spec.js, client/src/hooks/Input/__tests__/useSpeechToTextRealtime.spec.tsx, client/src/hooks/Input/__tests__/silence.spec.ts, client/src/components/Nav/SettingsTabs/Speech/STT/__tests__/EngineSTTDropdown.spec.tsx
• Extended: STTService.spec.js, getCustomConfigSpeech.spec.js, config.spec.ts, useDictation.spec.tsx, useSpeechToText.spec.ts, useSpeechSettingsInit.spec.tsx
• Full Jest suites on this machine: client 12108 of 12112 passed, api 6346 of 6365 (18 skipped), packages/api with its test:ci ignore patterns 20824 of 20852 (25 skipped), and packages/data-provider 2884 of 2884. None of the failures involve files this PR changes. The 4 client failures (balance and TokenUsage number formatting) come from this machine's en-IN locale grouping (31,00,000) and pass under LANG=en_US.UTF-8. The 3 packages/api/src/agents/hooks process-group signal tests also fail on dev (73847f2) here. The api failure in streamTenant.spec.js is a keepalive timing test, and that spec fails intermittently on dev here too.
• npx tsc --noEmit passes in client, packages/api, and packages/data-provider
• npm run static-checks -- --against upstream/dev --full passes (the unused npm packages check was skipped because depcheck isn't installed)

Screenshots / recordings

Settings > Speech > Engine, before (dev) and after (this branch, with stt.openai.realtime configured):

Before	After

Live dictation is a new engine, so there is no "before". From left to right: connecting, listening, and the final transcript left in the composer.

Connecting	Listening	Transcript

Recording of the full flow (Azure gpt-4o-mini-transcribe, so the text arrives after Stop):

live-dictation.webm

Risk / compatibility

• Config: every new field is optional, and existing configs behave as before. engineSTT also accepts 'realtime'. If a user's saved or admin-default realtime engine isn't offered by the server, the client falls back to External, or to Browser when External isn't available either.
• Provider scope: Live requires the same single-provider STT config the recorded path accepts, with that provider being openai. Azure works through the openai provider with an Azure /openai/v1 realtime URL, as tested above. The azureOpenAI provider block doesn't enable Live.
• Content filters: live audio and transcripts go straight from the provider to the browser, so filters rules for the transcript field don't apply to them. Message filters still apply when the turn is sent. This is documented next to the new config in librechat.example.yaml.
• Cost: Realtime sessions are billed per minute while the microphone is open. This is noted in the example config, and Live is never enabled implicitly.
• No migrations.

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published
• Required documentation PR: 🗣️ docs: Document gpt-transcribe Hints and Live Dictation docs#831