🗣️ docs: Document gpt-transcribe Hints and Live Dictation

content/docs/configuration/librechat_yaml/object_structure/speech.mdx

@@ -3,6 +3,13 @@ title: "Speech Configuration"
 icon: Mic
 ---
 
+{/*
+  Pending items: `stt.openai.prompt`, `keywords`, `languages`, `realtime` and the `realtime`
+  value of `speechTab.speechToText.engineSTT` (LibreChat-AI/LibreChat#17190). Delete the
+  Pending markers once it merges to dev, and re-check the schema in
+  packages/data-provider/src/config.ts.
+*/}
+
 ## Overview
 
 The `speech` object allows you to configure Text-to-Speech (TTS) and Speech-to-Text (STT) providers directly in your `librechat.yaml` configuration file. This enables server-side speech services without requiring users to configure their own API keys.
@@ -203,22 +210,62 @@ The `stt` object configures Speech-to-Text providers.
 
 ### openai
 
-OpenAI Whisper STT configuration.
+OpenAI STT configuration for `whisper-1`, `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, `gpt-transcribe`, and OpenAI-compatible endpoints.
 
 <OptionTable
   options={[
     ['url', 'String', 'Custom API URL (optional). Use for OpenAI-compatible endpoints.', ''],
     ['apiKey', 'String', 'OpenAI API key. Use environment variable reference.', 'Required'],
-    ['model', 'String', 'STT model to use (e.g., "whisper-1").', 'Required'],
+    ['model', 'String', 'STT model for recorded dictation (e.g., "whisper-1", "gpt-4o-transcribe", "gpt-transcribe").', 'Required'],
+    ['prompt', 'String', 'Pending. Free-form context about the audio, such as the subject or vocabulary to expect. Sent with every transcription, to every model.', ''],
+    ['keywords', 'Array of Strings', 'Pending. Names, acronyms and other literal terms the model should spell exactly. Sent only to `gpt-transcribe` and `gpt-live-transcribe` models. Entries cannot contain `<`, `>` or line breaks.', ''],
+    ['languages', 'Array of Strings', 'Pending. Languages to expect, as ISO 639-1 or 639-3 codes with an optional region (e.g., "de", "zh-tw"). Sent only to `gpt-transcribe` and `gpt-live-transcribe` models. When a user picks a language in their speech settings, that language replaces this list.', ''],
+    ['realtime', 'Object', 'Pending. Enables live dictation over the Realtime API. See [realtime](#realtime) below.', ''],
   ]}
 />
 
+`gpt-transcribe` and `gpt-live-transcribe` models, including dated snapshots, take `languages` instead of the singular `language` field. LibreChat sends them `prompt`, `keywords` and `languages`. Older models (Whisper, `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`) receive `prompt` and the user's speech language as `language`, as before.
+
 **Example:**
 ```yaml filename="speech / stt / openai"
 stt:
   openai:
     apiKey: "${STT_API_KEY}"
-    model: "whisper-1"
+    model: "gpt-transcribe"
+    prompt: "Internal retail and software terminology may be used."
+    keywords: ["LibreChat", "OpenShift"]
+    languages: ["de", "en"]
+```
+
+#### realtime
+
+**Pending.** Live dictation streams microphone audio to a Realtime transcription session over WebRTC, so text appears in the composer while the user speaks. With this block set, users can pick **Live** under **Settings > Speech > Engine**. Recorded dictation (the **External** engine) keeps using `model`.
+
+<OptionTable
+  options={[
+    ['model', 'String', 'Realtime transcription model, such as "gpt-live-transcribe". Older models such as "gpt-4o-transcribe" also work, but return text only after the user stops.', 'Required'],
+    ['url', 'String', 'Realtime API base URL. LibreChat calls `{url}/realtime/client_secrets` and `{url}/realtime/calls`. For Azure OpenAI, use `https://<resource>.openai.azure.com/openai/v1`.', 'https://api.openai.com/v1'],
+    ['delay', 'String', 'Latency preset for `gpt-live-transcribe` models: "minimal", "low", "medium", "high" or "xhigh". Not sent to other models.', ''],
+  ]}
+/>
+
+- Live dictation uses `stt.openai.apiKey`, `prompt`, `keywords` and `languages`. `stt.openai.url` applies only to recorded dictation.
+- **Live** is offered only when `stt` configures exactly one provider and that provider is `openai`. The `azureOpenAI` provider does not enable it.
+- The API key never reaches the browser. For each recording, the server uses it to create a client secret that expires after 60 seconds, then spends that secret on the WebRTC connection setup itself. The browser receives only the connection answer.
+- `stt.allowedAddresses` and the proxy settings apply to both requests the server makes.
+- Live audio and transcripts travel directly between the browser and the provider. `filters` rules for the `transcript` field do not see them; message filters still apply once the text is sent.
+- Realtime sessions are billed per minute while the microphone is open.
+
+**Example:**
+```yaml filename="speech / stt / openai / realtime"
+stt:
+  openai:
+    apiKey: "${STT_API_KEY}"
+    model: "gpt-transcribe"
+    keywords: ["LibreChat", "OpenShift"]
+    realtime:
+      model: "gpt-live-transcribe"
+      delay: "low"
 ```
 
 ### azureOpenAI
@@ -268,7 +315,7 @@ When using an object instead of a boolean:
     [
       'engineSTT',
       'String',
-      'Default STT engine. Use `"browser"` or `"external"`. Legacy `"openai"` and `"azureOpenAI"` values are accepted and normalized to `"external"`.',
+      'Default STT engine. Use `"browser"`, `"external"` or `"realtime"` (Pending; the **Live** engine, which requires `stt.openai.realtime`). Legacy `"openai"` and `"azureOpenAI"` values are accepted and normalized to `"external"`.',
       'browser',
     ],
     ['languageSTT', 'String', 'Default language for STT.', ''],
@@ -359,4 +406,5 @@ speech:
 - The `speechTab` settings define defaults that users can override in their personal settings
 - `browser` uses the browser's built-in speech support; `external` uses the server-side providers configured under `speech.stt` or `speech.tts`
 - Existing `openai` and `azureOpenAI` STT defaults, and `openai`, `azureOpenAI`, `elevenlabs`, and `localai` TTS defaults, are migrated to `external`; an unavailable external engine falls back to `browser`
+- `realtime` (**Pending**) is the **Live** engine. If live dictation is not available, a `realtime` default or a user's saved **Live** choice falls back to `external`, or to `browser` when no external engine is available either
 - For detailed feature documentation, see [Speech to Text & Text to Speech](/docs/configuration/stt_tts)

content/docs/configuration/stt_tts.mdx

@@ -4,6 +4,12 @@ icon: Mic
 description: Configuration of the Speech to Text (STT) and Text to Speech (TTS) features
 ---
 
+{/*
+  Pending items: transcription hints, live dictation and the `realtime` engine
+  (LibreChat-AI/LibreChat#17190). Delete the Pending markers once it merges to dev, and
+  re-check the schema in packages/data-provider/src/config.ts.
+*/}
+
 <Callout type="info" title="Upcoming STT/TTS Enhancements" collapsible>
 The Google Cloud STT/TTS and Deepgram services are being planned for future integration.
 </Callout>
@@ -48,7 +54,7 @@ Entries are bare `host:port` pairs: no scheme or path, port required, IPv6 brack
 
 The `speechTab` menu provides customizable options for conversation and advanced modes, as well as detailed settings for STT and TTS. This will set the default settings for users
 
-Use `browser` for built-in browser speech or `external` for a server-side provider configured below. Older provider-specific defaults remain accepted for compatibility and are normalized to `external`; if the matching external service is unavailable, LibreChat falls back to `browser`.
+Use `browser` for built-in browser speech or `external` for a server-side provider configured below. For speech to text, `realtime` (**Pending**) selects [live dictation](#live-dictation) and falls back to `external` when the server does not offer it. Older provider-specific defaults remain accepted for compatibility and are normalized to `external`; if the matching external service is unavailable, LibreChat falls back to `browser`.
 
 example:
 
@@ -89,7 +95,7 @@ speech:
 
 <OptionTable
   options={[
-    ['engineSTT', 'String', 'Which transcription engine to use. `browser` uses the built-in Web Speech API and needs no server config; the others use the matching provider block under `speech.stt`.', 'Options: "browser", "external", "openai", "azureOpenAI"'],
+    ['engineSTT', 'String', 'Which transcription engine to use. `browser` uses the built-in Web Speech API and needs no server config; the others use the matching provider block under `speech.stt`. `realtime` (Pending) is the **Live** engine and needs `speech.stt.openai.realtime`.', 'Options: "browser", "external", "realtime", "openai", "azureOpenAI"'],
     ['languageSTT', 'String', 'Language the transcriber should expect, as shown in the speech settings dropdown.', 'Example: "English (US)"'],
     ['autoTranscribeAudio', 'Boolean', 'Keep the microphone listening instead of stopping at the first pause. With an external engine it also turns on silence detection, which uses `decibelValue` to decide when you have stopped speaking and ends the recording. You still start the recording yourself.', ''],
     ['decibelValue', 'Number', 'Silence threshold in dB used by that silence detection. Lower values are more sensitive to quiet speech.', 'Range: -100 to -30. Default: -45'],
@@ -127,6 +133,8 @@ The Speech-to-Text (STT) feature converts spoken words into written text. To ena
   - Whisper (tested on LocalAI)
 - **Cloud STT**
   - OpenAI Whisper
+  - OpenAI `gpt-4o-transcribe`, `gpt-4o-mini-transcribe` and `gpt-transcribe`
+  - Live dictation with `gpt-live-transcribe` (**Pending**)
   - Azure Whisper
   - Other OpenAI-compatible STT services
 
@@ -158,6 +166,23 @@ speech:
       model: 'whisper-1'
 ```
 
+- #### OpenAI gpt-transcribe
+
+`gpt-transcribe` accepts hints for names, terminology and the languages to expect. `prompt`, `keywords` and `languages` are **Pending** and optional.
+
+```yaml
+speech:
+  stt:
+    openai:
+      apiKey: '${STT_API_KEY}'
+      model: 'gpt-transcribe'
+      prompt: 'Internal retail and software terminology may be used.'
+      keywords: ['LibreChat', 'OpenShift']
+      languages: ['de', 'en']
+```
+
+`keywords` and `languages` go only to `gpt-transcribe` and `gpt-live-transcribe` models. When a user picks a language in their speech settings, that language replaces `languages`. Older models such as `whisper-1` and `gpt-4o-transcribe` receive `prompt` and the user's language as before.
+
 - #### Azure Whisper
 
 ```yaml
@@ -192,6 +217,50 @@ speech:
       model: 'whisper'
   ```
 
+### Live dictation
+
+<Callout type="important" title="Pending: not yet released">
+  Live dictation is part of LibreChat-AI/LibreChat#17190.
+</Callout>
+
+Live dictation shows text in the composer while the user speaks, instead of after the recording ends. It streams microphone audio to an OpenAI Realtime transcription session over WebRTC. Add a `realtime` block under `speech.stt.openai`:
+
+```yaml
+speech:
+  stt:
+    openai:
+      apiKey: '${STT_API_KEY}'
+      model: 'gpt-transcribe'
+      realtime:
+        model: 'gpt-live-transcribe'
+        delay: 'low' # optional, gpt-live-transcribe only: minimal | low | medium | high | xhigh
+        # url: 'https://api.openai.com/v1' # default
+  speechTab:
+    speechToText:
+      engineSTT: 'realtime' # optional: make Live the default engine
+```
+
+Users then pick **Live** under **Settings > Speech > Engine**. Stop, or the silence detection from `autoTranscribeAudio`, ends the take, and the transcript stays in the composer to edit or send. Auto send, cancel and the recording timer work as they do for the **External** engine.
+
+- **Requirements:** `speech.stt` must configure exactly one provider, `openai`. Live dictation uses its `apiKey`, `prompt`, `keywords` and `languages`; `stt.openai.url` applies only to recorded dictation. The browser must support WebRTC.
+- **Models:** `gpt-live-transcribe` streams text while the user speaks. Older realtime transcription models such as `gpt-4o-transcribe` also work, but return text only after the user stops.
+- **Security:** the API key never reaches the browser. For each recording, the server creates a client secret that expires after 60 seconds, spends it on the WebRTC connection setup, and returns only the connection answer. `stt.allowedAddresses` and the proxy settings apply to both requests.
+- **Content filters:** live audio and transcripts travel directly between the browser and the provider, so `filters` rules for the `transcript` field do not see them. Message filters still apply once the text is sent.
+- **Cost:** Realtime sessions are billed per minute while the microphone is open.
+
+To use Azure OpenAI, point the `openai` provider at your Azure resource. Recorded dictation uses the deployment's transcription URL, and live dictation uses the resource's `/openai/v1` base URL:
+
+```yaml
+speech:
+  stt:
+    openai:
+      url: 'https://<resource>.openai.azure.com/openai/deployments/<deployment>/audio/transcriptions?api-version=2025-04-01-preview'
+      apiKey: '${STT_API_KEY}'
+      model: 'gpt-4o-transcribe'
+      realtime:
+        url: 'https://<resource>.openai.azure.com/openai/v1'
+        model: 'gpt-4o-transcribe' # Azure deployment name
+```
 
 ## TTS (Text-to-Speech)