diff --git a/CHANGELOG.md b/CHANGELOG.md index c71385d..c5f333f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## 0.10.0 · Text actions on your ChatGPT subscription (on `main`, ships with 1.0) + +- Add **Settings → AI → Provider**: OpenAI key or **ChatGPT**. With ChatGPT, Clean, Polish, Summarize, Prompt, Email and your own actions run on your ChatGPT subscription through the Codex CLI login (`codex login`) at no extra cost. Transcription keeps using the OpenAI key or the local model. +- Mark it **Experimental**: OpenAI does not support third-party apps on this path and may block it. There is no automatic fallback to the API key; a failure is shown and the plain transcript is delivered, as with any failed action. +- Choose the ChatGPT model under **Text model** from the models the Codex CLI lists; **Auto** takes the fastest one (GPT-6-Luna today). Effort offers Auto, Low, Medium and High. +- Renew an expiring ChatGPT sign-in and write it back for the Codex CLI, so both stay signed in. Say when the login is missing before a recording starts, and when the usage limit is reached. + +See [0.10.0 release notes](docs/releases/v0.10.0.md). + ## 0.9.0 · Settings, found at once (on `main`, ships with 1.0) - Rebuild Settings as one page: a search field that finds a setting in any category, five categories (Recording, Transcription, AI, Output, App) and one row per setting. diff --git a/README.md b/README.md index 288910b..02d8483 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ and the matching rows from every category appear, ready to change right there. | --- | --- | | Recording | Microphone, shortcut, sounds | | Transcription | Engine (Cloud, Live, Local), OpenAI key, language, vocabulary, context, local model | -| AI | Actions (built-in and your own prompts), text model and thinking effort | +| AI | Actions (built-in and your own prompts), provider (OpenAI key or ChatGPT), text model and thinking effort | | Output | Typing method, keystroke delay, file folder, history | | App | Theme, start at login, diagnostics | @@ -252,6 +252,12 @@ Only what differs from the shipped text is stored, so a prompt you left alone st Under **Settings → AI → Text model**: the model that runs these prompts (`gpt-5-mini` by default) and its **thinking effort** — Auto, Minimal, Low, Medium or High. Auto leaves the model its own default; lower is faster and cheaper. Not every model offers every level, and one that does not know the level you chose refuses the request, in which case the plain transcript is delivered and the reason is shown. +### On your ChatGPT subscription (experimental) + +**Settings → AI → Provider → ChatGPT** runs these prompts on your ChatGPT subscription instead of the API, at no extra cost. It uses the login of the [Codex CLI](https://developers.openai.com/codex/cli): install it and run `codex login` once; Utterform reads that login from `~/.codex/auth.json` (or `$CODEX_HOME`) and renews it when it expires. Transcription still uses your OpenAI key or the local model. **Text model** then lists the models the Codex CLI offers; *Auto* takes the fastest. + +OpenAI does not officially support other apps on this path and may block it, which is why it is marked *Experimental*. Utterform does not quietly fall back to your API key: if ChatGPT refuses, the reason is shown and the plain transcript is delivered. + ## Vocabulary Names, products and spellings a model would otherwise guess at go under **Settings → Transcription → Vocabulary**, one word at a time: type it and press Enter, remove it with ×. A pasted list keeps its lines. Write *Careum* there and it stops coming back as *Kareum*. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index befa430..ad79454 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -11,6 +11,7 @@ Utterform uses Tauri 2 as its desktop shell, Rust for all privileged or compute- - `tray.rs` — Tauri's native tray on Windows/macOS, an own StatusNotifierItem on Linux so a left click arrives; both show a red dot while recording and a five-second green dot after successful delivery - `actions.rs` — the prompts Utterform ships with, and how a user's replacement resolves against them - `transcription/openai.rs` — GPT Transcribe and Responses API calls +- `transcription/codex.rs` — text actions on the ChatGPT subscription through the Codex CLI login (experimental); see [Text actions on ChatGPT](#text-actions-on-chatgpt-0100) - `transcription/local.rs` — blocking local Whisper inference - `packaging/cmake/portable-cpu.cmake` — the instruction set whisper.cpp is compiled for, and why it is not the build machine's - `models.rs` — curated model catalog, downloads, progress events, and SHA-256 verification @@ -105,6 +106,35 @@ Settings (since 0.9.0) is one page over the main window, using the same theme to `plain` is in the same list with no prompt at all: it is shown in the editor as the action that never reaches a text model, and has nothing to edit. +## Text actions on ChatGPT (0.10.0) + +`AppSettings::text_provider` chooses who runs a text action: `openai_key` (the Responses +API with the keyring key) or `codex` (the user's ChatGPT subscription). Transcription is +never affected. `transcription::transform` dispatches; the instructions resolve the same +way for both (`openai::action_instructions`). + +`codex.rs` has no login of its own. It reads `auth.json` from `$CODEX_HOME` or +`~/.codex` (`%USERPROFILE%\.codex` on Windows), sends the access token and account id to +`https://chatgpt.com/backend-api/codex/responses` with `originator: utterform`, and reads +the answer from the server-sent-event stream the endpoint insists on (`stream: true`, +`input` as a message list). An access token within five minutes of its `exp`, or one the +server answers with 401, is renewed once at `auth.openai.com/oauth/token` under the Codex +CLI's public client id. Refresh tokens are single-use, so the renewal is written back to +the same file — atomically, mode 0600 — or the CLI would be signed out; before renewing, +the file is read again and a token the CLI renewed in the meantime is taken instead. +Approach taken from `kit` (jli-software/kit). + +Models come from the CLI's `models_cache.json`: only `visibility: list`, in its priority +order. `codex_model: None` means Auto: the first `…-luna`/`…-mini`, else the first listed. +Effort `minimal` is sent as `low`, the Codex models' lowest level. + +Decided with Jonas on 2026-11-07: marked *Experimental* in Settings, and **no fallback to +the API key** — a ChatGPT failure is the usual `transform.fallback` to the plain transcript +with ChatGPT's reason. `begin_recording` checks the login like a missing key, before anything +is said. `transform.*` events carry `provider=openai_key|codex`; nothing from `auth.json` +is ever logged or sent to the interface, which gets only `codex_status` (signed in, model +ids and names). `open_codex_setup` opens one fixed URL. + ## What the local engine is compiled for whisper.cpp is built from source by `whisper-rs-sys`, and ggml's CMake targets the machine doing the compiling unless told otherwise. For a release built on someone else's server that is a trap rather than an optimization: the 0.7.2 Linux binary carried the runner's AVX-512 and AMX instructions and was stopped with SIGILL on an Intel N300 while loading a model — inside `ggml_backend_cpu_device_get_extra_buffers_type`, plain C++ frame code, which is before ggml's own runtime dispatch can choose a kernel and therefore beyond its help. The bug had been there since the first release; only the pairing of build machine and user machine decided whether it fired. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 5dc401b..2263d0a 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -19,7 +19,18 @@ proposals are in [plans/v1.0-roadmap.md](plans/v1.0-roadmap.md). The plan of 202 (reliability, assistant, finish) is superseded; 0.7.13 shipped, 0.7.14 (`feat/simple-settings`) was rejected and will not be merged. -## Current handoff — 0.9.0 Settings from scratch +## Current handoff — 0.10.0 Codex login for text actions + +Stage 0.10.0 of [the roadmap](plans/v1.0-roadmap.md), branch `work/utterform-0.10.0-codex`. +Settings → AI → Provider chooses OpenAI key or ChatGPT; with ChatGPT the text actions run +on the subscription through the Codex CLI login (`transcription/codex.rs`, approach from +`kit`). Jonas decided: "Experimental" label, no fallback to the API key; mock-up accepted. +Design in [ARCHITECTURE.md → Text actions on ChatGPT](ARCHITECTURE.md#text-actions-on-chatgpt-0100), +measurements and what was not exercised live in [TESTING.md](TESTING.md). Merged into +`main` without tag or release; ships with 1.0. Next stage: 0.11.0 context — it can build on +either provider through `transcription::transform`. + +## Previous handoff — 0.9.0 Settings from scratch Stage 0.9.0 of [the roadmap](plans/v1.0-roadmap.md), branch `work/utterform-0.9.0-settings` (worktree `~/git/utterform-settings` on `asus16`), rebased on main/v0.8.0. diff --git a/docs/TESTING.md b/docs/TESTING.md index ba9b9ff..890ef28 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -10,6 +10,26 @@ `PLAYWRIGHT_CHROMIUM_EXECUTABLE=/usr/bin/chromium npm run test:e2e`. - `cmake` is needed for whisper.cpp. Without root, `mise exec cmake@latest -- cargo test` works. +## 0.10.0 Text actions on ChatGPT + +Unit tests in `transcription/codex.rs`: renewal only near expiry, a renewal replacing the +tokens and nothing else, the atomic 0600 write-back, the missing-login message, +`$CODEX_HOME`, the listed models and Auto, the request body and effort mapping, the +event-stream answer, and the refusal messages (401, 429 with reset time, others). +Interface: switching the provider saves at once and shows *Experimental*, *Set up* when +not signed in, the model list with Auto first and no Minimal. Playwright covers the row +at 920×720. + +Live, on `asus16` with the real `codex login`: +`dbus-run-session -- cargo test codex_live -- --ignored --nocapture` → GPT-6-Luna, 1.5 s. +A spike with the same Clean prompt: ChatGPT 1.6–2.2 s (GPT-6-Luna, GPT-6.1-Sol, GPT-5.6-Luna, +any effort), API `gpt-5-mini` 5.2–7.5 s. The endpoint refuses `stream: false` and a string +`input`; `originator` and `chatgpt-account-id` are accepted as sent. + +Not exercised live: a real token renewal (the current access token lives about ten days; +renewing it on purpose would rotate Jonas's refresh token). It is the path `kit` uses daily. +To check at 1.0: an action with the installed app on Linux, Windows and macOS. + ## 0.8.0 Omarchy dictation backend `scripts/test-omarchy-dictation.sh` (also in CI) installs a stripped release with the real diff --git a/docs/plans/v1.0-roadmap.md b/docs/plans/v1.0-roadmap.md index 8a99ec9..3abf857 100644 --- a/docs/plans/v1.0-roadmap.md +++ b/docs/plans/v1.0-roadmap.md @@ -57,6 +57,7 @@ and without its subscription: | 2026-11-07 | Settings rules: as little text as possible, **no Save button** (every change applies at once), as simple as possible but clear, everything findable. Do not spend long on it — it has to work. | | 2026-11-07 | Jonas tests only once, at 1.0, everything together; until then no preview builds for him. Stages go through on their own: merge into `main` when tests and CI are green, no tag or release before 1.0. Style samples (mock-ups, images) are still welcome, and so are occasional questions. The two parallel sessions coordinate with each other. | | 2026-11-07 | Settings mock-up accepted: search field on top, categories as tabs below, one row per setting. The "Context" category appears only once 0.11.0 gives it rows. | +| 2026-11-07 | 0.10.0: ChatGPT is marked "Experimental" in Settings; **no fallback** to the OpenAI key — a ChatGPT failure is shown and the plain transcript delivered. Mock-up (AI → Provider: OpenAI key \| ChatGPT) accepted. | Proposals not yet confirmed by Jonas are marked **Proposal**. @@ -66,7 +67,7 @@ Proposals not yet confirmed by Jonas are marked **Proposal**. |---|---|---| | 0.8.0 | Omarchy dictation backend | released | | 0.9.0 | Settings from scratch | merged, ships with 1.0 | -| 0.10.0 | Codex login for text actions | planned | +| 0.10.0 | Codex login for text actions | merged, ships with 1.0 | | 0.11.0 | Context: selected text and clipboard | planned | | 0.12.0 | Action per application | to discuss first | | 0.13.0 | Audio files | planned | @@ -129,9 +130,9 @@ Rules from the decisions above. Design contract: - Jonas's `kit` (jli-software/kit, Rust) already does this: it reads the Codex CLI login from `~/.codex/auth.json` and calls the Responses API at `chatgpt.com/backend-api/codex`. Reuse that approach; verify token refresh and expiry behaviour before relying on it. -- **Proposal:** mark it "experimental" in the interface. OpenAI does not officially support - third-party apps using the subscription and may block it; Utterform is public. If Codex - fails, fall back to the OpenAI key when one is stored, and say so once. +- Marked "experimental" in the interface (confirmed). OpenAI does not officially support + third-party apps using the subscription and may block it; Utterform is public. + **No fallback** to the OpenAI key (Jonas, 2026-11-07): a failure is shown as such. - Model list from `~/.codex/models_cache.json` (as in kit); default the smallest fast model. - Measure latency against the API path; dictation must stay quick. - Requires the Codex CLI installed and logged in; Settings says so in one line with a link. diff --git a/docs/releases/v0.10.0.md b/docs/releases/v0.10.0.md new file mode 100644 index 0000000..e2595ab --- /dev/null +++ b/docs/releases/v0.10.0.md @@ -0,0 +1,32 @@ +# Utterform 0.10.0 — Text actions on your ChatGPT subscription + +*Stage of the road to 1.0, merged into `main`; not released on its own. It ships with 1.0.* + +Clean, Polish, Email and the other actions need a text model. Until now that was always +the OpenAI API, paid per request with your key. If you already pay for ChatGPT, that +subscription can now do the work instead. + +**Settings → AI → Provider: OpenAI key | ChatGPT.** Choose ChatGPT and the actions run +on your subscription, through the login the Codex CLI keeps on your computer. Utterform +has no login of its own: install the [Codex CLI](https://developers.openai.com/codex/cli) +and run `codex login` once. Until then the row says **Needs `codex login`** with a +**Set up** link, and a recording with an action is stopped before it starts. + +**Transcription does not change.** GPT Transcribe with your OpenAI key, or the local +Whisper model, stays as it is. Only the rewriting afterwards moves. + +**Experimental.** OpenAI does not officially support other apps on this path and may +close it. Utterform says so next to the choice and does not hide a failure behind your +API key: if ChatGPT refuses, you see why and get the plain transcript, as with any +failed action. Switch back to *OpenAI key* to use the API again. + +**Quick.** On `asus16` a Clean through ChatGPT took about 1.5–2 seconds, against 5–7 +seconds for `gpt-5-mini` over the API with the same prompt. + +**Text model** lists the models the Codex CLI offers. *Auto* takes the fastest one +(GPT-6-Luna today) and follows when OpenAI replaces it. Effort is Auto, Low, Medium or +High; the API's *Minimal* runs as Low here. + +When the ChatGPT sign-in is about to expire, Utterform renews it and writes it back to +the Codex CLI's file, so the CLI stays signed in too. When the subscription's usage +limit is reached, the message says when it resets. diff --git a/src-tauri/src/commands.rs b/src-tauri/src/commands.rs index 309c0c8..7a8a6e0 100644 --- a/src-tauri/src/commands.rs +++ b/src-tauri/src/commands.rs @@ -13,7 +13,7 @@ use crate::{ audio::{self, AudioCaptureState, RecordingArtifact}, diagnostics::{self, Failure}, domain::{ - AppSettings, AudioDeviceInfo, LocalModelInfo, ProcessRequest, ProcessResult, + AppSettings, AudioDeviceInfo, LocalModelInfo, ProcessRequest, ProcessResult, TextProvider, TranscriptionEngine, }, feedback::{self, Cue}, @@ -223,7 +223,21 @@ fn begin_recording( mode = "batch" ) }; - if engine == TranscriptionEngine::OpenAi || action != actions::PLAIN { + let current_settings = settings::load(app).map_err(|failure| { + diagnostics::failure!( + "settings.load_failed", + &failure, + during = "recording.start", + recording = recording + ) + })?; + let codex_text = + action != actions::PLAIN && current_settings.text_provider == TextProvider::Codex; + if codex_text { + // Like a missing key: say so while nothing has been said yet. + transcription::codex::ensure_signed_in().map_err(rejected)?; + } + if engine == TranscriptionEngine::OpenAi || (action != actions::PLAIN && !codex_text) { // A missing key stops here, while nothing has been said yet. A keyring // that cannot answer does not: the recording is kept if it still // cannot answer afterwards, and the words survive the outage. @@ -258,14 +272,6 @@ fn begin_recording( ))); } } - let current_settings = settings::load(app).map_err(|failure| { - diagnostics::failure!( - "settings.load_failed", - &failure, - during = "recording.start", - recording = recording - ) - })?; // The previous recording's Done cue ends here if it is still sounding, // before the start cue is scheduled: a chime finishing inside the next // dictation would confirm the wrong recording, and its tail must not be @@ -578,12 +584,19 @@ async fn process_recording( let effort = current_settings .text_effort .map_or("auto", |effort| effort.as_str()); + let provider = current_settings.text_provider.as_str(); + let text_model = if transforms { + transcription::text_model(¤t_settings) + } else { + String::new() + }; if transforms { diagnostics::info!( "transform.started", recording = recording, action = action, - model = text_model_name(¤t_settings.text_model), + provider = provider, + model = text_model_name(&text_model), effort = effort ); } @@ -616,7 +629,8 @@ async fn process_recording( &failure, recording = recording, action = action, - model = text_model_name(¤t_settings.text_model), + provider = provider, + model = text_model_name(&text_model), effort = effort, elapsed_ms = transforming.elapsed().as_millis() ) @@ -816,6 +830,31 @@ pub fn copy_text(app: AppHandle, text: String) -> Result<(), String> { } } +/// Whether the Codex CLI is signed in to ChatGPT, and the models it offers. +/// Never a token or an account. +#[tauri::command] +pub async fn codex_status() -> transcription::codex::CodexStatus { + tauri::async_runtime::spawn_blocking(transcription::codex::status) + .await + .unwrap_or(transcription::codex::CodexStatus { + signed_in: false, + models: Vec::new(), + default_model: None, + }) +} + +/// The Codex CLI's setup page. A fixed address: the interface cannot ask the +/// backend to open anything else. +#[tauri::command] +pub fn open_codex_setup() -> Result<(), String> { + tauri_plugin_opener::open_url(transcription::codex::SETUP_URL, None::<&str>).map_err(|error| { + diagnostics::failure!( + "codex.setup_open_failed", + &Failure::new("opener", "The Codex setup page could not be opened").detail(error) + ) + }) +} + #[tauri::command] pub fn has_openai_api_key() -> bool { secrets::has_openai_api_key() diff --git a/src-tauri/src/domain.rs b/src-tauri/src/domain.rs index b5c8adb..6cffc91 100644 --- a/src-tauri/src/domain.rs +++ b/src-tauri/src/domain.rs @@ -90,6 +90,30 @@ pub enum Theme { System, } +/// Who runs the text actions. Transcription always uses the OpenAI key or a +/// local model; only the rewriting afterwards can go through ChatGPT. +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Default)] +pub enum TextProvider { + /// The OpenAI API, paid per request with the stored key. + #[default] + #[serde(rename = "openai_key")] + OpenAiKey, + /// The user's ChatGPT subscription through the Codex CLI login. + /// Experimental: OpenAI does not support it for third-party apps. + #[serde(rename = "codex")] + Codex, +} + +impl TextProvider { + /// The stored name, for diagnostic events. + pub fn as_str(self) -> &'static str { + match self { + Self::OpenAiKey => "openai_key", + Self::Codex => "codex", + } + } +} + /// How much the text model thinks before rewriting a transcript. Lower is /// faster and cheaper; `None` sends nothing and leaves the model its own /// default, so a model that does not know a level cannot reject the request. @@ -127,6 +151,11 @@ pub struct AppSettings { pub language_hints: Vec, pub type_at_cursor: bool, pub text_model: String, + /// Where the text actions run. The OpenAI key stays in use for transcription. + pub text_provider: TextProvider, + /// The ChatGPT model for text actions. `None` takes the fastest model the + /// Codex CLI lists, so a model OpenAI retires is never stuck in Settings. + pub codex_model: Option, /// Reasoning effort for the text transformation, where the model offers it. pub text_effort: Option, /// Literal terms to expect in the audio: names, products, spellings the @@ -171,6 +200,8 @@ impl Default for AppSettings { language_hints: Vec::new(), type_at_cursor: false, text_model: "gpt-5-mini".into(), + text_provider: TextProvider::OpenAiKey, + codex_model: None, text_effort: None, vocabulary: Vec::new(), transcription_context: String::new(), diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 03e15ff..e0ec4ea 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -262,6 +262,8 @@ pub fn run() { commands::list_history, commands::clear_history, commands::copy_text, + commands::codex_status, + commands::open_codex_setup, commands::has_openai_api_key, commands::set_openai_api_key, commands::delete_openai_api_key, diff --git a/src-tauri/src/transcription/codex.rs b/src-tauri/src/transcription/codex.rs new file mode 100644 index 0000000..049e6dd --- /dev/null +++ b/src-tauri/src/transcription/codex.rs @@ -0,0 +1,705 @@ +//! Text actions through the user's ChatGPT subscription (experimental). +//! +//! Utterform has no login of its own: the Codex CLI's `codex login` leaves its +//! tokens in `~/.codex/auth.json` (or `$CODEX_HOME`), and the subscription +//! answers on the Responses API at `chatgpt.com/backend-api/codex`. When the +//! access token is about to expire, Utterform renews it and writes the result +//! back in the same format, because OpenAI's refresh tokens are single-use: a +//! renewal the CLI never sees would sign the CLI out. +//! +//! OpenAI does not support third-party apps on this path and may close it. +//! There is deliberately no fallback to the API key (Jonas, 2026-11-07): a +//! failure here is reported as such, and the transcript is delivered plain. +//! +//! Nothing from `auth.json` — token, account, e-mail — ever reaches the log. + +use std::{ + fs, + path::{Path, PathBuf}, + time::{Duration, SystemTime, UNIX_EPOCH}, +}; + +use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD}; +use reqwest::{Client, StatusCode}; +use serde::Serialize; +use serde_json::{Value, json}; + +use crate::{ + diagnostics::Failure, + domain::{AppSettings, TextEffort}, +}; + +const RESPONSES_URL: &str = "https://chatgpt.com/backend-api/codex/responses"; +const TOKEN_URL: &str = "https://auth.openai.com/oauth/token"; +/// The Codex CLI's public OAuth client. Refreshing under it keeps the tokens +/// usable by the CLI that issued them. +const CLIENT_ID: &str = "app_EMoamEEZ73f0CkXaXp7hrann"; +/// Renew a token that has less than this left, so it cannot expire mid-request. +const EXPIRY_MARGIN_SECONDS: i64 = 300; +pub const SETUP_URL: &str = "https://developers.openai.com/codex/cli"; + +const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); +const REQUEST_TIMEOUT: Duration = Duration::from_secs(120); + +const NOT_SIGNED_IN: &str = + "ChatGPT is not signed in. Install the Codex CLI and run `codex login` once."; +const SIGN_IN_AGAIN: &str = "The ChatGPT sign-in has expired. Run `codex login` again."; + +/// What Settings may know about the Codex login: whether there is one, and +/// which models it offers. Never a token or an account. +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct CodexStatus { + pub signed_in: bool, + pub models: Vec, + /// The model used while the user has not chosen one. + pub default_model: Option, +} + +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct CodexModel { + pub id: String, + pub name: String, +} + +pub fn status() -> CodexStatus { + let home = codex_home(); + let models = home + .as_deref() + .and_then(|home| fs::read_to_string(home.join("models_cache.json")).ok()) + .and_then(|text| serde_json::from_str::(&text).ok()) + .map(|cache| listed_models(&cache)) + .unwrap_or_default(); + CodexStatus { + signed_in: home + .and_then(|home| Auth::load(&home.join("auth.json")).ok()) + .is_some(), + default_model: default_model(&models), + models, + } +} + +/// The model a text action runs on: the user's choice, else the default. +pub fn effective_model(settings: &AppSettings) -> String { + settings + .codex_model + .as_deref() + .map(str::trim) + .filter(|model| !model.is_empty()) + .map(str::to_string) + .or_else(|| status().default_model) + .unwrap_or_else(|| FALLBACK_MODEL.to_string()) +} + +/// Used only when the CLI has not cached its model list yet. +const FALLBACK_MODEL: &str = "gpt-6-luna"; + +/// Checked before a recording starts, like a missing API key. +pub fn ensure_signed_in() -> Result<(), Failure> { + let path = auth_path()?; + Auth::load(&path).map(|_| ()) +} + +pub async fn transform( + instructions: &str, + transcript: &str, + settings: &AppSettings, +) -> Result { + let path = auth_path()?; + let mut auth = Auth::load(&path)?; + let body = request_body( + &effective_model(settings), + instructions, + transcript, + settings.text_effort, + ); + let client = client()?; + if auth.needs_refresh(now()) { + auth.refresh(&client, &path).await?; + } + let mut renewed = false; + loop { + let mut request = client + .post(RESPONSES_URL) + .bearer_auth(auth.access_token()) + .header("originator", "utterform") + .header("accept", "text/event-stream") + .json(&body); + if let Some(account) = auth.account_id() { + request = request.header("chatgpt-account-id", account); + } + let response = request.send().await.map_err(|error| { + Failure::new( + super::openai::transport_class(&error), + format!("ChatGPT could not be reached: {error}"), + ) + })?; + let status = response.status(); + // A token the server no longer accepts gets one renewal, then counts + // as a lost sign-in. + if status == StatusCode::UNAUTHORIZED && !renewed { + renewed = true; + auth.refresh(&client, &path).await?; + continue; + } + let text = response.text().await.map_err(|error| { + Failure::new( + super::openai::transport_class(&error), + format!("ChatGPT's answer was interrupted: {error}"), + ) + })?; + if !status.is_success() { + return Err(status_failure(status, &text)); + } + return answer_text(&text); + } +} + +fn client() -> Result { + Client::builder() + .connect_timeout(CONNECT_TIMEOUT) + .timeout(REQUEST_TIMEOUT) + .build() + .map_err(|error| { + Failure::new( + "client_init", + format!("Could not initialize the ChatGPT client: {error}"), + ) + }) +} + +fn request_body( + model: &str, + instructions: &str, + transcript: &str, + effort: Option, +) -> Value { + let mut body = json!({ + "model": model, + "instructions": instructions, + // The subscription endpoint takes a list of messages only. + "input": [{ + "type": "message", + "role": "user", + "content": [{ "type": "input_text", "text": transcript }], + }], + "store": false, + // …and answers only as a stream. + "stream": true, + }); + if let Some(effort) = effort { + // The Codex models start at "low"; "minimal" is the API's level. + let level = match effort { + TextEffort::Minimal | TextEffort::Low => "low", + TextEffort::Medium => "medium", + TextEffort::High => "high", + }; + body["reasoning"] = json!({ "effort": level }); + } + body +} + +/// The text of a finished server-sent-event stream. +fn answer_text(stream: &str) -> Result { + let mut deltas = String::new(); + let mut done = None; + for line in stream.lines() { + let Some(data) = line.strip_prefix("data:") else { + continue; + }; + let Ok(event) = serde_json::from_str::(data.trim()) else { + continue; + }; + match event.get("type").and_then(Value::as_str).unwrap_or("") { + "response.output_text.delta" => { + deltas.push_str(event.get("delta").and_then(Value::as_str).unwrap_or("")); + } + "response.output_text.done" => { + done = event + .get("text") + .and_then(Value::as_str) + .map(str::to_string); + } + "response.failed" | "error" => { + let message = event + .get("response") + .and_then(error_message) + .or_else(|| error_message(&event)) + .unwrap_or_else(|| "ChatGPT could not finish the answer".into()); + return Err(Failure::new("codex_failed", format!("ChatGPT: {message}"))); + } + _ => {} + } + } + let text = done + .filter(|text| !text.trim().is_empty()) + .unwrap_or(deltas); + let text = text.trim(); + if text.is_empty() { + return Err(Failure::new( + "empty_result", + "ChatGPT returned an empty transformation", + )); + } + Ok(text.to_string()) +} + +fn error_message(value: &Value) -> Option { + value + .get("error") + .and_then(|error| error.get("message")) + .or_else(|| value.get("detail")) + .or_else(|| value.get("message")) + .and_then(Value::as_str) + .map(str::trim) + .filter(|message| !message.is_empty()) + .map(str::to_string) +} + +/// The user reads what ChatGPT said; the log only ever gets the status. +fn status_failure(status: StatusCode, body: &str) -> Failure { + let parsed = serde_json::from_str::(body).unwrap_or(Value::Null); + let failure = match status { + StatusCode::UNAUTHORIZED => Failure::guidance("codex_signed_out", SIGN_IN_AGAIN), + StatusCode::TOO_MANY_REQUESTS => { + let minutes = parsed + .get("error") + .and_then(|error| error.get("resets_in_seconds")) + .and_then(Value::as_i64) + .map(|seconds| (seconds + 59) / 60); + Failure::guidance( + "codex_limit", + match minutes { + Some(minutes) => format!( + "The ChatGPT usage limit is reached. It resets in about {minutes} min." + ), + None => "The ChatGPT usage limit is reached.".to_string(), + }, + ) + } + _ => Failure::new( + "http", + error_message(&parsed) + .map(|message| format!("ChatGPT: {message}")) + .unwrap_or_else(|| format!("ChatGPT request failed with HTTP {status}")), + ), + }; + failure.status(status.as_u16()) +} + +fn codex_home() -> Option { + codex_home_from( + std::env::var_os("CODEX_HOME"), + std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE")), + ) +} + +fn codex_home_from( + codex_home: Option, + home: Option, +) -> Option { + match codex_home { + Some(dir) if !dir.is_empty() => Some(PathBuf::from(dir)), + _ => home + .filter(|home| !home.is_empty()) + .map(|home| PathBuf::from(home).join(".codex")), + } +} + +fn auth_path() -> Result { + codex_home() + .map(|home| home.join("auth.json")) + .ok_or_else(|| Failure::guidance("codex_signed_out", NOT_SIGNED_IN)) +} + +/// The models the Codex CLI shows in its own picker, in its order. +fn listed_models(cache: &Value) -> Vec { + let mut models: Vec<(i64, CodexModel)> = cache + .get("models") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter(|model| model.get("visibility").and_then(Value::as_str) == Some("list")) + .filter_map(|model| { + let id = model.get("slug")?.as_str()?.trim(); + if id.is_empty() { + return None; + } + let name = model + .get("display_name") + .and_then(Value::as_str) + .filter(|name| !name.trim().is_empty()) + .unwrap_or(id); + let priority = model + .get("priority") + .and_then(Value::as_i64) + .unwrap_or(i64::MAX); + Some(( + priority, + CodexModel { + id: id.to_string(), + name: name.to_string(), + }, + )) + }) + .collect(); + models.sort_by_key(|(priority, _)| *priority); + models.into_iter().map(|(_, model)| model).collect() +} + +/// Dictation has to stay quick: the newest of the small, fast models +/// ("…-luna", "…-mini"), else whatever the CLI lists first. +fn default_model(models: &[CodexModel]) -> Option { + models + .iter() + .find(|model| { + let id = model.id.to_ascii_lowercase(); + id.ends_with("-luna") || id.ends_with("-mini") + }) + .or_else(|| models.first()) + .map(|model| model.id.clone()) +} + +fn now() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs() as i64) +} + +/// The expiry of a JWT, read without verifying it: the server does that. +fn jwt_expiry(token: &str) -> Option { + let payload = token.split('.').nth(1)?; + let bytes = URL_SAFE_NO_PAD.decode(payload.trim_end_matches('=')).ok()?; + serde_json::from_slice::(&bytes) + .ok()? + .get("exp")? + .as_i64() +} + +/// `auth.json` as the Codex CLI wrote it, kept whole so a write-back changes +/// only the tokens. +struct Auth { + data: Value, +} + +impl Auth { + fn load(path: &Path) -> Result { + let text = fs::read_to_string(path) + .map_err(|_| Failure::guidance("codex_signed_out", NOT_SIGNED_IN))?; + let data = serde_json::from_str::(&text) + .map_err(|_| Failure::guidance("codex_signed_out", NOT_SIGNED_IN))?; + let auth = Self { data }; + if auth.token("access_token").is_none() || auth.token("refresh_token").is_none() { + return Err(Failure::guidance("codex_signed_out", NOT_SIGNED_IN)); + } + Ok(auth) + } + + fn token(&self, key: &str) -> Option<&str> { + self.data + .get("tokens")? + .get(key)? + .as_str() + .filter(|value| !value.is_empty()) + } + + fn access_token(&self) -> &str { + self.token("access_token").unwrap_or_default() + } + + fn account_id(&self) -> Option<&str> { + self.token("account_id") + } + + fn needs_refresh(&self, now: i64) -> bool { + jwt_expiry(self.access_token()).is_some_and(|exp| exp - now < EXPIRY_MARGIN_SECONDS) + } + + async fn refresh(&mut self, client: &Client, path: &Path) -> Result<(), Failure> { + // The CLI may have renewed in the meantime, and its refresh token is + // then the only one still valid: take its token rather than renew again. + let stale = self.access_token().to_string(); + *self = Self::load(path)?; + if self.access_token() != stale && !self.needs_refresh(now()) { + return Ok(()); + } + let response = client + .post(TOKEN_URL) + .json(&json!({ + "client_id": CLIENT_ID, + "grant_type": "refresh_token", + "refresh_token": self.token("refresh_token").unwrap_or_default(), + "scope": "openid profile email", + })) + .send() + .await + .map_err(|error| { + Failure::new( + super::openai::transport_class(&error), + format!("The ChatGPT sign-in could not be renewed: {error}"), + ) + })?; + let status = response.status(); + if !status.is_success() { + return Err( + Failure::guidance("codex_signed_out", SIGN_IN_AGAIN).status(status.as_u16()) + ); + } + let renewed = response.json::().await.map_err(|_| { + Failure::new( + "invalid_response", + "The ChatGPT sign-in renewal returned an unreadable answer", + ) + })?; + self.merge(&renewed, &chrono::Utc::now().to_rfc3339()); + write_back(path, &self.data) + } + + /// Takes the tokens a renewal returned; everything else stays as written. + fn merge(&mut self, renewed: &Value, stamp: &str) { + if let Some(tokens) = self.data.get_mut("tokens").and_then(Value::as_object_mut) { + for key in ["access_token", "refresh_token", "id_token"] { + if let Some(value) = renewed + .get(key) + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + { + tokens.insert(key.into(), Value::String(value.into())); + } + } + } + self.data["last_refresh"] = Value::String(stamp.into()); + } +} + +/// Replace `auth.json` in one step, readable by the user alone, so neither a +/// crash nor the CLI reading at the same moment ever sees half a file. +fn write_back(path: &Path, data: &Value) -> Result<(), Failure> { + let failed = |error: std::io::Error| { + Failure::io( + "codex_auth_write", + &error, + "The renewed ChatGPT sign-in could not be saved", + ) + }; + let text = serde_json::to_string_pretty(data).map_err(|_| { + Failure::new( + "codex_auth_write", + "The renewed ChatGPT sign-in could not be saved", + ) + })?; + let temporary = path.with_extension("json.utterform-tmp"); + { + let mut options = fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + use std::io::Write; + let mut file = options.open(&temporary).map_err(failed)?; + file.write_all(text.as_bytes()).map_err(failed)?; + file.sync_all().map_err(failed)?; + } + fs::rename(&temporary, path).map_err(|error| { + let _ = fs::remove_file(&temporary); + failed(error) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn jwt(exp: i64) -> String { + format!( + "header.{}.signature", + URL_SAFE_NO_PAD.encode(format!(r#"{{"exp":{exp}}}"#)) + ) + } + + fn auth(access: &str) -> Auth { + Auth { + data: json!({ + "auth_mode": "chatgpt", + "OPENAI_API_KEY": null, + "tokens": { "access_token": access, "refresh_token": "r1", "id_token": "i1", "account_id": "acc" }, + "last_refresh": "2026-10-29T16:33:30Z", + }), + } + } + + #[test] + fn a_token_is_renewed_only_shortly_before_it_expires() { + let now = 1_000_000; + assert!(!auth(&jwt(now + 3600)).needs_refresh(now)); + assert!(auth(&jwt(now + 60)).needs_refresh(now)); + assert!(auth(&jwt(now - 1)).needs_refresh(now)); + // Not a JWT: let the server decide, then renew on its 401. + assert!(!auth("opaque").needs_refresh(now)); + } + + #[test] + fn a_renewal_replaces_the_tokens_and_keeps_everything_else() { + let mut auth = auth("old"); + auth.merge( + &json!({ "access_token": "new", "refresh_token": "r2", "id_token": "" }), + "2026-11-07T10:00:00Z", + ); + assert_eq!(auth.access_token(), "new"); + assert_eq!(auth.token("refresh_token"), Some("r2")); + // An empty value in the answer never erases a token. + assert_eq!(auth.token("id_token"), Some("i1")); + assert_eq!(auth.account_id(), Some("acc")); + assert_eq!(auth.data["auth_mode"], "chatgpt"); + assert_eq!(auth.data["last_refresh"], "2026-11-07T10:00:00Z"); + } + + #[test] + fn the_write_back_leaves_a_whole_file_only_the_user_can_read() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("auth.json"); + fs::write(&path, "{}").unwrap(); + write_back(&path, &auth("written").data).unwrap(); + assert_eq!(Auth::load(&path).unwrap().access_token(), "written"); + assert!(!dir.path().join("auth.json.utterform-tmp").exists()); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = fs::metadata(&path).unwrap().permissions().mode(); + assert_eq!(mode & 0o777, 0o600); + } + } + + #[test] + fn without_a_chatgpt_login_the_user_is_told_how_to_sign_in() { + let dir = tempfile::tempdir().unwrap(); + let missing = Auth::load(&dir.path().join("auth.json")).err().unwrap(); + assert_eq!(missing.class(), "codex_signed_out"); + assert!(missing.message().contains("codex login")); + + // Signed in with an API key only: there are no ChatGPT tokens. + let path = dir.path().join("auth.json"); + fs::write(&path, r#"{"auth_mode":"apikey","OPENAI_API_KEY":"sk-x"}"#).unwrap(); + assert!(Auth::load(&path).is_err()); + } + + #[test] + fn the_codex_folder_follows_codex_home_then_the_home_folder() { + assert_eq!( + codex_home_from(Some("/custom".into()), Some("/home/u".into())), + Some(PathBuf::from("/custom")) + ); + assert_eq!( + codex_home_from(Some("".into()), Some("/home/u".into())), + Some(PathBuf::from("/home/u/.codex")) + ); + assert_eq!(codex_home_from(None, None), None); + } + + #[test] + fn only_the_models_the_cli_lists_are_offered_in_its_order() { + let cache = json!({ "models": [ + { "slug": "gpt-6-luna", "display_name": "GPT-6-Luna", "visibility": "list", "priority": 7 }, + { "slug": "hidden", "display_name": "Hidden", "visibility": "hide", "priority": 1 }, + { "slug": "gpt-6.1-sol", "display_name": "GPT-6.1-Sol", "visibility": "list", "priority": 1 }, + { "slug": "gpt-5.6-luna", "visibility": "list", "priority": 10 }, + ]}); + let models = listed_models(&cache); + assert_eq!( + models + .iter() + .map(|model| model.id.as_str()) + .collect::>(), + ["gpt-6.1-sol", "gpt-6-luna", "gpt-5.6-luna"] + ); + assert_eq!(models[2].name, "gpt-5.6-luna"); + assert_eq!(default_model(&models).as_deref(), Some("gpt-6-luna")); + assert_eq!(default_model(&models[..1]).as_deref(), Some("gpt-6.1-sol")); + assert_eq!(default_model(&[]), None); + } + + #[test] + fn the_request_is_a_stream_of_one_message_with_codex_effort_levels() { + let body = request_body("gpt-6-luna", "Fix it.", "hallo", None); + assert_eq!(body["stream"], true); + assert_eq!(body["store"], false); + assert_eq!(body["input"][0]["content"][0]["text"], "hallo"); + assert!(body.get("reasoning").is_none()); + let minimal = request_body("m", "i", "t", Some(TextEffort::Minimal)); + assert_eq!(minimal["reasoning"]["effort"], "low"); + let high = request_body("m", "i", "t", Some(TextEffort::High)); + assert_eq!(high["reasoning"]["effort"], "high"); + } + + #[test] + fn the_answer_is_read_from_the_event_stream() { + let stream = "event: response.created\ndata: {\"type\":\"response.created\"}\n\n\ + data: {\"type\":\"response.output_text.delta\",\"delta\":\"Hallo \"}\n\n\ + data: {\"type\":\"response.output_text.delta\",\"delta\":\"Welt.\"}\n\n\ + data: {\"type\":\"response.completed\"}\n\n"; + assert_eq!(answer_text(stream).unwrap(), "Hallo Welt."); + + let done = "data: {\"type\":\"response.output_text.delta\",\"delta\":\"Hal\"}\n\ + data: {\"type\":\"response.output_text.done\",\"text\":\"Hallo Welt.\"}\n"; + assert_eq!(answer_text(done).unwrap(), "Hallo Welt."); + + let failed = "data: {\"type\":\"response.failed\",\"response\":{\"error\":{\"message\":\"model busy\"}}}\n"; + assert_eq!( + answer_text(failed).unwrap_err().message(), + "ChatGPT: model busy" + ); + assert_eq!(answer_text("").unwrap_err().class(), "empty_result"); + } + + #[test] + fn refusals_say_what_the_user_can_do() { + let limit = status_failure( + StatusCode::TOO_MANY_REQUESTS, + r#"{"error":{"resets_in_seconds":600}}"#, + ); + assert_eq!(limit.class(), "codex_limit"); + assert!(limit.message().contains("10 min")); + let expired = status_failure(StatusCode::UNAUTHORIZED, ""); + assert!(expired.message().contains("codex login")); + let other = status_failure( + StatusCode::BAD_REQUEST, + r#"{"detail":"Stream must be set to true"}"#, + ); + assert_eq!(other.message(), "ChatGPT: Stream must be set to true"); + assert_eq!( + status_failure(StatusCode::BAD_GATEWAY, "").message(), + "ChatGPT request failed with HTTP 502 Bad Gateway" + ); + } +} + +/// Against the real subscription, with this machine's `codex login`: +/// `cargo test codex_live -- --ignored --nocapture`. +#[cfg(test)] +mod live { + #[tokio::test] + #[ignore = "uses the real ChatGPT login and subscription"] + async fn codex_live_transform() { + let settings = crate::domain::AppSettings { + text_provider: crate::domain::TextProvider::Codex, + ..Default::default() + }; + let started = std::time::Instant::now(); + let text = super::transform( + crate::actions::find("clean").unwrap().prompt, + "also ich habe am montag zeit bitte sag mir ob zehn uhr passt", + &settings, + ) + .await + .unwrap(); + println!( + "{} in {:?}: {text}", + super::effective_model(&settings), + started.elapsed() + ); + assert!(!text.is_empty()); + } +} diff --git a/src-tauri/src/transcription/mod.rs b/src-tauri/src/transcription/mod.rs index cb6a016..2568aa2 100644 --- a/src-tauri/src/transcription/mod.rs +++ b/src-tauri/src/transcription/mod.rs @@ -1,3 +1,4 @@ +pub mod codex; mod local; mod openai; @@ -7,7 +8,7 @@ use crate::{ actions, audio::RecordingArtifact, diagnostics::Failure, - domain::{AppSettings, TranscriptionEngine}, + domain::{AppSettings, TextProvider, TranscriptionEngine}, }; /// Most often a key pressed by accident or a recording stopped too soon: it is @@ -36,5 +37,22 @@ pub async fn transform( if action == actions::PLAIN { return Ok(transcript.trim().to_string()); } - openai::transform(transcript, action, custom_prompt, settings).await + match settings.text_provider { + TextProvider::OpenAiKey => { + openai::transform(transcript, action, custom_prompt, settings).await + } + // No fallback to the API key: a failure here is ChatGPT's to report. + TextProvider::Codex => { + let instructions = openai::action_instructions(action, custom_prompt, settings)?; + codex::transform(&instructions, transcript, settings).await + } + } +} + +/// The text model an action runs on, for diagnostic events. +pub fn text_model(settings: &AppSettings) -> String { + match settings.text_provider { + TextProvider::OpenAiKey => settings.text_model.clone(), + TextProvider::Codex => codex::effective_model(settings), + } } diff --git a/src-tauri/src/transcription/openai.rs b/src-tauri/src/transcription/openai.rs index 95fcb3e..8d637df 100644 --- a/src-tauri/src/transcription/openai.rs +++ b/src-tauri/src/transcription/openai.rs @@ -198,7 +198,7 @@ fn client() -> Result { } /// What kind of transport failure a request met, without its URL or body. -fn transport_class(error: &reqwest::Error) -> &'static str { +pub(super) fn transport_class(error: &reqwest::Error) -> &'static str { if error.is_timeout() { "timeout" } else if error.is_connect() { @@ -260,7 +260,7 @@ fn usable_keywords(vocabulary: &[String]) -> Vec { .collect() } -fn action_instructions( +pub(super) fn action_instructions( action: &str, custom_prompt: Option<&str>, settings: &AppSettings, diff --git a/src/App.svelte b/src/App.svelte index e0d4c29..1a12a5f 100644 --- a/src/App.svelte +++ b/src/App.svelte @@ -17,6 +17,7 @@ AudioDevice, BuiltInAction, CloudModel, + CodexStatus, DownloadProgress, HistoryEntry, HotkeySupport, @@ -103,6 +104,9 @@ // Typed as written rather than reassembled from the parsed terms, so a blank // line being typed is not swallowed under the cursor. let hasApiKey = false; + /// Read each time Settings opens: `codex login` happens in a terminal while + /// Utterform stays open. + let codexStatus: CodexStatus = { signedIn: false, models: [], defaultModel: null }; let apiKeyInput = ""; // Until the backend answers, assume the session grants nothing: a field that // appears and then turns out to be dead is worse than one that arrives late. @@ -220,7 +224,22 @@ $: localModelSummary = selectedModel ? `${selectedModel.name}${selectedModel.downloaded ? "" : " · not downloaded"} · ${downloadedModels.length} on device` : `None · ${downloadedModels.length} on device`; - $: effortLabel = EFFORTS.find((effort) => effort.value === settings.text_effort)?.label ?? "Auto"; + $: usesCodex = settings.text_provider === "codex"; + // The Codex models start at Low; a Minimal chosen for the API runs as Low there. + $: shownEffort = usesCodex && settings.text_effort === "minimal" ? "low" : settings.text_effort; + $: effortChoices = usesCodex ? EFFORTS.filter((effort) => effort.value !== "minimal") : EFFORTS; + $: effortLabel = EFFORTS.find((effort) => effort.value === shownEffort)?.label ?? "Auto"; + $: codexDefaultName = codexStatus.models.find((model) => model.id === codexStatus.defaultModel)?.name ?? codexStatus.defaultModel ?? ""; + $: codexModelOptions = [ + { value: "", label: "Auto", hint: codexDefaultName || undefined }, + ...codexStatus.models.map((model) => ({ value: model.id, label: model.name })), + // A model the CLI no longer lists stays visible rather than silently replaced. + ...(settings.codex_model && !codexStatus.models.some((model) => model.id === settings.codex_model) + ? [{ value: settings.codex_model, label: settings.codex_model }] : []), + ]; + $: textModelSummary = usesCodex + ? (codexStatus.models.find((model) => model.id === settings.codex_model)?.name ?? settings.codex_model ?? (codexDefaultName || "Auto")) + : settings.text_model; $: historyOptions = history.map((entry) => ({ value: entry.id, label: entry.title, hint: formatHistoryTime(historyTimestamp(entry), now) })); $: displayedTimestamp = selectedHistory ? historyTimestamp(selectedHistory) : resultTimestamp; $: recordingActive = phase === "recording" || phase === "paused"; @@ -1079,6 +1098,15 @@ void refreshAutostart(); void refreshDiagnosticsInfo(); void refreshInputState(); + void refreshCodexStatus(); + } + } + + async function refreshCodexStatus() { + try { + codexStatus = await api.codexStatus(); + } catch { + /* Keep what is showing. */ } } @@ -1392,12 +1420,18 @@ {:else if settingsDetail === "text-model"}

Text model

-
- change({ text_model: event.currentTarget.value }, TYPING_SAVE_DELAY_MS)} />
+ {#if usesCodex} +
Model
+ change({ codex_model: value || null })} />
+ {:else} +
+ change({ text_model: event.currentTarget.value }, TYPING_SAVE_DELAY_MS)} />
+ {/if}
Effort
- {#each EFFORTS as effort} - + {#each effortChoices as effort} + {/each}
{:else if settingsDetail === "action"} @@ -1518,9 +1552,23 @@ {#if liveMode}

Live uses Plain only.

{/if} {/if} + {#if shownRows.has("text-provider")} +
{@render rowHead("text-provider", "Provider")} +
+ {#if usesCodex}Experimental{/if} +
+ + +
+
+ {#if usesCodex && !codexStatus.signedIn} +

Needs codex login.

+ {/if} +
+ {/if} {#if shownRows.has("text-model")} + {textModelSummary} · {effortLabel} {/if} diff --git a/src/App.test.ts b/src/App.test.ts index 964610f..1cc2802 100644 --- a/src/App.test.ts +++ b/src/App.test.ts @@ -29,7 +29,7 @@ vi.mock("./lib/api", () => ({ api: { takeStartupIntent: vi.fn(async () => null), getSettings: vi.fn(), listInputDevices: vi.fn(async () => []), listLocalModels: vi.fn(async () => []), listBuiltInActions: vi.fn(), liveSupport: vi.fn(), getLiveStatus: vi.fn(), typingSupport: vi.fn(async () => ({ supported: true, explanation: "" })), - hasOpenAiApiKey: vi.fn(async () => true), listHistory: vi.fn(), saveSettings: vi.fn(), + hasOpenAiApiKey: vi.fn(async () => true), codexStatus: vi.fn(), openCodexSetup: vi.fn(async () => {}), listHistory: vi.fn(), saveSettings: vi.fn(), globalHotkeySupport: vi.fn(async () => ({ supported: true, default: "Ctrl+Alt+D", explanation: "", failure: null })), applyGlobalHotkey: vi.fn(), autostartEnabled: vi.fn(), enableAutostart: vi.fn(), disableAutostart: vi.fn(), @@ -72,6 +72,7 @@ beforeEach(() => { vi.mocked(api.listBuiltInActions).mockResolvedValue(structuredClone(SHIPPED_ACTIONS)); vi.mocked(api.listHistory).mockResolvedValue([latest, older]); vi.mocked(api.listLocalModels).mockResolvedValue([]); + vi.mocked(api.codexStatus).mockResolvedValue({ signedIn: false, models: [], defaultModel: null }); let paused = false; const status = () => ({ recording: true, paused, limitReached: false, elapsedSeconds: 12, level: paused ? 0 : .3 }); vi.mocked(api.getRecordingStatus).mockImplementation(async () => status()); @@ -787,6 +788,60 @@ describe("editing the prompts Utterform ships with", () => { }); }); +describe("text actions through ChatGPT", () => { + const CODEX = { + signedIn: true, + models: [{ id: "gpt-6.1-sol", name: "GPT-6.1-Sol" }, { id: "gpt-6-luna", name: "GPT-6-Luna" }], + defaultModel: "gpt-6-luna", + }; + + async function openAi() { + const view = render(App); + await waitFor(() => expect(view.queryByRole("button", { name: "Open settings" })).not.toBeNull()); + await fireEvent.click(view.getByRole("button", { name: "Open settings" })); + await showSettingsTab(view, "AI"); + return view; + } + + it("switches the provider at once, marked experimental, and says when the login is missing", async () => { + const view = await openAi(); + expect(view.queryByText("Experimental")).toBeNull(); + await fireEvent.click(view.getByRole("button", { name: "ChatGPT" })); + await waitFor(() => expect(api.saveSettings).toHaveBeenLastCalledWith( + expect.objectContaining({ text_provider: "codex" }), + )); + expect(view.queryByText("Experimental")).not.toBeNull(); + await waitFor(() => expect(view.queryByRole("button", { name: "Set up" })).not.toBeNull()); + await fireEvent.click(view.getByRole("button", { name: "Set up" })); + expect(api.openCodexSetup).toHaveBeenCalled(); + + await fireEvent.click(view.getByRole("button", { name: "OpenAI key" })); + await waitFor(() => expect(api.saveSettings).toHaveBeenLastCalledWith( + expect.objectContaining({ text_provider: "openai_key" }), + )); + expect(view.queryByText("Experimental")).toBeNull(); + }); + + it("offers the models the Codex CLI lists, Auto first, and no Minimal effort", async () => { + vi.mocked(api.codexStatus).mockResolvedValue(structuredClone(CODEX)); + vi.mocked(api.getSettings).mockResolvedValue({ ...structuredClone(DEFAULT_SETTINGS), text_provider: "codex", text_effort: "minimal" }); + const view = await openAi(); + await waitFor(() => expect(view.queryByRole("button", { name: /^Text model/ })?.textContent).toContain("GPT-6-Luna · Low")); + expect(view.queryByRole("button", { name: "Set up" })).toBeNull(); + await fireEvent.click(view.getByRole("button", { name: /^Text model/ })); + expect(view.queryByRole("button", { name: "Minimal" })).toBeNull(); + expect(view.queryByRole("textbox", { name: "Model" })).toBeNull(); + await fireEvent.click(view.getByRole("combobox", { name: "ChatGPT model" })); + const options = [...document.querySelectorAll("[role=option]")].map((node) => node.textContent?.trim()); + expect(options[0]).toContain("Auto"); + expect(options.slice(1)).toEqual(["GPT-6.1-Sol", "GPT-6-Luna"]); + await fireEvent.click(view.getByRole("option", { name: "GPT-6.1-Sol" })); + await waitFor(() => expect(api.saveSettings).toHaveBeenLastCalledWith( + expect.objectContaining({ codex_model: "gpt-6.1-sol" }), + )); + }); +}); + describe("vocabulary", () => { async function openVocabulary() { const view = render(App); diff --git a/src/app.css b/src/app.css index ed621d0..99fc1fb 100644 --- a/src/app.css +++ b/src/app.css @@ -119,6 +119,9 @@ main:not(.has-result) footer { margin-top:20px; } .row-button:hover:not(:disabled) { background:var(--accent-soft); } .row-button.quiet { color:var(--muted); } .danger-text { color:var(--danger); } +.badge { padding:2px 7px; border:1px solid var(--line); border-radius:999px; color:var(--muted); font-size:11px; letter-spacing:.02em; white-space:nowrap; } +.row-message .link-button { padding:0; border:0; background:none; color:inherit; font:inherit; text-decoration:underline; cursor:pointer; } +.row-message code { font-size:.95em; } .row-path { min-width:0; max-width:300px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; direction:rtl; text-align:left; color:var(--muted); font-size:12px; } .key-state { color:#4f9a62; font-size:12px; } .more-row { width:100%; border:0; border-bottom:1px solid var(--line); background:transparent; text-align:left; cursor:pointer; } diff --git a/src/lib/api.ts b/src/lib/api.ts index e6a3784..958c15e 100644 --- a/src/lib/api.ts +++ b/src/lib/api.ts @@ -4,6 +4,7 @@ import type { AppSettings, AudioDevice, BuiltInAction, + CodexStatus, CopiedDiagnostics, DiagnosticsInfo, Engine, @@ -83,6 +84,9 @@ export const api = { typeAtCursor: boolean; outputFormat: OutputFormat; }) => invoke("finish_recording", { request }), + // Whether `codex login` signed the Codex CLI in to ChatGPT, and its models. + codexStatus: () => invoke("codex_status"), + openCodexSetup: () => invoke("open_codex_setup"), hasOpenAiApiKey: () => invoke("has_openai_api_key"), setOpenAiApiKey: (apiKey: string) => invoke("set_openai_api_key", { apiKey }), deleteOpenAiApiKey: () => invoke("delete_openai_api_key"), diff --git a/src/lib/settings-catalog.test.ts b/src/lib/settings-catalog.test.ts index 12a4c7e..561727d 100644 --- a/src/lib/settings-catalog.test.ts +++ b/src/lib/settings-catalog.test.ts @@ -17,6 +17,8 @@ describe("the settings catalog", () => { expect([...visibleRows("app", "mic")]).toEqual(["microphone"]); expect([...visibleRows("app", "output folder")]).toEqual(["folder"]); expect(visibleRows("app", "xylophone").size).toBe(0); + expect([...visibleRows("app", "chatgpt")]).toEqual(["text-provider"]); + expect([...visibleRows("app", "codex")]).toEqual(["text-provider"]); }); it("needs every word of a search", () => { diff --git a/src/lib/settings-catalog.ts b/src/lib/settings-catalog.ts index e26367b..5b82fd9 100644 --- a/src/lib/settings-catalog.ts +++ b/src/lib/settings-catalog.ts @@ -15,7 +15,7 @@ export const SETTINGS_CATEGORIES: Array<{ id: SettingsCategory; label: string }> export type SettingsRowId = | "microphone" | "shortcut" | "sounds" | "engine" | "api-key" | "language" | "vocabulary" | "context" | "local-model" - | "actions" | "text-model" + | "actions" | "text-provider" | "text-model" | "typing" | "typing-delay" | "folder" | "history" | "theme" | "autostart" | "diagnostics"; @@ -38,6 +38,7 @@ export const SETTINGS_ROWS: SettingsRow[] = [ { id: "context", category: "transcription", label: "Context", keywords: ["topic", "recording context", "hint"] }, { id: "local-model", category: "transcription", label: "Local model", keywords: ["whisper", "download", "offline", "model", "on device"] }, { id: "actions", category: "ai", label: "Actions", keywords: ["prompts", "prompt", "clean", "polish", "summarize", "email", "custom", "rewrite"] }, + { id: "text-provider", category: "ai", label: "Provider", keywords: ["chatgpt", "codex", "subscription", "plus", "pro", "abo", "login", "openai", "experimental"] }, { id: "text-model", category: "ai", label: "Text model", keywords: ["model", "gpt", "effort", "thinking", "reasoning", "llm"] }, { id: "typing", category: "output", label: "Type at cursor", keywords: ["paste", "keystrokes", "typing", "insert"] }, { id: "typing-delay", category: "output", label: "Keystroke delay", keywords: ["delay", "milliseconds", "slow", "typing"] }, diff --git a/src/lib/types.ts b/src/lib/types.ts index 252b1be..decd108 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -5,6 +5,17 @@ export type Theme = "light" | "dark" | "system"; /// How hard the text model thinks before rewriting a transcript. `null` sends /// nothing and leaves the model its own default. export type TextEffort = "minimal" | "low" | "medium" | "high"; +/// Who runs the text actions: the OpenAI API with the stored key, or the +/// user's ChatGPT subscription through the Codex CLI login (experimental). +/// Transcription is not affected. +export type TextProvider = "openai_key" | "codex"; + +/// What Settings may know about the Codex login. Never a token or an account. +export interface CodexStatus { + signedIn: boolean; + models: Array<{ id: string; name: string }>; + defaultModel: string | null; +} /// How the finished text reaches the focused window. A paste moves it whole; /// keystrokes travel one character at a time and can be dropped on the way. export type TypingMethod = "paste" | "keystrokes"; @@ -23,6 +34,9 @@ export interface AppSettings { local_model_id: string | null; language_hints: string[]; text_model: string; + text_provider: TextProvider; + /// `null` takes the fastest model the Codex CLI lists. + codex_model: string | null; text_effort: TextEffort | null; vocabulary: string[]; transcription_context: string; @@ -138,6 +152,8 @@ export const DEFAULT_SETTINGS: AppSettings = { local_model_id: "base", language_hints: [], text_model: "gpt-5-mini", + text_provider: "openai_key", + codex_model: null, text_effort: null, vocabulary: [], transcription_context: "", diff --git a/tests/ui.e2e.ts b/tests/ui.e2e.ts index 9b0a634..04dab00 100644 --- a/tests/ui.e2e.ts +++ b/tests/ui.e2e.ts @@ -64,6 +64,8 @@ test.beforeEach(async ({ page }) => { case "get_settings": return settings; case "save_settings": Object.assign(window, { __savedSettings: args.value }); return; case "has_openai_api_key": return true; + case "codex_status": return { signedIn: !Reflect.get(window, "__codexSignedOut"), models: [{ id: "gpt-6.1-sol", name: "GPT-6.1-Sol" }, { id: "gpt-6-luna", name: "GPT-6-Luna" }], defaultModel: "gpt-6-luna" }; + case "open_codex_setup": (Reflect.get(window, "__supportCalls") as string[]).push("codex-setup"); return; case "list_input_devices": return [{ id: "test-mic", name: "Studio microphone", isDefault: true }]; case "list_local_models": return models; case "download_local_model": case "delete_local_model": { @@ -858,3 +860,31 @@ for (const viewport of [{ width: 920, height: 720 }, { width: 360, height: 400 } expect(errors).toEqual([]); }); } + +test("text actions can run on the ChatGPT subscription, marked experimental", async ({ page }, testInfo) => { + await page.addInitScript(() => Object.assign(window, { __codexSignedOut: true })); + await page.goto("/"); + await page.getByRole("button", { name: "Open settings" }).click(); + await page.getByRole("tab", { name: "AI" }).click(); + await page.getByRole("button", { name: "ChatGPT" }).click(); + await expect(page.getByText("Experimental", { exact: true })).toBeVisible(); + await expect.poll(() => page.evaluate(() => (Reflect.get(window, "__savedSettings") as { text_provider?: string } | null)?.text_provider)).toBe("codex"); + await page.getByRole("button", { name: "Set up" }).click(); + expect(await page.evaluate(() => Reflect.get(window, "__supportCalls"))).toContain("codex-setup"); + expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true); + await page.mouse.move(0, 0); + await page.waitForTimeout(300); + await page.screenshot({ path: testInfo.outputPath("ai-chatgpt-signed-out.png") }); + await page.evaluate(() => Object.assign(window, { __codexSignedOut: false })); + await page.getByRole("button", { name: "OpenAI key" }).click(); + await page.getByRole("button", { name: "ChatGPT" }).click(); + await expect(page.getByRole("button", { name: "Set up" })).toHaveCount(0); + await expect(page.getByRole("button", { name: /^Text model/ })).toContainText("GPT-6-Luna"); + await page.mouse.move(0, 0); + await page.waitForTimeout(300); + await page.screenshot({ path: testInfo.outputPath("ai-chatgpt.png") }); + await page.getByRole("button", { name: /^Text model/ }).click(); + await page.getByRole("combobox", { name: "ChatGPT model" }).click(); + await expect(page.getByRole("option", { name: "GPT-6.1-Sol" })).toBeVisible(); + await page.screenshot({ path: testInfo.outputPath("ai-chatgpt-model.png") }); +});