diff --git a/AGENTS.md b/AGENTS.md index 9a3d9ba2..b636aae2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -197,7 +197,7 @@ Each was tried the other way and reverted. If a task seems to require one, stop | Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | | Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | | Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/v1/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. | | Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | | Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. | | Send focus context to the API again | The transcription prompt is switched off at `TranscriptionPrompt.isEnabled`, so `config.prompt` is omitted and no window title, field label, surrounding text or key term leaves the machine. Keep the builder and its wiring — don't route context on by another path. | @@ -247,7 +247,7 @@ framework, or notarization rejects the build; roll-forward-only for a bad releas ```text DictationKeyTap (CGEventTap + DictationKeyGate) → AppCoordinator → DictationSession (actor) → MicCapture ↓ - AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /transcribe) + AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /v1/transcribe) ↓ KeyInjector → focused app (clipboard paste via a synthesized ⌘V CGEvent) ``` @@ -275,7 +275,7 @@ the seam they inject. ### `AssemblyAITranscriber` — `Sources/BlurtEngine/STT/AssemblyAITranscriber.swift` Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single -`POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob +`POST https://dictation.assemblyai.com/v1/transcribe` with the captured audio as a raw S16LE PCM blob in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and an `llm` block). No model header — the service pins the STT model server-side. The config also has a `prompt` field for steering _transcription_, but nothing is put in it: `TranscriptionPrompt.build` returns diff --git a/BLURTENGINE.md b/BLURTENGINE.md index 831255ef..8ab366b0 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -51,7 +51,7 @@ Before the first dictation can succeed the host must have: ```text press() ──▶ MicCapture.start() release() ──▶ MicCapture.stop() → Data (raw S16LE PCM) (16 kHz mono 16-bit PCM) AssemblyAITranscriber.transcribe(pcm:sampleRate:context:) - + focus/context capture (one POST dictation.assemblyai.com/transcribe: STT + LLM rewrite) + + focus/context capture (one POST dictation.assemblyai.com/v1/transcribe: STT + LLM rewrite) + connection warm-up KeyInjector.insert(text, after: priorText) (clipboard paste via synthesized ⌘V) ``` @@ -133,7 +133,7 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn func warmUp() async // optional; no-op default ``` -`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before. +`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/v1/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before. The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift. diff --git a/README.md b/README.md index 6874fbef..627a4ddf 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ what each script does, signing — and how changes land; Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dependencies Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs - STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe + STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/v1/transcribe (STT + LLM rewrite); TranscriptionPrompt (contextual priming, built and tested but currently switched off — nothing is sent) Pipeline/ DictationSession actor: press/release/cancel commands, phase diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index 27a19ddd..ced2247b 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -9,7 +9,7 @@ private let transcriberLog = Logger(subsystem: BlurtIdentity.subsystem, category /// `TranscriberProtocol` backed by AssemblyAI's **dictation** API. /// -/// A single `POST dictation.assemblyai.com/transcribe` carries the captured +/// A single `POST dictation.assemblyai.com/v1/transcribe` carries the captured /// audio (raw S16LE PCM, exactly the bytes the mic recorded — there is no /// re-encoding pass) plus a JSON `config` part, and the response body carries /// both the verbatim transcript and — when the config requests one via its @@ -70,7 +70,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt) let boundary = "blurt-\(UUID().uuidString)" - var request = URLRequest(url: baseURL.appendingPathComponent("transcribe")) + var request = URLRequest(url: baseURL.appendingPathComponent("v1/transcribe")) request.httpMethod = "POST" // Bounds a stalled connection; see `requestTimeoutSeconds` for why an idle // timeout is the right shape here. @@ -103,7 +103,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { /// `transcribe` reuses it instead of paying DNS+TCP+TLS on the hot path /// (~170 ms cold, more on mobile — measured). A throwaway GET to the host /// root is enough to establish the HTTP/2 connection `URLSession` then reuses - /// for the POST to `/transcribe`; the response (an auth-less 4xx) is + /// for the POST to `/v1/transcribe`; the response (an auth-less 4xx) is /// discarded. No key, so it never counts as a transcription. A short timeout /// keeps a dead network from leaving the task hanging. Fire-and-forget: any /// error is swallowed — a failed warm-up just means the next request pays diff --git a/Sources/BlurtEngine/STT/CleanupInstruction.swift b/Sources/BlurtEngine/STT/CleanupInstruction.swift index 800c3eba..8c0192fe 100644 --- a/Sources/BlurtEngine/STT/CleanupInstruction.swift +++ b/Sources/BlurtEngine/STT/CleanupInstruction.swift @@ -1,7 +1,7 @@ /// The cleanup instruction sent as `config.llm.instruction` (see /// `AssemblyAITranscriber.DictationConfig`). The service applies it to the /// verbatim transcript with its own rewrite model, inside the same -/// `/transcribe` call — this is the *server-side* rewrite instruction, not a +/// `/v1/transcribe` call — this is the *server-side* rewrite instruction, not a /// client-side cleanup pass, and not a `TranscriptionPrompt` change (that /// prompt steers transcription and deliberately carries no filler-word clause, /// because disfluency removal is this rewrite's job). diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index 0cd306a1..7d44011f 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -16,7 +16,7 @@ struct HTTPClientTests { let hits = Counter() let transport = FakeHTTPTransport { request in _ = hits.next() - guard request.url?.path.hasSuffix("/transcribe") == true, + guard request.url?.path == "/v1/transcribe", request.httpMethod == "POST" else { return (404, Data()) } return (200, json(["text": "um hello world", "llm_response": "Hello world."])) @@ -55,7 +55,7 @@ struct HTTPClientTests { @Test("transcriber succeeds with a real context (which builds no prompt today)") func transcribeWithContext() async throws { let transport = FakeHTTPTransport { request in - guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) } + guard request.url?.path == "/v1/transcribe" else { return (404, Data()) } return (200, json(["text": "hello world"])) } diff --git a/evals/dictation-prompt/README.md b/evals/dictation-prompt/README.md index 51e50d44..0a564522 100644 --- a/evals/dictation-prompt/README.md +++ b/evals/dictation-prompt/README.md @@ -1,7 +1,7 @@ # Dictation cleanup-prompt eval A DSPy eval that searches for the best **cleanup instruction** for the dictation API's -LLM rewrite — the `config.llm` block Blurt sends on every `/transcribe` request. +LLM rewrite — the `config.llm` block Blurt sends on every `/v1/transcribe` request. Blurt sends `llm` as an empty object today, which selects the service's own default cleanup instruction. This harness answers the question that comes next: is there an explicit @@ -297,7 +297,7 @@ Two consequences worth holding onto when reading a result: - A winner is only as transferable as `--model` is representative of the service's rewrite model, which runs under a ~5s budget and is probably much smaller. - Confirming a win against the live default would mean sending real audio to - `dictation.assemblyai.com/transcribe` with an empty `llm` block and comparing. That is a + `dictation.assemblyai.com/v1/transcribe` with an empty `llm` block and comparing. That is a separate exercise, not this one. ## Reading the results @@ -481,7 +481,7 @@ real thing. `--verify-live N` closes that loop. After a winner is picked it takes N **held-out** rows, speaks the disfluent side with `say`, converts to 16 kHz mono PCM, and POSTs it to the real -`/transcribe` with the winner as `config.llm.instruction`: +`/v1/transcribe` with the winner as `config.llm.instruction`: ```bash export ASSEMBLYAI_API_KEY=... @@ -511,7 +511,7 @@ between them hold. macOS only, and off by default — it costs real transcriptio | `corpus.py` | Sources, loading, de-tagging, splitting, the echo floor. | | `disfluency.py` | The seeded, additive disfluency injector. | | `metrics.py` | Token alignment, the two word-error-rate axes, GEPA feedback text. | -| `live.py` | Synthesis + the real `/transcribe` round trip, for `--verify-live`. | +| `live.py` | Synthesis + the real `/v1/transcribe` round trip, for `--verify-live`. | | `program.py` | Everything that imports DSPy — the program, metrics adapters, optimizers. | | `test_eval.py` | Offline tests for injection, scoring, de-tagging, loading, and splitting. | diff --git a/evals/dictation-prompt/live.py b/evals/dictation-prompt/live.py index 5ee0cb74..a1636dfc 100644 --- a/evals/dictation-prompt/live.py +++ b/evals/dictation-prompt/live.py @@ -8,7 +8,7 @@ There is something to do about it. The dictation API takes audio, so: - reference text -> `say` -> 16 kHz mono PCM -> POST /transcribe with the + reference text -> `say` -> 16 kHz mono PCM -> POST /v1/transcribe with the candidate as config.llm.instruction -> score `llm_response` The response carries both sides of the question. `text` is the verbatim transcript, @@ -44,7 +44,7 @@ from corpus import Utterance #: The dictation endpoint. Same host `AssemblyAITranscriber` posts to. -DICTATION_URL = "https://dictation.assemblyai.com/transcribe" +DICTATION_URL = "https://dictation.assemblyai.com/v1/transcribe" #: What the service expects, and what `SyncSTTLimits` records on the Swift side. SAMPLE_RATE = 16_000 @@ -103,7 +103,7 @@ def _multipart(pcm: bytes, config: dict) -> tuple[bytes, str]: def transcribe(pcm: bytes, api_key: str, instruction: str | None, url: str = DICTATION_URL) -> dict: - """One `/transcribe` round trip. `instruction=None` asks for the service default. + """One `/v1/transcribe` round trip. `instruction=None` asks for the service default. That `None` is the comparison the text harness has never been able to make: an empty `llm` block selects the service's own default wording, so it is the real diff --git a/evals/dictation-prompt/optimize_cleanup_prompt.py b/evals/dictation-prompt/optimize_cleanup_prompt.py index 57c44a7c..d8761a30 100644 --- a/evals/dictation-prompt/optimize_cleanup_prompt.py +++ b/evals/dictation-prompt/optimize_cleanup_prompt.py @@ -7,7 +7,7 @@ What this optimizes ------------------- -Blurt sends one `POST /transcribe` per utterance. The request's `config.prompt` +Blurt sends one `POST /v1/transcribe` per utterance. The request's `config.prompt` steers *transcription*; the `config.llm` block asks the service to run an LLM rewrite over the verbatim transcript — that rewrite is what removes disfluencies and fixes punctuation before the text is pasted. Blurt sends `candidates.PRIOR_WINNER`