Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ Each was tried the other way and reverted. If a task seems to require one, stop
| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. |
| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. |
| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. |
| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. |
| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/v1/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. |
| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. |
| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. |
| Send focus context to the API again | The transcription prompt is switched off at `TranscriptionPrompt.isEnabled`, so `config.prompt` is omitted and no window title, field label, surrounding text or key term leaves the machine. Keep the builder and its wiring — don't route context on by another path. |
Expand Down Expand Up @@ -247,7 +247,7 @@ framework, or notarization rejects the build; roll-forward-only for a bad releas
```text
DictationKeyTap (CGEventTap + DictationKeyGate) → AppCoordinator → DictationSession (actor) → MicCapture
AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /transcribe)
AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /v1/transcribe)
KeyInjector → focused app (clipboard paste via a synthesized ⌘V CGEvent)
```
Expand Down Expand Up @@ -275,7 +275,7 @@ the seam they inject.
### `AssemblyAITranscriber` — `Sources/BlurtEngine/STT/AssemblyAITranscriber.swift`

Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single
`POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob
`POST https://dictation.assemblyai.com/v1/transcribe` with the captured audio as a raw S16LE PCM blob
in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and an `llm`
block). No model header — the service pins the STT model server-side. The config also has a `prompt`
field for steering _transcription_, but nothing is put in it: `TranscriptionPrompt.build` returns
Expand Down
4 changes: 2 additions & 2 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Before the first dictation can succeed the host must have:
```text
press() ──▶ MicCapture.start() release() ──▶ MicCapture.stop() → Data (raw S16LE PCM)
(16 kHz mono 16-bit PCM) AssemblyAITranscriber.transcribe(pcm:sampleRate:context:)
+ focus/context capture (one POST dictation.assemblyai.com/transcribe: STT + LLM rewrite)
+ focus/context capture (one POST dictation.assemblyai.com/v1/transcribe: STT + LLM rewrite)
+ connection warm-up KeyInjector.insert(text, after: priorText)
(clipboard paste via synthesized ⌘V)
```
Expand Down Expand Up @@ -133,7 +133,7 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn
func warmUp() async // optional; no-op default
```

`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.
`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/v1/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.

The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ what each script does, signing — and how changes land;
Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dependencies
Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM,
live level meter; DX7/Juno-106 sound packs
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/v1/transcribe
(STT + LLM rewrite); TranscriptionPrompt (contextual priming,
built and tested but currently switched off — nothing is sent)
Pipeline/ DictationSession actor: press/release/cancel commands, phase
Expand Down
6 changes: 3 additions & 3 deletions Sources/BlurtEngine/STT/AssemblyAITranscriber.swift
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ private let transcriberLog = Logger(subsystem: BlurtIdentity.subsystem, category

/// `TranscriberProtocol` backed by AssemblyAI's **dictation** API.
///
/// A single `POST dictation.assemblyai.com/transcribe` carries the captured
/// A single `POST dictation.assemblyai.com/v1/transcribe` carries the captured
/// audio (raw S16LE PCM, exactly the bytes the mic recorded — there is no
/// re-encoding pass) plus a JSON `config` part, and the response body carries
/// both the verbatim transcript and — when the config requests one via its
Expand Down Expand Up @@ -70,7 +70,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt)
let boundary = "blurt-\(UUID().uuidString)"

var request = URLRequest(url: baseURL.appendingPathComponent("transcribe"))
var request = URLRequest(url: baseURL.appendingPathComponent("v1/transcribe"))
request.httpMethod = "POST"
// Bounds a stalled connection; see `requestTimeoutSeconds` for why an idle
// timeout is the right shape here.
Expand Down Expand Up @@ -103,7 +103,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
/// `transcribe` reuses it instead of paying DNS+TCP+TLS on the hot path
/// (~170 ms cold, more on mobile — measured). A throwaway GET to the host
/// root is enough to establish the HTTP/2 connection `URLSession` then reuses
/// for the POST to `/transcribe`; the response (an auth-less 4xx) is
/// for the POST to `/v1/transcribe`; the response (an auth-less 4xx) is
/// discarded. No key, so it never counts as a transcription. A short timeout
/// keeps a dead network from leaving the task hanging. Fire-and-forget: any
/// error is swallowed — a failed warm-up just means the next request pays
Expand Down
2 changes: 1 addition & 1 deletion Sources/BlurtEngine/STT/CleanupInstruction.swift
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/// The cleanup instruction sent as `config.llm.instruction` (see
/// `AssemblyAITranscriber.DictationConfig`). The service applies it to the
/// verbatim transcript with its own rewrite model, inside the same
/// `/transcribe` call — this is the *server-side* rewrite instruction, not a
/// `/v1/transcribe` call — this is the *server-side* rewrite instruction, not a
/// client-side cleanup pass, and not a `TranscriptionPrompt` change (that
/// prompt steers transcription and deliberately carries no filler-word clause,
/// because disfluency removal is this rewrite's job).
Expand Down
4 changes: 2 additions & 2 deletions Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ struct HTTPClientTests {
let hits = Counter()
let transport = FakeHTTPTransport { request in
_ = hits.next()
guard request.url?.path.hasSuffix("/transcribe") == true,
guard request.url?.path == "/v1/transcribe",
request.httpMethod == "POST"
else { return (404, Data()) }
return (200, json(["text": "um hello world", "llm_response": "Hello world."]))
Expand Down Expand Up @@ -55,7 +55,7 @@ struct HTTPClientTests {
@Test("transcriber succeeds with a real context (which builds no prompt today)")
func transcribeWithContext() async throws {
let transport = FakeHTTPTransport { request in
guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) }
guard request.url?.path == "/v1/transcribe" else { return (404, Data()) }
return (200, json(["text": "hello world"]))
}

Expand Down
8 changes: 4 additions & 4 deletions evals/dictation-prompt/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Dictation cleanup-prompt eval

A DSPy eval that searches for the best **cleanup instruction** for the dictation API's
LLM rewrite — the `config.llm` block Blurt sends on every `/transcribe` request.
LLM rewrite — the `config.llm` block Blurt sends on every `/v1/transcribe` request.

Blurt sends `llm` as an empty object today, which selects the service's own default cleanup
instruction. This harness answers the question that comes next: is there an explicit
Expand Down Expand Up @@ -297,7 +297,7 @@ Two consequences worth holding onto when reading a result:
- A winner is only as transferable as `--model` is representative of the service's rewrite
model, which runs under a ~5s budget and is probably much smaller.
- Confirming a win against the live default would mean sending real audio to
`dictation.assemblyai.com/transcribe` with an empty `llm` block and comparing. That is a
`dictation.assemblyai.com/v1/transcribe` with an empty `llm` block and comparing. That is a
separate exercise, not this one.

## Reading the results
Expand Down Expand Up @@ -481,7 +481,7 @@ real thing.

`--verify-live N` closes that loop. After a winner is picked it takes N **held-out** rows,
speaks the disfluent side with `say`, converts to 16 kHz mono PCM, and POSTs it to the real
`/transcribe` with the winner as `config.llm.instruction`:
`/v1/transcribe` with the winner as `config.llm.instruction`:

```bash
export ASSEMBLYAI_API_KEY=...
Expand Down Expand Up @@ -511,7 +511,7 @@ between them hold. macOS only, and off by default — it costs real transcriptio
| `corpus.py` | Sources, loading, de-tagging, splitting, the echo floor. |
| `disfluency.py` | The seeded, additive disfluency injector. |
| `metrics.py` | Token alignment, the two word-error-rate axes, GEPA feedback text. |
| `live.py` | Synthesis + the real `/transcribe` round trip, for `--verify-live`. |
| `live.py` | Synthesis + the real `/v1/transcribe` round trip, for `--verify-live`. |
| `program.py` | Everything that imports DSPy — the program, metrics adapters, optimizers. |
| `test_eval.py` | Offline tests for injection, scoring, de-tagging, loading, and splitting. |

Expand Down
6 changes: 3 additions & 3 deletions evals/dictation-prompt/live.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

There is something to do about it. The dictation API takes audio, so:

reference text -> `say` -> 16 kHz mono PCM -> POST /transcribe with the
reference text -> `say` -> 16 kHz mono PCM -> POST /v1/transcribe with the
candidate as config.llm.instruction -> score `llm_response`

The response carries both sides of the question. `text` is the verbatim transcript,
Expand Down Expand Up @@ -44,7 +44,7 @@
from corpus import Utterance

#: The dictation endpoint. Same host `AssemblyAITranscriber` posts to.
DICTATION_URL = "https://dictation.assemblyai.com/transcribe"
DICTATION_URL = "https://dictation.assemblyai.com/v1/transcribe"

#: What the service expects, and what `SyncSTTLimits` records on the Swift side.
SAMPLE_RATE = 16_000
Expand Down Expand Up @@ -103,7 +103,7 @@ def _multipart(pcm: bytes, config: dict) -> tuple[bytes, str]:


def transcribe(pcm: bytes, api_key: str, instruction: str | None, url: str = DICTATION_URL) -> dict:
"""One `/transcribe` round trip. `instruction=None` asks for the service default.
"""One `/v1/transcribe` round trip. `instruction=None` asks for the service default.

That `None` is the comparison the text harness has never been able to make: an
empty `llm` block selects the service's own default wording, so it is the real
Expand Down
2 changes: 1 addition & 1 deletion evals/dictation-prompt/optimize_cleanup_prompt.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

What this optimizes
-------------------
Blurt sends one `POST /transcribe` per utterance. The request's `config.prompt`
Blurt sends one `POST /v1/transcribe` per utterance. The request's `config.prompt`
steers *transcription*; the `config.llm` block asks the service to run an LLM
rewrite over the verbatim transcript — that rewrite is what removes disfluencies
and fixes punctuation before the text is pasted. Blurt sends `candidates.PRIOR_WINNER`
Expand Down
Loading