From bc512f2375adc966726d34e5890ab7e76c44a7c7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 01:52:33 +0000 Subject: [PATCH 01/14] Cut AirPods dictation lag at both ends of an utterance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opening the mic on a Bluetooth input makes the system renegotiate the link into its mic-capable mode — hundreds of milliseconds, sometimes over a second — and that link then buffers audio in both directions. Blurt put all of it on the visible hot path, so an AirPods user pressed the key and watched nothing happen, then lost the last word of what they said. Four changes, none of which touch a settled decision (capture stays a fresh AVAudioRecorder per session): 1. MicCapture re-arms its prepared recorder after every capture, not just at launch. The cost is paid at prepareToRecord(), i.e. per session, so warming only the first one hid it for one dictation out of N. The warm recorder is validated against the live default input's UID before reuse — AVAudioRecorder resolves its device once and never re-resolves — and expires after 60s idle, because holding the input open is what pins AirPods in the profile where output audio is degraded. 2. stop() keeps capturing for a further 220ms when the session's input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated. It runs after .transcribing is claimed, so it delays the transcript, never the "it heard me" cue. Cancels take the new MicCaptureProtocol.cancelCapture() instead, which skips both the linger and the file read-back. 3. A new PipelinePhase.starting is claimed before mic.start(), so the pill answers the key-down rather than the hardware route. It is presented as "Starting…", never as live capture, so the rule that the UI must not claim audio is being recorded before it is still holds. 4. RecordingCueGate rides PipelinePhase.isCapturing, so the start chime fires at the press. CueSoundPlayer re-primes its players on output route changes, since opening the mic drops the format its pre-roll was made against — the first chime after that flip is the one that stalls, and it is the chime at the start of a dictation. The CoreAudio routing reads behind 1, 2 and 4 live in AudioRoute (internal) and AudioRouteMonitor (public, for the cue players); both are excluded from the coverage gate for the same reason MicCapture is. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 69 +++++- App/Blurt/Blurt/CueSoundPlayer.swift | 38 ++++ App/Blurt/Blurt/Overlay/OverlayView.swift | 15 +- App/Blurt/Blurt/UITestSupport.swift | 4 +- BLURTENGINE.md | 23 +- Sources/BlurtEngine/Audio/AudioRoute.swift | 110 +++++++++ .../BlurtEngine/Audio/AudioRouteMonitor.swift | 164 ++++++++++++++ Sources/BlurtEngine/Audio/MicCapture.swift | 211 ++++++++++++++++-- .../Audio/MicCaptureProtocol.swift | 16 ++ .../Pipeline/DictationSession.swift | 15 +- .../BlurtEngine/Pipeline/MenuBarStatus.swift | 6 +- .../BlurtEngine/Pipeline/OverlayUIState.swift | 15 +- .../BlurtEngine/Pipeline/PipelinePhase.swift | 32 ++- .../Pipeline/RecordingCueGate.swift | 25 ++- .../DictationSessionTests.swift | 47 ++++ .../BlurtEngineTests/MenuBarStatusTests.swift | 3 + .../MicCaptureProtocolDefaultsTests.swift | 38 +++- .../OverlayUIStateTests.swift | 12 +- .../BlurtEngineTests/PipelinePhaseTests.swift | 28 +++ .../RecordingCueGateTests.swift | 59 +++-- .../Stubs/StubMicCapture.swift | 10 + scripts/check.sh | 9 +- 22 files changed, 875 insertions(+), 74 deletions(-) create mode 100644 Sources/BlurtEngine/Audio/AudioRoute.swift create mode 100644 Sources/BlurtEngine/Audio/AudioRouteMonitor.swift diff --git a/AGENTS.md b/AGENTS.md index 1d497c98..5c8610c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,8 @@ the design; BLURTENGINE.md covers the _what_ of the API surface. ```text Sources/BlurtEngine/ the engine (dependency-free Swift package) - Audio/ MicCapture (+meter), SoundPack/Catalog/Store — record cues + Audio/ MicCapture (+meter), AudioRoute(+Monitor) — CoreAudio routing facts, + SoundPack/Catalog/Store — record cues Config/ Keychain-backed API key, key terms, developer mode, DefaultsKey + PersistedSettings (every defaults key, and the reset sweep over them) FocusCapture/ Accessibility reads of the frontmost app / focused field @@ -382,13 +383,45 @@ A **fresh recorder per session** resolves the current default input device at `r is deliberate — see [Settled decisions](#settled-decisions--dont-reintroduce-these) for the `AVAudioEngine` failure it replaced. +**Bluetooth inputs are the reason for three of this actor's moving parts.** Opening the mic on +AirPods (or any Bluetooth headset) makes the system renegotiate the link into its mic-capable mode — +hundreds of milliseconds, sometimes over a second — and that link then buffers audio in both +directions. So: + +- **The warm recorder is re-armed after every capture**, not just at launch. The cost above is paid + at `prepareToRecord()`, i.e. per session, so warming only the first one hid it for one dictation + out of N. `stop()`/`cancelCapture()` schedule a re-warm; `start()` consumes it. +- **A warm recorder is validated before reuse.** `AVAudioRecorder` resolves its device once and + never re-resolves, so `MicCapture` records the default input's UID (`AudioRoute.currentInput()`) + alongside the warm recorder and discards it when the device has changed — otherwise a recorder + warmed before the user connected their AirPods would keep recording the built-in mic. Unknown + counts as changed. +- **The warm recorder expires** (`preparedRecorderLifetime`, 60 s). A prepared recorder holds the + input device open, which is exactly what pins AirPods in the profile where _output_ audio is + degraded — so it is not held indefinitely. Back-to-back dictations land inside the window; a press + past it just prepares lazily, which is the pre-re-warm behavior. + +`stop()` also waits out `bluetoothTailLinger` (220 ms) before ending the recording **when the +session's input is Bluetooth**, so speech still travelling over the link lands in the file instead of +being truncated — the missing last word. It runs after `.transcribing` is claimed, so it delays the +transcript, never the "it heard me" cue. Cancels take `cancelCapture()` instead, which skips both the +linger and the file read-back: the audio is being discarded, so neither is worth delaying the user's +cancel for. + +The routing facts behind all of that live in **`AudioRoute`** (`Audio/AudioRoute.swift`, internal): +which device is the default input, its UID, and whether its transport is Bluetooth. Its sibling +**`AudioRouteMonitor`** (public) publishes output-route changes for the cue players — see +[Settings, persistence, and cues](#settings-persistence-and-cues). Both are excluded from the +coverage gate for the same reason +`MicCapture` is: they answer questions only real hardware can answer. + The overlay meter (`levels`) comes from the recorder's dBFS power on a ~20 Hz timer (`MicCapture.meterIntervalSeconds` — public because the pill caps its animation redraws to the same cadence and reads it from here rather than restating it), mapped to `0…1` by `linearLevel(fromPowerDB:)` and floored so room ambient reads as empty bars rather than a meter that -never rests. `levels` and `warmUp()` are part of `MicCaptureProtocol` itself with an empty-stream / -no-op default, so stubs conform with just `start()`/`stop()` while hosts still read the meter through -the seam they inject. +never rests. `levels`, `warmUp()` and `cancelCapture()` are part of `MicCaptureProtocol` itself with +empty-stream / no-op / stop-and-discard defaults, so stubs conform with just `start()`/`stop()` while +hosts still read the meter through the seam they inject. ### `AssemblyAITranscriber` — `Sources/BlurtEngine/STT/AssemblyAITranscriber.swift` @@ -453,8 +486,9 @@ stream and signposts) and `+Pipeline.swift` (the post-release transcribe→injec It exposes `press()` / `release()` / `cancel()` / `cancelRecording()`, a synchronous fire-and-forget `submit(_: Command)` mirroring those four for callback-shaped hosts (commands run in exact emit order — the tap wires straight into it, no per-callback `Task` spawning), and a -`phase: PipelinePhase` (`idle | recording | transcribing | injecting | failed | cancelled`, plus the -terminal successes `pasted` / `noTarget`). `phaseStream()` yields the current phase immediately then +`phase: PipelinePhase` (`idle | starting | recording | transcribing | injecting | failed | +cancelled`, plus the terminal successes `pasted` / `noTarget`). `phaseStream()` yields the current +phase immediately then every transition, and is **multi-observer** (one continuation per call), though hosts should still render from one consumer and project the phase into their own state. @@ -472,14 +506,21 @@ round trip, and the log wrote to the user's real `~/Library/Logs/Blurt`. STT err are wrapped in `.sttFailed`. The pipeline is just transcribe → inject, and an empty transcript returns to `.idle` without injecting. -Three perceived-latency choices to preserve: +Four perceived-latency choices to preserve: +- `press()` claims `.starting` _before_ `mic.start()`, so the pill and the start chime answer the + key-down rather than waiting on the hardware route. `.starting` is presented as "Starting…", never + as live capture — the rule that the UI must not claim audio is being recorded before it is holds, + and `.recording` is still only claimed once `mic.start()` has succeeded. `RecordingCueGate` rides + `PipelinePhase.isCapturing` (`.starting || .recording`) for the same reason, so the chime fires at + the press and stays silent across `.starting` → `.recording`. - `.injecting` projects to `OverlayUIState.processing`, **not** `.idle` — the shell reads an idle projection as "dismiss", so mapping this working phase to idle faded the pill out mid-dictation and blinked it back for "Pasted". - `release()` claims `.transcribing` _before_ `mic.stop()`, so the stop chime and pill switch fire at key-up rather than after the recording is read back. This ordering also closes the double-release - window `ReleaseRaceTests` pins. + window `ReleaseRaceTests` pins — and it is what lets `MicCapture`'s Bluetooth tail linger sit + inside `stop()` without the user ever waiting on it. - The press-time Accessibility context read is consumed with a bounded wait (`contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall. @@ -651,8 +692,16 @@ Record cues: **`SoundPack`** is a selectable start/stop chime voice (vintage syn `App/Blurt/Blurt/Resources/Sounds/`), listed by **`SoundPackCatalog.swift`**, which is _generated_ by `scripts/generate-sounds.swift` alongside the audio. Regenerate both halves together — `check.sh`'s sound-catalog guard exists because a drift plays silence with no error. **`RecordingCueGate`** is the -pure edge detector deciding when the chimes fire; the AppKit `CueSoundPlayer` just plays what it -resolves. +pure edge detector deciding when the chimes fire — on the _capture_ edge +(`PipelinePhase.isCapturing`), so the start chime lands at the press rather than after the mic +route comes up; the AppKit `CueSoundPlayer` just plays what it resolves. + +`CueSoundPlayer` decodes and pre-rolls the players once so the first chime never stalls the pill, and +that pre-roll is bound to the output route it was made against. Blurt's own capture invalidates it: +opening the mic flips AirPods out of their output-only profile, dropping the output format underneath +the primed players. So the player observes **`AudioRouteMonitor.outputRouteChanges`** and reloads — +the monitor watches both the default output _device_ (a user switch) and the current device's nominal +sample rate (the profile flip), re-targeting the second listener whenever the first fires. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never written to disk). **`DictationLog`** appends each completed dictation with its context snapshot to diff --git a/App/Blurt/Blurt/CueSoundPlayer.swift b/App/Blurt/Blurt/CueSoundPlayer.swift index 333c735c..078cd032 100644 --- a/App/Blurt/Blurt/CueSoundPlayer.swift +++ b/App/Blurt/Blurt/CueSoundPlayer.swift @@ -15,6 +15,15 @@ final class CueSoundPlayer { /// pipeline phase to a cue lives in the engine (`RecordingCueGate`), where /// `swift test` covers it; this player just plays whatever it resolves to. private var cueGate = RecordingCueGate() + /// Fires when the output route changes under the pre-rolled players. Blurt's + /// own capture is the usual cause: opening the mic flips AirPods out of their + /// output-only profile, which drops the output format the players were primed + /// against — so the very next chime is the one that stalls, and that's the + /// chime at the start of a dictation. + private let routeMonitor = AudioRouteMonitor() + /// Kept alive for the app's lifetime; assignment is the use. Nil until + /// `prime()` starts it, which is also what makes starting idempotent. + private var routeObserver: Task? /// The cues are deliberate UI accents, not music — they are normalized to a /// hot peak, so play them well below full scale so they read as a soft chime @@ -36,6 +45,35 @@ final class CueSoundPlayer { /// decode never sits on the main thread during startup. func prime() { Task { await loadCurrentPack() } + startObservingRoute() + } + + /// The player is owned for the whole app session, so this never runs in + /// practice — but cancelling the observer mirrors the `[weak self]` care below + /// and documents that its lifetime is owned rather than leaked. + deinit { + routeObserver?.cancel() + } + + /// Re-primes the players whenever the output route changes, so the pre-roll + /// `prime()` bought at launch survives a device switch or a profile flip. + /// Idempotent — `prime()` runs on every "app is ready" transition, and only + /// the first call installs the observer. + /// + /// A full reload rather than a bare `prepareToPlay()`: route changes are rare + /// (a handful an hour at most), the decode runs off the main actor like every + /// other load, and re-creating the players is the one thing guaranteed to + /// leave them primed against the *current* route. + private func startObservingRoute() { + guard routeObserver == nil else { return } + let changes = routeMonitor.outputRouteChanges + routeObserver = Task { [weak self] in + for await _ in changes { + guard let self else { return } + self.loadedPack = nil + await self.loadCurrentPack() + } + } } /// Reloads the players for a newly selected pack and previews the new voice diff --git a/App/Blurt/Blurt/Overlay/OverlayView.swift b/App/Blurt/Blurt/Overlay/OverlayView.swift index 559a4fba..9741ed92 100644 --- a/App/Blurt/Blurt/Overlay/OverlayView.swift +++ b/App/Blurt/Blurt/Overlay/OverlayView.swift @@ -32,7 +32,7 @@ struct OverlayView: View { switch state { case .error: return Color(red: 0.62, green: 0.13, blue: 0.13) - case .recording, .processing, .pasted, .noTarget, .idle: + case .starting, .recording, .processing, .pasted, .noTarget, .idle: return Color(white: 0.16) } } @@ -84,6 +84,19 @@ struct OverlayView: View { // background would collapse with it) keeps the pill's shape intact for // `hide()`'s pre-hide reset. Color.clear + case .starting: + // The mic is opening; nothing is being captured yet. Styled exactly like + // "Transcribing…"/"Pasted" (same status-line type, tracking, and cyan + // --ice) so the starting → recording hand-off reads as one status line + // rather than a new kind of alert — and deliberately *not* the `● REC` + // tag or the meter, which would claim live capture. + // + // On a built-in mic this is on screen for a frame or two, inside the + // pill's own 0.08 s fade-in, so it blends into the appearance rather than + // flashing; on a Bluetooth input it holds for as long as the link takes, + // which is the whole point. + StatusLineText("Starting…") + .transition(.opacity) case .recording: // "● REC" tag beside the live waveform, mirroring the site demo's recording // pill (magenta tag + bars). The bars fill the width left of the tag. diff --git a/App/Blurt/Blurt/UITestSupport.swift b/App/Blurt/Blurt/UITestSupport.swift index 57d407d2..e0de29f7 100644 --- a/App/Blurt/Blurt/UITestSupport.swift +++ b/App/Blurt/Blurt/UITestSupport.swift @@ -159,8 +159,8 @@ extension DictationComponents { /// The all-stub pipeline used under UI testing: no mic, no network, no /// Accessibility paste. `UITestMic` inherits `MicCaptureProtocol`'s default - /// empty `levels` stream (the overlay meter isn't asserted) and no-op - /// `warmUp()`. + /// empty `levels` stream (the overlay meter isn't asserted), no-op + /// `warmUp()`, and stop-and-discard `cancelCapture()`. static func uiTest() -> DictationComponents { DictationComponents( mic: UITestMic(), diff --git a/BLURTENGINE.md b/BLURTENGINE.md index a0ac2609..8d6dd986 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -60,7 +60,7 @@ Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. - **Cleanup happens server-side, and it's optional.** The request's `llm` block asks the service to apply our own cleanup instruction (`CleanupInstruction.text` — delete disfluencies, change nothing else) to the verbatim transcript inside the same call. It is the only instruction on the request: the separate `config.prompt` field, which primes the _transcription_, is switched off at `TranscriptionPrompt.isEnabled` and omitted from every request. The block is gated by the **enhanced transcripts** setting (`EnhancedTranscriptsStore`, on by default): turned off, the config omits `llm` and the verbatim transcript is pasted as spoken. The user's **custom style instructions** (`CustomStyleStore`, empty by default) are appended to that instruction via `CleanupInstruction.sendable(appending:)`, trimmed to the headroom the API's 2048 instruction cap leaves (measured in UTF-8 bytes, the conservative bound — the cap's own unit is unmeasured); blank means the base instruction goes out unchanged. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. -- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. +- **Latency is pre-paid where possible.** `press()` claims `.starting` before it touches the mic, so a host's pill and start cue answer the keypress rather than the hardware route (which on a Bluetooth input is the slowest thing in the press path). `MicCapture` re-arms its prepared recorder after every capture so that route activation is paid between dictations rather than during one. `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. ## DictationSession @@ -83,14 +83,20 @@ For callback-shaped hosts that can't `await` — an event tap, a button action `phase` / `phaseStream()` expose the pipeline's `PipelinePhase`: ```text -idle → recording → transcribing → injecting → pasted | noTarget - │ │ - └── failed(BlurtError) / cancelled (from any stage) +idle → starting → recording → transcribing → injecting → pasted | noTarget + │ │ + └── failed(BlurtError) / cancelled (from any stage) ``` +`starting` is claimed the moment a press is accepted, _before_ the mic is opened, so your UI can +answer the keypress instead of the hardware. It is not live capture — opening a Bluetooth input takes +hundreds of milliseconds — so present it as "starting", never as recording; `recording` follows only +once audio is genuinely being captured. `PipelinePhase.isCapturing` covers both if you want the +"a dictation is in progress" bit rather than the distinction. + - `phaseStream()` yields the current phase immediately, then every transition. It is a **multi-observer** stream: every call gets its own continuation and all of them see later transitions, so an extra consumer (a diagnostic, a second window) is safe. Still, prefer one renderer that projects the phase into your own state over a fan-out of long-lived consumers — one source of UI truth is easier to reason about than several. - `.pasted` and `.noTarget` are terminal _success_ states, not errors. `.noTarget` means transcription worked but nothing editable was focused (or the target app quit), so the text was left on the clipboard — show a quiet "copied" notice, not a failure. -- Two ready-made projections keep UI mapping out of your shell: `phase.overlayState` (`OverlayUIState`: idle / recording / processing / error(message:) / pasted / noTarget, with accessibility labels and — for the transient notices — `noticeDwellSeconds`, how long to hold one before reverting to idle) and `phase.menuBarStatus` (coarser: idle / recording / transcribing, never shows errors, with `symbolName`/`accessibilityLabel` presentation). +- Two ready-made projections keep UI mapping out of your shell: `phase.overlayState` (`OverlayUIState`: idle / starting / recording / processing / error(message:) / pasted / noTarget, with accessibility labels and — for the transient notices — `noticeDwellSeconds`, how long to hold one before reverting to idle) and `phase.menuBarStatus` (coarser: idle / recording / transcribing, never shows errors, with `symbolName`/`accessibilityLabel` presentation — `starting` rests at idle there rather than claiming a live mic). - Pill geometry is available too, if you're drawing something like Blurt's overlay: `OverlayPlacement` resolves how big the panel is (`panelSize(pillSize:shadowMargin:)`, sized to hold the pill plus room for its shadow) and where it goes (clearance, clamping a dragged origin back on screen), and `MeterBarGeometry` gives the level meter its shape. Build a `MeterBarRow(availableSize:)` once per layout — it resolves how many bars fit and how tall they may be — then ask it for `height(at:level:time:animated:)` per bar; `MeterBarGeometry.breathingOpacity(time:period:minOpacity:)` is the pulse the record dot and status label share. All pure math; pass `animated: false` to honor Reduce Motion. ### Errors @@ -116,16 +122,21 @@ All cases are `LocalizedError` with user-ready `errorDescription` strings, and ` ```swift func start() async throws func stop() async throws -> Data // raw S16LE mono PCM, 16 kHz, in order +func cancelCapture() async throws // stop and discard; default: stop-and-drop var levels: AsyncStream { get } // 0…1 meter; default: empty stream func warmUp() async // pre-open the device; default: no-op ``` -Only `start()`/`stop()` must be implemented — `levels` and `warmUp()` have defaults, so a stub or headless capture conforms for free while hosts still read the meter and warm the device through the same seam they inject. +Only `start()`/`stop()` must be implemented — `cancelCapture()`, `levels` and `warmUp()` have defaults, so a stub or headless capture conforms for free while hosts still read the meter and warm the device through the same seam they inject. + +`cancelCapture()` is what the session calls when a dictation is cancelled, and it exists because the two teardowns want opposite things: `stop()` may legitimately spend time preserving the audio, while a cancel has nothing to preserve and must take effect at once. Override it only if stopping cheaply differs from stopping carefully in your capture. `MicCapture` records with `AVAudioRecorder` straight to a temp 16 kHz / mono / 16-bit PCM WAV — exactly the geometry the dictation API wants — and reads it back as raw S16LE bytes on `stop()` (no float detour; the blob uploads as-is). A **fresh recorder per session** resolves the current default input device at `record()` time, which is why device switches (headset ↔ built-in) just work. Do **not** replace this with a long-lived `AVAudioEngine`/`installTap` graph: that design was tried, bound itself to one device, and failed with `-10868` or all-zero buffers on device switches. `MicCapture`'s `levels` is a ~20 Hz meter of the recorder's dBFS power mapped to `0…1` (floored at −50 dBFS so room ambient reads as silence) — feed it to a voice-bars view; it costs nothing when unobserved. Its `warmUp()` pre-creates and prepares a recorder so the first `start()` skips hardware route discovery (Blurt calls it at launch, once mic permission is granted, so warming never triggers the permission prompt). +**Bluetooth inputs get three accommodations**, because opening the mic on AirPods makes the system renegotiate the link into its mic-capable mode — hundreds of milliseconds — and that link then buffers audio. `MicCapture` re-arms the warm recorder after _every_ capture rather than only at launch (the cost is per session, paid at `prepareToRecord()`); it records the default input's UID alongside the warm recorder and discards it when the device has changed, since `AVAudioRecorder` resolves its device once and never re-resolves; and it releases an unused warm recorder after 60 s, because holding the input device open is what pins AirPods in the profile where output audio is degraded. On top of that, `stop()` keeps capturing for a further 220 ms when the input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. `cancelCapture()` skips both that linger and the file read-back. + ### `TranscriberProtocol` → `AssemblyAITranscriber` ```swift diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift new file mode 100644 index 00000000..58afd878 --- /dev/null +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -0,0 +1,110 @@ +import CoreAudio +import Foundation + +/// Read-only queries against the system's current audio routing — the two facts +/// about the mic that `AVFoundation` doesn't expose but the capture path needs: +/// +/// 1. **Which device is the default input**, so `MicCapture` can tell whether a +/// recorder it prepared earlier is still bound to the device the user is +/// about to speak into. `AVAudioRecorder` resolves the route at +/// `prepareToRecord()` time and never re-resolves it, so a recorder warmed +/// before the user connected their AirPods would silently record from the +/// built-in mic. +/// 2. **Whether that device is a Bluetooth one**, whose link buffers audio for +/// a couple of hundred milliseconds — the tail `MicCapture.stop()` waits for +/// rather than truncating (see `bluetoothTailLinger`). +/// +/// Internal, not public: the app never asks these directly (it observes route +/// *changes* through `AudioRouteMonitor`), and `.periphery.yml` runs with +/// `retain_public: false`, so a `public` symbol only the engine reaches fails +/// the unused-code scan. +enum AudioRoute { + /// Identity plus link character of the default input device, read together in + /// one pass so the capture path makes a single trip through CoreAudio per + /// session rather than one per question. + struct InputSnapshot: Equatable, Sendable { + /// The device's persistent UID. Non-optional on purpose: an unreadable UID + /// means "we can't tell which device this is", which must not compare equal + /// to another unknown — so `currentInput()` returns nil instead, and callers + /// treat that as "assume it changed". + let uid: String + /// Whether the device's transport is Bluetooth, i.e. whether its capture + /// path carries link latency worth lingering for. + let isBluetooth: Bool + } + + /// The default input device as an `InputSnapshot`, or nil when there is no + /// input device (all of them unplugged or asleep) or CoreAudio refused either + /// read. Nil is the conservative answer everywhere it's consumed: an unknown + /// input invalidates a warm recorder rather than silently keeping one bound to + /// a device that may have gone away. + static func currentInput() -> InputSnapshot? { + guard let deviceID = defaultDeviceID(for: kAudioHardwarePropertyDefaultInputDevice), + let uid = uid(of: deviceID) + else { return nil } + return InputSnapshot(uid: uid, isBluetooth: isBluetooth(deviceID)) + } + + /// The system's current default *output* device — what `AudioRouteMonitor` + /// hangs its format listener on. Nil when there is none, or the read failed. + static func defaultOutputDeviceID() -> AudioDeviceID? { + defaultDeviceID(for: kAudioHardwarePropertyDefaultOutputDevice) + } + + // MARK: - CoreAudio reads + + /// The device the system object reports for `selector` (a default-device + /// property). Nil covers both a failed read and the "no such device" sentinel, + /// which callers treat identically. + private static func defaultDeviceID(for selector: AudioObjectPropertySelector) -> AudioDeviceID? { + var address = AudioObjectPropertyAddress( + mSelector: selector, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + var deviceID = AudioDeviceID(0) + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData( + AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, &deviceID) + // 0 is `kAudioObjectUnknown` — "there is no such device" — spelled as the + // literal so this doesn't depend on how the constant imports. + guard status == noErr, deviceID != 0 else { return nil } + return deviceID + } + + /// The device's persistent UID string. `Unmanaged` rather than a + /// bridged `CFString?`: the property returns a +1 reference, so the ownership + /// transfer has to be spelled out (`takeRetainedValue`) instead of left to an + /// implicit bridge that would over-release it. + private static func uid(of deviceID: AudioDeviceID) -> String? { + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyDeviceUID, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + var value: Unmanaged? + var size = UInt32(MemoryLayout?>.size) + let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &value) + guard status == noErr, let value else { return nil } + return value.takeRetainedValue() as String + } + + /// Whether the device is reached over Bluetooth. Both transport types count: + /// AirPods and other wireless headsets report the classic `bluetooth` + /// transport, and LE Audio devices report `bluetoothLE` — the link-latency + /// characteristic the callers care about is the same either way. + /// + /// A failed read answers `false`: the conservative default is "no linger", + /// since padding every wired capture with a delay would be a worse regression + /// than losing the tail on a device we couldn't classify. + private static func isBluetooth(_ deviceID: AudioDeviceID) -> Bool { + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyTransportType, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + var transport = UInt32(0) + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &transport) + guard status == noErr else { return false } + return transport == kAudioDeviceTransportTypeBluetooth + || transport == kAudioDeviceTransportTypeBluetoothLE + } +} diff --git a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift new file mode 100644 index 00000000..394816bd --- /dev/null +++ b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift @@ -0,0 +1,164 @@ +import CoreAudio +import Foundation +import os + +/// Ticks whenever the system's audio **output** route changes in a way that +/// invalidates an already-pre-rolled `AVAudioPlayer`. +/// +/// This exists for the record cue chimes. `CueSoundPlayer` decodes and +/// `prepareToPlay()`s them once at launch so the first chime never stalls the +/// pill — but that pre-roll is bound to the output route it was made against, +/// and Blurt's own capture is what invalidates it: opening the mic flips AirPods +/// out of their output-only profile into the bidirectional one, which drops the +/// output format underneath the primed players. The first chime after such a +/// flip is exactly the one that stalls, which is the chime at the start of a +/// dictation. +/// +/// Two properties are watched, because "the route changed" has two shapes: +/// +/// - `kAudioHardwarePropertyDefaultOutputDevice` on the system object — the user +/// switched output devices (built-in speakers → AirPods). +/// - `kAudioDevicePropertyNominalSampleRate` on whichever device is *currently* +/// default — the same device renegotiated its format, which is what the +/// profile flip looks like from CoreAudio. That listener is re-targeted +/// whenever the first one fires, so it always tracks the live device. +/// +/// Public because the app owns the cue players; the engine's own use of +/// CoreAudio routing (`AudioRoute`) stays internal. +/// +/// `@unchecked Sendable` because the listener registrations below are confined +/// to `queue` rather than protected by a lock — see their declarations. +public final class AudioRouteMonitor: @unchecked Sendable { + private static let logger = Logger(subsystem: BlurtIdentity.subsystem, category: "AudioRoute") + + /// Fires once per observed route change. `.bufferingNewest(1)` because this is + /// an invalidation signal, not a log: a consumer that was busy through three + /// changes needs to re-prime once, not three times. + public let outputRouteChanges: AsyncStream + private let continuation: AsyncStream.Continuation + + /// The queue CoreAudio delivers every listener callback on, and the one place + /// the registrations below are touched. Serial, so a re-target triggered by a + /// default-device change can't interleave with itself. + private let queue: DispatchQueue + + /// The registered listener blocks, kept so they can be handed back to + /// CoreAudio — removal matches on block identity, so a re-created block would + /// deregister nothing. + /// + /// `nonisolated(unsafe)` rather than lock-guarded: every read and write happens + /// inside a `queue` block, including the initial registration (`init` wraps it + /// in `queue.sync` precisely so a listener can't fire before the property + /// recording it has been written). Dispatch's serial ordering supplies both the + /// exclusion and the memory barriers a lock would. + private nonisolated(unsafe) var systemListener: AudioObjectPropertyListenerBlock? + private nonisolated(unsafe) var deviceListener: + (id: AudioDeviceID, block: AudioObjectPropertyListenerBlock)? + + public init() { + let (stream, continuation) = AsyncStream.makeStream(bufferingPolicy: .bufferingNewest(1)) + self.outputRouteChanges = stream + self.continuation = continuation + self.queue = DispatchQueue(label: "\(BlurtIdentity.subsystem).AudioRoute") + queue.sync { + installDefaultDeviceListener() + retargetFormatListener() + } + } + + /// The monitor is owned for the app's lifetime, so this never runs in + /// practice — but deregistering mirrors the `[weak self]` care below and + /// documents that the CoreAudio registrations are owned rather than leaked: a + /// listener left behind outlives the monitor, since CoreAudio retains the block + /// and nothing else would ever hand it back. + /// + /// The `queue.sync` cannot deadlock: the listener blocks hold `self` weakly, so + /// `queue` never owns the last reference and this deinit never runs on it. + deinit { + continuation.finish() + queue.sync { + if let systemListener { + var address = Self.defaultOutputDeviceAddress + _ = AudioObjectRemovePropertyListenerBlock( + AudioObjectID(kAudioObjectSystemObject), &address, queue, systemListener) + } + if let deviceListener { + var address = Self.sampleRateAddress + _ = AudioObjectRemovePropertyListenerBlock( + deviceListener.id, &address, queue, deviceListener.block) + } + systemListener = nil + deviceListener = nil + } + } + + // MARK: - Registration (queue-confined) + + /// Watches for the default output device itself changing. Registered once and + /// never re-targeted — the system object is always there. + private func installDefaultDeviceListener() { + var address = Self.defaultOutputDeviceAddress + // `[weak self]`, so CoreAudio's strong hold on the block doesn't keep the + // monitor alive forever — and so a callback landing during teardown finds + // nil rather than a half-destroyed object. + let block: AudioObjectPropertyListenerBlock = { [weak self] _, _ in + guard let self else { return } + // Re-target first, then publish: a consumer that re-primes on this tick + // should already be behind a listener pointed at the new device. + self.retargetFormatListener() + self.continuation.yield() + } + let status = AudioObjectAddPropertyListenerBlock( + AudioObjectID(kAudioObjectSystemObject), &address, queue, block) + guard status == noErr else { + Self.logger.error("default-output listener failed: \(status)") + return + } + systemListener = block + } + + /// Points the format listener at the current default output device, removing + /// the one on the previous device. A no-op when the device hasn't actually + /// changed, so a notification that resolves to the same device doesn't churn + /// the registration. + private func retargetFormatListener() { + let device = AudioRoute.defaultOutputDeviceID() + if let existing = deviceListener { + guard existing.id != device else { return } + var address = Self.sampleRateAddress + _ = AudioObjectRemovePropertyListenerBlock(existing.id, &address, queue, existing.block) + deviceListener = nil + } + guard let device else { return } + let block: AudioObjectPropertyListenerBlock = { [weak self] _, _ in + self?.continuation.yield() + } + var address = Self.sampleRateAddress + let status = AudioObjectAddPropertyListenerBlock(device, &address, queue, block) + guard status == noErr else { + Self.logger.error("output-format listener failed: \(status)") + return + } + deviceListener = (id: device, block: block) + } + + // MARK: - Property addresses + + // Computed, not stored: each caller needs its own mutable copy to pass `inout` + // to CoreAudio anyway, so a shared constant would only add a global whose + // `Sendable`-ness depends on how the C struct imports. + + private static var defaultOutputDeviceAddress: AudioObjectPropertyAddress { + AudioObjectPropertyAddress( + mSelector: kAudioHardwarePropertyDefaultOutputDevice, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + } + + private static var sampleRateAddress: AudioObjectPropertyAddress { + AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyNominalSampleRate, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + } +} diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 11a8bfd4..41b26a5f 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -26,15 +26,39 @@ public actor MicCapture: MicCaptureProtocol { /// resampling or re-encoding pass. private static let targetSampleRate = Double(SyncSTTLimits.sampleRate) - /// Pre-prepared in `warmUp()` so the *first* dictation doesn't pay hardware - /// route discovery on the hot path; consumed by the first `start()`. Every - /// later session creates a fresh recorder instead, so a device switch is always - /// reflected. (A switch in the brief launch→first-dictation window is not worth - /// guarding against — it would cost a CoreAudio device listener for a case that - /// effectively never happens.) + /// A recorder prepared ahead of the press so `start()` doesn't pay hardware + /// route activation on the hot path. Filled by `warmUp()` at launch and + /// **re-filled after every capture** (see `scheduleRewarm`), because that cost + /// is paid per session, not once: `prepareToRecord()` is where the route is + /// resolved and opened, and on a Bluetooth input that means renegotiating the + /// link into its mic-capable mode — hundreds of milliseconds, sometimes over a + /// second, during which the user has pressed the key and nothing has happened. + /// Warming only the first session (the previous behavior) hid that cost for one + /// dictation out of every N. + /// + /// Still a *fresh recorder per session*, which is the invariant the + /// `AVAudioEngine` rewrite bought: the warm recorder is validated against the + /// live default input before it is used (`takeWarmRecorder`) and discarded + /// rather than reused when the device has changed underneath it. private var preparedRecorder: AVAudioRecorder? + /// The default input `preparedRecorder` was built against. `AVAudioRecorder` + /// resolves its device once, at `prepareToRecord()`, and never re-resolves — + /// so without this a recorder warmed while the built-in mic was default would + /// keep recording from it after the user connected their AirPods. + private var preparedInput: AudioRoute.InputSnapshot? + /// Releases `preparedRecorder` once it has gone unused for + /// `preparedRecorderLifetime`. See that constant for why holding one open + /// forever is not an option. + private var preparedExpiry: Task? + /// The recorder for the in-flight session; nil between `stop()` and `start()`. private var activeRecorder: AVAudioRecorder? + /// Whether the in-flight session's input is a Bluetooth device, sampled once + /// at `start()`. Read by `stop()` to decide on the tail linger — sampled at + /// start rather than re-read at stop so a device switch mid-utterance can't + /// make the two halves of one capture disagree. + private var activeInputIsBluetooth = false + /// Polls the active recorder's meter and feeds `levels` while recording. private var meterTask: Task? /// The last value `emitLevel` put on the stream, so an unchanged tick can be @@ -56,6 +80,35 @@ public actor MicCapture: MicCaptureProtocol { /// `meterIntervalSeconds` as the `Duration` the meter task sleeps for. private static let meterInterval = Duration.seconds(meterIntervalSeconds) + /// How much longer capture runs after the key-up that ends it, when the input + /// is a Bluetooth device. + /// + /// A Bluetooth link buffers: audio the user has already spoken is still in + /// flight when `stop()` is called, and `recorder.stop()` drops it — which is + /// why the last word of a dictation goes missing on AirPods and the app reads + /// as running behind the speaker. The linger is deliberately shorter than a + /// typical link's worst case: it buys back the common tail without making + /// every dictation feel sluggish, and it costs nothing on a wired input, where + /// it is skipped entirely. + /// + /// The delay lands *after* `.transcribing` is claimed (see + /// `DictationSession.performRelease`), so it delays the transcript, never the + /// user's "it heard me" cue. Cancels skip it — see `cancelCapture()`. + static let bluetoothTailLinger = Duration.milliseconds(220) + + /// How long a prepared-but-unused recorder is held before being torn down. + /// + /// The warm recorder is the fix for per-session route activation, but it is + /// not free to hold: a prepared recorder keeps the input device open, and an + /// open input is exactly what pins AirPods in their mic-capable profile, where + /// *output* audio is degraded. Holding one indefinitely would trade dictation + /// latency for permanently worse music. This bounds that: back-to-back + /// dictations — the case the warm recorder exists for — land well inside the + /// window, and a user who stops dictating gets their output route back shortly + /// after. The next press past the window simply prepares lazily, which is the + /// behavior that shipped before the re-warm existed. + private static let preparedRecorderLifetime = Duration.seconds(60) + public init() { // The continuation is fed from a ~20 Hz meter timer; the levels stream is a // meter, not the captured signal — the consumer only renders the most recent @@ -69,22 +122,19 @@ public actor MicCapture: MicCaptureProtocol { /// hardware route discovery. Does NOT begin capture — no mic indicator. Safe to /// call multiple times; a failure here just leaves `start()` to prepare lazily. public func warmUp() { - guard preparedRecorder == nil else { return } - do { - preparedRecorder = try Self.makeRecorder() - Self.logger.info("warmUp prepared recorder") - } catch { - Self.logger.error("warmUp failed: \(error.localizedDescription, privacy: .public)") - } + guard preparedRecorder == nil, activeRecorder == nil else { return } + prepareWarmRecorder() } public func start() async throws { - // Reuse the warm recorder for the first session; otherwise build a fresh one - // bound to the current default input device. `??` only evaluates - // `makeRecorder()` when nothing was warmed, so the clear below can be - // unconditional — it is a no-op in exactly the case that could have thrown. - let recorder = try preparedRecorder ?? Self.makeRecorder() - preparedRecorder = nil + // One CoreAudio read per session, answering both questions this capture has + // about its input: whether the warm recorder is still bound to it, and + // whether its link buffers a tail worth waiting for at stop. + let input = AudioRoute.currentInput() + // Reuse the warm recorder when it is still bound to the current default + // input; otherwise build a fresh one. `??` only evaluates `makeRecorder()` + // when nothing usable was warmed. + let recorder = try takeWarmRecorder(matching: input) ?? Self.makeRecorder() // record() returns false when no usable input device is available (unplugged, // asleep, route lost). Surface that as a thrown Swift error so @@ -98,6 +148,7 @@ public actor MicCapture: MicCaptureProtocol { } activeRecorder = recorder + activeInputIsBluetooth = input?.isBluetooth ?? false lastEmittedLevel = nil Self.logger.info("start recording to \(recorder.url.lastPathComponent, privacy: .public)") startMeterTimer() @@ -108,18 +159,136 @@ public actor MicCapture: MicCaptureProtocol { meterTask = nil guard let recorder = activeRecorder else { return Data() } activeRecorder = nil + // Read before the suspension below, so this capture's decision can't be + // rewritten by whatever a later `start()` sets. + let lingerForTail = activeInputIsBluetooth + if lingerForTail { + // Keep capturing for a moment past key-up so the audio still travelling + // over the link lands in the file instead of being truncated. See + // `bluetoothTailLinger`. + try? await Task.sleep(for: Self.bluetoothTailLinger) + } recorder.stop() let url = recorder.url - defer { Self.removeFile(at: url) } + defer { + Self.removeFile(at: url) + // Re-arm for the *next* press now that the device is free, so the route + // activation this session just paid for isn't paid again. Scheduled rather + // than done inline: preparing re-opens the input, which is the slow part, + // and `stop()` is on the release path the transcript waits behind. + scheduleRewarm() + } let pcm = try Self.decodePCM(fromFileAt: url) let sampleCount = pcm.count / SyncSTTLimits.bytesPerSample let durationMs = SyncSTTLimits.durationMs(ofPCMBytes: pcm.count) - Self.logger.info("stop samples=\(sampleCount) durationMs=\(durationMs)") + Self.logger.info("stop samples=\(sampleCount) durationMs=\(durationMs) linger=\(lingerForTail)") return pcm } + /// Ends the capture and throws the audio away — the teardown behind + /// `DictationSession`'s cancels. + /// + /// Deliberately *not* `stop()`-and-discard: the user asked for nothing to + /// happen, so neither of `stop()`'s costs is worth paying. The Bluetooth tail + /// linger would delay the `.cancelled` phase (and with it the pill's dismissal) + /// to preserve audio about to be deleted, and reading the whole recording back + /// off disk would decode a blob with no consumer. + /// + /// Not marked `throws`, because nothing on this path can fail — a + /// non-throwing implementation satisfies the `throws` requirement fine. The + /// *requirement* keeps it, so that the protocol's stop-and-discard default + /// (which can throw, via `stop()`) still conforms and `stopAndCancel`'s + /// developer-mode failure log keeps working for hosts that take it. + public func cancelCapture() { + meterTask?.cancel() + meterTask = nil + guard let recorder = activeRecorder else { return } + activeRecorder = nil + recorder.stop() + Self.removeFile(at: recorder.url) + Self.logger.info("cancelled capture, discarded audio") + scheduleRewarm() + } + + // MARK: - Warm recorder + + /// The warm recorder if it is still bound to `input`, else nil — discarding + /// (and cleaning up after) one that isn't. + /// + /// Reuse requires *positively* confirming the device is unchanged: an + /// unreadable route on either side leaves us unable to tell, and a recorder + /// bound to the wrong device doesn't fail loudly — it records the wrong mic, or + /// silence. Paying route activation is the cheaper mistake, so unknown means + /// discard. + private func takeWarmRecorder(matching input: AudioRoute.InputSnapshot?) -> AVAudioRecorder? { + preparedExpiry?.cancel() + preparedExpiry = nil + guard let recorder = preparedRecorder else { return nil } + let warmed = preparedInput + preparedRecorder = nil + preparedInput = nil + guard let warmed, let input, warmed.uid == input.uid else { + Self.removeFile(at: recorder.url) + Self.logger.info("discarded warm recorder — input device changed since warm-up") + return nil + } + return recorder + } + + /// Queues a re-warm to run once the current actor turn finishes, so the caller + /// (`stop()` / `cancelCapture()`) returns before the input is re-opened. + private func scheduleRewarm() { + Task { [weak self] in + await self?.rewarm() + } + } + + /// Prepares the next session's recorder, unless a capture has already started + /// or a warm one is already held — both of which mean this re-warm has been + /// overtaken and has nothing to do. + private func rewarm() { + guard activeRecorder == nil, preparedRecorder == nil else { return } + prepareWarmRecorder() + } + + /// Builds a recorder, records the input it is bound to, and starts its idle + /// countdown. A failure is non-fatal: `start()` then prepares lazily, exactly + /// as it did before any warm recorder existed. + private func prepareWarmRecorder() { + do { + let recorder = try Self.makeRecorder() + preparedRecorder = recorder + preparedInput = AudioRoute.currentInput() + armPreparedRecorderExpiry() + Self.logger.info("prepared a warm recorder") + } catch { + Self.logger.error("warm-up failed: \(error.localizedDescription, privacy: .public)") + } + } + + private func armPreparedRecorderExpiry() { + preparedExpiry?.cancel() + preparedExpiry = Task { [weak self] in + try? await Task.sleep(for: Self.preparedRecorderLifetime) + guard !Task.isCancelled else { return } + await self?.releasePreparedRecorder() + } + } + + /// Tears down an idle warm recorder, freeing the input device — which is what + /// lets a Bluetooth output route return to its full-quality profile. See + /// `preparedRecorderLifetime`. + private func releasePreparedRecorder() { + preparedExpiry = nil + guard let recorder = preparedRecorder else { return } + preparedRecorder = nil + preparedInput = nil + Self.removeFile(at: recorder.url) + Self.logger.info("released idle warm recorder") + } + // MARK: - Recorder construction /// Build a recorder that writes mono 16-bit little-endian PCM at the target diff --git a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift index 9323daed..244485d4 100644 --- a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift +++ b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift @@ -8,6 +8,14 @@ public protocol MicCaptureProtocol: Sendable { /// the release hot path. Throws if the captured audio couldn't be read back, /// so the pipeline can surface an error instead of silently dropping speech. func stop() async throws -> Data + /// Stop capture and discard the audio — the teardown behind a *cancel*, where + /// the user asked for nothing to happen. Split from `stop()` because the two + /// want opposite things: `stop()` may legitimately spend time preserving the + /// captured audio (`MicCapture` waits out a Bluetooth link's tail before + /// ending the recording), whereas a cancel must take effect immediately and + /// has nothing to preserve. Declared here (not only in the default extension) + /// so it dispatches dynamically through `any MicCaptureProtocol`. + func cancelCapture() async throws /// Loudness feed for a meter UI: `0…1`, emitted while recording. Declared on /// the protocol (with an empty-stream default below) so hosts read the meter /// through the same seam they inject — a stub without a meter satisfies it @@ -29,4 +37,12 @@ extension MicCaptureProtocol { /// No-op default: a capture with nothing to pre-open inherits this, mirroring /// `TranscriberProtocol.warmUp`. public func warmUp() async {} + + /// Stop-and-discard default, so a capture with no cancel-specific teardown + /// (every stub) conforms for free and still records the stop the way it always + /// did. Implementations override it when stopping cheaply differs from + /// stopping carefully — `MicCapture` does. + public func cancelCapture() async throws { + _ = try await stop() + } } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index f264cdf4..3ffd39ca 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -221,6 +221,16 @@ public actor DictationSession { // the only throwing call, and it precedes `.recording`, so the two ends are // mutually exclusive). let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) + // Claim `.starting` before anything slow, so the overlay answers the + // keypress now rather than when the mic finishes opening. `mic.start()` + // below is the slow step — it resolves and activates the hardware route, + // which on a Bluetooth input means renegotiating the link into its + // mic-capable mode. Until this phase existed, all of that sat between the + // user's key-down and the first thing they could see or hear, and read as + // the app lagging behind them. `.starting` is presented as "starting", never + // as live capture, so the phase still flips to `.recording` only once audio + // is genuinely being recorded. + setPhase(.starting) do { // Pre-open the dictation connection while the user speaks, so the first dictation after an idle // gap doesn't pay DNS+TCP+TLS on the transcribe hot path (~170 ms cold, measured). Detached @@ -353,7 +363,10 @@ public actor DictationSession { func stopAndCancel() async { cancelAutoRelease() do { - _ = try await mic.stop() + // `cancelCapture`, not `stop`: the audio is being thrown away, so neither + // preserving it (the Bluetooth tail linger) nor reading it back off disk + // is worth delaying the user's cancel for. + try await mic.cancelCapture() } catch { // Stays out of the UI: the user asked for nothing to happen, and a cancel // must not flash red (same rule as `performRelease`'s "a cancel wins over diff --git a/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift b/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift index 7bdc0bba..18848a43 100644 --- a/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift +++ b/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift @@ -40,7 +40,11 @@ extension PipelinePhase { switch self { case .recording: .recording case .transcribing: .transcribing - case .idle, .injecting, .cancelled, .failed, .pasted, .noTarget: .idle + // `.starting` rests at idle. The status item is the coarse indicator, and + // showing "recording" while the mic is still opening would be the one thing + // the phase exists to prevent; a distinct fourth state isn't worth a glyph + // for something that is usually gone within a frame. + case .idle, .starting, .injecting, .cancelled, .failed, .pasted, .noTarget: .idle } } } diff --git a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift index 19b1ca9a..e8a65247 100644 --- a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift +++ b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift @@ -3,6 +3,13 @@ /// is unit-testable; the shell just renders whatever this resolves to. public enum OverlayUIState: Equatable, Sendable { case idle + /// The press landed and the mic is opening — the pill is up, but nothing is + /// being captured yet. A steady state, not a notice: it holds for exactly as + /// long as the hardware takes, which is a frame or two on the built-in mic and + /// noticeably longer on a Bluetooth input. The shell renders it as a plain + /// "Starting…" status line rather than the `● REC` tag, so the pill answers + /// the keypress without claiming to be recording. + case starting case recording case processing /// A dictation attempt failed. The shell shows this as a brief red flash on @@ -27,6 +34,7 @@ public enum OverlayUIState: Equatable, Sendable { public var accessibilityLabel: String { switch self { case .idle: "Blurt." + case .starting: "Starting." case .recording: "Recording." case .processing: "Processing." case .error(let message): message @@ -46,7 +54,7 @@ public enum OverlayUIState: Equatable, Sendable { switch self { case .pasted: 0.8 case .error, .noTarget: 1.6 - case .idle, .recording, .processing: nil + case .idle, .starting, .recording, .processing: nil } } } @@ -64,6 +72,11 @@ extension PipelinePhase { // content — then `.pasted` arrived and faded it back in. A visible blink at // the end of a dictation, worst when the user has switched apps mid-transcribe. case .injecting: .processing + // Its own pill state, NOT `.recording`: the phase exists precisely because + // capture hasn't begun, so projecting it onto the recording pill would put + // the `● REC` tag and a live meter on screen over a mic that isn't open yet + // — the lie the split was made to avoid. + case .starting: .starting case .recording: .recording case .transcribing: .processing // A setup blocker (a missing API key) is an expected state, not a fault: the diff --git a/Sources/BlurtEngine/Pipeline/PipelinePhase.swift b/Sources/BlurtEngine/Pipeline/PipelinePhase.swift index 93d68089..b8768242 100644 --- a/Sources/BlurtEngine/Pipeline/PipelinePhase.swift +++ b/Sources/BlurtEngine/Pipeline/PipelinePhase.swift @@ -2,6 +2,18 @@ import Foundation public enum PipelinePhase: Equatable, Sendable { case idle + /// The press was accepted and the mic is being opened, but no audio is being + /// captured yet. Claimed *before* `mic.start()` so the overlay answers the + /// keypress immediately instead of at whatever moment the hardware route + /// finishes coming up — on a Bluetooth input that is hundreds of milliseconds, + /// sometimes over a second, of a press that looked like it did nothing. + /// + /// Deliberately distinct from `.recording` rather than folded into it: the + /// projections below present it as "starting", never as live capture, which + /// keeps the rule that **the UI never claims audio is being recorded before it + /// is**. Non-terminal, so a second press during it is refused like one during + /// `.recording`. + case starting case recording case transcribing case injecting @@ -26,7 +38,25 @@ public enum PipelinePhase: Equatable, Sendable { public var isTerminal: Bool { switch self { case .idle, .failed, .cancelled, .pasted, .noTarget: true - case .recording, .transcribing, .injecting: false + case .starting, .recording, .transcribing, .injecting: false + } + } + + /// Whether this phase is part of a live capture attempt — the mic is open, or + /// on its way to being open. + /// + /// The single definition of "a dictation is being captured right now", so the + /// consumers that key off it can't drift apart. Today that's `RecordingCueGate` + /// (the start chime fires on the *press*, i.e. entering `.starting`, so the + /// user hears the app respond at key-down rather than after the route comes + /// up). Internal: nothing outside the engine asks, and `.periphery.yml` runs + /// with `retain_public: false`. + /// + /// Exhaustive for the same reason as `isTerminal`. + var isCapturing: Bool { + switch self { + case .starting, .recording: true + case .idle, .transcribing, .injecting, .failed, .cancelled, .pasted, .noTarget: false } } diff --git a/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift b/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift index fd74569e..582ecfd6 100644 --- a/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift +++ b/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift @@ -10,20 +10,27 @@ public enum RecordingCue: Equatable, Sendable { /// Edge-detector deciding when the record start/stop chimes fire. The host calls /// `cue(for:)` on *every* pipeline phase, so the gate fires `.start` only on the -/// idle→recording edge and `.stop` only on the recording→not-recording edge, -/// staying silent while a phase repeats and across transitions between two -/// non-recording phases. Value type holding a single edge bit; the host owns one -/// instance for the app's lifetime. +/// edge into a capture and `.stop` only on the edge out of one, staying silent +/// while a phase repeats and across transitions between two non-capturing +/// phases. Value type holding a single edge bit; the host owns one instance for +/// the app's lifetime. +/// +/// The edge is `PipelinePhase.isCapturing`, not `== .recording`, so the start +/// chime fires when the user presses the key (`.starting`) rather than when the +/// mic finishes opening. On a Bluetooth input those are hundreds of milliseconds +/// apart, and the chime is the fastest feedback the app has — holding it until +/// the route is live wasted exactly the interval it was there to cover. The +/// `.starting`→`.recording` step is inside one capture, so it stays silent. public struct RecordingCueGate: Sendable { - private var wasRecording = false + private var wasCapturing = false public init() {} - /// The cue to play for `phase`, or `nil` when the recording edge didn't move. + /// The cue to play for `phase`, or `nil` when the capture edge didn't move. public mutating func cue(for phase: PipelinePhase) -> RecordingCue? { - let isRecording = phase == .recording - defer { wasRecording = isRecording } - switch (wasRecording, isRecording) { + let isCapturing = phase.isCapturing + defer { wasCapturing = isCapturing } + switch (wasCapturing, isCapturing) { case (false, true): return .start case (true, false): return .stop default: return nil diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index 29ffa690..d7d603d6 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -238,6 +238,32 @@ extension DictationSessionTests { #expect(await terminal == .pasted) } + @Test("press claims .starting before the mic opens, then .recording") + func pressPublishesStartingBeforeRecording() async throws { + // The whole point of the phase: the overlay gets something to show at + // key-down instead of at whatever moment `mic.start()` returns. On a + // Bluetooth input that gap is hundreds of milliseconds of a press that + // looked like it did nothing, so `.starting` must be *published*, not just + // passed through — a `setPhase` skipped here would put the pill back to + // appearing only once the hardware route was up. + let fixture = makeSession() + + let stream = await fixture.session.phaseStream() + fixture.session.submit(.press) + + var seen: [PipelinePhase] = [] + for await phase in stream { + seen.append(phase) + if phase == .recording { break } + } + + // The subscription's initial yield is the current phase (.idle), then the + // press's two transitions in order. + #expect(seen == [.idle, .starting, .recording]) + + await fixture.session.cancel() + } + @Test("cancel during active recording stops mic, discards audio, and transitions to .cancelled") func cancelDuringRecording() async throws { let fixture = makeSession(mode: .transcript("Hello")) @@ -250,6 +276,27 @@ extension DictationSessionTests { #expect(await fixture.mic.stopCalls == 1) #expect(await fixture.injector.inserted.isEmpty) } + + @Test("a cancel tears the mic down through cancelCapture, a release through stop") + func cancelUsesTheDiscardingTeardown() async throws { + // The two teardowns want opposite things, so the session must not conflate + // them. `stop()` may legitimately spend time preserving the audio — + // `MicCapture` waits out a Bluetooth link's tail before ending the + // recording — while a cancel has nothing to preserve and must take effect at + // once. Routing a cancel through `stop()` would make the user's cancel pay + // that linger to save audio it is about to delete. + let cancelled = makeSession() + await cancelled.session.press() + await cancelled.session.cancel() + #expect(await cancelled.mic.cancelCaptureCalls == 1) + + let released = makeSession() + await released.session.press() + await released.session.release() + await released.session.waitForIdle() + #expect(await released.mic.cancelCaptureCalls == 0) + #expect(await released.mic.stopCalls == 1) + } } // Guard/no-op behaviors and phase-stream supersession live in diff --git a/Tests/BlurtEngineTests/MenuBarStatusTests.swift b/Tests/BlurtEngineTests/MenuBarStatusTests.swift index c66aa177..0ec8fd58 100644 --- a/Tests/BlurtEngineTests/MenuBarStatusTests.swift +++ b/Tests/BlurtEngineTests/MenuBarStatusTests.swift @@ -16,6 +16,9 @@ struct MenuBarStatusTests { (.recording, .recording), (.transcribing, .transcribing), (.idle, .idle), + // The mic is still opening, so the coarse indicator rests at idle rather + // than claiming "recording" — the one thing `.starting` exists to prevent. + (.starting, .idle), // Injection happens silently; the indicator rests at idle through the brief // paste rather than showing a distinct state. (.injecting, .idle), diff --git a/Tests/BlurtEngineTests/MicCaptureProtocolDefaultsTests.swift b/Tests/BlurtEngineTests/MicCaptureProtocolDefaultsTests.swift index e7a33c50..e9884469 100644 --- a/Tests/BlurtEngineTests/MicCaptureProtocolDefaultsTests.swift +++ b/Tests/BlurtEngineTests/MicCaptureProtocolDefaultsTests.swift @@ -1,18 +1,31 @@ import Foundation +import Synchronization import Testing @testable import BlurtEngine -/// The protocol's default meter and warm-up, which let a capture without either -/// (test stubs, headless hosts) conform with just `start()`/`stop()`. +/// The protocol's default meter, warm-up, and cancel teardown, which let a +/// capture without any of them (test stubs, headless hosts) conform with just +/// `start()`/`stop()`. @Suite("MicCaptureProtocol defaults") struct MicCaptureProtocolDefaultsTests { - /// Supplies only the two required capture calls, so `levels` and `warmUp()` - /// resolve to the protocol's defaults. - struct BareMic: MicCaptureProtocol { + /// Supplies only the two required capture calls, so `levels`, `warmUp()` and + /// `cancelCapture()` resolve to the protocol's defaults. + /// A `final class` over a `Mutex` rather than a struct (what this stub used to + /// be) so it can count calls: the protocol's methods are non-mutating, and + /// `Mutex` is non-copyable, so a struct can't hold one. Same shape as the test + /// support's `RecordedLog`. + final class BareMic: MicCaptureProtocol { + /// Counts the `stop()` calls the defaults route through, so the cancel + /// default can be observed. + let stops = Mutex(0) + func start() async throws {} - func stop() async throws -> Data { Data() } + func stop() async throws -> Data { + stops.withLock { $0 += 1 } + return Data() + } } @Test("default levels stream is empty and finishes immediately; warmUp is a no-op") @@ -27,4 +40,17 @@ struct MicCaptureProtocolDefaultsTests { for await _ in mic.levels { count += 1 } #expect(count == 0) } + + @Test("default cancelCapture stops and discards") + func cancelCaptureDefault() async throws { + // A capture with no cancel-specific teardown must still *end* on a cancel — + // the default is stop-and-discard, so a conformance that never heard of + // `cancelCapture` keeps the behavior it had when the session called `stop()` + // directly. (`MicCapture` overrides it to skip the tail linger and the + // read-back; that path needs real hardware, so it isn't covered here.) + let mic = BareMic() + try await mic.cancelCapture() + let stops = mic.stops.withLock { $0 } + #expect(stops == 1) + } } diff --git a/Tests/BlurtEngineTests/OverlayUIStateTests.swift b/Tests/BlurtEngineTests/OverlayUIStateTests.swift index f3ffb5b3..8c358e6b 100644 --- a/Tests/BlurtEngineTests/OverlayUIStateTests.swift +++ b/Tests/BlurtEngineTests/OverlayUIStateTests.swift @@ -15,6 +15,10 @@ struct OverlayUIStateTests { /// carve-out — stay as their own tests below. static let projections: [(phase: PipelinePhase, expected: OverlayUIState)] = [ (.idle, .idle), + // Its own pill state, not `.recording`: `.starting` exists because capture + // hasn't begun, so projecting it onto the recording pill would show the + // `● REC` tag and a live meter over a mic that isn't open yet. + (.starting, .starting), (.recording, .recording), (.transcribing, .processing), // `.injecting` is a *working* phase, so it must not project to `.idle`: the @@ -71,7 +75,9 @@ struct OverlayUIStateTests { // A genuine failure is not a setup state. #expect(PipelinePhase.failed(.targetAppLost).setupBlocker == nil) // Neither is any non-failed phase. - for phase in [PipelinePhase.idle, .recording, .transcribing, .injecting, .pasted, .noTarget] { + for phase in [ + PipelinePhase.idle, .starting, .recording, .transcribing, .injecting, .pasted, .noTarget, + ] { #expect(phase.setupBlocker == nil) } // And the pill projection agrees with the classification. @@ -98,6 +104,7 @@ struct OverlayUIStateAccessibilityLabelTests { /// (echo the carried message verbatim), not a constant, so it keeps its own test. static let labels: [(state: OverlayUIState, spoken: String)] = [ (.idle, "Blurt."), + (.starting, "Starting."), (.recording, "Recording."), (.processing, "Processing."), (.pasted, "Your dictation was pasted."), @@ -137,6 +144,9 @@ struct OverlayUIStateNoticeDwellTests { @Test func steadyStatesHaveNoDwell() { // Held for as long as the pipeline is in them — no auto-revert. #expect(OverlayUIState.idle.noticeDwellSeconds == nil) + // `.starting` in particular: a dwell would auto-revert the pill to idle + // mid-press, dismissing it while the mic was still opening. + #expect(OverlayUIState.starting.noticeDwellSeconds == nil) #expect(OverlayUIState.recording.noticeDwellSeconds == nil) #expect(OverlayUIState.processing.noticeDwellSeconds == nil) } diff --git a/Tests/BlurtEngineTests/PipelinePhaseTests.swift b/Tests/BlurtEngineTests/PipelinePhaseTests.swift index be7373de..c431b721 100644 --- a/Tests/BlurtEngineTests/PipelinePhaseTests.swift +++ b/Tests/BlurtEngineTests/PipelinePhaseTests.swift @@ -29,8 +29,36 @@ struct PipelinePhaseTests { @Test("active phases are not terminal") func activePhasesAreNotTerminal() { + // `.starting` included: the press guard keys off terminality, so a terminal + // `.starting` would let a second key-down start a second capture while the + // first is still opening the mic — the exact window it was added to cover. + #expect(!PipelinePhase.starting.isTerminal) #expect(!PipelinePhase.recording.isTerminal) #expect(!PipelinePhase.transcribing.isTerminal) #expect(!PipelinePhase.injecting.isTerminal) } } + +/// `isCapturing` is the single definition of "a dictation is being captured +/// right now" — the edge the start/stop chimes ride. Pinned per case: a phase +/// wrongly reading as capturing would chime at the wrong moment, and `.starting` +/// wrongly reading as *not* capturing would put the start chime back where it +/// was, after the hardware route comes up. +@Suite("PipelinePhase.isCapturing") +struct PipelinePhaseCapturingTests { + @Test("the mic is open, or opening") + func capturingPhases() { + #expect(PipelinePhase.starting.isCapturing) + #expect(PipelinePhase.recording.isCapturing) + } + + @Test("every other phase is not capturing") + func nonCapturingPhases() { + for phase: PipelinePhase in [ + .idle, .transcribing, .injecting, .cancelled, .pasted, .noTarget, + .failed(.apiKeyMissing), + ] { + #expect(!phase.isCapturing, "\(phase) must not read as capturing") + } + } +} diff --git a/Tests/BlurtEngineTests/RecordingCueGateTests.swift b/Tests/BlurtEngineTests/RecordingCueGateTests.swift index 8efdc0d0..0bf4fb85 100644 --- a/Tests/BlurtEngineTests/RecordingCueGateTests.swift +++ b/Tests/BlurtEngineTests/RecordingCueGateTests.swift @@ -2,18 +2,38 @@ import Testing @testable import BlurtEngine -/// The record start/stop chimes fire on the *edges* of the recording phase, not -/// on every phase tick. `AppCoordinator.render` calls the cue gate on every -/// pipeline phase (idle, recording, transcribing, injecting, pasted, …), so the -/// gate must fire `.start` only on the idle→recording edge and `.stop` only on -/// the recording→not-recording edge, staying silent on repeats and on -/// transitions between two non-recording phases. Lifting that edge detection out -/// of the AppKit `CueSoundPlayer` lets `swift test` cover it — the same split as +/// The record start/stop chimes fire on the *edges* of a capture, not on every +/// phase tick. `AppCoordinator.render` calls the cue gate on every pipeline +/// phase (idle, starting, recording, transcribing, injecting, pasted, …), so the +/// gate must fire `.start` only on the edge into a capture and `.stop` only on +/// the edge out of one, staying silent on repeats and on transitions between two +/// non-capturing phases. Lifting that edge detection out of the AppKit +/// `CueSoundPlayer` lets `swift test` cover it — the same split as /// `OverlayUIState`/`MenuBarStatus`. @Suite("RecordingCueGate") struct RecordingCueGateTests { + @Test("the press fires the start cue, before the mic is open") + func startOnPressEdge() { + // The edge is `.starting`, not `.recording`. The chime is the app's fastest + // feedback, and on a Bluetooth input those two phases are hundreds of + // milliseconds apart — holding the chime until the route came up wasted + // exactly the interval it was there to cover. + var gate = RecordingCueGate() + #expect(gate.cue(for: .starting) == .start) + } + + @Test("the mic coming up mid-capture is silent") + func noCueOnStartingToRecording() { + // `.starting` → `.recording` is one capture continuing, not a new one. + var gate = RecordingCueGate() + #expect(gate.cue(for: .starting) == .start) + #expect(gate.cue(for: .recording) == nil) + } + @Test("entering recording from idle fires the start cue") func startOnRisingEdge() { + // A host that never observes `.starting` (a phase stream joined late) still + // gets its start chime on the first capturing phase it does see. var gate = RecordingCueGate() #expect(gate.cue(for: .recording) == .start) } @@ -32,11 +52,23 @@ struct RecordingCueGateTests { #expect(gate.cue(for: .recording) == nil) } - @Test("transitions between two non-recording phases are silent") - func silentBetweenNonRecordingPhases() { + @Test("a press whose mic never opens still chimes closed") + func failedStartClosesTheCue() { + // `mic.start()` throwing takes the pipeline `.starting` → `.failed`. The + // start chime has already played, so the stop chime is what keeps the pair + // balanced — and it leaves the gate ready for the next press rather than + // latched as if a capture were still running. var gate = RecordingCueGate() - // From the initial (non-recording) state through a run of non-recording - // phases, nothing chimes — only a recording edge does. + #expect(gate.cue(for: .starting) == .start) + #expect(gate.cue(for: .failed(.audioCaptureFailed(underlying: MicCaptureError.noInputDevice))) == .stop) + #expect(gate.cue(for: .starting) == .start) + } + + @Test("transitions between two non-capturing phases are silent") + func silentBetweenNonCapturingPhases() { + var gate = RecordingCueGate() + // From the initial (non-capturing) state through a run of non-capturing + // phases, nothing chimes — only a capture edge does. #expect(gate.cue(for: .idle) == nil) #expect(gate.cue(for: .transcribing) == nil) #expect(gate.cue(for: .injecting) == nil) @@ -47,9 +79,10 @@ struct RecordingCueGateTests { @Test("a full record→stop→record cycle chimes start, stop, start again") func fullCycle() { var gate = RecordingCueGate() - #expect(gate.cue(for: .recording) == .start) + #expect(gate.cue(for: .starting) == .start) + #expect(gate.cue(for: .recording) == nil) #expect(gate.cue(for: .injecting) == .stop) #expect(gate.cue(for: .idle) == nil) - #expect(gate.cue(for: .recording) == .start) + #expect(gate.cue(for: .starting) == .start) } } diff --git a/Tests/BlurtEngineTests/Stubs/StubMicCapture.swift b/Tests/BlurtEngineTests/Stubs/StubMicCapture.swift index d22466a6..bec698a8 100644 --- a/Tests/BlurtEngineTests/Stubs/StubMicCapture.swift +++ b/Tests/BlurtEngineTests/Stubs/StubMicCapture.swift @@ -5,6 +5,7 @@ import Foundation actor StubMicCapture: MicCaptureProtocol { var startCalls = 0 var stopCalls = 0 + var cancelCaptureCalls = 0 var pcmToReturn = StubPCM.aboveMinimum var startError: (any Error & Sendable)? var stopError: (any Error & Sendable)? @@ -20,6 +21,15 @@ actor StubMicCapture: MicCaptureProtocol { if let stopError { throw stopError } return pcmToReturn } + /// Overrides the protocol's stop-and-discard default only to *count* the call, + /// then delegates to `stop()` so `stopCalls` and `stopError` keep meaning what + /// they did before the cancel path had its own entry point — the suites that + /// assert a cancel stopped the mic (and that a failing stop is logged) are + /// unchanged. `MicCaptureProtocolDefaultsTests` covers the bare default. + func cancelCapture() async throws { + cancelCaptureCalls += 1 + _ = try await stop() + } func setPCM(_ pcm: Data) { pcmToReturn = pcm } func setStartError(_ error: (any Error & Sendable)?) { startError = error } func setStopError(_ error: (any Error & Sendable)?) { stopError = error } diff --git a/scripts/check.sh b/scripts/check.sh index 2b88bbf2..679f30f5 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -470,8 +470,15 @@ else # MicCapture+Meter.swift, which IS covered. Keep this # list tight — exclude only code that genuinely cannot # be exercised without hardware. + # - AudioRoute*.swift : the CoreAudio routing reads (AudioRoute) and the + # property listeners (AudioRouteMonitor). Both answer + # questions only real hardware can answer — which device + # is default, whether its transport is Bluetooth, and + # when the user switches output — and the listener half + # can only fire on an actual route change. Same + # justification as MicCapture.swift above. COVERAGE="$(xcrun llvm-cov export -summary-only -instr-profile "$PROFDATA" "$XCTEST_BIN" \ - -ignore-filename-regex='Tests/|Audio/MicCapture\.swift' \ + -ignore-filename-regex='Tests/|Audio/MicCapture\.swift|Audio/AudioRoute' \ | python3 -c 'import sys,json; print(round(json.load(sys.stdin)["data"][0]["totals"]["lines"]["percent"],2))')" echo "engine line coverage: ${COVERAGE}%" if ! awk -v c="$COVERAGE" -v min="$MIN_COVERAGE" 'BEGIN{ exit (c+0 < min+0) }'; then From b086e9c5e628698eca3a9656caae8e51ab1b3fa6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 02:04:07 +0000 Subject: [PATCH 02/14] Satisfy swift-format in the new CoreAudio files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check.sh is fail-fast, so this also joins every other line in AudioRoute / AudioRouteMonitor that swift-format would have flagged on the next run — the two AudioObject*PropertyListenerBlock calls and the transport-type comparison all fit inside the 120-column budget once the system-object expression is hoisted behind a constant. deinit lifts the registrations into locals before the queue.sync, so the closure captures only those rather than a self that is already being torn down. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- Sources/BlurtEngine/Audio/AudioRoute.swift | 7 ++-- .../BlurtEngine/Audio/AudioRouteMonitor.swift | 32 +++++++++++-------- 2 files changed, 21 insertions(+), 18 deletions(-) diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift index 58afd878..a12a5e4c 100644 --- a/Sources/BlurtEngine/Audio/AudioRoute.swift +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -63,8 +63,8 @@ enum AudioRoute { mElement: kAudioObjectPropertyElementMain) var deviceID = AudioDeviceID(0) var size = UInt32(MemoryLayout.size) - let status = AudioObjectGetPropertyData( - AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, &deviceID) + let system = AudioObjectID(kAudioObjectSystemObject) + let status = AudioObjectGetPropertyData(system, &address, 0, nil, &size, &deviceID) // 0 is `kAudioObjectUnknown` — "there is no such device" — spelled as the // literal so this doesn't depend on how the constant imports. guard status == noErr, deviceID != 0 else { return nil } @@ -104,7 +104,6 @@ enum AudioRoute { var size = UInt32(MemoryLayout.size) let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &transport) guard status == noErr else { return false } - return transport == kAudioDeviceTransportTypeBluetooth - || transport == kAudioDeviceTransportTypeBluetoothLE + return transport == kAudioDeviceTransportTypeBluetooth || transport == kAudioDeviceTransportTypeBluetoothLE } } diff --git a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift index 394816bd..111d7af9 100644 --- a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift +++ b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift @@ -52,8 +52,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { /// recording it has been written). Dispatch's serial ordering supplies both the /// exclusion and the memory barriers a lock would. private nonisolated(unsafe) var systemListener: AudioObjectPropertyListenerBlock? - private nonisolated(unsafe) var deviceListener: - (id: AudioDeviceID, block: AudioObjectPropertyListenerBlock)? + private nonisolated(unsafe) var deviceListener: (id: AudioDeviceID, block: AudioObjectPropertyListenerBlock)? public init() { let (stream, continuation) = AsyncStream.makeStream(bufferingPolicy: .bufferingNewest(1)) @@ -73,22 +72,25 @@ public final class AudioRouteMonitor: @unchecked Sendable { /// and nothing else would ever hand it back. /// /// The `queue.sync` cannot deadlock: the listener blocks hold `self` weakly, so - /// `queue` never owns the last reference and this deinit never runs on it. + /// `queue` never owns the last reference and this deinit never runs on it. The + /// registrations are lifted into locals first so the closure captures only + /// those, never a `self` that is already being torn down. deinit { continuation.finish() + let system = systemListener + let device = deviceListener + let queue = queue + systemListener = nil + deviceListener = nil queue.sync { - if let systemListener { + if let system { var address = Self.defaultOutputDeviceAddress - _ = AudioObjectRemovePropertyListenerBlock( - AudioObjectID(kAudioObjectSystemObject), &address, queue, systemListener) + _ = AudioObjectRemovePropertyListenerBlock(Self.systemObject, &address, queue, system) } - if let deviceListener { + if let device { var address = Self.sampleRateAddress - _ = AudioObjectRemovePropertyListenerBlock( - deviceListener.id, &address, queue, deviceListener.block) + _ = AudioObjectRemovePropertyListenerBlock(device.id, &address, queue, device.block) } - systemListener = nil - deviceListener = nil } } @@ -108,8 +110,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { self.retargetFormatListener() self.continuation.yield() } - let status = AudioObjectAddPropertyListenerBlock( - AudioObjectID(kAudioObjectSystemObject), &address, queue, block) + let status = AudioObjectAddPropertyListenerBlock(Self.systemObject, &address, queue, block) guard status == noErr else { Self.logger.error("default-output listener failed: \(status)") return @@ -142,7 +143,10 @@ public final class AudioRouteMonitor: @unchecked Sendable { deviceListener = (id: device, block: block) } - // MARK: - Property addresses + // MARK: - CoreAudio addressing + + /// The system-wide audio object, which owns the default-device properties. + private static var systemObject: AudioObjectID { AudioObjectID(kAudioObjectSystemObject) } // Computed, not stored: each caller needs its own mutable copy to pass `inout` // to CoreAudio anyway, so a shared constant would only add a global whose From bdc7957ed5a222dd14eae4a9cf4932757b8daa6b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 02:23:49 +0000 Subject: [PATCH 03/14] Drop the unused Foundation imports from the CoreAudio files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit swiftlint analyze flagged `import Foundation` in both new files. Each had a different real cause, so neither is suppressed: AudioRouteMonitor only ever wanted Dispatch — Foundation was supplying DispatchQueue by re-export. Imports Dispatch directly now. AudioRoute pulled Foundation in for exactly one thing: bridging the device UID's CFString to String. Identity is now the AudioDeviceID, which removes the bridge, the Unmanaged dance, and the import together. IDs are in principle reusable across an unplug/replug where UIDs are not, but the two disagree in one case only — the warmed device was removed and a new one took its ID inside the 60s warm window — and that case fails loudly (the recorder is bound to a device that no longer exists, so record() returns false and the press surfaces .audioCaptureFailed). It cannot produce the failure the check exists to prevent, which is silently recording the wrong mic. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 4 +- BLURTENGINE.md | 2 +- Sources/BlurtEngine/Audio/AudioRoute.swift | 47 ++++++++----------- .../BlurtEngine/Audio/AudioRouteMonitor.swift | 2 +- Sources/BlurtEngine/Audio/MicCapture.swift | 5 +- 5 files changed, 27 insertions(+), 33 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5c8610c7..beefffca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -392,7 +392,7 @@ directions. So: at `prepareToRecord()`, i.e. per session, so warming only the first one hid it for one dictation out of N. `stop()`/`cancelCapture()` schedule a re-warm; `start()` consumes it. - **A warm recorder is validated before reuse.** `AVAudioRecorder` resolves its device once and - never re-resolves, so `MicCapture` records the default input's UID (`AudioRoute.currentInput()`) + never re-resolves, so `MicCapture` records the default input's identity (`AudioRoute.currentInput()`) alongside the warm recorder and discards it when the device has changed — otherwise a recorder warmed before the user connected their AirPods would keep recording the built-in mic. Unknown counts as changed. @@ -409,7 +409,7 @@ linger and the file read-back: the audio is being discarded, so neither is worth cancel for. The routing facts behind all of that live in **`AudioRoute`** (`Audio/AudioRoute.swift`, internal): -which device is the default input, its UID, and whether its transport is Bluetooth. Its sibling +which device is the default input and whether its transport is Bluetooth. Its sibling **`AudioRouteMonitor`** (public) publishes output-route changes for the cue players — see [Settings, persistence, and cues](#settings-persistence-and-cues). Both are excluded from the coverage gate for the same reason diff --git a/BLURTENGINE.md b/BLURTENGINE.md index 8d6dd986..cd95e54a 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -135,7 +135,7 @@ Only `start()`/`stop()` must be implemented — `cancelCapture()`, `levels` and `MicCapture`'s `levels` is a ~20 Hz meter of the recorder's dBFS power mapped to `0…1` (floored at −50 dBFS so room ambient reads as silence) — feed it to a voice-bars view; it costs nothing when unobserved. Its `warmUp()` pre-creates and prepares a recorder so the first `start()` skips hardware route discovery (Blurt calls it at launch, once mic permission is granted, so warming never triggers the permission prompt). -**Bluetooth inputs get three accommodations**, because opening the mic on AirPods makes the system renegotiate the link into its mic-capable mode — hundreds of milliseconds — and that link then buffers audio. `MicCapture` re-arms the warm recorder after _every_ capture rather than only at launch (the cost is per session, paid at `prepareToRecord()`); it records the default input's UID alongside the warm recorder and discards it when the device has changed, since `AVAudioRecorder` resolves its device once and never re-resolves; and it releases an unused warm recorder after 60 s, because holding the input device open is what pins AirPods in the profile where output audio is degraded. On top of that, `stop()` keeps capturing for a further 220 ms when the input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. `cancelCapture()` skips both that linger and the file read-back. +**Bluetooth inputs get three accommodations**, because opening the mic on AirPods makes the system renegotiate the link into its mic-capable mode — hundreds of milliseconds — and that link then buffers audio. `MicCapture` re-arms the warm recorder after _every_ capture rather than only at launch (the cost is per session, paid at `prepareToRecord()`); it records which device the warm recorder was built against and discards it when the default input has changed, since `AVAudioRecorder` resolves its device once and never re-resolves; and it releases an unused warm recorder after 60 s, because holding the input device open is what pins AirPods in the profile where output audio is degraded. On top of that, `stop()` keeps capturing for a further 220 ms when the input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. `cancelCapture()` skips both that linger and the file read-back. ### `TranscriberProtocol` → `AssemblyAITranscriber` diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift index a12a5e4c..0a4675d7 100644 --- a/Sources/BlurtEngine/Audio/AudioRoute.swift +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -1,5 +1,4 @@ import CoreAudio -import Foundation /// Read-only queries against the system's current audio routing — the two facts /// about the mic that `AVFoundation` doesn't expose but the capture path needs: @@ -23,26 +22,36 @@ enum AudioRoute { /// one pass so the capture path makes a single trip through CoreAudio per /// session rather than one per question. struct InputSnapshot: Equatable, Sendable { - /// The device's persistent UID. Non-optional on purpose: an unreadable UID - /// means "we can't tell which device this is", which must not compare equal - /// to another unknown — so `currentInput()` returns nil instead, and callers - /// treat that as "assume it changed". - let uid: String + /// Which device this is. + /// + /// The `AudioDeviceID` rather than the device's persistent UID string, even + /// though IDs are in principle reusable across an unplug/replug while UIDs + /// are not. The only consumer is "is the warm recorder still bound to the + /// device about to be recorded from", and the two disagree in exactly one + /// case: the warmed device was removed and a new one took its ID inside the + /// 60 s warm window. That case fails *loudly* — the recorder is bound to a + /// device that no longer exists, so `record()` returns false and the press + /// surfaces `.audioCaptureFailed`. It cannot produce the failure the check + /// exists to prevent, which is silently recording the wrong mic. Reading the + /// UID instead would mean bridging a `CFString`, i.e. pulling Foundation + /// into a file that otherwise needs only CoreAudio, to buy a distinction + /// that changes a loud failure into a slightly louder one. + let deviceID: AudioDeviceID /// Whether the device's transport is Bluetooth, i.e. whether its capture /// path carries link latency worth lingering for. let isBluetooth: Bool } /// The default input device as an `InputSnapshot`, or nil when there is no - /// input device (all of them unplugged or asleep) or CoreAudio refused either + /// input device (all of them unplugged or asleep) or CoreAudio refused the /// read. Nil is the conservative answer everywhere it's consumed: an unknown /// input invalidates a warm recorder rather than silently keeping one bound to /// a device that may have gone away. static func currentInput() -> InputSnapshot? { - guard let deviceID = defaultDeviceID(for: kAudioHardwarePropertyDefaultInputDevice), - let uid = uid(of: deviceID) - else { return nil } - return InputSnapshot(uid: uid, isBluetooth: isBluetooth(deviceID)) + guard let deviceID = defaultDeviceID(for: kAudioHardwarePropertyDefaultInputDevice) else { + return nil + } + return InputSnapshot(deviceID: deviceID, isBluetooth: isBluetooth(deviceID)) } /// The system's current default *output* device — what `AudioRouteMonitor` @@ -71,22 +80,6 @@ enum AudioRoute { return deviceID } - /// The device's persistent UID string. `Unmanaged` rather than a - /// bridged `CFString?`: the property returns a +1 reference, so the ownership - /// transfer has to be spelled out (`takeRetainedValue`) instead of left to an - /// implicit bridge that would over-release it. - private static func uid(of deviceID: AudioDeviceID) -> String? { - var address = AudioObjectPropertyAddress( - mSelector: kAudioDevicePropertyDeviceUID, - mScope: kAudioObjectPropertyScopeGlobal, - mElement: kAudioObjectPropertyElementMain) - var value: Unmanaged? - var size = UInt32(MemoryLayout?>.size) - let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &value) - guard status == noErr, let value else { return nil } - return value.takeRetainedValue() as String - } - /// Whether the device is reached over Bluetooth. Both transport types count: /// AirPods and other wireless headsets report the classic `bluetooth` /// transport, and LE Audio devices report `bluetoothLE` — the link-latency diff --git a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift index 111d7af9..c4556c07 100644 --- a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift +++ b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift @@ -1,5 +1,5 @@ import CoreAudio -import Foundation +import Dispatch import os /// Ticks whenever the system's audio **output** route changes in a way that diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 41b26a5f..e6361bfc 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -44,7 +44,8 @@ public actor MicCapture: MicCaptureProtocol { /// The default input `preparedRecorder` was built against. `AVAudioRecorder` /// resolves its device once, at `prepareToRecord()`, and never re-resolves — /// so without this a recorder warmed while the built-in mic was default would - /// keep recording from it after the user connected their AirPods. + /// keep recording from it after the user connected their AirPods. See + /// `AudioRoute.InputSnapshot.deviceID` for why identity is the device ID. private var preparedInput: AudioRoute.InputSnapshot? /// Releases `preparedRecorder` once it has gone unused for /// `preparedRecorderLifetime`. See that constant for why holding one open @@ -229,7 +230,7 @@ public actor MicCapture: MicCaptureProtocol { let warmed = preparedInput preparedRecorder = nil preparedInput = nil - guard let warmed, let input, warmed.uid == input.uid else { + guard let warmed, let input, warmed.deviceID == input.deviceID else { Self.removeFile(at: recorder.url) Self.logger.info("discarded warm recorder — input device changed since warm-up") return nil From a0b6428af9902544faffec1fc902b298393bd19c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 18:11:18 +0000 Subject: [PATCH 04/14] Gate recording on mic liveness, and stop chiming at the press MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopts the central fix from #134 and drops the one change here that contradicted it. `record()` returning true only means the AudioQueue started, not that the input route is delivering frames. A Bluetooth mic spends ~1-2s switching into its mic-capable profile first and the OS captures nothing in that window, so returning immediately cues the user to speak into a dead mic and the first words never reach the transcript. start() now polls recorder.currentTime until it advances past 0 — the recorder's clock only moves once frames arrive, which distinguishes "route still switching" from "user is silent" (a level meter can't). Capped per transport by the new pure MicLiveness (2.5s Bluetooth, 300ms otherwise) and failing open on timeout, so a broken mic degrades to the old behavior rather than bricking the press. stopGeneration covers the suspension this introduces: a teardown landing mid-wait wins and the recorder is torn down rather than installed. The chime change is reverted. RecordingCueGate keys on .recording again, so it rides the connecting->recording edge by construction and fires only once audio actually flows. Firing it at the press — what this branch did before — moved the "speak now" cue *earlier* into the dead window, making the lost-first-words symptom worse. PipelinePhase.isCapturing existed only to serve that, and is gone. .starting is renamed .connecting throughout to match #134, and the pill now breathes "Connecting…" on the same curve as "Transcribing…" since the wait can last a second or two. Kept from this branch, none of which #134 covers: the per-session recorder re-warm (which composes with the gate — it keeps the route open so the honest wait usually returns immediately), the Bluetooth tail linger for the last word, cancelCapture(), and the cue re-prime on output route change. Transport classification moves to a pure, tested AudioTransport, leaving AudioRoute as raw CoreAudio reads only — policy shouldn't hide in a file the coverage gate can't reach. The warm-recorder lifecycle moves to MicCapture+Warm.swift to stay inside the lint file-length budget. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- .claude/skills/project-guardrails/SKILL.md | 2 +- AGENTS.md | 59 ++++--- .../Blurt/Overlay/OverlayPillContent.swift | 25 +++ App/Blurt/Blurt/Overlay/OverlayView.swift | 19 +- BLURTENGINE.md | 27 +-- Sources/BlurtEngine/Audio/AudioRoute.swift | 38 ++-- .../BlurtEngine/Audio/AudioTransport.swift | 30 ++++ .../BlurtEngine/Audio/MicCapture+Warm.swift | 93 ++++++++++ Sources/BlurtEngine/Audio/MicCapture.swift | 162 ++++++++---------- .../Audio/MicCaptureProtocol.swift | 9 + Sources/BlurtEngine/Audio/MicLiveness.swift | 61 +++++++ .../BlurtEngine/Duration+Milliseconds.swift | 8 + .../Pipeline/DictationSession.swift | 18 +- .../BlurtEngine/Pipeline/MenuBarStatus.swift | 11 +- .../BlurtEngine/Pipeline/OverlayUIState.swift | 25 +-- .../BlurtEngine/Pipeline/PipelinePhase.swift | 43 ++--- .../Pipeline/RecordingCueGate.swift | 32 ++-- .../DictationSessionTests.swift | 18 +- .../BlurtEngineTests/MenuBarStatusTests.swift | 4 +- Tests/BlurtEngineTests/MicLivenessTests.swift | 115 +++++++++++++ .../OverlayUIStateTests.swift | 12 +- .../BlurtEngineTests/PipelinePhaseTests.swift | 33 +--- .../RecordingCueGateTests.swift | 82 +++++---- 23 files changed, 612 insertions(+), 314 deletions(-) create mode 100644 Sources/BlurtEngine/Audio/AudioTransport.swift create mode 100644 Sources/BlurtEngine/Audio/MicCapture+Warm.swift create mode 100644 Sources/BlurtEngine/Audio/MicLiveness.swift create mode 100644 Sources/BlurtEngine/Duration+Milliseconds.swift create mode 100644 Tests/BlurtEngineTests/MicLivenessTests.swift diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 1b45e8ba..4ed8ecb8 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -67,7 +67,7 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's generated from `project.yml`; edit that and run `xcodegen generate`. check.sh fails on pbxproj drift (a PreToolUse hook also blocks edits to it). - The engine has **no external SPM dependencies** (Foundation/Security/ - AVFoundation only). Don't add one to `Sources/BlurtEngine/`. + AVFoundation/CoreAudio only). Don't add one to `Sources/BlurtEngine/`. - Unit tests use **Swift Testing**, not XCTest (the `BlurtUITests` XCUITest bundle is the one exception — XCUIAutomation requires XCTest). **Never touch the real Keychain in tests** — `APIKeyStore` is the production item; use an diff --git a/AGENTS.md b/AGENTS.md index beefffca..d4d06453 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,10 +29,10 @@ Four reflexes before you touch anything: ### The two layers -| Layer | What it is | -| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Sources/BlurtEngine/` | Swift package (`swift-tools-version:6.2`, `platforms: [.macOS(.v15)]`) owning the pipeline. Pure logic behind protocol seams, no AppKit-shell deps, **no external SPM deps** — Foundation/Security/AVFoundation plus toolchain modules like Synchronization, with AppKit types only at the seams. | -| `App/Blurt/` | AppKit/SwiftUI shell (Xcode project generated by XcodeGen) that wires the engine to an overlay window, the main window, a Settings scene, a menu bar item, and the trigger key. Its only package is the local `BlurtEngine` (declared in `App/Blurt/project.yml`). | +| Layer | What it is | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Sources/BlurtEngine/` | Swift package (`swift-tools-version:6.2`, `platforms: [.macOS(.v15)]`) owning the pipeline. Pure logic behind protocol seams, no AppKit-shell deps, **no external SPM deps** — Foundation/Security/AVFoundation/CoreAudio plus toolchain modules like Synchronization, with AppKit types only at the seams. | +| `App/Blurt/` | AppKit/SwiftUI shell (Xcode project generated by XcodeGen) that wires the engine to an overlay window, the main window, a Settings scene, a menu bar item, and the trigger key. Its only package is the local `BlurtEngine` (declared in `App/Blurt/project.yml`). | The engine's dependency-free rule is a **rule**; the app merely happens to carry none today. The former `mxcl/AppUpdater` dependency and its in-place self-updater were removed (see [Updates](#updates)), @@ -47,8 +47,8 @@ the design; BLURTENGINE.md covers the _what_ of the API surface. ```text Sources/BlurtEngine/ the engine (dependency-free Swift package) - Audio/ MicCapture (+meter), AudioRoute(+Monitor) — CoreAudio routing facts, - SoundPack/Catalog/Store — record cues + Audio/ MicCapture (+meter), MicLiveness (mic bring-up gate), AudioRoute + (+Monitor)/AudioTransport — CoreAudio routing, SoundPack/Catalog/Store Config/ Keychain-backed API key, key terms, developer mode, DefaultsKey + PersistedSettings (every defaults key, and the reset sweep over them) FocusCapture/ Accessibility reads of the frontmost app / focused field @@ -383,11 +383,21 @@ A **fresh recorder per session** resolves the current default input device at `r is deliberate — see [Settled decisions](#settled-decisions--dont-reintroduce-these) for the `AVAudioEngine` failure it replaced. -**Bluetooth inputs are the reason for three of this actor's moving parts.** Opening the mic on +**Bluetooth inputs are the reason for four of this actor's moving parts.** Opening the mic on AirPods (or any Bluetooth headset) makes the system renegotiate the link into its mic-capable mode — -hundreds of milliseconds, sometimes over a second — and that link then buffers audio in both -directions. So: - +one to two seconds, during which the OS receives no audio at all — and that link then buffers audio +in both directions. So: + +- **`start()` does not return until the input is live.** `record()` returning `true` only means the + AudioQueue started, not that frames are arriving, so `start()` polls `recorder.currentTime` until + it advances past 0 — the recorder's clock only moves once the device delivers audio, which is what + distinguishes "route still switching" from "user is silent" (a level meter can't). Capped per + transport by **`MicLiveness`** (2.5 s Bluetooth, 300 ms otherwise) and **failing open** on timeout, + so a broken or silent mic degrades to the old behavior rather than bricking the press. This is what + stops the app cueing the user to speak into a dead mic; audio spoken during the switch cannot be + recovered by anything, because nothing ever receives it. `stopGeneration` covers the one suspension + this introduces — a teardown landing mid-wait wins, and the recorder is torn down rather than + installed. - **The warm recorder is re-armed after every capture**, not just at launch. The cost above is paid at `prepareToRecord()`, i.e. per session, so warming only the first one hid it for one dictation out of N. `stop()`/`cancelCapture()` schedule a re-warm; `start()` consumes it. @@ -401,6 +411,10 @@ directions. So: degraded — so it is not held indefinitely. Back-to-back dictations land inside the window; a press past it just prepares lazily, which is the pre-re-warm behavior. +The re-warm and the liveness gate are complements, not alternatives: the re-warm shortens how _often_ +the profile switch is paid (a warm recorder has already held the route open), and the gate is what +keeps the app honest on the presses that pay it anyway. + `stop()` also waits out `bluetoothTailLinger` (220 ms) before ending the recording **when the session's input is Bluetooth**, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. It runs after `.transcribing` is claimed, so it delays the @@ -409,7 +423,10 @@ linger and the file read-back: the audio is being discarded, so neither is worth cancel for. The routing facts behind all of that live in **`AudioRoute`** (`Audio/AudioRoute.swift`, internal): -which device is the default input and whether its transport is Bluetooth. Its sibling +which device is the default input, and its raw CoreAudio transport type. **Raw reads only** — what a +transport _means_ is **`AudioTransport.isBluetooth`** and **`MicLiveness.timeout`**, which are pure +and unit-tested, because `AudioRoute` itself needs real hardware and is excluded from the coverage +gate. Don't let a decision drift into it. Its sibling **`AudioRouteMonitor`** (public) publishes output-route changes for the cue players — see [Settings, persistence, and cues](#settings-persistence-and-cues). Both are excluded from the coverage gate for the same reason @@ -508,12 +525,14 @@ returns to `.idle` without injecting. Four perceived-latency choices to preserve: -- `press()` claims `.starting` _before_ `mic.start()`, so the pill and the start chime answer the - key-down rather than waiting on the hardware route. `.starting` is presented as "Starting…", never - as live capture — the rule that the UI must not claim audio is being recorded before it is holds, - and `.recording` is still only claimed once `mic.start()` has succeeded. `RecordingCueGate` rides - `PipelinePhase.isCapturing` (`.starting || .recording`) for the same reason, so the chime fires at - the press and stays silent across `.starting` → `.recording`. +- `press()` claims `.connecting` _before_ `mic.start()`, so the pill answers the key-down while the + mic is still coming up — but the **start chime deliberately does not**. `RecordingCueGate` keys on + `.recording`, which `MicCapture`'s liveness gate only reaches once the input route is delivering + frames, so the chime rides the connecting→recording edge by construction. That ordering is + load-bearing, not cosmetic: the chime is a "speak now" cue, on a Bluetooth route it and the press + are ~1–2 s apart, and speech in that window is **unrecoverable** (the OS receives nothing while the + profile switch is in flight). Chiming at the press is what lost the first words of the utterance. + Don't "fix" the chime to fire earlier. - `.injecting` projects to `OverlayUIState.processing`, **not** `.idle` — the shell reads an idle projection as "dismiss", so mapping this working phase to idle faded the pill out mid-dictation and blinked it back for "Pasted". @@ -692,9 +711,9 @@ Record cues: **`SoundPack`** is a selectable start/stop chime voice (vintage syn `App/Blurt/Blurt/Resources/Sounds/`), listed by **`SoundPackCatalog.swift`**, which is _generated_ by `scripts/generate-sounds.swift` alongside the audio. Regenerate both halves together — `check.sh`'s sound-catalog guard exists because a drift plays silence with no error. **`RecordingCueGate`** is the -pure edge detector deciding when the chimes fire — on the _capture_ edge -(`PipelinePhase.isCapturing`), so the start chime lands at the press rather than after the mic -route comes up; the AppKit `CueSoundPlayer` just plays what it resolves. +pure edge detector deciding when the chimes fire — on the `.recording` edge, i.e. once audio is +actually flowing, never at the press (see the latency notes above for why that ordering is +load-bearing); the AppKit `CueSoundPlayer` just plays what it resolves. `CueSoundPlayer` decodes and pre-rolls the players once so the first chime never stalls the pill, and that pre-roll is bound to the output route it was made against. Blurt's own capture invalidates it: diff --git a/App/Blurt/Blurt/Overlay/OverlayPillContent.swift b/App/Blurt/Blurt/Overlay/OverlayPillContent.swift index 57015493..8c088fcd 100644 --- a/App/Blurt/Blurt/Overlay/OverlayPillContent.swift +++ b/App/Blurt/Blurt/Overlay/OverlayPillContent.swift @@ -82,6 +82,31 @@ struct TranscribingLabel: View { } } +/// The "Connecting…" status line shown while `MicCapture`'s liveness gate waits +/// for the input route to deliver frames. Breathes on the same curve as +/// `TranscribingLabel` (the pill's other "working, hold on" state) so the two +/// waits read alike — this one can hold for a second or two on a Bluetooth +/// route, and a frozen line would read as a hung app. +/// +/// Deliberately *not* the `● REC` tag or the meter: those are the "speak now" +/// cues, and audio spoken during the bring-up is unrecoverable — the OS receives +/// nothing while the profile switch is in flight. The pill must not invite +/// speech it cannot capture. +struct ConnectingLabel: View { + /// Whether to run the breathing motion (off under Reduce Motion). + let animated: Bool + + // Matched to `TranscribingLabel`: the two are the same kind of wait, so they + // share one heartbeat rather than each picking a rate. + private let breathPeriod: Double = 1.8 + private let minOpacity: Double = 0.55 + + var body: some View { + StatusLineText("Connecting…") + .pulsingOpacity(period: breathPeriod, minOpacity: minOpacity, animated: animated) + } +} + /// The "● REC" recording tag: a pulsing magenta dot + "REC" caption, sitting to /// the left of the waveform — the native echo of the site demo's magenta pixel /// tag. Magenta (the brand --hot) stands in for the conventional red record dot; diff --git a/App/Blurt/Blurt/Overlay/OverlayView.swift b/App/Blurt/Blurt/Overlay/OverlayView.swift index 9741ed92..fc5ee1c9 100644 --- a/App/Blurt/Blurt/Overlay/OverlayView.swift +++ b/App/Blurt/Blurt/Overlay/OverlayView.swift @@ -32,7 +32,7 @@ struct OverlayView: View { switch state { case .error: return Color(red: 0.62, green: 0.13, blue: 0.13) - case .starting, .recording, .processing, .pasted, .noTarget, .idle: + case .connecting, .recording, .processing, .pasted, .noTarget, .idle: return Color(white: 0.16) } } @@ -84,18 +84,19 @@ struct OverlayView: View { // background would collapse with it) keeps the pill's shape intact for // `hide()`'s pre-hide reset. Color.clear - case .starting: - // The mic is opening; nothing is being captured yet. Styled exactly like - // "Transcribing…"/"Pasted" (same status-line type, tracking, and cyan - // --ice) so the starting → recording hand-off reads as one status line - // rather than a new kind of alert — and deliberately *not* the `● REC` - // tag or the meter, which would claim live capture. + case .connecting: + // The mic is coming up; nothing is being captured yet. Styled like + // "Transcribing…" (same status-line type, tracking, cyan --ice, and + // breathing pulse) so the connecting → recording hand-off reads as one + // status line rather than a new kind of alert — and deliberately *not* + // the `● REC` tag or the meter, which would cue the user to speak into a + // mic that isn't delivering yet. // // On a built-in mic this is on screen for a frame or two, inside the // pill's own 0.08 s fade-in, so it blends into the appearance rather than - // flashing; on a Bluetooth input it holds for as long as the link takes, + // flashing; on a Bluetooth route it holds for as long as the link takes, // which is the whole point. - StatusLineText("Starting…") + ConnectingLabel(animated: !reduceMotion) .transition(.opacity) case .recording: // "● REC" tag beside the live waveform, mirroring the site demo's recording diff --git a/BLURTENGINE.md b/BLURTENGINE.md index cd95e54a..d1ac225d 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -4,7 +4,7 @@ BlurtEngine is the Swift package that powers [Blurt](README.md)'s dictation pipe ## What you get -- **`Sources/BlurtEngine/`** — a Swift package (`swift-tools-version:6.2`, macOS 15+, Swift 6 strict concurrency) with **no external dependencies**: just Foundation, Security, AVFoundation, toolchain modules like Synchronization, and AppKit types at the seams. That dependency-free rule is deliberate and enforced — don't add SPM dependencies to the engine. +- **`Sources/BlurtEngine/`** — a Swift package (`swift-tools-version:6.2`, macOS 15+, Swift 6 strict concurrency) with **no external dependencies**: just Foundation, Security, AVFoundation, CoreAudio, toolchain modules like Synchronization, and AppKit types at the seams. That dependency-free rule is deliberate and enforced — don't add SPM dependencies to the engine. - Pure logic behind three protocol seams (`MicCaptureProtocol`, `TranscriberProtocol`, `InjectorProtocol`), so every collaborator can be stubbed in tests and replaced in a host app. - Production implementations of all three seams (`MicCapture`, `AssemblyAITranscriber`, `KeyInjector`), plus the supporting pieces a dictation product needs: Keychain-backed API-key storage, per-utterance contextual prompting (built and tested, currently switched off), a hotkey state machine, permission checks, and UI-state projections. @@ -60,7 +60,7 @@ Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. - **Cleanup happens server-side, and it's optional.** The request's `llm` block asks the service to apply our own cleanup instruction (`CleanupInstruction.text` — delete disfluencies, change nothing else) to the verbatim transcript inside the same call. It is the only instruction on the request: the separate `config.prompt` field, which primes the _transcription_, is switched off at `TranscriptionPrompt.isEnabled` and omitted from every request. The block is gated by the **enhanced transcripts** setting (`EnhancedTranscriptsStore`, on by default): turned off, the config omits `llm` and the verbatim transcript is pasted as spoken. The user's **custom style instructions** (`CustomStyleStore`, empty by default) are appended to that instruction via `CleanupInstruction.sendable(appending:)`, trimmed to the headroom the API's 2048 instruction cap leaves (measured in UTF-8 bytes, the conservative bound — the cap's own unit is unmeasured); blank means the base instruction goes out unchanged. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. -- **Latency is pre-paid where possible.** `press()` claims `.starting` before it touches the mic, so a host's pill and start cue answer the keypress rather than the hardware route (which on a Bluetooth input is the slowest thing in the press path). `MicCapture` re-arms its prepared recorder after every capture so that route activation is paid between dictations rather than during one. `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. +- **Latency is pre-paid where possible, and never faked.** `press()` claims `.connecting` before it touches the mic, so a host's pill answers the keypress — but `MicCapture.start()` deliberately holds until the input device is actually delivering frames, and the _start cue_ waits for `.recording`. On a Bluetooth route those are ~1–2 s apart and the OS captures nothing in between, so cueing at the press loses the first words. `MicCapture` re-arms its prepared recorder after every capture so that route activation is usually paid between dictations rather than during one. `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. ## DictationSession @@ -83,20 +83,21 @@ For callback-shaped hosts that can't `await` — an event tap, a button action `phase` / `phaseStream()` expose the pipeline's `PipelinePhase`: ```text -idle → starting → recording → transcribing → injecting → pasted | noTarget +idle → connecting → recording → transcribing → injecting → pasted | noTarget │ │ └── failed(BlurtError) / cancelled (from any stage) ``` -`starting` is claimed the moment a press is accepted, _before_ the mic is opened, so your UI can -answer the keypress instead of the hardware. It is not live capture — opening a Bluetooth input takes -hundreds of milliseconds — so present it as "starting", never as recording; `recording` follows only -once audio is genuinely being captured. `PipelinePhase.isCapturing` covers both if you want the -"a dictation is in progress" bit rather than the distinction. +`connecting` is claimed the moment a press is accepted, _before_ the mic is up, so your UI can answer +the keypress instead of the hardware. It is **not** live capture: `MicCapture.start()` holds until the +input device actually delivers frames, which on a Bluetooth route takes ~1–2 s during which the OS +receives no audio at all. Present it as a warming-up state and **withhold your "speak now" cues** — +the recording indicator, the start chime — until `recording`, which follows only once audio is +genuinely being captured. Cueing at the press invites speech that nothing can capture. - `phaseStream()` yields the current phase immediately, then every transition. It is a **multi-observer** stream: every call gets its own continuation and all of them see later transitions, so an extra consumer (a diagnostic, a second window) is safe. Still, prefer one renderer that projects the phase into your own state over a fan-out of long-lived consumers — one source of UI truth is easier to reason about than several. - `.pasted` and `.noTarget` are terminal _success_ states, not errors. `.noTarget` means transcription worked but nothing editable was focused (or the target app quit), so the text was left on the clipboard — show a quiet "copied" notice, not a failure. -- Two ready-made projections keep UI mapping out of your shell: `phase.overlayState` (`OverlayUIState`: idle / starting / recording / processing / error(message:) / pasted / noTarget, with accessibility labels and — for the transient notices — `noticeDwellSeconds`, how long to hold one before reverting to idle) and `phase.menuBarStatus` (coarser: idle / recording / transcribing, never shows errors, with `symbolName`/`accessibilityLabel` presentation — `starting` rests at idle there rather than claiming a live mic). +- Two ready-made projections keep UI mapping out of your shell: `phase.overlayState` (`OverlayUIState`: idle / connecting / recording / processing / error(message:) / pasted / noTarget, with accessibility labels and — for the transient notices — `noticeDwellSeconds`, how long to hold one before reverting to idle) and `phase.menuBarStatus` (coarser: idle / recording / transcribing, never shows errors, with `symbolName`/`accessibilityLabel` presentation — `connecting` rests at idle there rather than claiming a live mic). - Pill geometry is available too, if you're drawing something like Blurt's overlay: `OverlayPlacement` resolves how big the panel is (`panelSize(pillSize:shadowMargin:)`, sized to hold the pill plus room for its shadow) and where it goes (clearance, clamping a dragged origin back on screen), and `MeterBarGeometry` gives the level meter its shape. Build a `MeterBarRow(availableSize:)` once per layout — it resolves how many bars fit and how tall they may be — then ask it for `height(at:level:time:animated:)` per bar; `MeterBarGeometry.breathingOpacity(time:period:minOpacity:)` is the pulse the record dot and status label share. All pure math; pass `animated: false` to honor Reduce Motion. ### Errors @@ -135,7 +136,11 @@ Only `start()`/`stop()` must be implemented — `cancelCapture()`, `levels` and `MicCapture`'s `levels` is a ~20 Hz meter of the recorder's dBFS power mapped to `0…1` (floored at −50 dBFS so room ambient reads as silence) — feed it to a voice-bars view; it costs nothing when unobserved. Its `warmUp()` pre-creates and prepares a recorder so the first `start()` skips hardware route discovery (Blurt calls it at launch, once mic permission is granted, so warming never triggers the permission prompt). -**Bluetooth inputs get three accommodations**, because opening the mic on AirPods makes the system renegotiate the link into its mic-capable mode — hundreds of milliseconds — and that link then buffers audio. `MicCapture` re-arms the warm recorder after _every_ capture rather than only at launch (the cost is per session, paid at `prepareToRecord()`); it records which device the warm recorder was built against and discards it when the default input has changed, since `AVAudioRecorder` resolves its device once and never re-resolves; and it releases an unused warm recorder after 60 s, because holding the input device open is what pins AirPods in the profile where output audio is degraded. On top of that, `stop()` keeps capturing for a further 220 ms when the input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. `cancelCapture()` skips both that linger and the file read-back. +**Bluetooth inputs get four accommodations**, because opening the mic on AirPods makes the system renegotiate the link into its mic-capable mode — one to two seconds, during which the OS receives no audio at all — and that link then buffers audio. + +The load-bearing one is that **`start()` doesn't return until the input is live.** `record()` returning `true` only means the AudioQueue started, so `start()` polls `recorder.currentTime` until it advances past 0 — the recorder's clock only moves once frames arrive, which distinguishes "route still switching" from "user is silent" (a level meter can't). The wait is capped per transport by `MicLiveness` (2.5 s Bluetooth, 300 ms otherwise) and **fails open** on timeout, so a broken or silent mic degrades to the old behavior rather than bricking the press. Speech during the switch is not recovered by this — nothing receives it — the point is to stop inviting it. + +The other three reduce how often that wait is paid, and fix the tail. `MicCapture` re-arms the warm recorder after _every_ capture rather than only at launch (the cost is per session, paid at `prepareToRecord()`); it records which device the warm recorder was built against and discards it when the default input has changed, since `AVAudioRecorder` resolves its device once and never re-resolves; and it releases an unused warm recorder after 60 s, because holding the input device open is what pins AirPods in the profile where output audio is degraded. On top of that, `stop()` keeps capturing for a further 220 ms when the input is Bluetooth, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. `cancelCapture()` skips both that linger and the file read-back. ### `TranscriberProtocol` → `AssemblyAITranscriber` @@ -206,7 +211,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments: -- **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only. +- **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation/CoreAudio only. - **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`. - **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time. - **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets. diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift index 0a4675d7..fff0b8ed 100644 --- a/Sources/BlurtEngine/Audio/AudioRoute.swift +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -9,9 +9,14 @@ import CoreAudio /// `prepareToRecord()` time and never re-resolves it, so a recorder warmed /// before the user connected their AirPods would silently record from the /// built-in mic. -/// 2. **Whether that device is a Bluetooth one**, whose link buffers audio for -/// a couple of hundred milliseconds — the tail `MicCapture.stop()` waits for -/// rather than truncating (see `bluetoothTailLinger`). +/// 2. **What transport that device is on**, since a Bluetooth link is both slow +/// to bring up (the wait `MicLiveness` caps) and buffered at the tail (the +/// linger `MicCapture.stop()` grants rather than truncating). +/// +/// Raw reads only — no policy. What a transport type *means* lives in +/// `AudioTransport` and `MicLiveness`, which are pure and unit-tested; this file +/// needs real hardware to answer anything, so it is excluded from the coverage +/// gate and must not be where a decision hides. /// /// Internal, not public: the app never asks these directly (it observes route /// *changes* through `AudioRouteMonitor`), and `.periphery.yml` runs with @@ -37,9 +42,11 @@ enum AudioRoute { /// into a file that otherwise needs only CoreAudio, to buy a distinction /// that changes a loud failure into a slightly louder one. let deviceID: AudioDeviceID - /// Whether the device's transport is Bluetooth, i.e. whether its capture - /// path carries link latency worth lingering for. - let isBluetooth: Bool + /// The device's CoreAudio transport type, or nil when the read failed. + /// Interpreted by `AudioTransport.isBluetooth` and + /// `MicLiveness.timeout(forTransportType:)` — kept raw here so the policy + /// stays in the files `swift test` can reach. + let transportType: UInt32? } /// The default input device as an `InputSnapshot`, or nil when there is no @@ -51,7 +58,7 @@ enum AudioRoute { guard let deviceID = defaultDeviceID(for: kAudioHardwarePropertyDefaultInputDevice) else { return nil } - return InputSnapshot(deviceID: deviceID, isBluetooth: isBluetooth(deviceID)) + return InputSnapshot(deviceID: deviceID, transportType: transportType(of: deviceID)) } /// The system's current default *output* device — what `AudioRouteMonitor` @@ -80,15 +87,10 @@ enum AudioRoute { return deviceID } - /// Whether the device is reached over Bluetooth. Both transport types count: - /// AirPods and other wireless headsets report the classic `bluetooth` - /// transport, and LE Audio devices report `bluetoothLE` — the link-latency - /// characteristic the callers care about is the same either way. - /// - /// A failed read answers `false`: the conservative default is "no linger", - /// since padding every wired capture with a delay would be a worse regression - /// than losing the tail on a device we couldn't classify. - private static func isBluetooth(_ deviceID: AudioDeviceID) -> Bool { + /// The device's transport type, or nil when the read failed — + /// `AudioTransport` and `MicLiveness` both treat nil as "not Bluetooth", which + /// is the conservative direction for each. + private static func transportType(of deviceID: AudioDeviceID) -> UInt32? { var address = AudioObjectPropertyAddress( mSelector: kAudioDevicePropertyTransportType, mScope: kAudioObjectPropertyScopeGlobal, @@ -96,7 +98,7 @@ enum AudioRoute { var transport = UInt32(0) var size = UInt32(MemoryLayout.size) let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &transport) - guard status == noErr else { return false } - return transport == kAudioDeviceTransportTypeBluetooth || transport == kAudioDeviceTransportTypeBluetoothLE + guard status == noErr else { return nil } + return transport } } diff --git a/Sources/BlurtEngine/Audio/AudioTransport.swift b/Sources/BlurtEngine/Audio/AudioTransport.swift new file mode 100644 index 00000000..b83f033f --- /dev/null +++ b/Sources/BlurtEngine/Audio/AudioTransport.swift @@ -0,0 +1,30 @@ +import CoreAudio + +/// Classification of a CoreAudio device's transport type +/// (`kAudioDevicePropertyTransportType`). +/// +/// Split from `AudioRoute`, which reads the raw value off the hardware and is +/// excluded from the coverage gate for that reason. The *decision* — which +/// transports behave like a buffered wireless link — is pure, and two separate +/// behaviors hang off it, so it lives somewhere `swift test` can pin it: +/// +/// - `MicLiveness.timeout(forTransportType:)`, the wait cap for the mic +/// bring-up gate. +/// - `MicCapture`'s tail linger, which keeps capturing past key-up so the last +/// word doesn't get truncated by the link's buffering. +enum AudioTransport { + /// Whether the transport is a Bluetooth one. Both types count: AirPods and + /// other wireless headsets report the classic `bluetooth` transport, LE Audio + /// devices report `bluetoothLE`, and the link characteristic both callers care + /// about — a slow bring-up and a buffered tail — is the same either way. + /// + /// A nil transport (the read failed, or there is no device) answers `false`. + /// That is the conservative direction for both callers: the short wait cap, + /// and no linger. Padding every wired capture with a delay would be a worse + /// regression than losing the tail on a device we couldn't classify. + static func isBluetooth(_ transportType: UInt32?) -> Bool { + guard let transportType else { return false } + return transportType == kAudioDeviceTransportTypeBluetooth + || transportType == kAudioDeviceTransportTypeBluetoothLE + } +} diff --git a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift new file mode 100644 index 00000000..16b14624 --- /dev/null +++ b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift @@ -0,0 +1,93 @@ +import AVFoundation +import Foundation + +// The warm-recorder lifecycle — prepare ahead of the press, validate it still +// matches the live input, and let it expire — split from `MicCapture.swift` to +// stay within the lint file-length budget, like `MicCapture+Meter`. Members it +// reaches (the prepared-recorder state, `logger`, `removeFile`, `makeRecorder`) +// are internal rather than private for that reason: `private` is file-scoped and +// can't cross the split. +extension MicCapture { + /// Pre-create and prepare a recorder so the first `start()` skips first-time + /// hardware route discovery. Does NOT begin capture — no mic indicator. Safe to + /// call multiple times; a failure here just leaves `start()` to prepare lazily. + public func warmUp() { + guard preparedRecorder == nil, activeRecorder == nil else { return } + prepareWarmRecorder() + } + + /// The warm recorder if it is still bound to `input`, else nil — discarding + /// (and cleaning up after) one that isn't. + /// + /// Reuse requires *positively* confirming the device is unchanged: an + /// unreadable route on either side leaves us unable to tell, and a recorder + /// bound to the wrong device doesn't fail loudly — it records the wrong mic, or + /// silence. Paying route activation is the cheaper mistake, so unknown means + /// discard. + func takeWarmRecorder(matching input: AudioRoute.InputSnapshot?) -> AVAudioRecorder? { + preparedExpiry?.cancel() + preparedExpiry = nil + guard let recorder = preparedRecorder else { return nil } + let warmed = preparedInput + preparedRecorder = nil + preparedInput = nil + guard let warmed, let input, warmed.deviceID == input.deviceID else { + Self.removeFile(at: recorder.url) + Self.logger.info("discarded warm recorder — input device changed since warm-up") + return nil + } + return recorder + } + + /// Queues a re-warm to run once the current actor turn finishes, so the caller + /// (`stop()` / `cancelCapture()`) returns before the input is re-opened. + func scheduleRewarm() { + Task { [weak self] in + await self?.rewarm() + } + } + + /// Prepares the next session's recorder, unless a capture has already started + /// or a warm one is already held — both of which mean this re-warm has been + /// overtaken and has nothing to do. + func rewarm() { + guard activeRecorder == nil, preparedRecorder == nil else { return } + prepareWarmRecorder() + } + + /// Builds a recorder, records the input it is bound to, and starts its idle + /// countdown. A failure is non-fatal: `start()` then prepares lazily, exactly + /// as it did before any warm recorder existed. + func prepareWarmRecorder() { + do { + let recorder = try Self.makeRecorder() + preparedRecorder = recorder + preparedInput = AudioRoute.currentInput() + armPreparedRecorderExpiry() + Self.logger.info("prepared a warm recorder") + } catch { + Self.logger.error("warm-up failed: \(error.localizedDescription, privacy: .public)") + } + } + + func armPreparedRecorderExpiry() { + preparedExpiry?.cancel() + preparedExpiry = Task { [weak self] in + try? await Task.sleep(for: Self.preparedRecorderLifetime) + guard !Task.isCancelled else { return } + await self?.releasePreparedRecorder() + } + } + + /// Tears down an idle warm recorder, freeing the input device — which is what + /// lets a Bluetooth output route return to its full-quality profile. See + /// `preparedRecorderLifetime`. + func releasePreparedRecorder() { + preparedExpiry = nil + guard let recorder = preparedRecorder else { return } + preparedRecorder = nil + preparedInput = nil + Self.removeFile(at: recorder.url) + Self.logger.info("released idle warm recorder") + } +} diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index e6361bfc..f6e88a61 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -15,7 +15,7 @@ public actor MicCapture: MicCaptureProtocol { // log show --predicate 'subsystem == "dev.alex.blurt"' --last 1h // Stderr is unreachable for .app bundles launched via Finder/LaunchServices, // so go through the unified logging system instead. - private static let logger = Logger(subsystem: BlurtIdentity.subsystem, category: "MicCapture") + static let logger = Logger(subsystem: BlurtIdentity.subsystem, category: "MicCapture") public nonisolated let levels: AsyncStream private nonisolated let levelsContinuation: AsyncStream.Continuation @@ -40,26 +40,37 @@ public actor MicCapture: MicCaptureProtocol { /// `AVAudioEngine` rewrite bought: the warm recorder is validated against the /// live default input before it is used (`takeWarmRecorder`) and discarded /// rather than reused when the device has changed underneath it. - private var preparedRecorder: AVAudioRecorder? + var preparedRecorder: AVAudioRecorder? /// The default input `preparedRecorder` was built against. `AVAudioRecorder` /// resolves its device once, at `prepareToRecord()`, and never re-resolves — /// so without this a recorder warmed while the built-in mic was default would /// keep recording from it after the user connected their AirPods. See /// `AudioRoute.InputSnapshot.deviceID` for why identity is the device ID. - private var preparedInput: AudioRoute.InputSnapshot? + var preparedInput: AudioRoute.InputSnapshot? /// Releases `preparedRecorder` once it has gone unused for /// `preparedRecorderLifetime`. See that constant for why holding one open /// forever is not an option. - private var preparedExpiry: Task? + var preparedExpiry: Task? /// The recorder for the in-flight session; nil between `stop()` and `start()`. - private var activeRecorder: AVAudioRecorder? + var activeRecorder: AVAudioRecorder? /// Whether the in-flight session's input is a Bluetooth device, sampled once /// at `start()`. Read by `stop()` to decide on the tail linger — sampled at /// start rather than re-read at stop so a device switch mid-utterance can't /// make the two halves of one capture disagree. private var activeInputIsBluetooth = false + /// Incremented by every `stop()` / `cancelCapture()`. `start()` snapshots it + /// before suspending in the liveness wait — its one internal suspension — and + /// re-checks after, so a teardown that interleaves during the wait wins. + /// Without it, a reentrant caller's stop saw `activeRecorder == nil`, returned + /// an empty "clean stop", and the not-yet-installed recorder kept capturing + /// (mic indicator hot, temp WAV leaked) until `start()` resumed. Unreachable + /// through `DictationSession` — its serial command queue runs release/cancel + /// only after the press turn completes — but this is a public actor, and any + /// host can call it unqueued. + private var stopGeneration = 0 + /// Polls the active recorder's meter and feeds `levels` while recording. private var meterTask: Task? /// The last value `emitLevel` put on the stream, so an unchanged tick can be @@ -108,7 +119,7 @@ public actor MicCapture: MicCaptureProtocol { /// window, and a user who stops dictating gets their output route back shortly /// after. The next press past the window simply prepares lazily, which is the /// behavior that shipped before the re-warm existed. - private static let preparedRecorderLifetime = Duration.seconds(60) + static let preparedRecorderLifetime = Duration.seconds(60) public init() { // The continuation is fed from a ~20 Hz meter timer; the levels stream is a @@ -119,14 +130,6 @@ public actor MicCapture: MicCaptureProtocol { self.levelsContinuation = continuation } - /// Pre-create and prepare a recorder so the first `start()` skips first-time - /// hardware route discovery. Does NOT begin capture — no mic indicator. Safe to - /// call multiple times; a failure here just leaves `start()` to prepare lazily. - public func warmUp() { - guard preparedRecorder == nil, activeRecorder == nil else { return } - prepareWarmRecorder() - } - public func start() async throws { // One CoreAudio read per session, answering both questions this capture has // about its input: whether the warm recorder is still bound to it, and @@ -148,14 +151,55 @@ public actor MicCapture: MicCaptureProtocol { throw BlurtError.audioCaptureFailed(underlying: MicCaptureError.noInputDevice) } + // `record()` returning true only means the AudioQueue started — not that the + // input route is delivering frames. A Bluetooth mic spends up to a couple of + // seconds switching into its mic-capable profile first, and the OS captures + // nothing in that window, so returning here immediately cues the user to + // speak into a dead mic and the first words never reach the transcript. + // Hold — `DictationSession` keeps the pill in `.connecting` and the start + // chime waits — until the recorder's clock advances, capped per transport; + // on timeout proceed anyway (fail open), which is exactly the old behavior. + // + // The re-warm above is what makes this cheap in the common case: a warm + // recorder has already held the route open, so the wait usually returns + // immediately. This gate is what makes it *correct* when it hasn't. + let timeout = MicLiveness.timeout(forTransportType: input?.transportType) + let generationBeforeWait = stopGeneration + let gap = await MicLiveness.waitUntilLive(timeout: timeout, clock: ContinuousClock()) { + // Off-actor read (`waitUntilLive` is nonisolated), safe by confinement: + // the polls run sequentially within one task, and nothing else references + // this recorder while `start()` is suspended in the wait — it isn't + // `activeRecorder` yet, the warm slot was cleared above, and the meter + // task hasn't started. + recorder.currentTime + } + + // A teardown that landed while the wait was suspended wins: tear the + // recorder down instead of installing it, so the caller's stop stays a real + // stop (see `stopGeneration`). + guard stopGeneration == generationBeforeWait else { + recorder.stop() + Self.removeFile(at: recorder.url) + Self.logger.info("start aborted — teardown landed during the liveness wait") + throw CancellationError() + } + + if let gap { + Self.logger.info("input live after \(Int(gap.milliseconds.rounded())) ms") + } else { + Self.logger.error( + "input liveness unconfirmed after \(Int(timeout.milliseconds.rounded())) ms — proceeding") + } + activeRecorder = recorder - activeInputIsBluetooth = input?.isBluetooth ?? false + activeInputIsBluetooth = AudioTransport.isBluetooth(input?.transportType) lastEmittedLevel = nil Self.logger.info("start recording to \(recorder.url.lastPathComponent, privacy: .public)") startMeterTimer() } public func stop() async throws -> Data { + stopGeneration += 1 meterTask?.cancel() meterTask = nil guard let recorder = activeRecorder else { return Data() } @@ -203,6 +247,7 @@ public actor MicCapture: MicCaptureProtocol { /// (which can throw, via `stop()`) still conforms and `stopAndCancel`'s /// developer-mode failure log keeps working for hosts that take it. public func cancelCapture() { + stopGeneration += 1 meterTask?.cancel() meterTask = nil guard let recorder = activeRecorder else { return } @@ -213,89 +258,12 @@ public actor MicCapture: MicCaptureProtocol { scheduleRewarm() } - // MARK: - Warm recorder - - /// The warm recorder if it is still bound to `input`, else nil — discarding - /// (and cleaning up after) one that isn't. - /// - /// Reuse requires *positively* confirming the device is unchanged: an - /// unreadable route on either side leaves us unable to tell, and a recorder - /// bound to the wrong device doesn't fail loudly — it records the wrong mic, or - /// silence. Paying route activation is the cheaper mistake, so unknown means - /// discard. - private func takeWarmRecorder(matching input: AudioRoute.InputSnapshot?) -> AVAudioRecorder? { - preparedExpiry?.cancel() - preparedExpiry = nil - guard let recorder = preparedRecorder else { return nil } - let warmed = preparedInput - preparedRecorder = nil - preparedInput = nil - guard let warmed, let input, warmed.deviceID == input.deviceID else { - Self.removeFile(at: recorder.url) - Self.logger.info("discarded warm recorder — input device changed since warm-up") - return nil - } - return recorder - } - - /// Queues a re-warm to run once the current actor turn finishes, so the caller - /// (`stop()` / `cancelCapture()`) returns before the input is re-opened. - private func scheduleRewarm() { - Task { [weak self] in - await self?.rewarm() - } - } - - /// Prepares the next session's recorder, unless a capture has already started - /// or a warm one is already held — both of which mean this re-warm has been - /// overtaken and has nothing to do. - private func rewarm() { - guard activeRecorder == nil, preparedRecorder == nil else { return } - prepareWarmRecorder() - } - - /// Builds a recorder, records the input it is bound to, and starts its idle - /// countdown. A failure is non-fatal: `start()` then prepares lazily, exactly - /// as it did before any warm recorder existed. - private func prepareWarmRecorder() { - do { - let recorder = try Self.makeRecorder() - preparedRecorder = recorder - preparedInput = AudioRoute.currentInput() - armPreparedRecorderExpiry() - Self.logger.info("prepared a warm recorder") - } catch { - Self.logger.error("warm-up failed: \(error.localizedDescription, privacy: .public)") - } - } - - private func armPreparedRecorderExpiry() { - preparedExpiry?.cancel() - preparedExpiry = Task { [weak self] in - try? await Task.sleep(for: Self.preparedRecorderLifetime) - guard !Task.isCancelled else { return } - await self?.releasePreparedRecorder() - } - } - - /// Tears down an idle warm recorder, freeing the input device — which is what - /// lets a Bluetooth output route return to its full-quality profile. See - /// `preparedRecorderLifetime`. - private func releasePreparedRecorder() { - preparedExpiry = nil - guard let recorder = preparedRecorder else { return } - preparedRecorder = nil - preparedInput = nil - Self.removeFile(at: recorder.url) - Self.logger.info("released idle warm recorder") - } - // MARK: - Recorder construction /// Build a recorder that writes mono 16-bit little-endian PCM at the target /// rate into a unique temp file. `prepareToRecord()` does the heavy route/buffer /// setup so the subsequent `record()` starts promptly. - private static func makeRecorder() throws -> AVAudioRecorder { + static func makeRecorder() throws -> AVAudioRecorder { let url = FileManager.default.temporaryDirectory .appendingPathComponent("blurt-\(UUID().uuidString).wav") let settings: [String: Any] = [ @@ -343,7 +311,11 @@ public actor MicCapture: MicCaptureProtocol { } // The dB→0...1 conversion `emitLevel` uses lives in `MicCapture+Meter.swift` - // — pure math the coverage gate counts, unlike this hardware-bound actor. + // — pure math the coverage gate counts, unlike this hardware-bound actor. The + // warm-recorder lifecycle (`warmUp`, `takeWarmRecorder`, the re-warm and its + // expiry) lives in `MicCapture+Warm.swift`, split off for the lint + // file-length budget — which is why the prepared-recorder state, `logger`, + // `removeFile` and `makeRecorder` are internal rather than private. // MARK: - File helpers @@ -365,7 +337,7 @@ public actor MicCapture: MicCaptureProtocol { return Data(bytes: channel, count: Int(buffer.frameLength) * SyncSTTLimits.bytesPerSample) } - private static func removeFile(at url: URL) { + static func removeFile(at url: URL) { try? FileManager.default.removeItem(at: url) } } diff --git a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift index 244485d4..1adb65ca 100644 --- a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift +++ b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift @@ -2,6 +2,15 @@ import Foundation public protocol MicCaptureProtocol: Sendable { /// Begin capturing 16 kHz mono 16-bit PCM. Throws on permission/device failure. + /// + /// **May hold until the input device actually delivers frames.** A Bluetooth + /// route spends up to ~2.5 s switching into its mic-capable profile before any + /// audio flows, and the OS captures nothing in that window. Hosts render the + /// whole in-flight call as a distinct "connecting" state (the pipeline's + /// `.connecting` phase), with the "speak now" cues — the recording pill, the + /// start chime — arriving only on return. A conformer that returns before + /// frames flow cues the user to speak into a dead mic, and the first words of + /// the utterance are unrecoverable. func start() async throws /// Stop capture and return the captured audio as raw S16LE PCM bytes — the /// exact encoding the dictation request uploads, so no conversion pass sits on diff --git a/Sources/BlurtEngine/Audio/MicLiveness.swift b/Sources/BlurtEngine/Audio/MicLiveness.swift new file mode 100644 index 00000000..419d46a5 --- /dev/null +++ b/Sources/BlurtEngine/Audio/MicLiveness.swift @@ -0,0 +1,61 @@ +import Foundation + +/// The pure decision half of `MicCapture.start()`'s liveness gate: how long to +/// wait for the input device to actually deliver frames, and the polling loop +/// that detects when it has. Kept out of the hardware-bound capture actor (the +/// same split as `MicCapture+Meter`) so the timeout policy and the wait's edge +/// cases are unit-tested against an injected clock. +enum MicLiveness { + /// How often the recorder's clock is re-checked while waiting. Every press + /// whose first read is still 0 pays at least one quantum before `.recording` + /// and the start chime, so keep it short — 10 ms, the same cadence as + /// `KeyInjector.waitUntilFrontmost`'s poll-until-deadline loop — rather than + /// taxing the common (wired/built-in) case to wait for the Bluetooth one. + static let pollInterval: Duration = .milliseconds(10) + + /// Wait cap for Bluetooth inputs: bringing an AirPods mic up means a profile + /// switch into the mic-capable mode that takes ~1–2 s, and the link drops back + /// to the output-only profile after an idle gap — so a dictation after a pause + /// pays it again, not just the first one. + /// + /// `MicCapture`'s re-warm shortens how *often* this is paid (it keeps the + /// input open between dictations); this cap governs what happens when it is + /// paid anyway. + static let bluetoothTimeout: Duration = .milliseconds(2500) + + /// Wait cap for every other transport (and an unreadable one): wired and + /// built-in inputs deliver frames near-instantly, so a route that hasn't + /// within this budget is broken and the gate fails open without ever making a + /// healthy mic feel laggy. + static let defaultTimeout: Duration = .milliseconds(300) + + /// The wait cap for an input device of the given CoreAudio transport type + /// (`kAudioDevicePropertyTransportType`); nil means the type couldn't be read, + /// which gets the conservative default cap. + static func timeout(forTransportType transportType: UInt32?) -> Duration { + AudioTransport.isBluetooth(transportType) ? bluetoothTimeout : defaultTimeout + } + + /// Polls `currentTime` every `pollInterval` until it advances past zero. The + /// recorder's clock only moves once the input device delivers frames, which is + /// what distinguishes "route still switching" (clock stuck at 0) from "user is + /// silent" (clock advancing over quiet audio) — a meter level can't. + /// + /// Returns the elapsed wait once frames flow, or nil when `timeout` (or a task + /// cancellation) won the race. The caller FAILS OPEN on nil — proceeding + /// exactly as if live — because a silent or broken mic must degrade to the old + /// behavior, never brick the press. + static func waitUntilLive( + timeout: Duration, + clock: some Clock, + currentTime: @escaping @Sendable () -> TimeInterval + ) async -> Duration? { + let start = clock.now + let deadline = start.advanced(by: timeout) + while currentTime() <= 0 { + guard clock.now < deadline, !Task.isCancelled else { return nil } + try? await clock.sleep(for: pollInterval) + } + return start.duration(to: clock.now) + } +} diff --git a/Sources/BlurtEngine/Duration+Milliseconds.swift b/Sources/BlurtEngine/Duration+Milliseconds.swift new file mode 100644 index 00000000..75505ded --- /dev/null +++ b/Sources/BlurtEngine/Duration+Milliseconds.swift @@ -0,0 +1,8 @@ +extension Duration { + /// This duration in milliseconds as a Double (for latency logging). Expressed + /// as a ratio of two `Duration`s rather than reassembled from `components`, + /// which meant restating the attoseconds-per-millisecond constant by hand. + var milliseconds: Double { + self / Duration.milliseconds(1) + } +} diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index 3ffd39ca..7c187a88 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -221,16 +221,14 @@ public actor DictationSession { // the only throwing call, and it precedes `.recording`, so the two ends are // mutually exclusive). let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) - // Claim `.starting` before anything slow, so the overlay answers the - // keypress now rather than when the mic finishes opening. `mic.start()` - // below is the slow step — it resolves and activates the hardware route, - // which on a Bluetooth input means renegotiating the link into its - // mic-capable mode. Until this phase existed, all of that sat between the - // user's key-down and the first thing they could see or hear, and read as - // the app lagging behind them. `.starting` is presented as "starting", never - // as live capture, so the phase still flips to `.recording` only once audio - // is genuinely being recorded. - setPhase(.starting) + // Claim `.connecting` before `mic.start()`: its liveness gate holds until + // the input route actually delivers frames, which on a Bluetooth route is + // ~1–2 s. The press must be visibly acknowledged in that window without + // cueing the user to speak — the pill shows a warming-up state, and the + // start chime rides the connecting→recording edge (`RecordingCueGate`), so + // it fires only once audio is genuinely flowing. `.recording` therefore + // keeps meaning exactly what it says. + setPhase(.connecting) do { // Pre-open the dictation connection while the user speaks, so the first dictation after an idle // gap doesn't pay DNS+TCP+TLS on the transcribe hot path (~170 ms cold, measured). Detached diff --git a/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift b/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift index 18848a43..88db129c 100644 --- a/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift +++ b/Sources/BlurtEngine/Pipeline/MenuBarStatus.swift @@ -40,11 +40,12 @@ extension PipelinePhase { switch self { case .recording: .recording case .transcribing: .transcribing - // `.starting` rests at idle. The status item is the coarse indicator, and - // showing "recording" while the mic is still opening would be the one thing - // the phase exists to prevent; a distinct fourth state isn't worth a glyph - // for something that is usually gone within a frame. - case .idle, .starting, .injecting, .cancelled, .failed, .pasted, .noTarget: .idle + // `.connecting` reads as idle: the filled glyph means "audio is being + // captured", and during the mic bring-up it isn't yet — the same honesty + // rule that holds the start chime. The pill carries the warming-up state, + // and a distinct fourth glyph isn't worth it for something usually gone + // within a frame on a wired mic. + case .idle, .connecting, .injecting, .cancelled, .failed, .pasted, .noTarget: .idle } } } diff --git a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift index e8a65247..d20c3883 100644 --- a/Sources/BlurtEngine/Pipeline/OverlayUIState.swift +++ b/Sources/BlurtEngine/Pipeline/OverlayUIState.swift @@ -3,13 +3,14 @@ /// is unit-testable; the shell just renders whatever this resolves to. public enum OverlayUIState: Equatable, Sendable { case idle - /// The press landed and the mic is opening — the pill is up, but nothing is - /// being captured yet. A steady state, not a notice: it holds for exactly as - /// long as the hardware takes, which is a frame or two on the built-in mic and - /// noticeably longer on a Bluetooth input. The shell renders it as a plain - /// "Starting…" status line rather than the `● REC` tag, so the pill answers - /// the keypress without claiming to be recording. - case starting + /// The press landed but the mic isn't delivering audio yet (the pipeline's + /// `.connecting` phase — a Bluetooth route takes ~1–2 s to come up). A steady + /// state, not a notice: it holds for exactly as long as the hardware takes, + /// which is a frame or two on the built-in mic. The shell renders it as a + /// breathing "Connecting…" status line, deliberately *without* the `● REC` + /// tag or the meter — those are the "speak now" cues, and speech during the + /// bring-up is unrecoverable, so the pill must not invite it. + case connecting case recording case processing /// A dictation attempt failed. The shell shows this as a brief red flash on @@ -34,7 +35,7 @@ public enum OverlayUIState: Equatable, Sendable { public var accessibilityLabel: String { switch self { case .idle: "Blurt." - case .starting: "Starting." + case .connecting: "Connecting to the microphone." case .recording: "Recording." case .processing: "Processing." case .error(let message): message @@ -54,7 +55,7 @@ public enum OverlayUIState: Equatable, Sendable { switch self { case .pasted: 0.8 case .error, .noTarget: 1.6 - case .idle, .starting, .recording, .processing: nil + case .idle, .connecting, .recording, .processing: nil } } } @@ -74,9 +75,9 @@ extension PipelinePhase { case .injecting: .processing // Its own pill state, NOT `.recording`: the phase exists precisely because // capture hasn't begun, so projecting it onto the recording pill would put - // the `● REC` tag and a live meter on screen over a mic that isn't open yet - // — the lie the split was made to avoid. - case .starting: .starting + // the `● REC` tag and a live meter on screen over a mic that isn't + // delivering yet — the "speak now" cue the whole gate exists to withhold. + case .connecting: .connecting case .recording: .recording case .transcribing: .processing // A setup blocker (a missing API key) is an expected state, not a fault: the diff --git a/Sources/BlurtEngine/Pipeline/PipelinePhase.swift b/Sources/BlurtEngine/Pipeline/PipelinePhase.swift index b8768242..9c9f09f2 100644 --- a/Sources/BlurtEngine/Pipeline/PipelinePhase.swift +++ b/Sources/BlurtEngine/Pipeline/PipelinePhase.swift @@ -2,18 +2,19 @@ import Foundation public enum PipelinePhase: Equatable, Sendable { case idle - /// The press was accepted and the mic is being opened, but no audio is being - /// captured yet. Claimed *before* `mic.start()` so the overlay answers the - /// keypress immediately instead of at whatever moment the hardware route - /// finishes coming up — on a Bluetooth input that is hundreds of milliseconds, - /// sometimes over a second, of a press that looked like it did nothing. + /// The press was accepted and the mic is being brought up: `MicCapture`'s + /// liveness gate is still waiting for the input route to deliver frames (a + /// Bluetooth profile switch takes ~1–2 s). Claimed *before* `mic.start()`, so + /// the overlay answers the keypress immediately instead of at whatever moment + /// the hardware finishes coming up. /// - /// Deliberately distinct from `.recording` rather than folded into it: the - /// projections below present it as "starting", never as live capture, which - /// keeps the rule that **the UI never claims audio is being recorded before it - /// is**. Non-terminal, so a second press during it is refused like one during - /// `.recording`. - case starting + /// Deliberately distinct from `.recording` rather than folded into it. The + /// projections below present it as a warming-up state, never as live capture, + /// and the start chime waits for `.recording` — so **the user is never cued to + /// speak into a mic that isn't delivering yet**, which is how the first words + /// of an utterance went missing on AirPods. Non-terminal, so a second press + /// during it is refused like one during `.recording`. + case connecting case recording case transcribing case injecting @@ -38,25 +39,7 @@ public enum PipelinePhase: Equatable, Sendable { public var isTerminal: Bool { switch self { case .idle, .failed, .cancelled, .pasted, .noTarget: true - case .starting, .recording, .transcribing, .injecting: false - } - } - - /// Whether this phase is part of a live capture attempt — the mic is open, or - /// on its way to being open. - /// - /// The single definition of "a dictation is being captured right now", so the - /// consumers that key off it can't drift apart. Today that's `RecordingCueGate` - /// (the start chime fires on the *press*, i.e. entering `.starting`, so the - /// user hears the app respond at key-down rather than after the route comes - /// up). Internal: nothing outside the engine asks, and `.periphery.yml` runs - /// with `retain_public: false`. - /// - /// Exhaustive for the same reason as `isTerminal`. - var isCapturing: Bool { - switch self { - case .starting, .recording: true - case .idle, .transcribing, .injecting, .failed, .cancelled, .pasted, .noTarget: false + case .connecting, .recording, .transcribing, .injecting: false } } diff --git a/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift b/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift index 582ecfd6..6599add4 100644 --- a/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift +++ b/Sources/BlurtEngine/Pipeline/RecordingCueGate.swift @@ -10,27 +10,29 @@ public enum RecordingCue: Equatable, Sendable { /// Edge-detector deciding when the record start/stop chimes fire. The host calls /// `cue(for:)` on *every* pipeline phase, so the gate fires `.start` only on the -/// edge into a capture and `.stop` only on the edge out of one, staying silent -/// while a phase repeats and across transitions between two non-capturing -/// phases. Value type holding a single edge bit; the host owns one instance for -/// the app's lifetime. +/// edge into `.recording` and `.stop` only on the recording→not-recording edge, +/// staying silent while a phase repeats and across transitions between two +/// non-recording phases. Value type holding a single edge bit; the host owns one +/// instance for the app's lifetime. /// -/// The edge is `PipelinePhase.isCapturing`, not `== .recording`, so the start -/// chime fires when the user presses the key (`.starting`) rather than when the -/// mic finishes opening. On a Bluetooth input those are hundreds of milliseconds -/// apart, and the chime is the fastest feedback the app has — holding it until -/// the route is live wasted exactly the interval it was there to cover. The -/// `.starting`→`.recording` step is inside one capture, so it stays silent. +/// The edge is `.recording` specifically, **not** "a press happened". In +/// production the press first claims `.connecting` while `MicCapture`'s liveness +/// gate waits for the input route to deliver frames, so the chime rides the +/// connecting→recording edge by construction — it sounds when audio is actually +/// flowing, not when the key went down. That ordering is the point: the chime is +/// a "speak now" cue, and on a Bluetooth route the two moments are ~1–2 s apart, +/// during which nothing is captured. Chiming at the press invites the user to +/// speak into a dead mic and loses the first words of the utterance. public struct RecordingCueGate: Sendable { - private var wasCapturing = false + private var wasRecording = false public init() {} - /// The cue to play for `phase`, or `nil` when the capture edge didn't move. + /// The cue to play for `phase`, or `nil` when the recording edge didn't move. public mutating func cue(for phase: PipelinePhase) -> RecordingCue? { - let isCapturing = phase.isCapturing - defer { wasCapturing = isCapturing } - switch (wasCapturing, isCapturing) { + let isRecording = phase == .recording + defer { wasRecording = isRecording } + switch (wasRecording, isRecording) { case (false, true): return .start case (true, false): return .stop default: return nil diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index d7d603d6..bd852417 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -238,14 +238,14 @@ extension DictationSessionTests { #expect(await terminal == .pasted) } - @Test("press claims .starting before the mic opens, then .recording") - func pressPublishesStartingBeforeRecording() async throws { - // The whole point of the phase: the overlay gets something to show at - // key-down instead of at whatever moment `mic.start()` returns. On a - // Bluetooth input that gap is hundreds of milliseconds of a press that - // looked like it did nothing, so `.starting` must be *published*, not just - // passed through — a `setPhase` skipped here would put the pill back to - // appearing only once the hardware route was up. + @Test("press claims .connecting while the mic comes up, then .recording") + func pressPublishesConnectingBeforeRecording() async throws { + // The whole point of the phase: `mic.start()` now holds until the input + // route actually delivers frames (~1–2 s on a Bluetooth link), and the + // overlay needs something to show for that whole window — while the start + // chime deliberately waits for `.recording`. So `.connecting` must be + // *published*, not merely passed through: a `setPhase` skipped here leaves + // the pill absent for the entire bring-up, and the press looks ignored. let fixture = makeSession() let stream = await fixture.session.phaseStream() @@ -259,7 +259,7 @@ extension DictationSessionTests { // The subscription's initial yield is the current phase (.idle), then the // press's two transitions in order. - #expect(seen == [.idle, .starting, .recording]) + #expect(seen == [.idle, .connecting, .recording]) await fixture.session.cancel() } diff --git a/Tests/BlurtEngineTests/MenuBarStatusTests.swift b/Tests/BlurtEngineTests/MenuBarStatusTests.swift index 0ec8fd58..915c3e73 100644 --- a/Tests/BlurtEngineTests/MenuBarStatusTests.swift +++ b/Tests/BlurtEngineTests/MenuBarStatusTests.swift @@ -17,8 +17,8 @@ struct MenuBarStatusTests { (.transcribing, .transcribing), (.idle, .idle), // The mic is still opening, so the coarse indicator rests at idle rather - // than claiming "recording" — the one thing `.starting` exists to prevent. - (.starting, .idle), + // than claiming "recording" — the one thing `.connecting` exists to prevent. + (.connecting, .idle), // Injection happens silently; the indicator rests at idle through the brief // paste rather than showing a distinct state. (.injecting, .idle), diff --git a/Tests/BlurtEngineTests/MicLivenessTests.swift b/Tests/BlurtEngineTests/MicLivenessTests.swift new file mode 100644 index 00000000..125dd0be --- /dev/null +++ b/Tests/BlurtEngineTests/MicLivenessTests.swift @@ -0,0 +1,115 @@ +import CoreAudio +import Foundation +import Synchronization +import Testing + +@testable import BlurtEngine + +/// The pure half of `MicCapture.start()`'s liveness gate: the transport-aware +/// wait cap and the poll-until-the-recorder's-clock-advances loop. Driven +/// against `TestClock` so the Bluetooth cap is exercised without waiting real +/// seconds. +@Suite("MicLiveness", .timeLimit(.minutes(1))) +struct MicLivenessTests { + @Test("Bluetooth transports get the long cap; everything else the short one") + func transportTimeouts() { + // The profile switch is the whole reason the gate exists — both Bluetooth + // transport types must get the multi-second budget. + #expect( + MicLiveness.timeout(forTransportType: kAudioDeviceTransportTypeBluetooth) + == MicLiveness.bluetoothTimeout) + #expect( + MicLiveness.timeout(forTransportType: kAudioDeviceTransportTypeBluetoothLE) + == MicLiveness.bluetoothTimeout) + // Wired/built-in inputs deliver frames near-instantly; a long cap there + // would make a genuinely broken mic feel like a hang. + #expect( + MicLiveness.timeout(forTransportType: kAudioDeviceTransportTypeBuiltIn) + == MicLiveness.defaultTimeout) + #expect( + MicLiveness.timeout(forTransportType: kAudioDeviceTransportTypeUSB) + == MicLiveness.defaultTimeout) + // An unreadable transport must not be treated as Bluetooth. + #expect(MicLiveness.timeout(forTransportType: nil) == MicLiveness.defaultTimeout) + } + + @Test("already-advancing recorder clock confirms immediately, without sleeping") + func immediateLiveness() async { + let clock = TestClock() + // Never advanced: a sleep would park forever, so returning at all proves + // the fast path never sleeps (the suite's time limit backs that up). + let gap = await MicLiveness.waitUntilLive(timeout: .seconds(1), clock: clock) { 0.1 } + #expect(gap == .zero) + } + + @Test("a clock that advances after a few polls confirms with the elapsed gap") + func livenessAfterPolls() async { + let clock = TestClock() + let polls = Mutex(0) + async let gap = MicLiveness.waitUntilLive(timeout: MicLiveness.bluetoothTimeout, clock: clock) { + // Stuck at 0 for the first two checks — the route still switching — then + // the recorder clock starts moving. + polls.withLock { polls in + polls += 1 + return polls < 3 ? 0 : 0.05 + } + } + for _ in 1...2 { + await clock.waitUntilSleeping(for: MicLiveness.pollInterval) + clock.advance(by: MicLiveness.pollInterval) + } + #expect(await gap == MicLiveness.pollInterval * 2) + } + + @Test("a clock that never advances times out with nil — the fail-open signal") + func timeoutFailsOpen() async { + // nil is what tells `MicCapture` to proceed anyway: a silent or broken mic + // must degrade to the pre-gate behavior, never brick the press. + let clock = TestClock() + let timeout = MicLiveness.pollInterval * 2 + async let gap = MicLiveness.waitUntilLive(timeout: timeout, clock: clock) { 0 } + for _ in 1...2 { + await clock.waitUntilSleeping(for: MicLiveness.pollInterval) + clock.advance(by: MicLiveness.pollInterval) + } + #expect(await gap == nil) + } +} + +/// The transport classification both the liveness cap and `MicCapture`'s tail +/// linger hang off. Pure and pinned here because `AudioRoute`, which reads the +/// raw value, needs real hardware and is excluded from the coverage gate — so +/// this is the only place the decision can be tested. +@Suite("AudioTransport") +struct AudioTransportTests { + @Test("both Bluetooth transport types count") + func bluetoothTransports() { + #expect(AudioTransport.isBluetooth(kAudioDeviceTransportTypeBluetooth)) + #expect(AudioTransport.isBluetooth(kAudioDeviceTransportTypeBluetoothLE)) + } + + @Test("wired, built-in, and unreadable transports do not") + func nonBluetoothTransports() { + #expect(!AudioTransport.isBluetooth(kAudioDeviceTransportTypeBuiltIn)) + #expect(!AudioTransport.isBluetooth(kAudioDeviceTransportTypeUSB)) + #expect(!AudioTransport.isBluetooth(kAudioDeviceTransportTypeAggregate)) + // nil is the conservative answer for both consumers: the short wait cap and + // no tail linger. Padding every wired capture with a delay would be a worse + // regression than losing the tail on a device we couldn't classify. + #expect(!AudioTransport.isBluetooth(nil)) + } +} + +/// `Duration.milliseconds` backs the latency lines `MicCapture` logs for the +/// liveness gap — the field evidence for whether the gate is doing anything — +/// so a silently wrong conversion would make those logs lie. +@Suite("Duration.milliseconds") +struct DurationMillisecondsTests { + @Test func convertsWholeAndFractionalDurations() { + #expect(Duration.milliseconds(250).milliseconds == 250) + #expect(Duration.seconds(2).milliseconds == 2000) + #expect(Duration.zero.milliseconds == 0) + // Sub-millisecond durations keep their fraction rather than truncating to 0. + #expect(Duration.microseconds(500).milliseconds == 0.5) + } +} diff --git a/Tests/BlurtEngineTests/OverlayUIStateTests.swift b/Tests/BlurtEngineTests/OverlayUIStateTests.swift index 8c358e6b..7c9f3e43 100644 --- a/Tests/BlurtEngineTests/OverlayUIStateTests.swift +++ b/Tests/BlurtEngineTests/OverlayUIStateTests.swift @@ -15,10 +15,10 @@ struct OverlayUIStateTests { /// carve-out — stay as their own tests below. static let projections: [(phase: PipelinePhase, expected: OverlayUIState)] = [ (.idle, .idle), - // Its own pill state, not `.recording`: `.starting` exists because capture + // Its own pill state, not `.recording`: `.connecting` exists because capture // hasn't begun, so projecting it onto the recording pill would show the // `● REC` tag and a live meter over a mic that isn't open yet. - (.starting, .starting), + (.connecting, .connecting), (.recording, .recording), (.transcribing, .processing), // `.injecting` is a *working* phase, so it must not project to `.idle`: the @@ -76,7 +76,7 @@ struct OverlayUIStateTests { #expect(PipelinePhase.failed(.targetAppLost).setupBlocker == nil) // Neither is any non-failed phase. for phase in [ - PipelinePhase.idle, .starting, .recording, .transcribing, .injecting, .pasted, .noTarget, + PipelinePhase.idle, .connecting, .recording, .transcribing, .injecting, .pasted, .noTarget, ] { #expect(phase.setupBlocker == nil) } @@ -104,7 +104,7 @@ struct OverlayUIStateAccessibilityLabelTests { /// (echo the carried message verbatim), not a constant, so it keeps its own test. static let labels: [(state: OverlayUIState, spoken: String)] = [ (.idle, "Blurt."), - (.starting, "Starting."), + (.connecting, "Connecting to the microphone."), (.recording, "Recording."), (.processing, "Processing."), (.pasted, "Your dictation was pasted."), @@ -144,9 +144,9 @@ struct OverlayUIStateNoticeDwellTests { @Test func steadyStatesHaveNoDwell() { // Held for as long as the pipeline is in them — no auto-revert. #expect(OverlayUIState.idle.noticeDwellSeconds == nil) - // `.starting` in particular: a dwell would auto-revert the pill to idle + // `.connecting` in particular: a dwell would auto-revert the pill to idle // mid-press, dismissing it while the mic was still opening. - #expect(OverlayUIState.starting.noticeDwellSeconds == nil) + #expect(OverlayUIState.connecting.noticeDwellSeconds == nil) #expect(OverlayUIState.recording.noticeDwellSeconds == nil) #expect(OverlayUIState.processing.noticeDwellSeconds == nil) } diff --git a/Tests/BlurtEngineTests/PipelinePhaseTests.swift b/Tests/BlurtEngineTests/PipelinePhaseTests.swift index c431b721..ff9d211a 100644 --- a/Tests/BlurtEngineTests/PipelinePhaseTests.swift +++ b/Tests/BlurtEngineTests/PipelinePhaseTests.swift @@ -29,36 +29,13 @@ struct PipelinePhaseTests { @Test("active phases are not terminal") func activePhasesAreNotTerminal() { - // `.starting` included: the press guard keys off terminality, so a terminal - // `.starting` would let a second key-down start a second capture while the - // first is still opening the mic — the exact window it was added to cover. - #expect(!PipelinePhase.starting.isTerminal) + // `.connecting` included: the press guard keys off terminality, so a + // terminal `.connecting` would let a second key-down start a second capture + // while the first is still bringing the mic up — a window that lasts ~1–2 s + // on a Bluetooth route, so it is very reachable. + #expect(!PipelinePhase.connecting.isTerminal) #expect(!PipelinePhase.recording.isTerminal) #expect(!PipelinePhase.transcribing.isTerminal) #expect(!PipelinePhase.injecting.isTerminal) } } - -/// `isCapturing` is the single definition of "a dictation is being captured -/// right now" — the edge the start/stop chimes ride. Pinned per case: a phase -/// wrongly reading as capturing would chime at the wrong moment, and `.starting` -/// wrongly reading as *not* capturing would put the start chime back where it -/// was, after the hardware route comes up. -@Suite("PipelinePhase.isCapturing") -struct PipelinePhaseCapturingTests { - @Test("the mic is open, or opening") - func capturingPhases() { - #expect(PipelinePhase.starting.isCapturing) - #expect(PipelinePhase.recording.isCapturing) - } - - @Test("every other phase is not capturing") - func nonCapturingPhases() { - for phase: PipelinePhase in [ - .idle, .transcribing, .injecting, .cancelled, .pasted, .noTarget, - .failed(.apiKeyMissing), - ] { - #expect(!phase.isCapturing, "\(phase) must not read as capturing") - } - } -} diff --git a/Tests/BlurtEngineTests/RecordingCueGateTests.swift b/Tests/BlurtEngineTests/RecordingCueGateTests.swift index 0bf4fb85..b5644d15 100644 --- a/Tests/BlurtEngineTests/RecordingCueGateTests.swift +++ b/Tests/BlurtEngineTests/RecordingCueGateTests.swift @@ -2,39 +2,46 @@ import Testing @testable import BlurtEngine -/// The record start/stop chimes fire on the *edges* of a capture, not on every -/// phase tick. `AppCoordinator.render` calls the cue gate on every pipeline -/// phase (idle, starting, recording, transcribing, injecting, pasted, …), so the -/// gate must fire `.start` only on the edge into a capture and `.stop` only on -/// the edge out of one, staying silent on repeats and on transitions between two -/// non-capturing phases. Lifting that edge detection out of the AppKit -/// `CueSoundPlayer` lets `swift test` cover it — the same split as +/// The record start/stop chimes fire on the *edges* of `.recording`, not on +/// every phase tick. `AppCoordinator.render` calls the cue gate on every +/// pipeline phase (idle, connecting, recording, transcribing, injecting, pasted, +/// …), so the gate must fire `.start` only on the edge into `.recording` and +/// `.stop` only on the edge out of it, staying silent on repeats and on +/// transitions between two non-recording phases. Lifting that edge detection out +/// of the AppKit `CueSoundPlayer` lets `swift test` cover it — the same split as /// `OverlayUIState`/`MenuBarStatus`. @Suite("RecordingCueGate") struct RecordingCueGateTests { - @Test("the press fires the start cue, before the mic is open") - func startOnPressEdge() { - // The edge is `.starting`, not `.recording`. The chime is the app's fastest - // feedback, and on a Bluetooth input those two phases are hundreds of - // milliseconds apart — holding the chime until the route came up wasted - // exactly the interval it was there to cover. + @Test("entering recording fires the start cue") + func startOnRisingEdge() { var gate = RecordingCueGate() - #expect(gate.cue(for: .starting) == .start) + #expect(gate.cue(for: .recording) == .start) } - @Test("the mic coming up mid-capture is silent") - func noCueOnStartingToRecording() { - // `.starting` → `.recording` is one capture continuing, not a new one. + @Test("the mic bring-up is silent — the chime waits for audio to actually flow") + func connectingIsSilent() { + // The load-bearing case. `.connecting` is the window where `MicCapture`'s + // liveness gate is waiting for the input route, which on a Bluetooth link is + // ~1–2 s during which the OS captures nothing. A chime here is a "speak now" + // cue for a dead mic, and the first words of the utterance are lost — the + // exact bug this ordering exists to prevent. var gate = RecordingCueGate() - #expect(gate.cue(for: .starting) == .start) - #expect(gate.cue(for: .recording) == nil) + #expect(gate.cue(for: .connecting) == nil) + // It fires on the connecting→recording edge instead. + #expect(gate.cue(for: .recording) == .start) } - @Test("entering recording from idle fires the start cue") - func startOnRisingEdge() { - // A host that never observes `.starting` (a phase stream joined late) still - // gets its start chime on the first capturing phase it does see. + @Test("a press whose mic never comes up never chimes") + func failedBringUpIsSilent() { + // `mic.start()` throwing takes the pipeline `.connecting` → `.failed` + // without ever reaching `.recording`. No start cue was played, so no stop + // cue is owed either — and the gate must be left unlatched for the next + // press rather than thinking a capture is still running. var gate = RecordingCueGate() + #expect(gate.cue(for: .connecting) == nil) + let failure = PipelinePhase.failed(.audioCaptureFailed(underlying: MicCaptureError.noInputDevice)) + #expect(gate.cue(for: failure) == nil) + #expect(gate.cue(for: .connecting) == nil) #expect(gate.cue(for: .recording) == .start) } @@ -52,23 +59,11 @@ struct RecordingCueGateTests { #expect(gate.cue(for: .recording) == nil) } - @Test("a press whose mic never opens still chimes closed") - func failedStartClosesTheCue() { - // `mic.start()` throwing takes the pipeline `.starting` → `.failed`. The - // start chime has already played, so the stop chime is what keeps the pair - // balanced — and it leaves the gate ready for the next press rather than - // latched as if a capture were still running. + @Test("transitions between two non-recording phases are silent") + func silentBetweenNonRecordingPhases() { var gate = RecordingCueGate() - #expect(gate.cue(for: .starting) == .start) - #expect(gate.cue(for: .failed(.audioCaptureFailed(underlying: MicCaptureError.noInputDevice))) == .stop) - #expect(gate.cue(for: .starting) == .start) - } - - @Test("transitions between two non-capturing phases are silent") - func silentBetweenNonCapturingPhases() { - var gate = RecordingCueGate() - // From the initial (non-capturing) state through a run of non-capturing - // phases, nothing chimes — only a capture edge does. + // From the initial (non-recording) state through a run of non-recording + // phases, nothing chimes — only a recording edge does. #expect(gate.cue(for: .idle) == nil) #expect(gate.cue(for: .transcribing) == nil) #expect(gate.cue(for: .injecting) == nil) @@ -76,13 +71,14 @@ struct RecordingCueGateTests { #expect(gate.cue(for: .failed(.apiKeyMissing)) == nil) } - @Test("a full record→stop→record cycle chimes start, stop, start again") + @Test("a full press→record→stop→press cycle chimes start, stop, start again") func fullCycle() { var gate = RecordingCueGate() - #expect(gate.cue(for: .starting) == .start) - #expect(gate.cue(for: .recording) == nil) + #expect(gate.cue(for: .connecting) == nil) + #expect(gate.cue(for: .recording) == .start) #expect(gate.cue(for: .injecting) == .stop) #expect(gate.cue(for: .idle) == nil) - #expect(gate.cue(for: .starting) == .start) + #expect(gate.cue(for: .connecting) == nil) + #expect(gate.cue(for: .recording) == .start) } } From 6d25adae8862d95de143d209c6c66cff8181fd81 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 18:13:22 +0000 Subject: [PATCH 05/14] Remove the duplicate Duration.milliseconds extension MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AssemblyAITranscriber already carried a fileprivate copy of exactly this extension for its request-timing log. Adding the shared one without removing it is an invalid redeclaration, not a shadow — fileprivate and internal members of the same type collide within a module. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- Sources/BlurtEngine/STT/AssemblyAITranscriber.swift | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index a120df41..a7fc5d73 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -339,14 +339,10 @@ private final class MetricsLogger: NSObject, URLSessionTaskDelegate, @unchecked } } -extension Duration { - /// This duration in milliseconds as a Double (for latency logging). Expressed - /// as a ratio of two `Duration`s rather than reassembled from `components`, - /// which meant restating the attoseconds-per-millisecond constant by hand. - fileprivate var milliseconds: Double { - self / Duration.milliseconds(1) - } -} +// `Duration.milliseconds` — the latency-logging conversion this file's request +// timing uses — moved to `Duration+Milliseconds.swift` when `MicCapture` needed +// the same thing for its liveness-gap line. It was `fileprivate` here; a second +// copy is an "invalid redeclaration", not a shadow. /// Errors specific to the AssemblyAI transport. These get wrapped in /// `BlurtError.sttFailed` before reaching the UI. From 8039fd19f8abfd102c10bd12de0e9ba10e4faa97 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 19:07:06 +0000 Subject: [PATCH 06/14] Fix five review findings from the liveness-gate work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five are consequences of `start()` gaining a suspension: state that used to be readable as a snapshot is now observable mid-flight. - MicCapture: `bringingUpCapture`. Across the liveness wait BOTH recorder slots are nil — `activeRecorder` isn't installed until the wait returns (the recorder stays confined to `start()` so nothing touches it while the poll loop reads its clock off-actor), and the warm slot was consumed on the way in. `rewarm()`/`warmUp()` read that as "idle" and prepared a second recorder onto the live input, which `stop()`'s scheduled re-warm makes very reachable. Both now go through `canPrepareWarmRecorder`. - MicCapture+Warm: the prepared-recorder expiry carries a generation ticket. Cancelling the task isn't enough — an expiry that already passed its `!Task.isCancelled` check still gets its actor turn, where it nils the *live* expiry's handle (leaving the current warm recorder with no countdown) and tears down a recorder prepared moments earlier. - AudioRouteMonitor: `deinit` no longer does `queue.sync`. `guard let self` upgrades the blocks' weak capture to strong, so while a block runs `queue` IS an owner and can drop the last reference — running deinit on that queue, where the sync deadlocks. Removal is inline now; it's race-free because deinit only runs once no block can be inside its upgrade, and `queue` is still passed to CoreAudio as the identity it matches removal on. - CueSoundPlayer: in-flight decodes carry a monotonic ticket instead of being disambiguated by `loadedPack` equality, which can't tell two loads of the same pack apart — exactly what a route re-prime forces. A decode in flight when the route changed could land last and install players primed against the old route, i.e. the stall the re-prime exists to prevent. The `loadedPack = nil` hack is gone; the observer passes `force: true`. - DictationSession: a `CancellationError` out of `start()` is the user's cancel arriving via an unqueued `cancelCapture()`, not a fault. It was caught as `.failed(.audioCaptureFailed)` — red pill plus a developer-mode error-log entry for a non-fault. Now `.cancelled`, which is also terminal, so `.connecting` can't strand the trigger's gate. The sixth finding is only partly addressed and is documented as a known gap in AGENTS.md: `performPress` now consumes a recorded `cancelRequested` before claiming `.recording`, so a cancel during the bring-up no longer produces a phantom recording and chime — but it cannot preempt the wait, and for the app it isn't even recorded until the press returns, because `submit(_:)`'s consumer is serial. Closing that needs a preemptible `mic.start()` or a non-blocking command consumer. New `GatedStartMic` stub pins the cancel-during-bring-up path. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 12 ++++- App/Blurt/Blurt/CueSoundPlayer.swift | 33 +++++++++---- .../BlurtEngine/Audio/AudioRouteMonitor.swift | 43 +++++++++-------- .../BlurtEngine/Audio/MicCapture+Warm.swift | 39 ++++++++++++---- Sources/BlurtEngine/Audio/MicCapture.swift | 25 ++++++++++ .../Pipeline/DictationSession.swift | 32 +++++++++++++ .../DictationSessionTests.swift | 46 +++++++++++++++++++ .../Stubs/GatedStartMic.swift | 44 ++++++++++++++++++ 8 files changed, 237 insertions(+), 37 deletions(-) create mode 100644 Tests/BlurtEngineTests/Stubs/GatedStartMic.swift diff --git a/AGENTS.md b/AGENTS.md index 40c7edfd..086382c6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -402,7 +402,17 @@ in both directions. So: stops the app cueing the user to speak into a dead mic; audio spoken during the switch cannot be recovered by anything, because nothing ever receives it. `stopGeneration` covers the one suspension this introduces — a teardown landing mid-wait wins, and the recorder is torn down rather than - installed. + installed. `bringingUpCapture` covers the other consequence: across the wait **both** recorder + slots are nil, so the warm-up paths can't infer "no capture in flight" from them (see + `canPrepareWarmRecorder`) or they'd open a second recorder onto the live input. + + **Known gap — a cancel during the bring-up isn't visible until it finishes.** `performPress` + consumes a recorded `cancelRequested` before claiming `.recording`, so the cancel is honored and + never leaves a phantom recording or chime behind — but it can't preempt the wait. Worse for the + app specifically: `submit(_:)`'s consumer is serial, so a submitted `.cancel` isn't even + _recorded_ until the press returns. Closing this needs either a preemptible `mic.start()` or a + command consumer that doesn't block on the press — both design changes, neither attempted yet. + - **The warm recorder is re-armed after every capture**, not just at launch. The cost above is paid at `prepareToRecord()`, i.e. per session, so warming only the first one hid it for one dictation out of N. `stop()`/`cancelCapture()` schedule a re-warm; `start()` consumes it. diff --git a/App/Blurt/Blurt/CueSoundPlayer.swift b/App/Blurt/Blurt/CueSoundPlayer.swift index 078cd032..a235e684 100644 --- a/App/Blurt/Blurt/CueSoundPlayer.swift +++ b/App/Blurt/Blurt/CueSoundPlayer.swift @@ -11,6 +11,13 @@ final class CueSoundPlayer { /// The pack the current `startSound`/`stopSound` were decoded from, so `prime()` /// can skip re-decoding when nothing changed. `nil` until the first load. private var loadedPack: SoundPack? + /// Monotonic ticket for in-flight decodes: only the newest one installs its + /// players. This is the staleness check, **not** `loadedPack` equality — which + /// cannot discriminate two loads of the *same* pack, and a route change forces + /// exactly that. Without it, a decode already in flight when the route changed + /// could land after the re-prime and install players primed against the old + /// output route — the stall the re-prime exists to prevent. + private var loadGeneration = 0 /// Edge-detector deciding when the start/stop chimes fire. The mapping from a /// pipeline phase to a cue lives in the engine (`RecordingCueGate`), where /// `swift test` covers it; this player just plays whatever it resolves to. @@ -70,8 +77,9 @@ final class CueSoundPlayer { routeObserver = Task { [weak self] in for await _ in changes { guard let self else { return } - self.loadedPack = nil - await self.loadCurrentPack() + // `force`, because the pack hasn't changed — the *route* has, and the + // pre-roll is what went stale. + await self.loadCurrentPack(force: true) } } } @@ -92,17 +100,22 @@ final class CueSoundPlayer { /// Decodes and installs the cue players for the current selection if they aren't /// already loaded. The `loadedPack` guard makes repeat calls cheap; the decode /// itself hops off the main actor. Returns once the players are assigned. - private func loadCurrentPack() async { + /// `force` skips the already-loaded short-circuit, for a re-prime where the + /// selection is unchanged and only the output route moved. + private func loadCurrentPack(force: Bool = false) async { let pack = SoundPackStore().soundPack - guard pack != loadedPack else { return } + guard force || pack != loadedPack else { return } + loadGeneration += 1 + let generation = loadGeneration loadedPack = pack let players = await Self.decode(pack) - // Re-check after the off-actor decode: if a newer selection was claimed while - // we were decoding (rapid pack switches), its decode owns the players now — - // dropping this stale result avoids installing players that disagree with - // `loadedPack`. `loadedPack` is written synchronously above (no await between - // read and write), so only the newest-requested load passes this guard. - guard loadedPack == pack else { return } + // Re-check after the off-actor decode: whoever asked last owns the players. + // Both are written synchronously above (no await between read and write), so + // exactly one in-flight decode passes this guard — the newest. Ticketed + // rather than compared against `loadedPack`, so two loads of the same pack + // (a rapid pack switch back, or any forced route re-prime) can still tell + // each other apart. + guard generation == loadGeneration else { return } startSound = players.start stopSound = players.stop } diff --git a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift index c4556c07..764d4121 100644 --- a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift +++ b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift @@ -71,26 +71,33 @@ public final class AudioRouteMonitor: @unchecked Sendable { /// listener left behind outlives the monitor, since CoreAudio retains the block /// and nothing else would ever hand it back. /// - /// The `queue.sync` cannot deadlock: the listener blocks hold `self` weakly, so - /// `queue` never owns the last reference and this deinit never runs on it. The - /// registrations are lifted into locals first so the closure captures only - /// those, never a `self` that is already being torn down. + /// Removal happens **inline, with no hop onto `queue`** — deliberately. + /// + /// A `queue.sync` here can self-deadlock. The listener blocks capture `self` + /// weakly, but `guard let self` upgrades that to a strong reference for the + /// body's duration, so while a block is running `queue` *is* an owner. If the + /// last other reference is dropped in that window, the block's release is the + /// final one and this `deinit` runs **on `queue`** — where `queue.sync` + /// deadlocks against itself. + /// + /// Inline removal is also race-free without the hop. `deinit` only runs once + /// the last reference is gone, so no block can be *inside* its `guard let self` + /// concurrently with this — a block that starts now fails the upgrade and + /// touches nothing. That leaves these reads of the queue-confined + /// registrations unopposed. (`queue` is still passed to CoreAudio, because + /// removal matches on the queue the listener was added with; that's an argument, + /// not an execution context.) deinit { continuation.finish() - let system = systemListener - let device = deviceListener - let queue = queue - systemListener = nil - deviceListener = nil - queue.sync { - if let system { - var address = Self.defaultOutputDeviceAddress - _ = AudioObjectRemovePropertyListenerBlock(Self.systemObject, &address, queue, system) - } - if let device { - var address = Self.sampleRateAddress - _ = AudioObjectRemovePropertyListenerBlock(device.id, &address, queue, device.block) - } + if let systemListener { + var address = Self.defaultOutputDeviceAddress + _ = AudioObjectRemovePropertyListenerBlock( + Self.systemObject, &address, queue, systemListener) + } + if let deviceListener { + var address = Self.sampleRateAddress + _ = AudioObjectRemovePropertyListenerBlock( + deviceListener.id, &address, queue, deviceListener.block) } } diff --git a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift index 16b14624..623b997b 100644 --- a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift +++ b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift @@ -12,10 +12,21 @@ extension MicCapture { /// hardware route discovery. Does NOT begin capture — no mic indicator. Safe to /// call multiple times; a failure here just leaves `start()` to prepare lazily. public func warmUp() { - guard preparedRecorder == nil, activeRecorder == nil else { return } + guard canPrepareWarmRecorder else { return } prepareWarmRecorder() } + /// Whether it is safe to open the input for a *warm* recorder right now: + /// nothing is capturing, nothing is mid-bring-up, and no warm recorder is + /// already held. + /// + /// `bringingUpCapture` is the load-bearing term. The two recorder slots are + /// both nil across `start()`'s liveness wait, so testing them alone reads a + /// live capture as "idle" and prepares a second recorder onto the open input. + var canPrepareWarmRecorder: Bool { + activeRecorder == nil && preparedRecorder == nil && !bringingUpCapture + } + /// The warm recorder if it is still bound to `input`, else nil — discarding /// (and cleaning up after) one that isn't. /// @@ -47,11 +58,11 @@ extension MicCapture { } } - /// Prepares the next session's recorder, unless a capture has already started - /// or a warm one is already held — both of which mean this re-warm has been + /// Prepares the next session's recorder, unless a capture is running or coming + /// up, or a warm one is already held — all of which mean this re-warm has been /// overtaken and has nothing to do. func rewarm() { - guard activeRecorder == nil, preparedRecorder == nil else { return } + guard canPrepareWarmRecorder else { return } prepareWarmRecorder() } @@ -63,26 +74,38 @@ extension MicCapture { let recorder = try Self.makeRecorder() preparedRecorder = recorder preparedInput = AudioRoute.currentInput() - armPreparedRecorderExpiry() + preparedGeneration += 1 + armPreparedRecorderExpiry(generation: preparedGeneration) Self.logger.info("prepared a warm recorder") } catch { Self.logger.error("warm-up failed: \(error.localizedDescription, privacy: .public)") } } - func armPreparedRecorderExpiry() { + /// Arms the idle countdown for the warm recorder identified by `generation`. + /// The ticket is what makes a stale expiry harmless — see + /// `releasePreparedRecorder(generation:)`. + func armPreparedRecorderExpiry(generation: Int) { preparedExpiry?.cancel() preparedExpiry = Task { [weak self] in try? await Task.sleep(for: Self.preparedRecorderLifetime) guard !Task.isCancelled else { return } - await self?.releasePreparedRecorder() + await self?.releasePreparedRecorder(generation: generation) } } /// Tears down an idle warm recorder, freeing the input device — which is what /// lets a Bluetooth output route return to its full-quality profile. See /// `preparedRecorderLifetime`. - func releasePreparedRecorder() { + /// + /// A stale expiry — one whose recorder was consumed by a press, or replaced by + /// a later re-warm — must do nothing at all, which is what `generation` buys. + /// Cancellation alone doesn't cover it: an expiry that already passed its + /// `!Task.isCancelled` check still gets its actor turn, and would otherwise nil + /// out the *live* expiry's handle (leaving the current warm recorder with no + /// countdown at all) and tear down a recorder prepared a moment ago. + func releasePreparedRecorder(generation: Int) { + guard generation == preparedGeneration else { return } preparedExpiry = nil guard let recorder = preparedRecorder else { return } preparedRecorder = nil diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index f6e88a61..6976dc29 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -51,6 +51,12 @@ public actor MicCapture: MicCaptureProtocol { /// `preparedRecorderLifetime`. See that constant for why holding one open /// forever is not an option. var preparedExpiry: Task? + /// Bumped for each warm recorder prepared, and carried by that recorder's + /// expiry task, so an expiry whose recorder has since been consumed or + /// replaced can recognise itself as stale and do nothing. See + /// `releasePreparedRecorder(generation:)` for why cancelling the task isn't + /// sufficient on its own. + var preparedGeneration = 0 /// The recorder for the in-flight session; nil between `stop()` and `start()`. var activeRecorder: AVAudioRecorder? @@ -71,6 +77,19 @@ public actor MicCapture: MicCaptureProtocol { /// host can call it unqueued. private var stopGeneration = 0 + /// True from `record()` succeeding until the capture is installed or torn + /// down — i.e. across the liveness wait. + /// + /// Needed because during that window **both** recorder slots are nil: + /// `activeRecorder` isn't installed until the wait returns (the recorder stays + /// confined to `start()` so nothing can touch it while the poll loop reads its + /// clock off-actor), and the warm slot was consumed on the way in. Without + /// this, `rewarm()`/`warmUp()` read those two nils as "no capture in flight" + /// and prepare a *second* recorder onto the already-live input — which is very + /// reachable, since `stop()` schedules a re-warm that can land inside the next + /// press's bring-up. + var bringingUpCapture = false + /// Polls the active recorder's meter and feeds `levels` while recording. private var meterTask: Task? /// The last value `emitLevel` put on the stream, so an unchanged tick can be @@ -151,6 +170,12 @@ public actor MicCapture: MicCaptureProtocol { throw BlurtError.audioCaptureFailed(underlying: MicCaptureError.noInputDevice) } + // The input is open from here, so claim the bring-up window before the first + // suspension — see `bringingUpCapture`. `defer` clears it on every exit, + // including the abort throw below. + bringingUpCapture = true + defer { bringingUpCapture = false } + // `record()` returning true only means the AudioQueue started — not that the // input route is delivering frames. A Bluetooth mic spends up to a couple of // seconds switching into its mic-capable profile first, and the OS captures diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index 813f8c14..c4209011 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -304,6 +304,25 @@ public actor DictationSession { contextFeed.yield(context.isEmpty ? nil : context) contextFeed.finish() } + // A cancel that arrived during the bring-up. It couldn't act at the time — + // `cancel()` has no synchronous path for `.connecting`, so it recorded the + // intent and queued behind this press — and the mic is live by now, so + // honor it here rather than claiming `.recording` and chiming "speak now" + // for a capture the user has already abandoned. Without this the press + // completed normally and the queued `performCancel` only tore it down + // afterwards, so a cancel produced a full recording start (chime included) + // that was then immediately cancelled. + // + // Not a fix for the *latency*: the cancel is still not visible until the + // bring-up finishes, which on a Bluetooth route is up to + // `MicLiveness.bluetoothTimeout`. Interrupting the wait would mean making + // `mic.start()` cancellable from here, which is a larger change. + if cancelRequested { + try? await mic.cancelCapture() + _ = consumeCancelRequest() + Self.signposter.endInterval(Self.pressSignpostName, pressInterval) + return + } setPhase(.recording) Self.signposter.endInterval(Self.pressSignpostName, pressInterval) let timeout = maxRecordingSeconds @@ -317,6 +336,19 @@ public actor DictationSession { } } catch { Self.signposter.endInterval(Self.pressSignpostName, pressInterval) + // `MicCapture.start()` throws `CancellationError` when a teardown landed + // during its liveness wait — an unqueued `cancelCapture()`, which its own + // doc anticipates for hosts that don't drive the mic through this session. + // That's the user's cancel arriving by another door, not a fault: reporting + // it as `.audioCaptureFailed` would flash the pill red *and* write a + // developer-mode error-log entry for something nothing went wrong in. Same + // rule `transcribe` and `inject` already follow. `.cancelled` rather than a + // bare return, because `.connecting` is non-terminal — leaving it would + // strand the trigger's gate and swallow the next press. + if error is CancellationError { + setPhase(.cancelled) + return + } setPhase(.failed(.audioCaptureFailed(underlying: error))) } } diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index bd852417..7f9e9c00 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -304,3 +304,49 @@ extension DictationSessionTests { // to stay within the lint file-length budget. The `onTranscriptDelivered` // side-channel tests live in `DictationSessionTranscriptTests.swift` for the // same reason. + +/// Behavior in the `.connecting` window — the up-to-2.5 s stretch +/// `MicCapture.start()` holds while a Bluetooth route brings the mic up. Before +/// the liveness gate, `start()` returned in microseconds and nothing could land +/// mid-press; now things can, so pin what happens when they do. +@Suite("DictationSession mic bring-up", .timeLimit(.minutes(1))) +struct DictationSessionBringUpTests { + @Test("a cancel landing during the bring-up never reaches .recording") + func cancelDuringBringUpSkipsRecording() async throws { + // The regression this guards: the press used to finish the bring-up, claim + // `.recording` and fire the start chime — cueing "speak now" for a capture + // the user had already cancelled — and only then get torn down by the queued + // cancel. `.recording` must never be published on this path, because + // `RecordingCueGate` chimes on exactly that edge. + let mic = GatedStartMic() + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), + injector: StubInjector(), seams: .offline) + + let stream = await session.phaseStream() + let pressed = Task { await session.press() } + await mic.waitUntilStartEntered() // press() is suspended inside mic.start() + // Direct `cancel()`, not `submit(.cancel)`: `submit`'s consumer is serial, so + // a submitted cancel cannot even be *recorded* until the press returns. See + // the note in `performPress` — this covers callers that can `await`. + let cancelled = Task { await session.cancel() } + await session.awaitCancelRequest() + await mic.allowStartToFinish() + await pressed.value + await cancelled.value + + var seen: [PipelinePhase] = [] + for await phase in stream { + seen.append(phase) + if phase.isTerminal, phase != .idle { break } + } + + #expect(seen == [.idle, .connecting, .cancelled]) + // Spelled out separately so a failure names the actual regression rather + // than just an unequal array. + #expect(!seen.contains(.recording)) + // The mic came up during the wait, so it has to have been torn down — and + // through the discarding teardown, not `stop()`. + #expect(await mic.cancelCaptureCalls == 1) + } +} diff --git a/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift b/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift new file mode 100644 index 00000000..0d7a651d --- /dev/null +++ b/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift @@ -0,0 +1,44 @@ +import Foundation + +@testable import BlurtEngine + +/// Mic stub whose `start()` blocks until the test releases it, so a command can +/// be landed while the session sits in `.connecting`. +/// +/// That window is not hypothetical: `MicCapture.start()` holds until the input +/// route delivers frames, which on a Bluetooth route is up to +/// `MicLiveness.bluetoothTimeout`. Before the gate existed, `start()` returned in +/// microseconds and nothing could arrive during a press. +/// +/// The entry/finish choreography lives in the shared `Gate`; `GatedStopMic` is +/// the mirror image for the release path. `HotkeyRaceTests` keeps its own private +/// gated-start stub predating this one — left alone rather than folded in, since +/// swapping a stub under a passing race suite risks more than the duplication +/// costs. +actor GatedStartMic: MicCaptureProtocol { + private(set) var startCalls = 0 + private(set) var stopCalls = 0 + private(set) var cancelCaptureCalls = 0 + private let gate = Gate() + + func start() async throws { + startCalls += 1 + await gate.enter() + } + + func waitUntilStartEntered() async { await gate.waitUntilEntered() } + func allowStartToFinish() { gate.allowToFinish() } + + func stop() async throws -> Data { + stopCalls += 1 + return StubPCM.aboveMinimum + } + + /// Counts the call and delegates, so `stopCalls` keeps meaning "the mic was + /// stopped" however the teardown was reached — the same shape as + /// `StubMicCapture`. + func cancelCapture() async throws { + cancelCaptureCalls += 1 + _ = try await stop() + } +} From e96fc10bc363014f5879566d3a003c1b843ceb32 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 19:30:37 +0000 Subject: [PATCH 07/14] Let a cancel preempt the mic bring-up instead of queueing behind it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two pieces, closing the gap documented in the previous commit. `.connecting` now gets the same treatment `cancel()` already gives `.transcribing`/`.injecting`: it is in-flight work, so publish its task handle and cancel it. `press()` records `inFlightPress` the way `release()` records `pipelineTask`, and `cancel()` gains a `.connecting` branch. `MicLiveness.waitUntilLive` already returns early on task cancellation, so the wait unblocks at once; `MicCapture.start()` adds `!Task.isCancelled` to its post-wait guard, which tears the recorder down and throws CancellationError rather than installing it. That distinction matters — the timeout returns nil too, but nil means "fail open, proceed as if live". The cancel flag moves from actor state into a Mutex beside that handle, so `submit(.cancel)` can record and preempt without a turn. This is the part that fixes the app: its cancel door is `submit`, whose consumer is serial, so a submitted `.cancel` was not even *recorded* until the press it meant to cancel had finished — the Escape was invisible for the whole bring-up. `requestCancel()` is the single place both doors funnel through, so they cannot drift. Commands are still yielded and executed in order; only the preemption is new. `performPress` still consumes the flag before claiming `.recording`, for the narrow window where the cancel lands after the wait returned and there is nothing left to interrupt. The abort point is well chosen by accident of the existing ordering: everything with side effects — the target-app assignment, the AX context stream, the auto-release timer — happens after `mic.start()`, so a cancellation during the wait has nothing to unwind but the recorder, which `start()` already handles. DictationSession.swift hit 509 lines against the 400 file_length budget (nothing in the engine on main exceeds it), so `performPress` moves to `DictationSession+Press.swift`, mirroring `+Pipeline` on the release side, and the cancel-intent accessors move beside the commands they serve in `+Commands`. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 18 +- Sources/BlurtEngine/Audio/MicCapture.swift | 18 +- .../Pipeline/DictationSession+Commands.swift | 76 ++++++- .../Pipeline/DictationSession+Press.swift | 142 +++++++++++++ .../Pipeline/DictationSession.swift | 201 ++++-------------- .../DictationSessionTests.swift | 35 +++ 6 files changed, 320 insertions(+), 170 deletions(-) create mode 100644 Sources/BlurtEngine/Pipeline/DictationSession+Press.swift diff --git a/AGENTS.md b/AGENTS.md index 086382c6..f65e9e07 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -406,12 +406,18 @@ in both directions. So: slots are nil, so the warm-up paths can't infer "no capture in flight" from them (see `canPrepareWarmRecorder`) or they'd open a second recorder onto the live input. - **Known gap — a cancel during the bring-up isn't visible until it finishes.** `performPress` - consumes a recorded `cancelRequested` before claiming `.recording`, so the cancel is honored and - never leaves a phantom recording or chime behind — but it can't preempt the wait. Worse for the - app specifically: `submit(_:)`'s consumer is serial, so a submitted `.cancel` isn't even - _recorded_ until the press returns. Closing this needs either a preemptible `mic.start()` or a - command consumer that doesn't block on the press — both design changes, neither attempted yet. + **A cancel preempts the bring-up rather than queueing behind it**, which took two pieces. The + press publishes its task handle (`inFlightPress`) exactly as the pipeline publishes + `pipelineTask`, and `cancel()` treats `.connecting` like the other in-flight phases: cancel the + handle, claim `.cancelled`, return. `waitUntilLive` already honors task cancellation, so the wait + unblocks at once and `start()` throws `CancellationError` rather than installing the recorder. + The cancel flag then lives in a `Mutex` beside that handle rather than in actor state, so + **`submit(.cancel)` can record and preempt without waiting for a turn** — the app's cancel door is + `submit`, and its consumer is serial, so before this a submitted `.cancel` wasn't even _recorded_ + until the press it meant to cancel had finished. `requestCancel()` is the single place both doors + funnel through. Commands are still yielded and executed in order; only the preemption is new. + `performPress` still consumes the flag before claiming `.recording`, for the narrow window where + the cancel lands after the wait returned and there is nothing left to interrupt. - **The warm recorder is re-armed after every capture**, not just at launch. The cost above is paid at `prepareToRecord()`, i.e. per session, so warming only the first one hid it for one dictation diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 6976dc29..58d9c204 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -199,13 +199,21 @@ public actor MicCapture: MicCaptureProtocol { recorder.currentTime } - // A teardown that landed while the wait was suspended wins: tear the - // recorder down instead of installing it, so the caller's stop stays a real - // stop (see `stopGeneration`). - guard stopGeneration == generationBeforeWait else { + // Two ways the bring-up can be abandoned while suspended, both ending the + // same way — tear the recorder down instead of installing it, so nothing is + // left capturing and no temp file is orphaned: + // + // - A teardown landed (`stopGeneration` moved), so the caller's stop has to + // stay a real stop rather than returning an empty "clean" one. + // - The task was cancelled, which is how a cancel preempts the wait: + // `waitUntilLive` returns as soon as it sees it, so this is the difference + // between an Escape acting now and acting in `bluetoothTimeout`. It must be + // distinguished from the timeout, which returns nil too but means "fail + // open, proceed as if live". + guard stopGeneration == generationBeforeWait, !Task.isCancelled else { recorder.stop() Self.removeFile(at: recorder.url) - Self.logger.info("start aborted — teardown landed during the liveness wait") + Self.logger.info("start aborted — teardown or cancellation during the liveness wait") throw CancellationError() } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Commands.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Commands.swift index 8ec9bcab..23aba1b8 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Commands.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Commands.swift @@ -18,6 +18,16 @@ extension DictationSession { /// single serial consumer — see `init`). Hosts that can `await` may call the /// async methods directly instead; the two styles hit the same serial queue. public nonisolated func submit(_ command: Command) { + // A cancel acts *now*, not when its turn comes up. The consumer below is + // serial by design, so a `.cancel` submitted during a press waits for that + // press to finish before `cancel()` even runs — and since `MicCapture.start()` + // holds until the mic is delivering audio, on a Bluetooth route that is + // seconds of an Escape that appears to do nothing. Recording the intent and + // preempting the press here is what `cancel()` does for a live pipeline; + // this gives the submitted door the same reach. The command is still yielded + // in order, and still executes in order — `performCancel` is idempotent + // against a cancel already consumed. + if command == .cancel { requestCancel() } commandFeed.yield(command) } @@ -32,6 +42,58 @@ extension DictationSession { } } + /// Set synchronously by either entry point — `cancel()` or `submit(.cancel)` — + /// before taking a queue turn, so a cancel arriving while a queued release + /// hasn't yet claimed `.transcribing` deterministically wins: `performRelease` + /// consumes the request after its `mic.stop()`, before any pipeline is spawned. + /// (A release that already claimed `.transcribing` is handled by `cancel()`'s + /// synchronous path instead.) `performCancel` clears it whether or not it was + /// consumed early, and `performPress` consumes it before claiming `.recording` + /// so a cancel during the mic bring-up never yields a phantom recording. + /// + /// Internal, like `pipelineTask` and `autoReleaseTask`, because the moment the + /// request is recorded is otherwise unobservable: a test landing a cancel + /// against an in-flight press has to know it was recorded before it releases + /// that press, and the alternative — draining a fixed number of `Task.yield()`s + /// and hoping — is a budget that drains the calling task, not this actor. + nonisolated var cancelRequested: Bool { + get { cancelState.withLock { $0.requested } } + set { cancelState.withLock { $0.requested = newValue } } + } + + /// Handle to the press currently on the command queue, so a cancel can + /// **preempt** the mic bring-up rather than queue behind it — the same + /// treatment `pipelineTask` already gives `.transcribing`/`.injecting`. + /// `MicCapture.start()` can hold for seconds on a Bluetooth route, and + /// `MicLiveness.waitUntilLive` already returns early on task cancellation, so + /// cancelling this handle unblocks it at once. + /// + /// Beside `requested` under the same lock, and for the same reason: reachable + /// from `nonisolated` `submit(_:)`, whose whole job is to act without waiting + /// for a turn on an actor the in-flight press is holding. + nonisolated var inFlightPress: Task? { + get { cancelState.withLock { $0.press } } + set { cancelState.withLock { $0.press = newValue } } + } + + /// Records the cancel intent and preempts an in-flight press, synchronously + /// and from any isolation. The one place both doors funnel through, so + /// `submit(.cancel)` and `cancel()` can't drift apart. + nonisolated func requestCancel() { + let press = cancelState.withLock { state -> Task? in + state.requested = true + return state.press + } + press?.cancel() + } + + /// Drops the press handle if it is still `task`'s — a later press may already + /// have claimed the slot, and clearing that one would let its bring-up run + /// un-cancellable. + nonisolated func clearInFlightPress(_ task: Task) { + cancelState.withLock { if $0.press == task { $0.press = nil } } + } + public func cancel() async { // A cancel that lands once `.transcribing` is claimed — while the release // is still inside mic.stop(), or later with the transcribe→inject task in @@ -46,12 +108,24 @@ extension DictationSession { setPhase(.cancelled) return } + // `.connecting` gets the same treatment, for the same reason: it is in-flight + // work with a handle to cancel. The press is suspended inside + // `MicCapture.start()`'s liveness wait, which honors task cancellation and + // returns at once — so this unblocks a bring-up that could otherwise hold for + // `MicLiveness.bluetoothTimeout`. `start()` tears the recorder down and throws + // `CancellationError`, which `performPress` maps to `.cancelled` — already + // claimed here, so the pill answers the Escape immediately. + if phase == .connecting { + requestCancel() + setPhase(.cancelled) + return + } // Record the intent before taking a queue turn: a release queued ahead of // our turn consumes it the moment its mic.stop() returns (no pipeline is // ever spawned), and a press ahead in the queue is followed by our own // turn, which ends the freshly started recording. Either way the cancel is // honored in arrival order, never dropped. - cancelRequested = true + requestCancel() await enqueue { await self.performCancel() } } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift new file mode 100644 index 00000000..1d0c952f --- /dev/null +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift @@ -0,0 +1,142 @@ +import Dispatch + +// The press half of the pipeline — everything between the key going down and +// `.recording` being claimed, including the mic bring-up that `.connecting` +// covers. Split from `DictationSession.swift` to stay within the lint +// file-length budget, mirroring `+Pipeline` (the release half). Members it +// reaches are internal, not private: file-scoped access can't cross the split. +// `Dispatch`, not `Foundation`: the only thing here from outside the module is +// `contextQueue.async` (see `performPress` for why that read is off-pool). +extension DictationSession { + func performPress() async { + guard phase.isTerminal else { return } + // Refuse the press before any capture begins when the host reports a + // blocker (e.g. no API key saved): recording an utterance that can only + // fail at transcribe time would discard the user's words after the fact. + if let blocker = readinessCheck() { + setPhase(.failed(blocker)) + return + } + // Times the startup path — the concurrent focus capture + mic.start (and the + // detached connection warm-up kicked off below) — up to the moment recording + // actually begins. Ended on both the success and failure exits (mic.start is + // the only throwing call, and it precedes `.recording`, so the two ends are + // mutually exclusive). + let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) + // Claim `.connecting` before `mic.start()`: its liveness gate holds until + // the input route actually delivers frames, which on a Bluetooth route is + // ~1–2 s. The press must be visibly acknowledged in that window without + // cueing the user to speak — the pill shows a warming-up state, and the + // start chime rides the connecting→recording edge (`RecordingCueGate`), so + // it fires only once audio is genuinely flowing. `.recording` therefore + // keeps meaning exactly what it says. + setPhase(.connecting) + do { + // Pre-open the dictation connection while the user speaks, so the first dictation after an idle + // gap doesn't pay DNS+TCP+TLS on the transcribe hot path (~170 ms cold, measured). Detached + // + fire-and-forget: it must never delay recording, and a failure is harmless (the request + // just pays setup as before); warming every press is cheap since a hot pool just reuses it. + let transcriber = transcriber + Task.detached { await transcriber.warmUp() } + // Capture the frontmost app (paste target) concurrently with mic startup — + // a cheap in-process AppKit read on the main actor. The phase still flips + // to .recording only after mic.start succeeds, so the UI never lies about + // whether audio is being captured. Lifted out of the actor first (like + // `transcriber` above) so the child task calls a Sendable closure rather + // than reading isolated state. + let captureFrontmost = seams.captureFrontmost + async let frontmost = captureFrontmost() + try await mic.start() + let captured = await frontmost + await injector.setTargetApp(captured.flatMap { FocusCapture.runningApp(for: $0) }) + // Key terms are read synchronously at press (cheap UserDefaults read), so + // each dictation observably re-reads Settings edits at press time. + let keyTerms = keyTermsProvider() + // Session history, read on the actor for the same reason: the capture below + // runs off-actor, so what it carries has to be a value taken now. + let recentTranscripts = recentDictations.transcriptsOldestFirst + // Kick off the AX field-context read now, while the target field still + // holds focus, but don't await it here: it's cross-process IPC into the + // frontmost app (detached — off the main actor, where it froze the + // overlay, and off this actor, where it would wedge release()/cancel()). + // runTranscribeInject consumes the result right before transcription, + // bounded by `contextWaitBudget` — so a slow AX target delays the + // transcript by at most the budget, never the recording indicator. + let (stream, contextFeed) = AsyncStream.makeStream( + of: TranscriptionContext?.self, bufferingPolicy: .bufferingNewest(1)) + contextStream = stream + // A Dispatch queue, not `Task.detached`: `captureFieldContext` is documented + // as making ~6 synchronous cross-process AX round trips, each bounded only by + // the 1 s messaging timeout, so against a beachballing frontmost app one + // press can *block* a thread for seconds. The Swift cooperative pool is sized + // to the core count and does not overcommit, so a few press/cancel cycles + // against a hung app could park every cooperative thread and stall the whole + // non-main runtime — including this actor. Dispatch overcommits, so a blocked + // capture costs a thread instead of the pool. Same reasoning as + // `DictationLog`'s serial queue. Concurrent so a hung capture can't delay the + // next press's. The body is fully synchronous and captures only Sendable + // values, so it needs no task context. + let captureFieldContext = seams.captureFieldContext + Self.contextQueue.async { + let field = captureFieldContext() + let context = TranscriptionContext( + appName: captured?.processName, + windowTitle: field.windowTitle, + fieldLabel: field.fieldLabel, + priorText: field.priorText, + selectedText: field.selectedText, + recentTranscripts: recentTranscripts, + keyTerms: keyTerms, + targetIsSecure: field.isSecure) + contextFeed.yield(context.isEmpty ? nil : context) + contextFeed.finish() + } + // A cancel that arrived during the bring-up, on the path where `start()` + // still returned normally — the cancel landed in the window between the + // liveness wait finishing and `.recording` being claimed, so there was + // nothing left to interrupt. (When it lands *during* the wait, `start()` + // throws `CancellationError` instead and the catch below owns it.) + // + // Either way the mic is live by now, so tear it down rather than claiming + // `.recording` and chiming "speak now" for a capture the user has already + // abandoned. `Task.isCancelled` is checked alongside the flag because the + // preempting door cancels this task without setting anything else. + if cancelRequested || Task.isCancelled { + try? await mic.cancelCapture() + // `consumeCancelRequest` claims `.cancelled` when the flag route was + // used; the task-cancellation route already claimed it in `cancel()`, so + // only fall back when nothing has. + if !consumeCancelRequest() { setPhase(.cancelled) } + Self.signposter.endInterval(Self.pressSignpostName, pressInterval) + return + } + setPhase(.recording) + Self.signposter.endInterval(Self.pressSignpostName, pressInterval) + let timeout = maxRecordingSeconds + let clock = clock + autoReleaseTask = Task { [weak self] in + try? await clock.sleep(for: .seconds(timeout)) + guard let self, !Task.isCancelled else { return } + // Enqueues like a manual key-up. If a real release already ran, the + // queued performRelease sees a non-.recording phase and drops out. + await self.release() + } + } catch { + Self.signposter.endInterval(Self.pressSignpostName, pressInterval) + // `MicCapture.start()` throws `CancellationError` when a teardown landed + // during its liveness wait — an unqueued `cancelCapture()`, which its own + // doc anticipates for hosts that don't drive the mic through this session. + // That's the user's cancel arriving by another door, not a fault: reporting + // it as `.audioCaptureFailed` would flash the pill red *and* write a + // developer-mode error-log entry for something nothing went wrong in. Same + // rule `transcribe` and `inject` already follow. `.cancelled` rather than a + // bare return, because `.connecting` is non-terminal — leaving it would + // strand the trigger's gate and swallow the next press. + if error is CancellationError { + setPhase(.cancelled) + return + } + setPhase(.failed(.audioCaptureFailed(underlying: error))) + } + } +} diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index c4209011..316269b5 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -1,10 +1,11 @@ import Foundation +import Synchronization import os public actor DictationSession { /// Off-pool home for the press-time AX field read — see its use in /// `performPress` for why blocking IPC must not run on the cooperative pool. - private static let contextQueue = DispatchQueue( + static let contextQueue = DispatchQueue( label: "\(BlurtIdentity.subsystem).FieldContext", qos: .userInitiated, attributes: .concurrent) @@ -17,12 +18,14 @@ public actor DictationSession { /// the failure from the log. public internal(set) var phase: PipelinePhase = .idle - // Split for the lint file-length budget: `phaseStream()`/`setPhase`/os_signpost - // live in `+Observation`; `submit(_:)` and both cancel commands live in - // `+Commands`; the post-release transcribe→inject pipeline lives in - // `+Pipeline`; the non-protocol collaborators (focus capture, developer-mode - // log) live in `+Seams`. Members those files reach are internal, not private - // (file-scoped access can't cross the split) — including `phase`'s setter. + // Split for the lint file-length budget: `performPress` — the whole press half, + // including the mic bring-up — lives in `+Press`, mirroring the post-release + // transcribe→inject pipeline in `+Pipeline`. `submit(_:)`, both cancel commands + // and the cancel-intent accessors over `cancelState` live in `+Commands`; + // `phaseStream()`/`setPhase`/os_signpost live in `+Observation`; the + // non-protocol collaborators (focus capture, developer-mode log) live in + // `+Seams`. Members those files reach are internal, not private (file-scoped + // access can't cross the split) — including `phase`'s setter. /// Live feeds of phase changes. Each `phaseStream()` call yields the current /// phase plus every subsequent transition, so the production renderer and @@ -36,7 +39,7 @@ public actor DictationSession { /// one at a time by the task spawned in `init`. nonisolated let commandFeed: AsyncStream.Continuation - private let mic: MicCaptureProtocol + let mic: MicCaptureProtocol let transcriber: TranscriberProtocol let injector: InjectorProtocol /// Supplies the user's key terms (domain vocabulary) at press time, so each @@ -44,12 +47,12 @@ public actor DictationSession { /// (`KeytermsBoost`), not as part of the conversation context. A closure, rather /// than a stored list, so edits in Settings take effect on the next dictation /// without rebuilding the session. Defaults to reading `KeyTermsStore`. - private let keyTermsProvider: @Sendable () -> [String] + let keyTermsProvider: @Sendable () -> [String] /// Auto-releases the hotkey after this long so a held key can't run forever. /// Defaults to just under the dictation API's audio cap (see /// `SyncSTTLimits`) — recording past it would only produce audio the /// endpoint rejects, so we stop early and transcribe what we have. - private let maxRecordingSeconds: Double + let maxRecordingSeconds: Double /// Clock the auto-release timer and the context-wait budget (`+Pipeline`) /// sleep on; injectable so tests advance it. let clock: any Clock @@ -60,7 +63,7 @@ public actor DictationSession { /// key-presence check so a missing API key fails at press time, not after the /// user has spoken a whole utterance. Defaults to always-ready (no Keychain /// read), so tests and keyless hosts are unaffected unless they opt in. - private let readinessCheck: @Sendable () -> BlurtError? + let readinessCheck: @Sendable () -> BlurtError? /// Fired once with the final transcript as soon as it's produced — before /// injection, so pasted, copied, and failed-to-paste dictations all count. The /// second argument is `recentDictations` as it stands, pushed from its one owner @@ -103,19 +106,18 @@ public actor DictationSession { /// a time in arrival order — none observes another suspended mid-`mic` call. private var commandQueue: Task? - /// Set synchronously by `cancel()` before it takes its queue turn, so a - /// cancel arriving while a queued release hasn't yet claimed `.transcribing` - /// deterministically wins: `performRelease` consumes the request after its - /// `mic.stop()`, before any pipeline is spawned. (A release that already - /// claimed `.transcribing` is handled by `cancel()`'s synchronous path - /// instead.) `performCancel` clears it whether or not it was consumed early. - /// - /// Internal, like `pipelineTask` and `autoReleaseTask`, because the moment the - /// request is recorded is otherwise unobservable: a test landing a cancel - /// against an in-flight press has to know it was recorded before it releases - /// that press, and the alternative — draining a fixed number of `Task.yield()`s - /// and hoping — is a budget that drains the calling task, not this actor. - var cancelRequested = false + /// Backing store for `cancelRequested` and `inFlightPress`. A `Mutex` rather + /// than actor state because **both doors into a cancel must record it + /// synchronously**, and one of them is `nonisolated`: `submit(.cancel)` can't + /// take an actor turn, and waiting for one is exactly the bug — the command + /// consumer is serial, so a submitted cancel sits unread in the feed until the + /// press it means to cancel has finished. + let cancelState = Mutex(CancelState()) + + struct CancelState { + var requested = false + var press: Task? + } // Internal, like `pipelineTask`, so a test can witness the cancel teardown // *directly* — nil means disarmed. Asserting it through the timer's effects @@ -204,154 +206,37 @@ public actor DictationSession { /// synchronous read-then-write of `commandQueue` makes the chain order match /// the order the public methods executed their first actor turn. func enqueue(_ op: @escaping @Sendable () async -> Void) async { + await chain(op).value + } + + /// Appends `op` to the serial command queue and hands back its handle + /// *without* waiting — the half of `enqueue` a caller needs when something + /// else must be able to reach the task while it runs. The synchronous + /// read-then-write of `commandQueue` is what makes the chain order match the + /// order the public methods executed their first actor turn. + private func chain(_ op: @escaping @Sendable () async -> Void) -> Task { let previous = commandQueue let task = Task { await previous?.value await op() } commandQueue = task - await task.value + return task } public func press() async { - await enqueue { await self.performPress() } + // Published before awaiting so a cancel can preempt the mic bring-up — see + // `inFlightPress`. Cleared on the way out, but only if it's still ours. + let task = chain { await self.performPress() } + inFlightPress = task + await task.value + clearInFlightPress(task) } public func release() async { await enqueue { await self.performRelease() } } - private func performPress() async { - guard phase.isTerminal else { return } - // Refuse the press before any capture begins when the host reports a - // blocker (e.g. no API key saved): recording an utterance that can only - // fail at transcribe time would discard the user's words after the fact. - if let blocker = readinessCheck() { - setPhase(.failed(blocker)) - return - } - // Times the startup path — the concurrent focus capture + mic.start (and the - // detached connection warm-up kicked off below) — up to the moment recording - // actually begins. Ended on both the success and failure exits (mic.start is - // the only throwing call, and it precedes `.recording`, so the two ends are - // mutually exclusive). - let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) - // Claim `.connecting` before `mic.start()`: its liveness gate holds until - // the input route actually delivers frames, which on a Bluetooth route is - // ~1–2 s. The press must be visibly acknowledged in that window without - // cueing the user to speak — the pill shows a warming-up state, and the - // start chime rides the connecting→recording edge (`RecordingCueGate`), so - // it fires only once audio is genuinely flowing. `.recording` therefore - // keeps meaning exactly what it says. - setPhase(.connecting) - do { - // Pre-open the dictation connection while the user speaks, so the first dictation after an idle - // gap doesn't pay DNS+TCP+TLS on the transcribe hot path (~170 ms cold, measured). Detached - // + fire-and-forget: it must never delay recording, and a failure is harmless (the request - // just pays setup as before); warming every press is cheap since a hot pool just reuses it. - let transcriber = transcriber - Task.detached { await transcriber.warmUp() } - // Capture the frontmost app (paste target) concurrently with mic startup — - // a cheap in-process AppKit read on the main actor. The phase still flips - // to .recording only after mic.start succeeds, so the UI never lies about - // whether audio is being captured. Lifted out of the actor first (like - // `transcriber` above) so the child task calls a Sendable closure rather - // than reading isolated state. - let captureFrontmost = seams.captureFrontmost - async let frontmost = captureFrontmost() - try await mic.start() - let captured = await frontmost - await injector.setTargetApp(captured.flatMap { FocusCapture.runningApp(for: $0) }) - // Key terms are read synchronously at press (cheap UserDefaults read), so - // each dictation observably re-reads Settings edits at press time. - let keyTerms = keyTermsProvider() - // Session history, read on the actor for the same reason: the capture below - // runs off-actor, so what it carries has to be a value taken now. - let recentTranscripts = recentDictations.transcriptsOldestFirst - // Kick off the AX field-context read now, while the target field still - // holds focus, but don't await it here: it's cross-process IPC into the - // frontmost app (detached — off the main actor, where it froze the - // overlay, and off this actor, where it would wedge release()/cancel()). - // runTranscribeInject consumes the result right before transcription, - // bounded by `contextWaitBudget` — so a slow AX target delays the - // transcript by at most the budget, never the recording indicator. - let (stream, contextFeed) = AsyncStream.makeStream( - of: TranscriptionContext?.self, bufferingPolicy: .bufferingNewest(1)) - contextStream = stream - // A Dispatch queue, not `Task.detached`: `captureFieldContext` is documented - // as making ~6 synchronous cross-process AX round trips, each bounded only by - // the 1 s messaging timeout, so against a beachballing frontmost app one - // press can *block* a thread for seconds. The Swift cooperative pool is sized - // to the core count and does not overcommit, so a few press/cancel cycles - // against a hung app could park every cooperative thread and stall the whole - // non-main runtime — including this actor. Dispatch overcommits, so a blocked - // capture costs a thread instead of the pool. Same reasoning as - // `DictationLog`'s serial queue. Concurrent so a hung capture can't delay the - // next press's. The body is fully synchronous and captures only Sendable - // values, so it needs no task context. - let captureFieldContext = seams.captureFieldContext - Self.contextQueue.async { - let field = captureFieldContext() - let context = TranscriptionContext( - appName: captured?.processName, - windowTitle: field.windowTitle, - fieldLabel: field.fieldLabel, - priorText: field.priorText, - selectedText: field.selectedText, - recentTranscripts: recentTranscripts, - keyTerms: keyTerms, - targetIsSecure: field.isSecure) - contextFeed.yield(context.isEmpty ? nil : context) - contextFeed.finish() - } - // A cancel that arrived during the bring-up. It couldn't act at the time — - // `cancel()` has no synchronous path for `.connecting`, so it recorded the - // intent and queued behind this press — and the mic is live by now, so - // honor it here rather than claiming `.recording` and chiming "speak now" - // for a capture the user has already abandoned. Without this the press - // completed normally and the queued `performCancel` only tore it down - // afterwards, so a cancel produced a full recording start (chime included) - // that was then immediately cancelled. - // - // Not a fix for the *latency*: the cancel is still not visible until the - // bring-up finishes, which on a Bluetooth route is up to - // `MicLiveness.bluetoothTimeout`. Interrupting the wait would mean making - // `mic.start()` cancellable from here, which is a larger change. - if cancelRequested { - try? await mic.cancelCapture() - _ = consumeCancelRequest() - Self.signposter.endInterval(Self.pressSignpostName, pressInterval) - return - } - setPhase(.recording) - Self.signposter.endInterval(Self.pressSignpostName, pressInterval) - let timeout = maxRecordingSeconds - let clock = clock - autoReleaseTask = Task { [weak self] in - try? await clock.sleep(for: .seconds(timeout)) - guard let self, !Task.isCancelled else { return } - // Enqueues like a manual key-up. If a real release already ran, the - // queued performRelease sees a non-.recording phase and drops out. - await self.release() - } - } catch { - Self.signposter.endInterval(Self.pressSignpostName, pressInterval) - // `MicCapture.start()` throws `CancellationError` when a teardown landed - // during its liveness wait — an unqueued `cancelCapture()`, which its own - // doc anticipates for hosts that don't drive the mic through this session. - // That's the user's cancel arriving by another door, not a fault: reporting - // it as `.audioCaptureFailed` would flash the pill red *and* write a - // developer-mode error-log entry for something nothing went wrong in. Same - // rule `transcribe` and `inject` already follow. `.cancelled` rather than a - // bare return, because `.connecting` is non-terminal — leaving it would - // strand the trigger's gate and swallow the next press. - if error is CancellationError { - setPhase(.cancelled) - return - } - setPhase(.failed(.audioCaptureFailed(underlying: error))) - } - } private func performRelease() async { guard phase == .recording else { return } @@ -397,7 +282,7 @@ public actor DictationSession { /// Consumes a cancel requested while this release held the queue, claiming the /// phase for the user's cancel. Returns whether it fired. - private func consumeCancelRequest() -> Bool { + func consumeCancelRequest() -> Bool { guard cancelRequested else { return false } cancelRequested = false setPhase(.cancelled) diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index 7f9e9c00..10f1159e 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -349,4 +349,39 @@ struct DictationSessionBringUpTests { // through the discarding teardown, not `stop()`. #expect(await mic.cancelCaptureCalls == 1) } + + @Test("a submitted cancel preempts the bring-up instead of queueing behind it") + func submittedCancelPreemptsBringUp() async throws { + // The app's cancel door is `submit`, and its consumer is serial — so a + // `.cancel` submitted during a press cannot reach `cancel()` until that press + // returns. With `MicCapture.start()` holding until the mic is live, that made + // Escape invisible for up to `MicLiveness.bluetoothTimeout`. `submit` now + // records the intent and cancels the in-flight press itself, without waiting + // for a turn on an actor the press is holding. + let mic = GatedStartMic() + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), + injector: StubInjector(), seams: .offline) + + let stream = await session.phaseStream() + session.submit(.press) + await mic.waitUntilStartEntered() // the consumer is now blocked inside this press + + session.submit(.cancel) + // Both effects have to be observable *before* the press is released, which is + // the whole point — neither needs the consumer to get a turn. + #expect(session.cancelRequested) + #expect(session.inFlightPress?.isCancelled == true) + + await mic.allowStartToFinish() + + var seen: [PipelinePhase] = [] + for await phase in stream { + seen.append(phase) + if phase.isTerminal, phase != .idle { break } + } + #expect(seen.last == .cancelled) + #expect(!seen.contains(.recording)) + #expect(await mic.cancelCaptureCalls == 1) + } } From 7382f09ace05a3df9314ab863366dad9269029de Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 19:42:32 +0000 Subject: [PATCH 08/14] Cleanup pass, plus a leaked cancel request the pass surfaced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bug first, since it isn't cleanup: `cancel()`'s `.connecting` branch claims the phase and returns *without* enqueueing `performCancel`, and the press it preempted exits through the CancellationError catch, which only set the phase. So `cancelRequested` stayed true, and the NEXT press read it after a perfectly good `mic.start()` and cancelled itself — one dead dictation per cancelled bring-up. The catch now consumes the request the same way the other exit does. `GatedStartMic` grew an opt-in `throwsIfCancelled` so a test can drive the throwing route, which is the only one that reaches it. Cleanup applied: - `ConnectingLabel`/`TranscribingLabel` were the same view with one string changed, each holding its own copy of the breath constants that a comment asked to stay equal. One `BreathingStatusLine(text:animated:)`. - `rewarm()` was byte-identical to `warmUp()`; `scheduleRewarm` calls `warmUp()` and the guard lives once. - `stop()` and `cancelCapture()` shared a five-line teardown preamble including the `stopGeneration` bump that `start()` depends on across its wait — exactly the line a third exit would forget. Now `detachActiveRecorder()`. - The tail-linger policy moved from `MicCapture` (excluded from the coverage gate) to `AudioTransport`, beside `MicLiveness`'s wait cap, so both transport-conditional decisions are unit-tested rather than one of two. `MicCapture` stores the transport type instead of a Bool. - `AudioRouteMonitor` was restating the CoreAudio address literal and the system-object expression that `AudioRoute` already owns; `AudioRoute` now exposes `globalAddress(_:)`/`systemObject` and reads both its properties through one generic helper. - `HotkeyRaceTests`' private gated-start stub was a duplicate of the shared `GatedStartMic`; deleted, and the hedge in the shared stub's doc with it. - `enqueue`'s doc restated the mechanism that moved into `chain`, and the extraction left a double blank line (`maximumBlankLines: 1`). - The coverage exclusion `Audio/AudioRoute` silently covered any future `AudioRoute*.swift`, including a pure one that should count. Named the two files. Skipped, deliberately — see the reply for reasoning: hoisting the context capture above `mic.start()` (real win, but a behavior change to the press path), deferring the cue re-prime out of the press window, geometric backoff in the liveness poll, lazily constructing AudioRouteMonitor off the launch path, and the captureState enum / WarmRecorder struct refactors. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 4 +- .../Blurt/Overlay/OverlayPillContent.swift | 53 +++++--------- App/Blurt/Blurt/Overlay/OverlayView.swift | 4 +- Sources/BlurtEngine/Audio/AudioRoute.swift | 52 +++++++++----- .../BlurtEngine/Audio/AudioRouteMonitor.swift | 37 ++-------- .../BlurtEngine/Audio/AudioTransport.swift | 22 ++++++ .../BlurtEngine/Audio/MicCapture+Warm.swift | 18 +++-- Sources/BlurtEngine/Audio/MicCapture.swift | 69 +++++++++---------- .../Pipeline/DictationSession+Press.swift | 9 ++- .../Pipeline/DictationSession.swift | 6 +- .../DictationSessionTests.swift | 29 ++++++++ Tests/BlurtEngineTests/HotkeyRaceTests.swift | 31 ++------- Tests/BlurtEngineTests/MicLivenessTests.swift | 14 ++++ .../Stubs/GatedStartMic.swift | 14 ++-- scripts/check.sh | 4 +- 15 files changed, 193 insertions(+), 173 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5c3c9b88..d3db1636 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -457,7 +457,9 @@ The re-warm and the liveness gate are complements, not alternatives: the re-warm the profile switch is paid (a warm recorder has already held the route open), and the gate is what keeps the app honest on the presses that pay it anyway. -`stop()` also waits out `bluetoothTailLinger` (220 ms) before ending the recording **when the +`stop()` also waits out `AudioTransport.tailLinger(forTransportType:)` (220 ms on Bluetooth, `.zero` +otherwise — the policy sits beside `MicLiveness`'s wait cap so both are unit-tested) before ending +the recording **when the session's input is Bluetooth**, so speech still travelling over the link lands in the file instead of being truncated — the missing last word. It runs after `.transcribing` is claimed, so it delays the transcript, never the "it heard me" cue. Cancels take `cancelCapture()` instead, which skips both the diff --git a/App/Blurt/Blurt/Overlay/OverlayPillContent.swift b/App/Blurt/Blurt/Overlay/OverlayPillContent.swift index 8c088fcd..9c848c76 100644 --- a/App/Blurt/Blurt/Overlay/OverlayPillContent.swift +++ b/App/Blurt/Blurt/Overlay/OverlayPillContent.swift @@ -56,13 +56,21 @@ extension View { } } -/// The "Transcribing…" status line with a slow breathing pulse — the processing -/// counterpart of the recording bars' idle shimmer, so the pill keeps visibly -/// working while the app waits on the dictation API and pastes the result. Driven by -/// the same continuous-clock `TimelineView` pattern as `WaveformBars` (never a -/// one-shot state toggle). Under Reduce Motion the label holds steady at full -/// opacity — exactly the pre-animation rendering. -struct TranscribingLabel: View { +/// A status line that breathes — the pill's "working, hold on" treatment, used +/// for both waits it has: "Connecting…" while `MicCapture`'s liveness gate waits +/// for the input route to deliver frames, and "Transcribing…" while the app +/// waits on the dictation API and pastes the result. +/// +/// One view rather than two near-identical ones, so the heartbeat is shared by +/// construction instead of by a comment asking two copies of the constants to +/// stay equal. Driven by the same continuous-clock `TimelineView` pattern as +/// `WaveformBars` (never a one-shot state toggle); under Reduce Motion it holds +/// steady at full opacity, exactly the pre-animation rendering. +/// +/// `StatusLineText` is shared with the "Pasted"/"Copied" notices, so every +/// hand-off between these states reads as one continuous status line. +struct BreathingStatusLine: View { + let text: String /// Whether to run the breathing motion (off under Reduce Motion). let animated: Bool @@ -75,34 +83,7 @@ struct TranscribingLabel: View { private let minOpacity: Double = 0.55 var body: some View { - // StatusLineText is shared with the "Pasted" notice (OverlayView's `.pasted` - // case) so the processing → pasted hand-off reads as one status line. - StatusLineText("Transcribing…") - .pulsingOpacity(period: breathPeriod, minOpacity: minOpacity, animated: animated) - } -} - -/// The "Connecting…" status line shown while `MicCapture`'s liveness gate waits -/// for the input route to deliver frames. Breathes on the same curve as -/// `TranscribingLabel` (the pill's other "working, hold on" state) so the two -/// waits read alike — this one can hold for a second or two on a Bluetooth -/// route, and a frozen line would read as a hung app. -/// -/// Deliberately *not* the `● REC` tag or the meter: those are the "speak now" -/// cues, and audio spoken during the bring-up is unrecoverable — the OS receives -/// nothing while the profile switch is in flight. The pill must not invite -/// speech it cannot capture. -struct ConnectingLabel: View { - /// Whether to run the breathing motion (off under Reduce Motion). - let animated: Bool - - // Matched to `TranscribingLabel`: the two are the same kind of wait, so they - // share one heartbeat rather than each picking a rate. - private let breathPeriod: Double = 1.8 - private let minOpacity: Double = 0.55 - - var body: some View { - StatusLineText("Connecting…") + StatusLineText(text) .pulsingOpacity(period: breathPeriod, minOpacity: minOpacity, animated: animated) } } @@ -119,7 +100,7 @@ struct RecordingTag: View { // One pulse every ~1.2 s, dimming to 40% and back: the universal "recording, // right now" heartbeat. Since magenta stands in for the conventional red dot, // the pulse — not the hue — carries the live-capture cue. Driven by the same - // continuous-clock TimelineView as the waveform and TranscribingLabel (never a + // continuous-clock TimelineView as the waveform and BreathingStatusLine (never a // one-shot repeatForever toggle). private let pulsePeriod: Double = 1.2 private let minOpacity: Double = 0.4 diff --git a/App/Blurt/Blurt/Overlay/OverlayView.swift b/App/Blurt/Blurt/Overlay/OverlayView.swift index fc5ee1c9..ec5cf609 100644 --- a/App/Blurt/Blurt/Overlay/OverlayView.swift +++ b/App/Blurt/Blurt/Overlay/OverlayView.swift @@ -96,7 +96,7 @@ struct OverlayView: View { // pill's own 0.08 s fade-in, so it blends into the appearance rather than // flashing; on a Bluetooth route it holds for as long as the link takes, // which is the whole point. - ConnectingLabel(animated: !reduceMotion) + BreathingStatusLine(text: "Connecting…", animated: !reduceMotion) .transition(.opacity) case .recording: // "● REC" tag beside the live waveform, mirroring the site demo's recording @@ -112,7 +112,7 @@ struct OverlayView: View { // Transcribing); cyan echoes the demo's --ice. Cross-fades like the bars. // The label breathes (slow opacity pulse) so the wait for the dictation API + // paste reads as active work rather than a frozen pill. - TranscribingLabel(animated: !reduceMotion) + BreathingStatusLine(text: "Transcribing…", animated: !reduceMotion) .transition(.opacity) case .error(let message): // "Try again" tells the user what to do; the full failure reason is too diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift index fff0b8ed..e5aa5c17 100644 --- a/Sources/BlurtEngine/Audio/AudioRoute.swift +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -67,23 +67,47 @@ enum AudioRoute { defaultDeviceID(for: kAudioHardwarePropertyDefaultOutputDevice) } + // MARK: - CoreAudio addressing + + /// The system-wide audio object, which owns the default-device properties. + static var systemObject: AudioObjectID { AudioObjectID(kAudioObjectSystemObject) } + + /// A global-scope address for `selector`. Returned fresh per call rather than + /// stored, because every caller needs its own copy to pass `inout` to + /// CoreAudio. Shared with `AudioRouteMonitor`, which addresses the same object + /// graph and would otherwise restate this three-field literal. + static func globalAddress(_ selector: AudioObjectPropertySelector) -> AudioObjectPropertyAddress { + AudioObjectPropertyAddress( + mSelector: selector, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain) + } + // MARK: - CoreAudio reads + /// One `AudioObjectGetPropertyData` read of a fixed-size value, or nil when + /// CoreAudio refused. `initial` supplies both the type and the zero value, so + /// each caller states the property it wants and nothing else. + private static func read( + _ selector: AudioObjectPropertySelector, from object: AudioObjectID, initial: T + ) -> T? { + var address = globalAddress(selector) + var value = initial + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData(object, &address, 0, nil, &size, &value) + guard status == noErr else { return nil } + return value + } + /// The device the system object reports for `selector` (a default-device /// property). Nil covers both a failed read and the "no such device" sentinel, /// which callers treat identically. private static func defaultDeviceID(for selector: AudioObjectPropertySelector) -> AudioDeviceID? { - var address = AudioObjectPropertyAddress( - mSelector: selector, - mScope: kAudioObjectPropertyScopeGlobal, - mElement: kAudioObjectPropertyElementMain) - var deviceID = AudioDeviceID(0) - var size = UInt32(MemoryLayout.size) - let system = AudioObjectID(kAudioObjectSystemObject) - let status = AudioObjectGetPropertyData(system, &address, 0, nil, &size, &deviceID) // 0 is `kAudioObjectUnknown` — "there is no such device" — spelled as the // literal so this doesn't depend on how the constant imports. - guard status == noErr, deviceID != 0 else { return nil } + guard let deviceID = read(selector, from: systemObject, initial: AudioDeviceID(0)), + deviceID != 0 + else { return nil } return deviceID } @@ -91,14 +115,6 @@ enum AudioRoute { /// `AudioTransport` and `MicLiveness` both treat nil as "not Bluetooth", which /// is the conservative direction for each. private static func transportType(of deviceID: AudioDeviceID) -> UInt32? { - var address = AudioObjectPropertyAddress( - mSelector: kAudioDevicePropertyTransportType, - mScope: kAudioObjectPropertyScopeGlobal, - mElement: kAudioObjectPropertyElementMain) - var transport = UInt32(0) - var size = UInt32(MemoryLayout.size) - let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &transport) - guard status == noErr else { return nil } - return transport + read(kAudioDevicePropertyTransportType, from: deviceID, initial: UInt32(0)) } } diff --git a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift index 764d4121..4d17d1d9 100644 --- a/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift +++ b/Sources/BlurtEngine/Audio/AudioRouteMonitor.swift @@ -90,12 +90,12 @@ public final class AudioRouteMonitor: @unchecked Sendable { deinit { continuation.finish() if let systemListener { - var address = Self.defaultOutputDeviceAddress + var address = AudioRoute.globalAddress(kAudioHardwarePropertyDefaultOutputDevice) _ = AudioObjectRemovePropertyListenerBlock( - Self.systemObject, &address, queue, systemListener) + AudioRoute.systemObject, &address, queue, systemListener) } if let deviceListener { - var address = Self.sampleRateAddress + var address = AudioRoute.globalAddress(kAudioDevicePropertyNominalSampleRate) _ = AudioObjectRemovePropertyListenerBlock( deviceListener.id, &address, queue, deviceListener.block) } @@ -106,7 +106,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { /// Watches for the default output device itself changing. Registered once and /// never re-targeted — the system object is always there. private func installDefaultDeviceListener() { - var address = Self.defaultOutputDeviceAddress + var address = AudioRoute.globalAddress(kAudioHardwarePropertyDefaultOutputDevice) // `[weak self]`, so CoreAudio's strong hold on the block doesn't keep the // monitor alive forever — and so a callback landing during teardown finds // nil rather than a half-destroyed object. @@ -117,7 +117,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { self.retargetFormatListener() self.continuation.yield() } - let status = AudioObjectAddPropertyListenerBlock(Self.systemObject, &address, queue, block) + let status = AudioObjectAddPropertyListenerBlock(AudioRoute.systemObject, &address, queue, block) guard status == noErr else { Self.logger.error("default-output listener failed: \(status)") return @@ -133,7 +133,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { let device = AudioRoute.defaultOutputDeviceID() if let existing = deviceListener { guard existing.id != device else { return } - var address = Self.sampleRateAddress + var address = AudioRoute.globalAddress(kAudioDevicePropertyNominalSampleRate) _ = AudioObjectRemovePropertyListenerBlock(existing.id, &address, queue, existing.block) deviceListener = nil } @@ -141,7 +141,7 @@ public final class AudioRouteMonitor: @unchecked Sendable { let block: AudioObjectPropertyListenerBlock = { [weak self] _, _ in self?.continuation.yield() } - var address = Self.sampleRateAddress + var address = AudioRoute.globalAddress(kAudioDevicePropertyNominalSampleRate) let status = AudioObjectAddPropertyListenerBlock(device, &address, queue, block) guard status == noErr else { Self.logger.error("output-format listener failed: \(status)") @@ -149,27 +149,4 @@ public final class AudioRouteMonitor: @unchecked Sendable { } deviceListener = (id: device, block: block) } - - // MARK: - CoreAudio addressing - - /// The system-wide audio object, which owns the default-device properties. - private static var systemObject: AudioObjectID { AudioObjectID(kAudioObjectSystemObject) } - - // Computed, not stored: each caller needs its own mutable copy to pass `inout` - // to CoreAudio anyway, so a shared constant would only add a global whose - // `Sendable`-ness depends on how the C struct imports. - - private static var defaultOutputDeviceAddress: AudioObjectPropertyAddress { - AudioObjectPropertyAddress( - mSelector: kAudioHardwarePropertyDefaultOutputDevice, - mScope: kAudioObjectPropertyScopeGlobal, - mElement: kAudioObjectPropertyElementMain) - } - - private static var sampleRateAddress: AudioObjectPropertyAddress { - AudioObjectPropertyAddress( - mSelector: kAudioDevicePropertyNominalSampleRate, - mScope: kAudioObjectPropertyScopeGlobal, - mElement: kAudioObjectPropertyElementMain) - } } diff --git a/Sources/BlurtEngine/Audio/AudioTransport.swift b/Sources/BlurtEngine/Audio/AudioTransport.swift index b83f033f..d1540cc7 100644 --- a/Sources/BlurtEngine/Audio/AudioTransport.swift +++ b/Sources/BlurtEngine/Audio/AudioTransport.swift @@ -27,4 +27,26 @@ enum AudioTransport { return transportType == kAudioDeviceTransportTypeBluetooth || transportType == kAudioDeviceTransportTypeBluetoothLE } + + /// How much longer capture runs past the key-up that ends it, for a device of + /// this transport. `.zero` for everything but Bluetooth, so the wired path + /// skips the wait entirely rather than testing a flag at the call site. + /// + /// A Bluetooth link buffers: audio the user has already spoken is still in + /// flight when `stop()` is called, and `recorder.stop()` drops it — which is + /// why the last word of a dictation goes missing on AirPods. The value is + /// deliberately shorter than a typical link's worst case: it buys back the + /// common tail without making every dictation feel sluggish. + /// + /// Lives here beside `MicLiveness.timeout(forTransportType:)` — the other + /// transport-conditional policy — rather than inside `MicCapture`, so both are + /// reachable by `swift test`. `MicCapture` needs real hardware and is excluded + /// from the coverage gate. + static func tailLinger(forTransportType transportType: UInt32?) -> Duration { + isBluetooth(transportType) ? bluetoothTailLinger : .zero + } + + /// See `tailLinger(forTransportType:)`. Exposed so a test can name it rather + /// than restate the number. + static let bluetoothTailLinger = Duration.milliseconds(220) } diff --git a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift index 623b997b..db1130b4 100644 --- a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift +++ b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift @@ -18,7 +18,8 @@ extension MicCapture { /// Whether it is safe to open the input for a *warm* recorder right now: /// nothing is capturing, nothing is mid-bring-up, and no warm recorder is - /// already held. + /// already held. A scheduled re-warm that fails this has been overtaken — + /// a press got there first — and has nothing to do. /// /// `bringingUpCapture` is the load-bearing term. The two recorder slots are /// both nil across `start()`'s liveness wait, so testing them alone reads a @@ -52,20 +53,17 @@ extension MicCapture { /// Queues a re-warm to run once the current actor turn finishes, so the caller /// (`stop()` / `cancelCapture()`) returns before the input is re-opened. + /// + /// `warmUp()` rather than a separate re-warm entry point: the two differ only + /// in when they are called, and both answer the same question — is it safe to + /// open the input for a warm recorder right now — so they share the guard + /// rather than each keeping a copy of it. func scheduleRewarm() { Task { [weak self] in - await self?.rewarm() + await self?.warmUp() } } - /// Prepares the next session's recorder, unless a capture is running or coming - /// up, or a warm one is already held — all of which mean this re-warm has been - /// overtaken and has nothing to do. - func rewarm() { - guard canPrepareWarmRecorder else { return } - prepareWarmRecorder() - } - /// Builds a recorder, records the input it is bound to, and starts its idle /// countdown. A failure is non-fatal: `start()` then prepares lazily, exactly /// as it did before any warm recorder existed. diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 58d9c204..0b0f7427 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -60,11 +60,11 @@ public actor MicCapture: MicCaptureProtocol { /// The recorder for the in-flight session; nil between `stop()` and `start()`. var activeRecorder: AVAudioRecorder? - /// Whether the in-flight session's input is a Bluetooth device, sampled once - /// at `start()`. Read by `stop()` to decide on the tail linger — sampled at - /// start rather than re-read at stop so a device switch mid-utterance can't - /// make the two halves of one capture disagree. - private var activeInputIsBluetooth = false + /// The in-flight session's input transport, sampled once at `start()`. Read by + /// `stop()` to size the tail linger — sampled at start rather than re-read at + /// stop so a device switch mid-utterance can't make the two halves of one + /// capture disagree. + private var activeTransportType: UInt32? /// Incremented by every `stop()` / `cancelCapture()`. `start()` snapshots it /// before suspending in the liveness wait — its one internal suspension — and @@ -84,7 +84,8 @@ public actor MicCapture: MicCaptureProtocol { /// `activeRecorder` isn't installed until the wait returns (the recorder stays /// confined to `start()` so nothing can touch it while the poll loop reads its /// clock off-actor), and the warm slot was consumed on the way in. Without - /// this, `rewarm()`/`warmUp()` read those two nils as "no capture in flight" + /// this, `warmUp()` (which every re-warm goes through) reads those two nils as + /// "no capture in flight" /// and prepare a *second* recorder onto the already-live input — which is very /// reachable, since `stop()` schedules a re-warm that can land inside the next /// press's bring-up. @@ -111,22 +112,6 @@ public actor MicCapture: MicCaptureProtocol { /// `meterIntervalSeconds` as the `Duration` the meter task sleeps for. private static let meterInterval = Duration.seconds(meterIntervalSeconds) - /// How much longer capture runs after the key-up that ends it, when the input - /// is a Bluetooth device. - /// - /// A Bluetooth link buffers: audio the user has already spoken is still in - /// flight when `stop()` is called, and `recorder.stop()` drops it — which is - /// why the last word of a dictation goes missing on AirPods and the app reads - /// as running behind the speaker. The linger is deliberately shorter than a - /// typical link's worst case: it buys back the common tail without making - /// every dictation feel sluggish, and it costs nothing on a wired input, where - /// it is skipped entirely. - /// - /// The delay lands *after* `.transcribing` is claimed (see - /// `DictationSession.performRelease`), so it delays the transcript, never the - /// user's "it heard me" cue. Cancels skip it — see `cancelCapture()`. - static let bluetoothTailLinger = Duration.milliseconds(220) - /// How long a prepared-but-unused recorder is held before being torn down. /// /// The warm recorder is the fix for per-session route activation, but it is @@ -225,26 +210,22 @@ public actor MicCapture: MicCaptureProtocol { } activeRecorder = recorder - activeInputIsBluetooth = AudioTransport.isBluetooth(input?.transportType) + activeTransportType = input?.transportType lastEmittedLevel = nil Self.logger.info("start recording to \(recorder.url.lastPathComponent, privacy: .public)") startMeterTimer() } public func stop() async throws -> Data { - stopGeneration += 1 - meterTask?.cancel() - meterTask = nil - guard let recorder = activeRecorder else { return Data() } - activeRecorder = nil // Read before the suspension below, so this capture's decision can't be // rewritten by whatever a later `start()` sets. - let lingerForTail = activeInputIsBluetooth - if lingerForTail { + let linger = AudioTransport.tailLinger(forTransportType: activeTransportType) + guard let recorder = detachActiveRecorder() else { return Data() } + if linger > .zero { // Keep capturing for a moment past key-up so the audio still travelling // over the link lands in the file instead of being truncated. See - // `bluetoothTailLinger`. - try? await Task.sleep(for: Self.bluetoothTailLinger) + // `AudioTransport.tailLinger(forTransportType:)`. + try? await Task.sleep(for: linger) } recorder.stop() @@ -261,7 +242,8 @@ public actor MicCapture: MicCaptureProtocol { let sampleCount = pcm.count / SyncSTTLimits.bytesPerSample let durationMs = SyncSTTLimits.durationMs(ofPCMBytes: pcm.count) - Self.logger.info("stop samples=\(sampleCount) durationMs=\(durationMs) linger=\(lingerForTail)") + let lingerMs = Int(linger.milliseconds.rounded()) + Self.logger.info("stop samples=\(sampleCount) durationMs=\(durationMs) lingerMs=\(lingerMs)") return pcm } @@ -280,17 +262,28 @@ public actor MicCapture: MicCaptureProtocol { /// (which can throw, via `stop()`) still conforms and `stopAndCancel`'s /// developer-mode failure log keeps working for hosts that take it. public func cancelCapture() { - stopGeneration += 1 - meterTask?.cancel() - meterTask = nil - guard let recorder = activeRecorder else { return } - activeRecorder = nil + guard let recorder = detachActiveRecorder() else { return } recorder.stop() Self.removeFile(at: recorder.url) Self.logger.info("cancelled capture, discarded audio") scheduleRewarm() } + /// Ends the current capture's claim on the actor's state and hands back the + /// recorder to dispose of, or nil when there was none. + /// + /// Shared by `stop()` and `cancelCapture()` because the two must not diverge: + /// the generation bump in particular is what `start()` re-checks across its + /// liveness wait to know a teardown landed, so a third exit that forgot it + /// would let an abandoned bring-up install itself and keep capturing. + private func detachActiveRecorder() -> AVAudioRecorder? { + stopGeneration += 1 + meterTask?.cancel() + meterTask = nil + defer { activeRecorder = nil } + return activeRecorder + } + // MARK: - Recorder construction /// Build a recorder that writes mono 16-bit little-endian PCM at the target diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift index 1d0c952f..69a6f465 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift @@ -132,8 +132,15 @@ extension DictationSession { // rule `transcribe` and `inject` already follow. `.cancelled` rather than a // bare return, because `.connecting` is non-terminal — leaving it would // strand the trigger's gate and swallow the next press. + // + // Consumed the same way as the exit above, not just phase-set: the + // `.connecting` branch of `cancel()` claims the phase and returns + // *without* enqueueing `performCancel`, so this is the only place left + // that can clear the request it recorded. Leaving it set let the flag + // survive into the next press, which then read it after a perfectly good + // `mic.start()` and cancelled itself. if error is CancellationError { - setPhase(.cancelled) + if !consumeCancelRequest() { setPhase(.cancelled) } return } setPhase(.failed(.audioCaptureFailed(underlying: error))) diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index 316269b5..3ca55fd0 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -202,9 +202,8 @@ public actor DictationSession { } } - /// Appends `op` to the serial command queue and waits for it to run. The - /// synchronous read-then-write of `commandQueue` makes the chain order match - /// the order the public methods executed their first actor turn. + /// Appends `op` to the serial command queue and waits for it to run; the + /// ordering guarantee is `chain`'s. func enqueue(_ op: @escaping @Sendable () async -> Void) async { await chain(op).value } @@ -237,7 +236,6 @@ public actor DictationSession { await enqueue { await self.performRelease() } } - private func performRelease() async { guard phase == .recording else { return } cancelAutoRelease() diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index 10f1159e..017b2d12 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -384,4 +384,33 @@ struct DictationSessionBringUpTests { #expect(!seen.contains(.recording)) #expect(await mic.cancelCaptureCalls == 1) } + + @Test("a cancel that aborts the bring-up doesn't leak its request into the next press") + func abortedBringUpLeavesNoStandingCancel() async throws { + // The route where `start()` actually throws — a cancel landing *inside* the + // liveness wait, which is what the real `MicCapture` does. `cancel()`'s + // `.connecting` branch claims the phase and returns without enqueueing + // `performCancel`, so the press's catch is the only place left that can + // consume the request it recorded. When it didn't, the flag survived, and the + // *next* press read it after a perfectly good `mic.start()` and cancelled + // itself — one dead dictation for every cancelled bring-up. + let mic = GatedStartMic() + await mic.setThrowsIfCancelled(true) + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("Hello world.")), + injector: StubInjector(), seams: .offline) + + let pressed = Task { await session.press() } + await mic.waitUntilStartEntered() + await session.cancel() + await mic.allowStartToFinish() + await pressed.value + + #expect(await session.phase == .cancelled) + #expect(!session.cancelRequested) + + // The proof that matters: the next press records normally. + await session.press() + #expect(await session.phase == .recording) + } } diff --git a/Tests/BlurtEngineTests/HotkeyRaceTests.swift b/Tests/BlurtEngineTests/HotkeyRaceTests.swift index 2dc98775..b2e47e5e 100644 --- a/Tests/BlurtEngineTests/HotkeyRaceTests.swift +++ b/Tests/BlurtEngineTests/HotkeyRaceTests.swift @@ -15,7 +15,7 @@ struct HotkeyRaceTests { @Test("release during mic.start is honored, not dropped") func releaseDuringStartIsHonored() async throws { - let mic = GatedMicCapture() + let mic = GatedStartMic() let stt = StubTranscriber(mode: .transcript("hi")) let injector = StubInjector() let session = DictationSession( @@ -40,7 +40,7 @@ struct HotkeyRaceTests { @Test("cancel during mic.start is honored, not dropped") func cancelDuringStartIsHonored() async throws { - let mic = GatedMicCapture() + let mic = GatedStartMic() let stt = StubTranscriber(mode: .transcript("hi")) let injector = StubInjector() let session = DictationSession( @@ -69,7 +69,7 @@ struct HotkeyRaceTests { /// — the mic must never be started twice. @Test("a second press during mic.start is dropped, not double-started") func secondPressDuringStartIsDropped() async throws { - let mic = GatedMicCapture() + let mic = GatedStartMic() let stt = StubTranscriber(mode: .transcript("hi")) let injector = StubInjector() let session = DictationSession( @@ -98,7 +98,7 @@ struct HotkeyRaceTests { /// transcribe→inject run. @Test("cancel overrides a pending release during mic.start") func cancelOverridesPendingReleaseDuringStart() async throws { - let mic = GatedMicCapture() + let mic = GatedStartMic() let stt = StubTranscriber(mode: .transcript("hi")) let injector = StubInjector() let session = DictationSession( @@ -128,26 +128,3 @@ struct HotkeyRaceTests { #expect(await injector.inserted.isEmpty) } } - -/// Mic stub whose `start()` blocks until the test releases it, so `release()` can -/// be landed deterministically while `press()` is suspended inside `mic.start()`. -/// The entry/finish choreography lives in the shared `Gate`. -private actor GatedMicCapture: MicCaptureProtocol { - private(set) var startCalls = 0 - private(set) var stopCalls = 0 - private let gate = Gate() - - func start() async throws { - startCalls += 1 - await gate.enter() - } - - func waitUntilStartEntered() async { await gate.waitUntilEntered() } - func allowStartToFinish() async { gate.allowToFinish() } - - func stop() async throws -> Data { - stopCalls += 1 - // This suite exercises the press/release race, not the too-short guard. - return StubPCM.aboveMinimum - } -} diff --git a/Tests/BlurtEngineTests/MicLivenessTests.swift b/Tests/BlurtEngineTests/MicLivenessTests.swift index 125dd0be..b7366686 100644 --- a/Tests/BlurtEngineTests/MicLivenessTests.swift +++ b/Tests/BlurtEngineTests/MicLivenessTests.swift @@ -88,6 +88,20 @@ struct AudioTransportTests { #expect(AudioTransport.isBluetooth(kAudioDeviceTransportTypeBluetoothLE)) } + @Test("the tail linger is Bluetooth-only") + func tailLinger() { + // The other transport-conditional policy, here rather than in `MicCapture` + // so it is reachable by `swift test` at all — the capture actor needs real + // hardware and is excluded from the coverage gate. + #expect( + AudioTransport.tailLinger(forTransportType: kAudioDeviceTransportTypeBluetooth) + == AudioTransport.bluetoothTailLinger) + // `.zero`, not a small duration: `stop()` skips the sleep entirely on a + // wired input rather than awaiting a nominal one. + #expect(AudioTransport.tailLinger(forTransportType: kAudioDeviceTransportTypeBuiltIn) == .zero) + #expect(AudioTransport.tailLinger(forTransportType: nil) == .zero) + } + @Test("wired, built-in, and unreadable transports do not") func nonBluetoothTransports() { #expect(!AudioTransport.isBluetooth(kAudioDeviceTransportTypeBuiltIn)) diff --git a/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift b/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift index 0d7a651d..4a5c7c42 100644 --- a/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift +++ b/Tests/BlurtEngineTests/Stubs/GatedStartMic.swift @@ -11,19 +11,25 @@ import Foundation /// microseconds and nothing could arrive during a press. /// /// The entry/finish choreography lives in the shared `Gate`; `GatedStopMic` is -/// the mirror image for the release path. `HotkeyRaceTests` keeps its own private -/// gated-start stub predating this one — left alone rather than folded in, since -/// swapping a stub under a passing race suite risks more than the duplication -/// costs. +/// the mirror image for the release path. `HotkeyRaceTests` drives its +/// press/release races through this one too. actor GatedStartMic: MicCaptureProtocol { private(set) var startCalls = 0 private(set) var stopCalls = 0 private(set) var cancelCaptureCalls = 0 private let gate = Gate() + /// When set, `start()` throws `CancellationError` if the task was cancelled + /// while it was gated — what the real `MicCapture` does when a cancel lands + /// during its liveness wait. Off by default so the common gated-press tests + /// see a bring-up that simply completes. + private var throwsIfCancelled = false + + func setThrowsIfCancelled(_ value: Bool) { throwsIfCancelled = value } func start() async throws { startCalls += 1 await gate.enter() + if throwsIfCancelled, Task.isCancelled { throw CancellationError() } } func waitUntilStartEntered() async { await gate.waitUntilEntered() } diff --git a/scripts/check.sh b/scripts/check.sh index becb408c..b567700f 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -470,7 +470,7 @@ else # MicCapture+Meter.swift, which IS covered. Keep this # list tight — exclude only code that genuinely cannot # be exercised without hardware. - # - AudioRoute*.swift : the CoreAudio routing reads (AudioRoute) and the + # - AudioRoute(Monitor).swift : the CoreAudio routing reads (AudioRoute) and the # property listeners (AudioRouteMonitor). Both answer # questions only real hardware can answer — which device # is default, whether its transport is Bluetooth, and @@ -478,7 +478,7 @@ else # can only fire on an actual route change. Same # justification as MicCapture.swift above. COVERAGE="$(xcrun llvm-cov export -summary-only -instr-profile "$PROFDATA" "$XCTEST_BIN" \ - -ignore-filename-regex='Tests/|Audio/MicCapture\.swift|Audio/AudioRoute' \ + -ignore-filename-regex='Tests/|Audio/MicCapture\.swift|Audio/AudioRoute(Monitor)?\.swift' \ | python3 -c 'import sys,json; print(round(json.load(sys.stdin)["data"][0]["totals"]["lines"]["percent"],2))')" echo "engine line coverage: ${COVERAGE}%" if ! awk -v c="$COVERAGE" -v min="$MIN_COVERAGE" 'BEGIN{ exit (c+0 < min+0) }'; then From a61c975577c5feb61044b8114763be51d0774d27 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 19:52:35 +0000 Subject: [PATCH 09/14] Take the five deferred efficiency and altitude findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The context-capture chain now overlaps the mic bring-up instead of queueing behind it. `mic.start()` becomes an `async let`, and the frontmost read, target-app assignment and AX field-context dispatch run while the route comes up. That ordering used to be free (start() returned in microseconds); with the liveness gate it was up to 2.5s of dead time the AX read could have used, and instead the read landed on the release path where runTranscribeInject waits up to contextWaitBudget for it with the user watching. The `async let` child still inherits cancellation, so cancel()'s `.connecting` preemption is unaffected. Cost: a press whose mic fails has already set the injector target and dispatched one AX read — both harmless and overwritten by the next press. - The cue re-prime is deferred to the next terminal phase. The usual cause of a route change is Blurt opening the mic, so the tick arrives *during* the press: reloading there put two AAC decodes alongside the bring-up and swapped startSound out from under the very chime it protects, possibly releasing a player mid-play. Deferring puts it between dictations and coalesces a burst of flips into one decode. The "route changes are rare" justification was wrong and is corrected. - The liveness poll backs off geometrically (1ms doubling to 25ms) instead of a fixed 10ms. currentTime is 0 the instant record() returns, so every press slept a full quantum before .recording, the chime and the meter — on a wired mic that quantum is the whole wait. The Bluetooth path drops from ~250 wakeups to ~30. Tests walk the backoff explicitly: since waitUntilSleeping matches on the deadline, a loop that slept a fixed quantum now hangs rather than quietly passing. - AudioRouteMonitor is built off the main actor inside the route observer, not as a stored property. Constructing it registers two CoreAudio listeners and makes the process's first HAL call, which as a stored-property initializer on a MainActor type ran on the main thread during launch — on every run, including onboarding and UI tests where no chime is ever played. It is owned by the observer task rather than stored, so dropping the task deregisters it. - The four correlated prepared* fields collapse into one `warm: WarmRecorder?`, making "a recorder without its input snapshot" and "an expiry without its recorder" unrepresentable rather than merely never written. Not done: the full captureState enum. It would rewrite the same bring-up paths this commit just changed, in the one engine file the coverage gate can't check, on top of four other unverified changes — worth doing, but after CI has confirmed this batch, not stacked under it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- AGENTS.md | 4 +- App/Blurt/Blurt/CueSoundPlayer.swift | 51 +++++++++++--- .../BlurtEngine/Audio/MicCapture+Warm.swift | 45 ++++++------ Sources/BlurtEngine/Audio/MicCapture.swift | 70 +++++++++++-------- Sources/BlurtEngine/Audio/MicLiveness.swift | 36 +++++++--- .../Pipeline/DictationSession+Press.swift | 45 ++++++++---- Tests/BlurtEngineTests/MicLivenessTests.swift | 54 +++++++++++--- 7 files changed, 204 insertions(+), 101 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d3db1636..8d4d4d2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -423,8 +423,8 @@ in both directions. So: stops the app cueing the user to speak into a dead mic; audio spoken during the switch cannot be recovered by anything, because nothing ever receives it. `stopGeneration` covers the one suspension this introduces — a teardown landing mid-wait wins, and the recorder is torn down rather than - installed. `bringingUpCapture` covers the other consequence: across the wait **both** recorder - slots are nil, so the warm-up paths can't infer "no capture in flight" from them (see + installed. `bringingUpCapture` covers the other consequence: across the wait both `activeRecorder` + and `warm` are nil, so the warm-up paths can't infer "no capture in flight" from them (see `canPrepareWarmRecorder`) or they'd open a second recorder onto the live input. **A cancel preempts the bring-up rather than queueing behind it**, which took two pieces. The diff --git a/App/Blurt/Blurt/CueSoundPlayer.swift b/App/Blurt/Blurt/CueSoundPlayer.swift index a235e684..59deb02d 100644 --- a/App/Blurt/Blurt/CueSoundPlayer.swift +++ b/App/Blurt/Blurt/CueSoundPlayer.swift @@ -27,7 +27,9 @@ final class CueSoundPlayer { /// output-only profile, which drops the output format the players were primed /// against — so the very next chime is the one that stalls, and that's the /// chime at the start of a dictation. - private let routeMonitor = AudioRouteMonitor() + /// Set when the output route changes, cleared when the re-prime actually runs. + /// See `transition(for:)` for why the reload waits. + private var needsReprime = false /// Kept alive for the app's lifetime; assignment is the use. Nil until /// `prime()` starts it, which is also what makes starting idempotent. private var routeObserver: Task? @@ -67,23 +69,41 @@ final class CueSoundPlayer { /// Idempotent — `prime()` runs on every "app is ready" transition, and only /// the first call installs the observer. /// - /// A full reload rather than a bare `prepareToPlay()`: route changes are rare - /// (a handful an hour at most), the decode runs off the main actor like every - /// other load, and re-creating the players is the one thing guaranteed to - /// leave them primed against the *current* route. + /// A full reload rather than a bare `prepareToPlay()`: re-creating the players + /// is the one thing guaranteed to leave them primed against the *current* + /// route, and the decode runs off the main actor like every other load. Route + /// changes are not rare — on a Bluetooth output Blurt's own capture causes one + /// per dictation burst — which is exactly why the reload is deferred rather + /// than run on the tick; see `transition(for:)`. private func startObservingRoute() { guard routeObserver == nil else { return } - let changes = routeMonitor.outputRouteChanges routeObserver = Task { [weak self] in - for await _ in changes { + // The monitor is owned by this task, not stored: the local keeps it alive + // for as long as the loop runs, and dropping it when the task ends is what + // deregisters its CoreAudio listeners. + let monitor = await Self.makeRouteMonitor() + for await _ in monitor.outputRouteChanges { + // Rebound per tick rather than bound once above, so the observer never + // keeps the player alive — the same reason `MicCapture`'s meter task + // rebinds across its sleep. guard let self else { return } - // `force`, because the pack hasn't changed — the *route* has, and the - // pre-roll is what went stale. - await self.loadCurrentPack(force: true) + // Recorded, not acted on — see `transition(for:)`. + self.needsReprime = true } } } + /// Constructs the monitor off the main actor. `nonisolated` + `async` for the + /// same reason `decode` is: this type defaults to `MainActor`, so a plain + /// initializer call would run on the main thread — and constructing it + /// registers two CoreAudio listeners and makes the process's *first* HAL call, + /// paying the client-side HAL init and the coreaudiod connection. As a stored + /// property that landed during launch, before the first frame, on every run + /// including onboarding and UI tests where no chime is ever played. + private nonisolated static func makeRouteMonitor() async -> AudioRouteMonitor { + AudioRouteMonitor() + } + /// Reloads the players for a newly selected pack and previews the new voice /// (start, then stop a beat apart) so the choice is audible immediately — /// after the reload lands, so the preview uses the freshly decoded players. @@ -129,6 +149,17 @@ final class CueSoundPlayer { case .stop: play(stopSound) case nil: break } + // A pending route re-prime waits for the dictation to finish. The usual + // cause of a route change is Blurt *opening the mic* — so the tick arrives + // during the press, and reloading there would put two AAC decodes and two + // `prepareToPlay()`s alongside the bring-up, then swap `startSound` out from + // under the very chime it is meant to protect (releasing an `AVAudioPlayer` + // that may be mid-play). Deferring to the next terminal phase puts the + // reload between dictations, where it costs nothing anyone is waiting on, + // and coalesces a burst of flips into one decode. + guard phase.isTerminal, needsReprime else { return } + needsReprime = false + Task { await loadCurrentPack(force: true) } } /// The decoded, pre-rolled players for a pack. Non-`Sendable` (holds diff --git a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift index db1130b4..b5cc270f 100644 --- a/Sources/BlurtEngine/Audio/MicCapture+Warm.swift +++ b/Sources/BlurtEngine/Audio/MicCapture+Warm.swift @@ -4,7 +4,7 @@ import Foundation // The warm-recorder lifecycle — prepare ahead of the press, validate it still // matches the live input, and let it expire — split from `MicCapture.swift` to // stay within the lint file-length budget, like `MicCapture+Meter`. Members it -// reaches (the prepared-recorder state, `logger`, `removeFile`, `makeRecorder`) +// reaches (`warm`, `preparedGeneration`, `logger`, `removeFile`, `makeRecorder`) // are internal rather than private for that reason: `private` is file-scoped and // can't cross the split. extension MicCapture { @@ -21,11 +21,11 @@ extension MicCapture { /// already held. A scheduled re-warm that fails this has been overtaken — /// a press got there first — and has nothing to do. /// - /// `bringingUpCapture` is the load-bearing term. The two recorder slots are - /// both nil across `start()`'s liveness wait, so testing them alone reads a - /// live capture as "idle" and prepares a second recorder onto the open input. + /// `bringingUpCapture` is the load-bearing term. Both `activeRecorder` and + /// `warm` are nil across `start()`'s liveness wait, so testing them alone reads + /// a live capture as "idle" and prepares a second recorder onto the open input. var canPrepareWarmRecorder: Bool { - activeRecorder == nil && preparedRecorder == nil && !bringingUpCapture + activeRecorder == nil && warm == nil && !bringingUpCapture } /// The warm recorder if it is still bound to `input`, else nil — discarding @@ -37,18 +37,15 @@ extension MicCapture { /// silence. Paying route activation is the cheaper mistake, so unknown means /// discard. func takeWarmRecorder(matching input: AudioRoute.InputSnapshot?) -> AVAudioRecorder? { - preparedExpiry?.cancel() - preparedExpiry = nil - guard let recorder = preparedRecorder else { return nil } - let warmed = preparedInput - preparedRecorder = nil - preparedInput = nil - guard let warmed, let input, warmed.deviceID == input.deviceID else { - Self.removeFile(at: recorder.url) + guard let held = warm else { return nil } + held.expiry?.cancel() + warm = nil + guard let warmed = held.input, let input, warmed.deviceID == input.deviceID else { + Self.removeFile(at: held.recorder.url) Self.logger.info("discarded warm recorder — input device changed since warm-up") return nil } - return recorder + return held.recorder } /// Queues a re-warm to run once the current actor turn finishes, so the caller @@ -69,10 +66,11 @@ extension MicCapture { /// as it did before any warm recorder existed. func prepareWarmRecorder() { do { - let recorder = try Self.makeRecorder() - preparedRecorder = recorder - preparedInput = AudioRoute.currentInput() preparedGeneration += 1 + warm = WarmRecorder( + recorder: try Self.makeRecorder(), + input: AudioRoute.currentInput(), + generation: preparedGeneration) armPreparedRecorderExpiry(generation: preparedGeneration) Self.logger.info("prepared a warm recorder") } catch { @@ -84,8 +82,8 @@ extension MicCapture { /// The ticket is what makes a stale expiry harmless — see /// `releasePreparedRecorder(generation:)`. func armPreparedRecorderExpiry(generation: Int) { - preparedExpiry?.cancel() - preparedExpiry = Task { [weak self] in + warm?.expiry?.cancel() + warm?.expiry = Task { [weak self] in try? await Task.sleep(for: Self.preparedRecorderLifetime) guard !Task.isCancelled else { return } await self?.releasePreparedRecorder(generation: generation) @@ -103,12 +101,9 @@ extension MicCapture { /// out the *live* expiry's handle (leaving the current warm recorder with no /// countdown at all) and tear down a recorder prepared a moment ago. func releasePreparedRecorder(generation: Int) { - guard generation == preparedGeneration else { return } - preparedExpiry = nil - guard let recorder = preparedRecorder else { return } - preparedRecorder = nil - preparedInput = nil - Self.removeFile(at: recorder.url) + guard let held = warm, held.generation == generation else { return } + warm = nil + Self.removeFile(at: held.recorder.url) Self.logger.info("released idle warm recorder") } } diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 0b0f7427..94173490 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -27,37 +27,52 @@ public actor MicCapture: MicCaptureProtocol { private static let targetSampleRate = Double(SyncSTTLimits.sampleRate) /// A recorder prepared ahead of the press so `start()` doesn't pay hardware - /// route activation on the hot path. Filled by `warmUp()` at launch and - /// **re-filled after every capture** (see `scheduleRewarm`), because that cost - /// is paid per session, not once: `prepareToRecord()` is where the route is - /// resolved and opened, and on a Bluetooth input that means renegotiating the - /// link into its mic-capable mode — hundreds of milliseconds, sometimes over a - /// second, during which the user has pressed the key and nothing has happened. - /// Warming only the first session (the previous behavior) hid that cost for one - /// dictation out of every N. + /// route activation on the hot path, together with everything that belongs to + /// *that* recorder: the input it is bound to, its idle countdown, and the + /// ticket that countdown carries. + /// + /// One optional rather than four parallel fields, so "a recorder without its + /// input snapshot" and "an expiry without its recorder" are unrepresentable + /// instead of merely never written. + /// + /// Filled by `warmUp()` at launch and **re-filled after every capture** (see + /// `scheduleRewarm`), because that cost is paid per session, not once: + /// `prepareToRecord()` is where the route is resolved and opened, and on a + /// Bluetooth input that means renegotiating the link into its mic-capable + /// mode — hundreds of milliseconds, sometimes over a second, during which the + /// user has pressed the key and nothing has happened. Warming only the first + /// session (the previous behavior) hid that cost for one dictation out of N. /// /// Still a *fresh recorder per session*, which is the invariant the /// `AVAudioEngine` rewrite bought: the warm recorder is validated against the /// live default input before it is used (`takeWarmRecorder`) and discarded /// rather than reused when the device has changed underneath it. - var preparedRecorder: AVAudioRecorder? - /// The default input `preparedRecorder` was built against. `AVAudioRecorder` - /// resolves its device once, at `prepareToRecord()`, and never re-resolves — - /// so without this a recorder warmed while the built-in mic was default would - /// keep recording from it after the user connected their AirPods. See - /// `AudioRoute.InputSnapshot.deviceID` for why identity is the device ID. - var preparedInput: AudioRoute.InputSnapshot? - /// Releases `preparedRecorder` once it has gone unused for - /// `preparedRecorderLifetime`. See that constant for why holding one open - /// forever is not an option. - var preparedExpiry: Task? - /// Bumped for each warm recorder prepared, and carried by that recorder's - /// expiry task, so an expiry whose recorder has since been consumed or - /// replaced can recognise itself as stale and do nothing. See + var warm: WarmRecorder? + + /// Bumped for each warm recorder prepared, so an expiry whose recorder has + /// since been consumed or replaced can recognise itself as stale. See /// `releasePreparedRecorder(generation:)` for why cancelling the task isn't /// sufficient on its own. var preparedGeneration = 0 + /// A prepared-but-not-started recorder and the state that only makes sense + /// alongside it. + struct WarmRecorder { + let recorder: AVAudioRecorder + /// The default input it was built against. `AVAudioRecorder` resolves its + /// device once, at `prepareToRecord()`, and never re-resolves — so without + /// this a recorder warmed while the built-in mic was default would keep + /// recording from it after the user connected their AirPods. See + /// `AudioRoute.InputSnapshot.deviceID` for why identity is the device ID. + let input: AudioRoute.InputSnapshot? + /// Releases this recorder once it has gone unused for + /// `preparedRecorderLifetime`. See that constant for why holding one open + /// forever is not an option. + var expiry: Task? = nil + /// The ticket `expiry` carries, so a stale one can identify itself. + let generation: Int + } + /// The recorder for the in-flight session; nil between `stop()` and `start()`. var activeRecorder: AVAudioRecorder? /// The in-flight session's input transport, sampled once at `start()`. Read by @@ -83,12 +98,11 @@ public actor MicCapture: MicCaptureProtocol { /// Needed because during that window **both** recorder slots are nil: /// `activeRecorder` isn't installed until the wait returns (the recorder stays /// confined to `start()` so nothing can touch it while the poll loop reads its - /// clock off-actor), and the warm slot was consumed on the way in. Without - /// this, `warmUp()` (which every re-warm goes through) reads those two nils as - /// "no capture in flight" - /// and prepare a *second* recorder onto the already-live input — which is very - /// reachable, since `stop()` schedules a re-warm that can land inside the next - /// press's bring-up. + /// clock off-actor), and `warm` was consumed on the way in. Without this, + /// `warmUp()` — which every re-warm goes through — reads those two nils as "no + /// capture in flight" and prepares a *second* recorder onto the already-live + /// input, which is very reachable since `stop()` schedules a re-warm that can + /// land inside the next press's bring-up. var bringingUpCapture = false /// Polls the active recorder's meter and feeds `levels` while recording. diff --git a/Sources/BlurtEngine/Audio/MicLiveness.swift b/Sources/BlurtEngine/Audio/MicLiveness.swift index 419d46a5..78846cb4 100644 --- a/Sources/BlurtEngine/Audio/MicLiveness.swift +++ b/Sources/BlurtEngine/Audio/MicLiveness.swift @@ -6,12 +6,23 @@ import Foundation /// same split as `MicCapture+Meter`) so the timeout policy and the wait's edge /// cases are unit-tested against an injected clock. enum MicLiveness { - /// How often the recorder's clock is re-checked while waiting. Every press - /// whose first read is still 0 pays at least one quantum before `.recording` - /// and the start chime, so keep it short — 10 ms, the same cadence as - /// `KeyInjector.waitUntilFrontmost`'s poll-until-deadline loop — rather than - /// taxing the common (wired/built-in) case to wait for the Bluetooth one. - static let pollInterval: Duration = .milliseconds(10) + /// The first re-check delay, doubling up to `maxPollInterval`. + /// + /// Geometric rather than a fixed quantum because the two cases this loop + /// serves want opposite things. `AVAudioRecorder.currentTime` is 0 the instant + /// `record()` returns, so *every* press sleeps at least once before + /// `.recording`, the start chime and the meter — on a wired mic that quantum + /// is the entire wait, and it should be as small as possible. A real Bluetooth + /// bring-up, meanwhile, runs to seconds, where a small fixed quantum is + /// hundreds of timer wakeups each doing a clock read for an answer that hasn't + /// changed. Starting at 1 ms and doubling gives the common case a ~1 ms + /// acknowledgement and the slow case ~30 wakeups instead of 250, at the cost + /// of at most one `maxPollInterval` of detection granularity out of a 1–2 s + /// wait. + static let initialPollInterval: Duration = .milliseconds(1) + + /// Ceiling for the backoff — the coarsest the loop ever gets. + static let maxPollInterval: Duration = .milliseconds(25) /// Wait cap for Bluetooth inputs: bringing an AirPods mic up means a profile /// switch into the mic-capable mode that takes ~1–2 s, and the link drops back @@ -36,10 +47,11 @@ enum MicLiveness { AudioTransport.isBluetooth(transportType) ? bluetoothTimeout : defaultTimeout } - /// Polls `currentTime` every `pollInterval` until it advances past zero. The - /// recorder's clock only moves once the input device delivers frames, which is - /// what distinguishes "route still switching" (clock stuck at 0) from "user is - /// silent" (clock advancing over quiet audio) — a meter level can't. + /// Polls `currentTime` on a backoff (see `initialPollInterval`) until it + /// advances past zero. The recorder's clock only moves once the input device + /// delivers frames, which is what distinguishes "route still switching" (clock + /// stuck at 0) from "user is silent" (clock advancing over quiet audio) — a + /// meter level can't. /// /// Returns the elapsed wait once frames flow, or nil when `timeout` (or a task /// cancellation) won the race. The caller FAILS OPEN on nil — proceeding @@ -52,9 +64,11 @@ enum MicLiveness { ) async -> Duration? { let start = clock.now let deadline = start.advanced(by: timeout) + var interval = initialPollInterval while currentTime() <= 0 { guard clock.now < deadline, !Task.isCancelled else { return nil } - try? await clock.sleep(for: pollInterval) + try? await clock.sleep(for: interval) + interval = min(interval * 2, maxPollInterval) } return start.duration(to: clock.now) } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift index 69a6f465..18abbb32 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift @@ -17,11 +17,11 @@ extension DictationSession { setPhase(.failed(blocker)) return } - // Times the startup path — the concurrent focus capture + mic.start (and the - // detached connection warm-up kicked off below) — up to the moment recording - // actually begins. Ended on both the success and failure exits (mic.start is - // the only throwing call, and it precedes `.recording`, so the two ends are - // mutually exclusive). + // Times the startup path — the mic bring-up and the context capture that now + // runs alongside it (plus the detached connection warm-up) — up to the moment + // recording actually begins. Ended on both the success and failure exits + // (`mic.start()` is the only throwing call, and it precedes `.recording`, so + // the two ends are mutually exclusive). let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) // Claim `.connecting` before `mic.start()`: its liveness gate holds until // the input route actually delivers frames, which on a Bluetooth route is @@ -38,16 +38,25 @@ extension DictationSession { // just pays setup as before); warming every press is cheap since a hot pool just reuses it. let transcriber = transcriber Task.detached { await transcriber.warmUp() } - // Capture the frontmost app (paste target) concurrently with mic startup — - // a cheap in-process AppKit read on the main actor. The phase still flips - // to .recording only after mic.start succeeds, so the UI never lies about - // whether audio is being captured. Lifted out of the actor first (like - // `transcriber` above) so the child task calls a Sendable closure rather - // than reading isolated state. + // The mic bring-up runs as a child task so the whole context-capture chain + // below overlaps it instead of queueing behind it. That ordering used to be + // free — `mic.start()` returned in microseconds — but the liveness gate can + // now hold it for `MicLiveness.bluetoothTimeout`, and every one of those + // milliseconds was dead time the AX read could have used. Sequenced after + // it, the read instead landed on the *release* path, where + // `runTranscribeInject` waits up to `contextWaitBudget` for it with the + // user watching. Now it is almost always finished before the mic is even + // live. + // + // `async let`, so a cancel still reaches it: the child inherits this task's + // cancellation, which is what `cancel()`'s `.connecting` branch relies on + // to preempt the wait. + async let started: Void = mic.start() + // Capture the frontmost app (paste target). A cheap in-process AppKit read + // on the main actor. Lifted out of the actor first (like `transcriber` + // above) so the call is a Sendable closure rather than isolated state. let captureFrontmost = seams.captureFrontmost - async let frontmost = captureFrontmost() - try await mic.start() - let captured = await frontmost + let captured = await captureFrontmost() await injector.setTargetApp(captured.flatMap { FocusCapture.runningApp(for: $0) }) // Key terms are read synchronously at press (cheap UserDefaults read), so // each dictation observably re-reads Settings edits at press time. @@ -91,6 +100,14 @@ extension DictationSession { contextFeed.yield(context.isEmpty ? nil : context) contextFeed.finish() } + + // Only now join the bring-up. Everything above ran while the mic was + // coming up; the phase still flips to `.recording` only once `start()` + // returns, so the UI never claims capture that isn't live. The cost of + // starting the context work first is that a press whose mic fails has + // already set the injector's target and dispatched one AX read — both + // harmless and overwritten by the next press. + try await started // A cancel that arrived during the bring-up, on the path where `start()` // still returned normally — the cancel landed in the window between the // liveness wait finishing and `.recording` being claimed, so there was diff --git a/Tests/BlurtEngineTests/MicLivenessTests.swift b/Tests/BlurtEngineTests/MicLivenessTests.swift index b7366686..4c0f7c95 100644 --- a/Tests/BlurtEngineTests/MicLivenessTests.swift +++ b/Tests/BlurtEngineTests/MicLivenessTests.swift @@ -42,23 +42,53 @@ struct MicLivenessTests { #expect(gap == .zero) } - @Test("a clock that advances after a few polls confirms with the elapsed gap") - func livenessAfterPolls() async { + @Test("the poll interval doubles, so a slow bring-up isn't hundreds of wakeups") + func livenessBacksOff() async { let clock = TestClock() let polls = Mutex(0) async let gap = MicLiveness.waitUntilLive(timeout: MicLiveness.bluetoothTimeout, clock: clock) { - // Stuck at 0 for the first two checks — the route still switching — then + // Stuck at 0 for the first three checks — the route still switching — then // the recorder clock starts moving. polls.withLock { polls in polls += 1 - return polls < 3 ? 0 : 0.05 + return polls < 4 ? 0 : 0.05 } } - for _ in 1...2 { - await clock.waitUntilSleeping(for: MicLiveness.pollInterval) - clock.advance(by: MicLiveness.pollInterval) + // Each wait is twice the last: 1 ms, 2 ms, 4 ms. Driving the clock by the + // exact expected delay is also the assertion — `waitUntilSleeping` matches on + // the deadline, so a loop that slept a fixed quantum would hang here rather + // than quietly pass. + var expected = MicLiveness.initialPollInterval + var elapsed = Duration.zero + for _ in 1...3 { + await clock.waitUntilSleeping(for: expected) + clock.advance(by: expected) + elapsed += expected + expected = min(expected * 2, MicLiveness.maxPollInterval) + } + #expect(await gap == elapsed) + } + + @Test("the backoff is capped rather than doubling without limit") + func backoffIsCapped() async { + // Otherwise a 2.5 s Bluetooth cap would end in multi-second sleeps and + // overshoot the deadline it is supposed to respect. + let clock = TestClock() + let polls = Mutex(0) + async let gap = MicLiveness.waitUntilLive(timeout: .seconds(30), clock: clock) { + polls.withLock { polls in + polls += 1 + return polls < 12 ? 0 : 0.05 + } + } + var expected = MicLiveness.initialPollInterval + for _ in 1...11 { + await clock.waitUntilSleeping(for: expected) + clock.advance(by: expected) + expected = min(expected * 2, MicLiveness.maxPollInterval) } - #expect(await gap == MicLiveness.pollInterval * 2) + #expect(expected == MicLiveness.maxPollInterval) + #expect(await gap != nil) } @Test("a clock that never advances times out with nil — the fail-open signal") @@ -66,11 +96,13 @@ struct MicLivenessTests { // nil is what tells `MicCapture` to proceed anyway: a silent or broken mic // must degrade to the pre-gate behavior, never brick the press. let clock = TestClock() - let timeout = MicLiveness.pollInterval * 2 + let timeout = MicLiveness.initialPollInterval * 3 async let gap = MicLiveness.waitUntilLive(timeout: timeout, clock: clock) { 0 } + var expected = MicLiveness.initialPollInterval for _ in 1...2 { - await clock.waitUntilSleeping(for: MicLiveness.pollInterval) - clock.advance(by: MicLiveness.pollInterval) + await clock.waitUntilSleeping(for: expected) + clock.advance(by: expected) + expected = min(expected * 2, MicLiveness.maxPollInterval) } #expect(await gap == nil) } From 606aa38e18ab2e1226c47ab7a52813ac6e776aa8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 19:58:25 +0000 Subject: [PATCH 10/14] =?UTF-8?q?Drop=20the=20generic=20CoreAudio=20read?= =?UTF-8?q?=20helper=20=E2=80=94=20it=20doesn't=20compile?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `&value` on an unconstrained generic is rejected: "forming 'UnsafeMutableRawPointer' to a variable of type 'T'; this is likely incorrect because 'T' may contain an object reference." Making it work means constraining to BitwiseCopyable and going through withUnsafeMutableBytes — more machinery than two five-line reads are worth in a file the coverage gate can't check anyway, and it was the lowest-value item in the cleanup batch. `globalAddress(_:)` and `systemObject` stay: those carry the actual duplication the review found (four address literals across two files), and the monitor keeps using them. Nothing else in the batch was implicated — the module emitted and 82 of 84 files compiled, so the `async let` press restructure, the WarmRecorder collapse, the poll backoff, `detachActiveRecorder` and the app-side changes are all clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- Sources/BlurtEngine/Audio/AudioRoute.swift | 35 +++++++++++----------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/Sources/BlurtEngine/Audio/AudioRoute.swift b/Sources/BlurtEngine/Audio/AudioRoute.swift index e5aa5c17..bf7e8e98 100644 --- a/Sources/BlurtEngine/Audio/AudioRoute.swift +++ b/Sources/BlurtEngine/Audio/AudioRoute.swift @@ -85,29 +85,25 @@ enum AudioRoute { // MARK: - CoreAudio reads - /// One `AudioObjectGetPropertyData` read of a fixed-size value, or nil when - /// CoreAudio refused. `initial` supplies both the type and the zero value, so - /// each caller states the property it wants and nothing else. - private static func read( - _ selector: AudioObjectPropertySelector, from object: AudioObjectID, initial: T - ) -> T? { - var address = globalAddress(selector) - var value = initial - var size = UInt32(MemoryLayout.size) - let status = AudioObjectGetPropertyData(object, &address, 0, nil, &size, &value) - guard status == noErr else { return nil } - return value - } + // Spelled out per property rather than shared behind a generic + // `read(_:from:initial:)`. That reads better but doesn't compile: `&value` + // on an unconstrained `T` is "forming 'UnsafeMutableRawPointer' to a variable + // of type 'T'; this is likely incorrect because 'T' may contain an object + // reference". Making it work means constraining to `BitwiseCopyable` and going + // through `withUnsafeMutableBytes` — more machinery than two five-line reads + // are worth, in a file the coverage gate can't check anyway. /// The device the system object reports for `selector` (a default-device /// property). Nil covers both a failed read and the "no such device" sentinel, /// which callers treat identically. private static func defaultDeviceID(for selector: AudioObjectPropertySelector) -> AudioDeviceID? { + var address = globalAddress(selector) + var deviceID = AudioDeviceID(0) + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData(systemObject, &address, 0, nil, &size, &deviceID) // 0 is `kAudioObjectUnknown` — "there is no such device" — spelled as the // literal so this doesn't depend on how the constant imports. - guard let deviceID = read(selector, from: systemObject, initial: AudioDeviceID(0)), - deviceID != 0 - else { return nil } + guard status == noErr, deviceID != 0 else { return nil } return deviceID } @@ -115,6 +111,11 @@ enum AudioRoute { /// `AudioTransport` and `MicLiveness` both treat nil as "not Bluetooth", which /// is the conservative direction for each. private static func transportType(of deviceID: AudioDeviceID) -> UInt32? { - read(kAudioDevicePropertyTransportType, from: deviceID, initial: UInt32(0)) + var address = globalAddress(kAudioDevicePropertyTransportType) + var transport = UInt32(0) + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &transport) + guard status == noErr else { return nil } + return transport } } From f1aff9da32369bea711541698e19b4fbfe4befb1 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 20:00:55 +0000 Subject: [PATCH 11/14] Fix three swiftlint violations from the last two commits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `performPress` reached 58 code lines (limit 50) once the context chain moved above the bring-up join. The chain is now `beginContextCapture()` — one phase of the press, not a reusable step, extracted for the budget. performPress is back to 36. - `WarmRecorder.expiry` carried an explicit `= nil`, which implicit_optional_initialization rejects. I added it defensively, worried the memberwise initializer wouldn't default a middle optional; the rule's existence settles that it does. - `DictationSessionTests.swift` hit 416 lines (limit 400), so the `.connecting` bring-up suite moves to `DictationSessionBringUpTests.swift`, and the file's split note names it alongside the other two. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- Sources/BlurtEngine/Audio/MicCapture.swift | 2 +- .../Pipeline/DictationSession+Press.swift | 109 +++++++++-------- .../DictationSessionBringUpTests.swift | 114 ++++++++++++++++++ .../DictationSessionTests.swift | 113 +---------------- 4 files changed, 177 insertions(+), 161 deletions(-) create mode 100644 Tests/BlurtEngineTests/DictationSessionBringUpTests.swift diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index 94173490..5d7230c7 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -68,7 +68,7 @@ public actor MicCapture: MicCaptureProtocol { /// Releases this recorder once it has gone unused for /// `preparedRecorderLifetime`. See that constant for why holding one open /// forever is not an option. - var expiry: Task? = nil + var expiry: Task? /// The ticket `expiry` carries, so a stale one can identify itself. let generation: Int } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift index 18abbb32..3f9d68ec 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Press.swift @@ -52,55 +52,7 @@ extension DictationSession { // cancellation, which is what `cancel()`'s `.connecting` branch relies on // to preempt the wait. async let started: Void = mic.start() - // Capture the frontmost app (paste target). A cheap in-process AppKit read - // on the main actor. Lifted out of the actor first (like `transcriber` - // above) so the call is a Sendable closure rather than isolated state. - let captureFrontmost = seams.captureFrontmost - let captured = await captureFrontmost() - await injector.setTargetApp(captured.flatMap { FocusCapture.runningApp(for: $0) }) - // Key terms are read synchronously at press (cheap UserDefaults read), so - // each dictation observably re-reads Settings edits at press time. - let keyTerms = keyTermsProvider() - // Session history, read on the actor for the same reason: the capture below - // runs off-actor, so what it carries has to be a value taken now. - let recentTranscripts = recentDictations.transcriptsOldestFirst - // Kick off the AX field-context read now, while the target field still - // holds focus, but don't await it here: it's cross-process IPC into the - // frontmost app (detached — off the main actor, where it froze the - // overlay, and off this actor, where it would wedge release()/cancel()). - // runTranscribeInject consumes the result right before transcription, - // bounded by `contextWaitBudget` — so a slow AX target delays the - // transcript by at most the budget, never the recording indicator. - let (stream, contextFeed) = AsyncStream.makeStream( - of: TranscriptionContext?.self, bufferingPolicy: .bufferingNewest(1)) - contextStream = stream - // A Dispatch queue, not `Task.detached`: `captureFieldContext` is documented - // as making ~6 synchronous cross-process AX round trips, each bounded only by - // the 1 s messaging timeout, so against a beachballing frontmost app one - // press can *block* a thread for seconds. The Swift cooperative pool is sized - // to the core count and does not overcommit, so a few press/cancel cycles - // against a hung app could park every cooperative thread and stall the whole - // non-main runtime — including this actor. Dispatch overcommits, so a blocked - // capture costs a thread instead of the pool. Same reasoning as - // `DictationLog`'s serial queue. Concurrent so a hung capture can't delay the - // next press's. The body is fully synchronous and captures only Sendable - // values, so it needs no task context. - let captureFieldContext = seams.captureFieldContext - Self.contextQueue.async { - let field = captureFieldContext() - let context = TranscriptionContext( - appName: captured?.processName, - windowTitle: field.windowTitle, - fieldLabel: field.fieldLabel, - priorText: field.priorText, - selectedText: field.selectedText, - recentTranscripts: recentTranscripts, - keyTerms: keyTerms, - targetIsSecure: field.isSecure) - contextFeed.yield(context.isEmpty ? nil : context) - contextFeed.finish() - } - + await beginContextCapture() // Only now join the bring-up. Everything above ran while the mic was // coming up; the phase still flips to `.recording` only once `start()` // returns, so the UI never claims capture that isn't live. The cost of @@ -163,4 +115,63 @@ extension DictationSession { setPhase(.failed(.audioCaptureFailed(underlying: error))) } } + + /// Captures the paste target and kicks off the press-time AX field-context + /// read, leaving the result in `contextStream` for `runTranscribeInject` to + /// consume. + /// + /// Called *before* the bring-up is joined, so all of it — including the + /// cross-process AX read, the expensive part — overlaps the mic coming up + /// rather than queueing behind it. Split out of `performPress` for the lint + /// function-length budget; it is one phase of the press, not a reusable step. + private func beginContextCapture() async { + // Capture the frontmost app (paste target). A cheap in-process AppKit read + // on the main actor. Lifted out of the actor first (like `transcriber` + // above) so the call is a Sendable closure rather than isolated state. + let captureFrontmost = seams.captureFrontmost + let captured = await captureFrontmost() + await injector.setTargetApp(captured.flatMap { FocusCapture.runningApp(for: $0) }) + // Key terms are read synchronously at press (cheap UserDefaults read), so + // each dictation observably re-reads Settings edits at press time. + let keyTerms = keyTermsProvider() + // Session history, read on the actor for the same reason: the capture below + // runs off-actor, so what it carries has to be a value taken now. + let recentTranscripts = recentDictations.transcriptsOldestFirst + // Kick off the AX field-context read now, while the target field still + // holds focus, but don't await it here: it's cross-process IPC into the + // frontmost app (detached — off the main actor, where it froze the + // overlay, and off this actor, where it would wedge release()/cancel()). + // runTranscribeInject consumes the result right before transcription, + // bounded by `contextWaitBudget` — so a slow AX target delays the + // transcript by at most the budget, never the recording indicator. + let (stream, contextFeed) = AsyncStream.makeStream( + of: TranscriptionContext?.self, bufferingPolicy: .bufferingNewest(1)) + contextStream = stream + // A Dispatch queue, not `Task.detached`: `captureFieldContext` is documented + // as making ~6 synchronous cross-process AX round trips, each bounded only by + // the 1 s messaging timeout, so against a beachballing frontmost app one + // press can *block* a thread for seconds. The Swift cooperative pool is sized + // to the core count and does not overcommit, so a few press/cancel cycles + // against a hung app could park every cooperative thread and stall the whole + // non-main runtime — including this actor. Dispatch overcommits, so a blocked + // capture costs a thread instead of the pool. Same reasoning as + // `DictationLog`'s serial queue. Concurrent so a hung capture can't delay the + // next press's. The body is fully synchronous and captures only Sendable + // values, so it needs no task context. + let captureFieldContext = seams.captureFieldContext + Self.contextQueue.async { + let field = captureFieldContext() + let context = TranscriptionContext( + appName: captured?.processName, + windowTitle: field.windowTitle, + fieldLabel: field.fieldLabel, + priorText: field.priorText, + selectedText: field.selectedText, + recentTranscripts: recentTranscripts, + keyTerms: keyTerms, + targetIsSecure: field.isSecure) + contextFeed.yield(context.isEmpty ? nil : context) + contextFeed.finish() + } + } } diff --git a/Tests/BlurtEngineTests/DictationSessionBringUpTests.swift b/Tests/BlurtEngineTests/DictationSessionBringUpTests.swift new file mode 100644 index 00000000..23ea9118 --- /dev/null +++ b/Tests/BlurtEngineTests/DictationSessionBringUpTests.swift @@ -0,0 +1,114 @@ +import Foundation +import Testing + +@testable import BlurtEngine + +/// Behavior in the `.connecting` window — the up-to-2.5 s stretch +/// `MicCapture.start()` holds while a Bluetooth route brings the mic up. Before +/// the liveness gate, `start()` returned in microseconds and nothing could land +/// mid-press; now things can, so pin what happens when they do. +@Suite("DictationSession mic bring-up", .timeLimit(.minutes(1))) +struct DictationSessionBringUpTests { + @Test("a cancel landing during the bring-up never reaches .recording") + func cancelDuringBringUpSkipsRecording() async throws { + // The regression this guards: the press used to finish the bring-up, claim + // `.recording` and fire the start chime — cueing "speak now" for a capture + // the user had already cancelled — and only then get torn down by the queued + // cancel. `.recording` must never be published on this path, because + // `RecordingCueGate` chimes on exactly that edge. + let mic = GatedStartMic() + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), + injector: StubInjector(), seams: .offline) + + let stream = await session.phaseStream() + let pressed = Task { await session.press() } + await mic.waitUntilStartEntered() // press() is suspended inside mic.start() + // Direct `cancel()`, not `submit(.cancel)`: `submit`'s consumer is serial, so + // a submitted cancel cannot even be *recorded* until the press returns. See + // the note in `performPress` — this covers callers that can `await`. + let cancelled = Task { await session.cancel() } + await session.awaitCancelRequest() + await mic.allowStartToFinish() + await pressed.value + await cancelled.value + + var seen: [PipelinePhase] = [] + for await phase in stream { + seen.append(phase) + if phase.isTerminal, phase != .idle { break } + } + + #expect(seen == [.idle, .connecting, .cancelled]) + // Spelled out separately so a failure names the actual regression rather + // than just an unequal array. + #expect(!seen.contains(.recording)) + // The mic came up during the wait, so it has to have been torn down — and + // through the discarding teardown, not `stop()`. + #expect(await mic.cancelCaptureCalls == 1) + } + + @Test("a submitted cancel preempts the bring-up instead of queueing behind it") + func submittedCancelPreemptsBringUp() async throws { + // The app's cancel door is `submit`, and its consumer is serial — so a + // `.cancel` submitted during a press cannot reach `cancel()` until that press + // returns. With `MicCapture.start()` holding until the mic is live, that made + // Escape invisible for up to `MicLiveness.bluetoothTimeout`. `submit` now + // records the intent and cancels the in-flight press itself, without waiting + // for a turn on an actor the press is holding. + let mic = GatedStartMic() + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), + injector: StubInjector(), seams: .offline) + + let stream = await session.phaseStream() + session.submit(.press) + await mic.waitUntilStartEntered() // the consumer is now blocked inside this press + + session.submit(.cancel) + // Both effects have to be observable *before* the press is released, which is + // the whole point — neither needs the consumer to get a turn. + #expect(session.cancelRequested) + #expect(session.inFlightPress?.isCancelled == true) + + await mic.allowStartToFinish() + + var seen: [PipelinePhase] = [] + for await phase in stream { + seen.append(phase) + if phase.isTerminal, phase != .idle { break } + } + #expect(seen.last == .cancelled) + #expect(!seen.contains(.recording)) + #expect(await mic.cancelCaptureCalls == 1) + } + + @Test("a cancel that aborts the bring-up doesn't leak its request into the next press") + func abortedBringUpLeavesNoStandingCancel() async throws { + // The route where `start()` actually throws — a cancel landing *inside* the + // liveness wait, which is what the real `MicCapture` does. `cancel()`'s + // `.connecting` branch claims the phase and returns without enqueueing + // `performCancel`, so the press's catch is the only place left that can + // consume the request it recorded. When it didn't, the flag survived, and the + // *next* press read it after a perfectly good `mic.start()` and cancelled + // itself — one dead dictation for every cancelled bring-up. + let mic = GatedStartMic() + await mic.setThrowsIfCancelled(true) + let session = DictationSession( + mic: mic, transcriber: StubTranscriber(mode: .transcript("Hello world.")), + injector: StubInjector(), seams: .offline) + + let pressed = Task { await session.press() } + await mic.waitUntilStartEntered() + await session.cancel() + await mic.allowStartToFinish() + await pressed.value + + #expect(await session.phase == .cancelled) + #expect(!session.cancelRequested) + + // The proof that matters: the next press records normally. + await session.press() + #expect(await session.phase == .recording) + } +} diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index 017b2d12..dcb2f250 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -302,115 +302,6 @@ extension DictationSessionTests { // Guard/no-op behaviors and phase-stream supersession live in // `DictationSessionGuardTests.swift` (same collaborators and stubs), split out // to stay within the lint file-length budget. The `onTranscriptDelivered` -// side-channel tests live in `DictationSessionTranscriptTests.swift` for the +// side-channel tests live in `DictationSessionTranscriptTests.swift`, and the +// `.connecting` bring-up window in `DictationSessionBringUpTests.swift`, for the // same reason. - -/// Behavior in the `.connecting` window — the up-to-2.5 s stretch -/// `MicCapture.start()` holds while a Bluetooth route brings the mic up. Before -/// the liveness gate, `start()` returned in microseconds and nothing could land -/// mid-press; now things can, so pin what happens when they do. -@Suite("DictationSession mic bring-up", .timeLimit(.minutes(1))) -struct DictationSessionBringUpTests { - @Test("a cancel landing during the bring-up never reaches .recording") - func cancelDuringBringUpSkipsRecording() async throws { - // The regression this guards: the press used to finish the bring-up, claim - // `.recording` and fire the start chime — cueing "speak now" for a capture - // the user had already cancelled — and only then get torn down by the queued - // cancel. `.recording` must never be published on this path, because - // `RecordingCueGate` chimes on exactly that edge. - let mic = GatedStartMic() - let session = DictationSession( - mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), - injector: StubInjector(), seams: .offline) - - let stream = await session.phaseStream() - let pressed = Task { await session.press() } - await mic.waitUntilStartEntered() // press() is suspended inside mic.start() - // Direct `cancel()`, not `submit(.cancel)`: `submit`'s consumer is serial, so - // a submitted cancel cannot even be *recorded* until the press returns. See - // the note in `performPress` — this covers callers that can `await`. - let cancelled = Task { await session.cancel() } - await session.awaitCancelRequest() - await mic.allowStartToFinish() - await pressed.value - await cancelled.value - - var seen: [PipelinePhase] = [] - for await phase in stream { - seen.append(phase) - if phase.isTerminal, phase != .idle { break } - } - - #expect(seen == [.idle, .connecting, .cancelled]) - // Spelled out separately so a failure names the actual regression rather - // than just an unequal array. - #expect(!seen.contains(.recording)) - // The mic came up during the wait, so it has to have been torn down — and - // through the discarding teardown, not `stop()`. - #expect(await mic.cancelCaptureCalls == 1) - } - - @Test("a submitted cancel preempts the bring-up instead of queueing behind it") - func submittedCancelPreemptsBringUp() async throws { - // The app's cancel door is `submit`, and its consumer is serial — so a - // `.cancel` submitted during a press cannot reach `cancel()` until that press - // returns. With `MicCapture.start()` holding until the mic is live, that made - // Escape invisible for up to `MicLiveness.bluetoothTimeout`. `submit` now - // records the intent and cancels the in-flight press itself, without waiting - // for a turn on an actor the press is holding. - let mic = GatedStartMic() - let session = DictationSession( - mic: mic, transcriber: StubTranscriber(mode: .transcript("never")), - injector: StubInjector(), seams: .offline) - - let stream = await session.phaseStream() - session.submit(.press) - await mic.waitUntilStartEntered() // the consumer is now blocked inside this press - - session.submit(.cancel) - // Both effects have to be observable *before* the press is released, which is - // the whole point — neither needs the consumer to get a turn. - #expect(session.cancelRequested) - #expect(session.inFlightPress?.isCancelled == true) - - await mic.allowStartToFinish() - - var seen: [PipelinePhase] = [] - for await phase in stream { - seen.append(phase) - if phase.isTerminal, phase != .idle { break } - } - #expect(seen.last == .cancelled) - #expect(!seen.contains(.recording)) - #expect(await mic.cancelCaptureCalls == 1) - } - - @Test("a cancel that aborts the bring-up doesn't leak its request into the next press") - func abortedBringUpLeavesNoStandingCancel() async throws { - // The route where `start()` actually throws — a cancel landing *inside* the - // liveness wait, which is what the real `MicCapture` does. `cancel()`'s - // `.connecting` branch claims the phase and returns without enqueueing - // `performCancel`, so the press's catch is the only place left that can - // consume the request it recorded. When it didn't, the flag survived, and the - // *next* press read it after a perfectly good `mic.start()` and cancelled - // itself — one dead dictation for every cancelled bring-up. - let mic = GatedStartMic() - await mic.setThrowsIfCancelled(true) - let session = DictationSession( - mic: mic, transcriber: StubTranscriber(mode: .transcript("Hello world.")), - injector: StubInjector(), seams: .offline) - - let pressed = Task { await session.press() } - await mic.waitUntilStartEntered() - await session.cancel() - await mic.allowStartToFinish() - await pressed.value - - #expect(await session.phase == .cancelled) - #expect(!session.cancelRequested) - - // The proof that matters: the next press records normally. - await session.press() - #expect(await session.phase == .recording) - } -} From 59b5a1817c8a4d13aa72743e3a7b463c824b04c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 20:20:43 +0000 Subject: [PATCH 12/14] Drop the now-unused os import from DictationSession.swift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The only `os` symbols the file used were `performPress`'s signpost calls, and those moved to `+Press.swift` when the press half was split out for the lint file-length budget. `OSSignposter` itself is declared in `+Observation.swift`, which imports `os` for it — so nothing here needs the module any more, and `swiftlint analyze` (the unused_import analyzer rule) failed the run on it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- Sources/BlurtEngine/Pipeline/DictationSession.swift | 1 - 1 file changed, 1 deletion(-) diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index 3ca55fd0..de5c56b9 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -1,6 +1,5 @@ import Foundation import Synchronization -import os public actor DictationSession { /// Off-pool home for the press-time AX field read — see its use in From 0f86058dbe8f57d1960d34499ca87c64215c0e39 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 21:43:14 +0000 Subject: [PATCH 13/14] Add engine tests covering the new warm-recorder lifecycle MicCapture+Warm.swift is gate-counted (the coverage exclusion matches only Audio/MicCapture.swift itself) but had no tests, which is what holds the branch at 87.37% against the 88% floor. Cover the warm lifecycle's decisions off-hardware: prepare-once, the bring-up-window refusal, device-identity validation before reuse, the stale-expiry generation ticket, and the scheduled re-warm. Device identity is driven through an injected input snapshot so the test machine's real routing never decides a test. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CNFeP9D8k1HJ1piwyidUv3 --- .../MicCaptureWarmTests.swift | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 Tests/BlurtEngineTests/MicCaptureWarmTests.swift diff --git a/Tests/BlurtEngineTests/MicCaptureWarmTests.swift b/Tests/BlurtEngineTests/MicCaptureWarmTests.swift new file mode 100644 index 00000000..4fb6dc3f --- /dev/null +++ b/Tests/BlurtEngineTests/MicCaptureWarmTests.swift @@ -0,0 +1,171 @@ +import Testing + +@testable import BlurtEngine + +/// The warm-recorder lifecycle (`MicCapture+Warm`): prepare ahead of the press, +/// refuse to double-prepare, validate against the live input before reuse, and +/// tear down on a stale expiry ticket. +/// +/// Runs without hardware: none of this begins capture — `warmUp()` only +/// constructs and prepares a file-backed recorder, so unlike the capture +/// lifecycle in `MicCapture.swift` (excluded from the coverage gate, exercised +/// by the env-gated `MicCaptureLevelsTests`) the decisions here are reachable in +/// CI. The device-identity checks are driven through +/// `installWarmRecorder(boundTo:)` below rather than `warmUp()` itself, so what +/// the test machine's `AudioRoute.currentInput()` answers never decides a test. +@Suite("MicCapture warm recorder", .timeLimit(.minutes(1))) +struct MicCaptureWarmTests { + private let builtIn = AudioRoute.InputSnapshot(deviceID: 7, transportType: nil) + private let airPods = AudioRoute.InputSnapshot(deviceID: 8, transportType: nil) + + @Test("warmUp prepares a recorder once; a second call has been overtaken and no-ops") + func warmUpPreparesOnce() async throws { + let mic = MicCapture() + #expect(await mic.canPrepareWarmRecorder) + + await mic.warmUp() + #expect(await mic.hasWarmRecorder) + // The slot is taken, so it is no longer safe to open the input for another. + #expect(await mic.canPrepareWarmRecorder == false) + + // A second warm-up — e.g. a scheduled re-warm that lost its race with a + // launch-time warmUp — must not stack a second open recorder onto the input. + let generation = await mic.preparedGeneration + await mic.warmUp() + #expect(await mic.preparedGeneration == generation) + + await mic.discardWarmRecorder() + } + + @Test("warmUp is refused across the bring-up window, when both recorder slots are nil") + func warmUpRefusedDuringBringUp() async throws { + // The regression `bringingUpCapture` exists for: across `start()`'s liveness + // wait both `activeRecorder` and `warm` are nil, so a guard reading only + // those would call a live capture "idle" and prepare a second recorder onto + // the already-open input. + let mic = MicCapture() + await mic.setBringingUpCapture(true) + await mic.warmUp() + #expect(await mic.hasWarmRecorder == false) + + await mic.setBringingUpCapture(false) + await mic.warmUp() + #expect(await mic.hasWarmRecorder) + + await mic.discardWarmRecorder() + } + + @Test("a warm recorder is reused only while still bound to the live default input") + func warmRecorderReusedWhenDeviceUnchanged() async throws { + let mic = MicCapture() + try await mic.installWarmRecorder(boundTo: builtIn) + + #expect(await mic.takeWarm(matching: builtIn)) + // Consumed either way — the take empties the slot, so the next press can't + // double-dip and the re-warm guard sees it as free again. + #expect(await mic.hasWarmRecorder == false) + #expect(await mic.canPrepareWarmRecorder) + + // Nothing held: the take answers nil rather than conjuring a recorder. + #expect(await mic.takeWarm(matching: builtIn) == false) + } + + @Test("a device change — or an unreadable route on either side — discards the warm recorder") + func warmRecorderDiscardedOnDeviceChangeOrUnknown() async throws { + // Reuse requires *positively* confirming the device is unchanged. A recorder + // bound to the wrong device doesn't fail loudly — it records the wrong mic, + // or silence — so unknown means discard: paying route activation again is + // the cheaper mistake. + let mic = MicCapture() + + // The user connected their AirPods after the warm-up. + try await mic.installWarmRecorder(boundTo: builtIn) + #expect(await mic.takeWarm(matching: airPods) == false) + #expect(await mic.hasWarmRecorder == false) + + // The route was unreadable when the recorder was warmed. + try await mic.installWarmRecorder(boundTo: nil) + #expect(await mic.takeWarm(matching: builtIn) == false) + + // The route is unreadable now, at press time. + try await mic.installWarmRecorder(boundTo: builtIn) + #expect(await mic.takeWarm(matching: nil) == false) + } + + @Test("an expiry with a stale generation ticket leaves a later warm recorder alone") + func staleExpiryTicketDoesNothing() async throws { + // Cancellation alone doesn't cover this: an expiry already past its + // cancellation check still gets its actor turn, and without the ticket it + // would tear down a recorder prepared a moment ago. + let mic = MicCapture() + try await mic.installWarmRecorder(boundTo: builtIn) + let generation = await mic.preparedGeneration + + await mic.releasePreparedRecorder(generation: generation - 1) + #expect(await mic.hasWarmRecorder) + + // The live ticket is what tears the idle recorder down, freeing the input. + await mic.releasePreparedRecorder(generation: generation) + #expect(await mic.hasWarmRecorder == false) + } + + @Test("a scheduled re-warm eventually prepares a recorder on its own turn") + func scheduledRewarmPreparesARecorder() async throws { + // `stop()`/`cancelCapture()` schedule rather than prepare inline because + // preparing re-opens the input — the slow part — and both sit on paths the + // user is waiting behind. All this can pin deterministically is the other + // half of that contract: the scheduled task does land, and prepares. + // Condition-waited rather than yield-budgeted (see `awaitCancelRequest`); + // the suite's time limit turns a re-warm that never lands into a failure. + let mic = MicCapture() + let before = await mic.preparedGeneration + await mic.scheduleRewarm() + while await mic.preparedGeneration == before { await Task.yield() } + #expect(await mic.hasWarmRecorder) + + await mic.discardWarmRecorder() + } +} + +/// Actor-isolated test seams over `MicCapture`'s internal warm-recorder state. +/// Extensions because `AVAudioRecorder` is not `Sendable`, so neither the `warm` +/// slot nor `takeWarmRecorder`'s return can cross the actor boundary into a +/// test — each helper reduces it to a `Sendable` answer on the actor instead. +extension MicCapture { + /// Whether a warm recorder is currently held. + var hasWarmRecorder: Bool { warm != nil } + + /// Opens or closes the bring-up window `canPrepareWarmRecorder` guards on, + /// standing in for a `start()` suspended in its liveness wait (which needs + /// real hardware to reach). + func setBringingUpCapture(_ value: Bool) { + bringingUpCapture = value + } + + /// Installs a warm recorder bound to a *known* input — `prepareWarmRecorder` + /// with the `AudioRoute.currentInput()` read replaced by `input`, so the + /// device-identity tests don't depend on what the test machine's routing + /// happens to answer. + func installWarmRecorder(boundTo input: AudioRoute.InputSnapshot?) throws { + preparedGeneration += 1 + warm = WarmRecorder( + recorder: try Self.makeRecorder(), input: input, generation: preparedGeneration) + } + + /// Consumes the warm slot through the production validation, reporting whether + /// the recorder was reusable. A reused recorder's temp file is cleaned up here, + /// the job `start()` would otherwise inherit; the discard path already does so. + func takeWarm(matching input: AudioRoute.InputSnapshot?) -> Bool { + guard let recorder = takeWarmRecorder(matching: input) else { return false } + Self.removeFile(at: recorder.url) + return true + } + + /// Test teardown: drops the warm recorder (and its expiry countdown) so a + /// finished test doesn't leave a 60 s expiry task holding the suite's actor. + /// Through `takeWarmRecorder` — which cancels the expiry — rather than a bare + /// `warm = nil`, so teardown can't diverge from how production empties the slot. + func discardWarmRecorder() { + _ = takeWarmRecorder(matching: nil) + } +} From d3787787ab29f18ede0eac62c4dd0bea8293eda6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 22:33:50 +0000 Subject: [PATCH 14/14] Env-gate the warm-recorder tests; exclude MicCapture+Warm from the coverage gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first CI run of these tests didn't just fail — it deadlocked the suite. Three expectations failed (every one that called the real `warmUp()`), and then 171 unrelated tests froze mid-run until the job's 30-minute timeout killed it, with `swiftpm-testing` left as an orphan process. The cause is that every path in these tests reaches `MicCapture.makeRecorder()`, which calls `prepareToRecord()` — the route-activation call this entire change is built around. On a runner with no input device it blocks its thread rather than suspending, so several of these running concurrently drain the cooperative pool and nothing else can make progress. The suite's own `.timeLimit` can't help: the threads it would need are the ones that are stuck. So the suite is gated on BLURT_LIVE_AUDIO_TESTS=1 and tagged `.liveAudio`, exactly like `MicCaptureLevelsTests`, which is env-gated for this same reason. It still documents and locks the warm lifecycle for anyone running it on a Mac with a real microphone. The re-warm test's `Task.yield()` spin is replaced with a deadline-bounded poll that sleeps between reads — a hot spin competes for the thread the task it is waiting on needs, which is its own deadlock. That leaves `MicCapture+Warm.swift` uncovered, so it joins the gate's exclusion list. It belongs there on the merits: it is the capture actor's hardware path, split out of `MicCapture.swift` (already excluded) purely for the lint file-length budget. The exclusion pattern pins a literal filename, so the split silently moved hardware-bound code onto the counted side — the opposite of `+Meter`, which is pure math and stays covered. The transport and liveness policy this file consults remains covered, in `AudioTransport` and `MicLiveness`. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JU1Ln2MKMsF9PJf1FLMQRX --- .../MicCaptureWarmTests.swift | 44 ++++++++++++++----- scripts/check.sh | 17 ++++++- 2 files changed, 49 insertions(+), 12 deletions(-) diff --git a/Tests/BlurtEngineTests/MicCaptureWarmTests.swift b/Tests/BlurtEngineTests/MicCaptureWarmTests.swift index 4fb6dc3f..f63dbfea 100644 --- a/Tests/BlurtEngineTests/MicCaptureWarmTests.swift +++ b/Tests/BlurtEngineTests/MicCaptureWarmTests.swift @@ -1,3 +1,4 @@ +import Foundation import Testing @testable import BlurtEngine @@ -6,14 +7,26 @@ import Testing /// refuse to double-prepare, validate against the live input before reuse, and /// tear down on a stale expiry ticket. /// -/// Runs without hardware: none of this begins capture — `warmUp()` only -/// constructs and prepares a file-backed recorder, so unlike the capture -/// lifecycle in `MicCapture.swift` (excluded from the coverage gate, exercised -/// by the env-gated `MicCaptureLevelsTests`) the decisions here are reachable in -/// CI. The device-identity checks are driven through -/// `installWarmRecorder(boundTo:)` below rather than `warmUp()` itself, so what -/// the test machine's `AudioRoute.currentInput()` answers never decides a test. -@Suite("MicCapture warm recorder", .timeLimit(.minutes(1))) +/// Gated on BLURT_LIVE_AUDIO_TESTS=1, like `MicCaptureLevelsTests`, because every +/// test here goes through `MicCapture.makeRecorder()` — and that calls +/// `prepareToRecord()`, which is *the* route-activation call this whole change is +/// about. On a runner with no input device it is not merely unreliable, it is +/// hostile: it blocks the calling thread rather than suspending, so several of +/// these running concurrently occupy the cooperative pool and wedge the entire +/// `swift test` run, not just this suite. That is not a hypothetical — an +/// ungated first attempt failed three expectations here and then hung 171 +/// unrelated tests until the job's 30-minute timeout killed it. +/// +/// So this suite documents and locks the warm lifecycle for a human running it +/// on a real Mac with a real microphone; `MicCapture+Warm.swift` is excluded +/// from the coverage gate for the same reason `MicCapture.swift` is. +@Suite( + "MicCapture warm recorder (live)", + .enabled( + if: ProcessInfo.processInfo.environment["BLURT_LIVE_AUDIO_TESTS"] == "1", + "set BLURT_LIVE_AUDIO_TESTS=1 to run (needs a real microphone)"), + .tags(.liveAudio), + .timeLimit(.minutes(1))) struct MicCaptureWarmTests { private let builtIn = AudioRoute.InputSnapshot(deviceID: 7, transportType: nil) private let airPods = AudioRoute.InputSnapshot(deviceID: 8, transportType: nil) @@ -115,12 +128,21 @@ struct MicCaptureWarmTests { // preparing re-opens the input — the slow part — and both sit on paths the // user is waiting behind. All this can pin deterministically is the other // half of that contract: the scheduled task does land, and prepares. - // Condition-waited rather than yield-budgeted (see `awaitCancelRequest`); - // the suite's time limit turns a re-warm that never lands into a failure. + // Polled on a deadline with a real sleep between reads, NOT a bare + // `Task.yield()` spin. `scheduleRewarm` hands the work to a child task that + // needs a cooperative thread to run on, and `warmUp()` then blocks that + // thread inside CoreAudio — so a hot spin here competes with the very task + // it is waiting for, and on a machine where the recorder never materialises + // it never terminates at all. Sleeping yields the thread outright, and the + // deadline turns "never landed" into a failed expectation rather than a hang + // the suite's time limit has to clean up. let mic = MicCapture() let before = await mic.preparedGeneration await mic.scheduleRewarm() - while await mic.preparedGeneration == before { await Task.yield() } + let deadline = ContinuousClock().now.advanced(by: .seconds(5)) + while await mic.preparedGeneration == before, ContinuousClock().now < deadline { + try await Task.sleep(for: .milliseconds(10)) + } #expect(await mic.hasWarmRecorder) await mic.discardWarmRecorder() diff --git a/scripts/check.sh b/scripts/check.sh index 26ce117d..a4575b55 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -588,6 +588,21 @@ else # MicCapture+Meter.swift, which IS covered. Keep this # list tight — exclude only code that genuinely cannot # be exercised without hardware. + # - MicCapture+Warm.swift : the same actor's warm-recorder lifecycle, split + # out of MicCapture.swift only for the lint file-length + # budget. Every path through it runs makeRecorder(), + # i.e. prepareToRecord() — the route-activation call, the + # one thing here that genuinely needs a device. Unlike + # +Meter (pure math, covered), splitting this out moved + # hardware-bound code onto the counted side by accident: + # the pattern above pins a literal filename. Trying to + # cover it in CI didn't merely fail, it deadlocked the + # whole test run — prepareToRecord blocks its thread + # instead of suspending, so concurrent attempts drained + # the cooperative pool. Its suite is env-gated alongside + # MicCaptureLevelsTests. The transport and liveness + # *policy* it consults stays covered, in AudioTransport + # and MicLiveness. # - AudioRoute(Monitor).swift : the CoreAudio routing reads (AudioRoute) and the # property listeners (AudioRouteMonitor). Both answer # questions only real hardware can answer — which device @@ -596,7 +611,7 @@ else # can only fire on an actual route change. Same # justification as MicCapture.swift above. COVERAGE="$(xcrun llvm-cov export -summary-only -instr-profile "$PROFDATA" "$XCTEST_BIN" \ - -ignore-filename-regex='Tests/|Audio/MicCapture\.swift|Audio/AudioRoute(Monitor)?\.swift' \ + -ignore-filename-regex='Tests/|Audio/MicCapture(\+Warm)?\.swift|Audio/AudioRoute(Monitor)?\.swift' \ | python3 -c 'import sys,json; print(round(json.load(sys.stdin)["data"][0]["totals"]["lines"]["percent"],2))')" echo "engine line coverage: ${COVERAGE}%" if ! awk -v c="$COVERAGE" -v min="$MIN_COVERAGE" 'BEGIN{ exit (c+0 < min+0) }'; then