diff --git a/.gitignore b/.gitignore index 8fd416e..38c0bca 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ .build/ .swiftpm/ *.xcodeproj/ +*.profraw .DS_Store icon.iconset/ @@ -8,12 +9,5 @@ icon.iconset/ *.apk DoubaoIME-decompiled/ -# Third-party reference projects kept locally for inspiration -BorderBeam/ - # Per-developer signing config (Apple Developer ID, notary profile) Makefile.local - -# Internal docs (reverse-engineering notes, manual test logs) — not for public repo -docs/ -.playwright-mcp/ diff --git a/AGENTS.md b/AGENTS.md index dd9aba1..4dac6b2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,67 +1,66 @@ # Agent instructions — Dousha -## Architecture invariants (READ FIRST for pipeline/concurrency changes) +## Read first Before touching the recording pipeline, any `@unchecked Sendable` type, the ASR clients, or anything that looks like duplicated code worth abstracting, -read `ARCHITECTURE.md`. It records the load-bearing invariants (three-phase -start order, generation guards, flag-before-HUD ordering) and the list of -**deliberately rejected refactors** — do not re-propose those without new -evidence. +read [ARCHITECTURE.md](ARCHITECTURE.md). Its invariants and deliberately rejected +refactors are normative; do not re-propose rejected designs without new evidence. +`CLAUDE.md` is a symlink to this file; keep it that way. ## Testing workflow (REQUIRED) -This is a menu-bar / accessibility app on macOS. You cannot test changes by running `swift build` or `swift run` directly — TCC permissions (Microphone, Accessibility, Speech Recognition) are bound to the signed bundle at `/Applications/Dousha.app`, and global hotkeys only work for the installed bundle the user already granted permissions to. +TCC permissions (Microphone, Accessibility, Speech Recognition) belong to the +signed bundle at `/Applications/Dousha.app`. Test audio, hotkeys, HUD and +permissions through that installed app, never a `swift run` product. -`make build` / `make install` now use `xcodebuild` with full Xcode to compile MLX's Metal shaders and package its resource bundles. `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build` and `swift test` still check compilation and offline logic, but their products cannot run MLX inference without the metallib. For headless real-model checks, build the `smoke-cli` scheme with `xcodebuild` and run the products-dir `smoke-cli mlx-transcribe`; see `ARCHITECTURE.md` §6 for commands. Keep the committed `Package.resolved` and pinned mlx-audio-swift revision together when upgrading dependencies. +`make build` / `make install` use full Xcode's `xcodebuild` to compile MLX Metal +shaders and package resources. For offline logic checks, run: -Every time you want the user to test a change, you must: - -1. **Ask the user to quit the running Dousha** (menu bar → Quit, or `killall Dousha`). `make install` will not replace a running `.app` cleanly, and the user will end up testing the old build without noticing. -2. Run `make install` — this does an Xcode release build, codesigns with the configured Developer ID identity (see below), and copies to `/Applications/Dousha.app`. -3. Launch the new bundle (`open /Applications/Dousha.app`) or ask the user to launch it. -4. Then ask the user to reproduce / test. - -Do **not** report a feature as "ready to test" after only `swift build` or `swift test`. Unit tests are fine for logic, but anything touching audio capture, hotkeys, HUD, or permissions must go through `make install` + relaunch before the user can verify it. - -## Release workflow - -Public releases go to GitHub Releases as a signed + notarized DMG. The pipeline is fully wired in the `Makefile`: - -``` -make release VERSION=0.1.2 +```sh +DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test ``` -This target asserts the version matches `Resources/Info.plist`, requires a clean working tree, builds with hardened runtime + Developer ID, packs a DMG with an `/Applications` symlink, signs the DMG, submits to Apple notary, staples, and `gh release create`s the DMG at `vX.Y.Z`. +`swift build` checks compilation only; neither SwiftPM product supplies the +metallib for inference. See ARCHITECTURE.md §2.1 for MLX smoke commands and +dependency pins, and §6 for pipeline verification. `make install` does not run tests. + +Every time you want the user to test a change, you must: -Per-release checklist: +1. **Ask the user to quit Dousha** (menu bar → Quit, or `killall Dousha`), so they + cannot unknowingly keep running the old build. +2. Run `make install` (release build, signing, copy to `/Applications/Dousha.app`). +3. Launch with `open /Applications/Dousha.app`, or ask the user to launch it. +4. Ask the user to reproduce / test. -1. Bump `Resources/Info.plist` — both `CFBundleShortVersionString` (semver) and `CFBundleVersion` (monotonic integer; notary rejects re-used build numbers for the same short version). -2. Commit + push the bump. -3. Run `make release VERSION=X.Y.Z`. Notarization usually takes 2–5 min; the target blocks until Accepted. +Do not call a feature "ready to test" after only `swift build` or `swift test`. -One-time setup (per dev machine): +## Release workflow -- Apple Developer ID cert must be in the login keychain with its private key. -- Notary credentials stored as a keychain profile: - ``` - xcrun notarytool store-credentials \ - --apple-id --team-id \ - --password - ``` -- Create a `Makefile.local` (gitignored) overriding the public Makefile placeholders: - ``` - DEVELOPER_ID_IDENTITY := Developer ID Application: () - NOTARY_PROFILE := - ``` +1. Bump both `CFBundleShortVersionString` (semver) and `CFBundleVersion` + (monotonic integer) in `Resources/Info.plist`. +2. Commit and push the bump; the release target requires a clean tree and matching version. +3. Run `make release VERSION=X.Y.Z`. It builds with hardened runtime, notarizes + and staples the app, packs and signs the DMG, notarizes and staples the DMG, + publishes GitHub release `vX.Y.Z`, then updates the Homebrew cask. -### Release gotchas — do not regress +## Signing setup -- **Entitlements are mandatory for the dist build.** Hardened runtime auto-denies mic/camera access unless the relevant entitlement is present. `dist` passes `--entitlements Resources/Dousha.entitlements` (currently grants `com.apple.security.device.audio-input`). Symptom of the regression: TCC shows `kTCCServiceMicrophone com.dousha.app auth_value=2 auth_reason=4` (system_set deny), no prompt ever appears to the user. v0.1.0 shipped without this and was broken; v0.1.1 fixed it. -- The DMG must also be codesigned (not just the .app) — `dist` does this. Notary will reject an unsigned DMG. +Reuse the configured Developer ID Application certificate and private key in the +login keychain. Store notary credentials with `xcrun notarytool store-credentials`. +Each checkout/worktree needs a gitignored `Makefile.local` overriding the placeholders: -## Other notes +```make +DEVELOPER_ID_IDENTITY := Developer ID Application: () +NOTARY_PROFILE := +``` -- Single signing identity for everything: `Developer ID Application: ()`. Both `make install` (local dev) and `make release` (DMG) sign with it. TCC grants persist across both because the csreq is anchored to the team ID, not the cdhash — so a fresh local build and the latest DMG release share Mic / Speech / Accessibility approvals. -- If the Developer ID cert is missing from the keychain (fresh clone / CI), the Makefile falls back to ad-hoc signing and TCC grants reset on every rebuild. Flag this if you see the ad-hoc fallback message. -- `make reset-perms` wipes Mic / SpeechRecognition / Accessibility grants for `com.dousha.app`. Don't run it without asking. +- Local installs and releases use the same identity; the team-anchored TCC + requirement preserves grants across rebuilds. Flag any ad-hoc fallback: + missing configuration/certificate means grants will not persist. +- Preserve `Resources/Dousha.entitlements` with + `com.apple.security.device.audio-input`; hardened runtime otherwise denies the + mic without prompting (the v0.1.0 regression). +- Sign the DMG too, and staple **both** app and DMG: a DMG ticket does not travel + with the extracted app for offline Gatekeeper checks. +- Never run `make reset-perms` without asking; it wipes all three TCC grants. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d396c3b..bb4a660 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -31,30 +31,23 @@ settings Doubao servers; mlx-transcribe with local MLX weights ``` -macOS-only. (A tier-2 Windows shell, `DoushaWin`, existed under QUA-209; it -was removed in 2026-07 when its one user stopped using it — its platform -gates, the hand-rolled `InsecureMD5`, and the `OpusEncoding` seam went with -it. Resurrect from git history if a port is ever needed again.) +macOS-only; retired Windows code is recoverable from git history. Rules: - Lower layers never import higher ones. The three ASR libraries are peers, and the two entry points never import each other. -- `MLXASR` depends on mlx-audio-swift's `MLXAudioSTT`, pinned to revision - `244458d7b61e54ff135073ddc9ed76782bb20758`. Local model support is restricted - to real-time streaming models by product decision; the current catalog is - Qwen3-ASR 0.6B / 1.7B (8-bit), with 0.6B as the default. Do not introduce a - batch-only local model behind the streaming engine contract. +- Local-engine layering, lifecycle and verification are defined in §2.1. - Platform APIs (AppKit) live only in the entry points under `Apps/` and `Tools/`. - Library targets never read `Preferences` (or any app singleton). The app snapshots config at recording start and passes it in - (`SonioxBackend`/`DoubaoBackend` snapshot glossary + language and `MLXBackend` - snapshots the language identifier and glossary in `beginSession`). This keeps mid-recording - Settings changes from mutating a live session, and keeps the libraries testable. + (backends snapshot language + glossary in `beginSession`). Mid-recording + Settings changes must not mutate a live session. - All logging goes through `doushaLog` (ConcurrencySupport). Log strings are part of the observable surface — field debugging greps for them — so don't reword - them casually. + them casually. Log lengths, flags and timings, never user content (transcripts, + prompts, glossary/context or credentials). File logging is disabled under XCTest. ## 2. Recording pipeline data flow @@ -107,17 +100,6 @@ engine is still alive (`ErrorGate`); a primary failure hands the HUD to the secondary immediately. Only all-engines-dead (or capture failure) becomes a fatal `.error` stream event. -`MLXBackend` passes the snapshotted Dousha language identifier (`zh-CN`, -`en-US`, `auto`) unchanged; `MLXStreamingASR` alone maps it to the Qwen3 -language name. Nil, auto and unknown identifiers mean auto-detect. Local -streaming text is a HUD preview; stop transcribes the entire retained recording -with Qwen3's batch API, and that nonempty result becomes the final text. Empty, -failed or overdue batch output falls back to the filtered streaming text. -`MLXBackend` snapshots the enabled, normalized glossary as a short plain-text -vocabulary list for the batch system context. Streaming still cannot receive -context ([giraphant/dousha#53](https://github.com/giraphant/dousha/issues/53)); -downstream `TranscriptCorrector` also continues to apply local corrections. - Fatal ≠ session end during live capture (issue #46): a fatal `.error` arriving while the controller is still `.recording` is latched (`pendingErrorMessage`), NOT transitioned — the mic + WAV don't need the network, so capture runs until @@ -134,23 +116,138 @@ thread reads it), then status + HUD glow, then dispatcher reset, then HUD visibility. Timing constants live as statics on the controller (`cancelTeardownGuard` 0.25s, `injectGreenFlash` 0.25s, `errorAutoReset` 3.0s). +### 2.1 Local MLX engine + +**Layering and input.** `MLXBackend` (`Engine.qwen3`, `PCMCaptureEngine`) adapts +app sessions to `MLXStreamingASR` → `MLXModelStore` → `MLXAudioSTT` from +mlx-audio-swift. Only streaming-capable models belong in this contract: Qwen3-ASR +0.6B / 1.7B (8-bit), default 0.6B. The backend snapshots Dousha's language +identifier and enabled, normalized glossary in `beginSession`; only the library +maps identifiers to Qwen language names (nil/auto/unknown → auto-detect). +Keep the user-selected language for **both** passes: the 2026-10-12 mixed-language +comparison retained a Chinese lead-in with forced Chinese that batch auto lost; +earlier English garbling was streaming-only evidence. Glossary is a short +plain-text vocabulary list for the final batch context only; streaming context +is unavailable ([#53](https://github.com/giraphant/dousha/issues/53)). +`TranscriptCorrector` still applies downstream; `prepareSession` context defaults to empty. + +**Preview and final.** Streaming text previews the HUD. Stop transcribes all +retained audio in one batch `generate` call; nonempty batch text wins, while +empty/failed/overdue output falls back to filtered streaming text. Batch runs +off main, after streaming cleanup, with temperature 0, repetition penalty 1.15 +and 32-token repetition context; upstream splits only very long audio at 1200s +low-energy boundaries. `finalPass: false` is for A/B smoke checks. +`MLXTranscriptState` strips complete Qwen headers from partials/finals, hides +incomplete tails only in auto mode, and preserves a safe confirmed prefix. +Preview/fallback reduce five-or-more consecutive repeats of a 1–4 character +unit (optional punctuation) to four; ordinary 哈哈哈 / 很多很多 stays intact. +Never apply that repetition guard to batch text, but always apply +`QwenTranscriptFilter`, even though upstream already strips auto-language headers. + +**Audio and generations.** `MLXStreamingASR` lifecycle is actor-isolated; +nonisolated `ingest` only appends PCM under the `MLXSessionInput` lock: no +conversion, MLX calls, tasks, logging or actor hops on the tap thread. That lock +protects the drain buffer and append-only full-session copy; prepare/cancel/ +completion clear both. A frozen final-pass PCM copy may survive delivery/cancel +until the non-cancellable batch returns. The detached feeder wakes every 100ms; +conversion, `feedAudio`, final drain and native stop/cancel share a private +serial queue off audio/main threads, so teardown cannot overtake feed. Carry +odd bytes into the next feed; discard only a malformed final odd byte. +Each recording owns fresh setup/final/event-finished channels; stale setup/cleanup +retains captured objects, never a newer lease. `.ended` signals final before +the generation guard. Only cumulative `.displayUpdate` replaces text; never +append confirmed/provisional events again. Partials, errors and exactly one +stop completion dispatch on main and recheck generation; cancel immediately +quarantines input and callbacks. + +**Model ownership.** The store actor coalesces loading/warmup and caches weights; +`QwenLoadedModel` is `@unchecked Sendable` only because a process-wide lease +(shared across separately constructed stores) covers loading/replacement, +prewarm, the whole streaming session and batch. Never overlap streaming and +batch inference. The store never mutates a published model; only its lease +holder uses it. Construction/loading runs off main; caller cancellation does +not cancel shared downloads, and setup retains its lease until loading settles. + +**Preload/prewarm.** Launch preloads after hotkey installation when Qwen3 is +active; routing/menu/model changes use the same fire-and-forget `warmup` path. +Only a preload-owned new load runs one off-main, 1s silent batch (8-token limit) +under the lease and discards output. Cached models and recording-owned loads +never prewarm, even when warmup joins or queues behind them. Failure logs an +error but leaves weights `.ready`; this state means loaded, not prewarm finished. +Recordings wait on the lease, released only after prewarm returns. +`[MLXModelStore] prewarm repoID=… ms=…` records cost. Keep batch-only prewarm: +a separate streaming prewarm on two 2026-10-12 clips showed no useful preview +benefit and roughly doubled prewarm cost. + +**Stop deadlines and shutdown.** Stop allows 30s for setup, 10s for `.ended` +(saving latest display text on timeout), 3s for streaming cleanup, then +`MLXFinalPassDeadline` defaults to `max(5 s, 0.10 × audioSeconds)` for batch +(injectable). The `finalPass` log includes the deadline; batch timing starts at +generation, not cleanup. Deadline expiry delivers fallback exactly once; the detached owner +retains the lease until batch returns and discards late text. Cancel suppresses +completion without releasing the lease early. +Cleanup waits 3s for queued native cancel; a stuck synchronous feed retains +the lease until cancel returns, then event closure gets 3s before consumer +cancellation/release. Upstream cancel has **no native decoder completion +barrier**: event closure does not prove detached GPU work ended. The final-pass +owner must therefore wait for natural `.ended` before cancel/cleanup/batch, +even after the initial 10s timeout. If natural stop exceeds the cleanup +allowance, deliver fallback, abandon batch and retain the lease until natural +stop ends. A stuck decoder can block subsequent sessions; never substitute +cancel's event closure or weaken the lease to allow concurrent inference. + +**Build and smoke verification.** Full Xcode's `xcodebuild` compiles Metal and +places `mlx-swift_Cmlx.bundle` beside the executable; the Makefile copies +resources into the app. SwiftPM builds/tests check compilation/offline logic +but do not supply the inference metallib. Build and run the headless scheme: + +```sh +export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer +xcodebuild -scheme smoke-cli -configuration Release \ + -destination 'platform=macOS,arch=arm64' -derivedDataPath .build/xcode \ + CODE_SIGNING_ALLOWED=NO build +.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ + ~/Library/Caches/Dousha/last_recording.wav --language zh-CN +.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ + ~/Library/Caches/Dousha/last_recording.wav --language auto --realtime +``` + +`WavReader` accepts 16kHz mono s16le WAVs. The harness uses the real lifecycle, +100ms chunks and 10ms sleeps (audio-rate with `--realtime`), downloads the default +model on first use, and accepts compatible repos via `--model`. Options: +`--final-pass on|off` (default on), `--context` (batch only), `--prewarm` (awaits +launch preload). Compare prewarm in fresh processes. `preload_ms` includes +load/prewarm; `model_load_ms` samples open→ready every 20ms, including downloads; +first-partial latency starts at ingest. `stop_to_final_ms` includes drain, +cleanup and batch; `batch_ms` is generation time (elapsed at fallback delivery +on timeout), `deadline_seconds` the allowance, `final_source` batch/stream. +Partials/final text are printed: keep private transcripts out of shared reports. +Async waits service main callbacks; watchdogs are 600s for model load and +max(80s, 48s + batch deadline) for stop. Inspect `finalTimedOut` logs even when +fallback makes the command succeed. + +Keep `Package.resolved` tracked with mlx-audio-swift revision +`244458d7b61e54ff135073ddc9ed76782bb20758` in `Package.swift`. Upgrade it, +mlx-swift, mlx-swift-lm and resolved dependencies together; rerun offline tests +and both language smoke checks. + ## 3. Protocol seams & testing | Seam | Purpose | Faked by | |---|---|---| | `SpeechBackend` | one session = one `AsyncStream`; `stop()` yields exactly one terminal `.final`, `cancel()` finishes with none | `MockSpeechBackend` (RecordingControllerTests) | | `PushCaptureEngine` (+ `PCMCaptureEngine` / `BufferCaptureEngine`) | engines receive pushed audio from the hub; two-phase `beginSession`/`openStream` | `MockBackend` (MultiEngineBackendTests, hub = nil) | -| `MLXBackend` (`Engine.local`, `PCMCaptureEngine`) | adapts prepare / open / ingest / stop / cancel to `MLXStreamingASR`; snapshots the unmapped language identifier and glossary context | injected prepare closure + isolated Preferences (MLXEngineWiringTests) | -| `MLXModelStore` loader + process-wide session lease | loads/caches Qwen3 weights once; serializes local inference sessions | injected failing/suspended loader (MLXASRLifecycleTests); lease tests never allocate MLX arrays | -| `MLXPreparedSession` stream + final-pass runner | exercises real stop/cancel/deadline/lease ownership without a model | fake event stream + suspended runner (MLXFinalPassTests) | +| `MLXBackend` | app adapter (§2.1) | injected prepare closure + isolated Preferences (MLXEngineWiringTests) | +| `MLXModelStore` loader + lease | model ownership (§2.1) | injected loader / `MLXLoadedModel` fake (MLXASRLifecycleTests, MLXModelPrewarmTests); lease tests never allocate MLX arrays | +| `MLXPreparedSession` stream + final-pass runner | stop/cancel/deadline ownership (§2.1) | fake event stream + suspended runner (MLXFinalPassTests) | | `RecordingEnvironment` | struct-of-closures: ALL side effects the controller performs (HUD, inject, refine, scheduling, clock) | closure fakes + virtual clock | | `SettingsActions` | struct-of-closures: system-touching settings effects (login item, dock icon, credentials) | closure fakes | Rules: a new side effect enters through a seam, never through a singleton reached from inside the controller/backends. Tests use `@testable import`, so `internal` visibility is test-reachable — never widen to `public` for tests. -Run `swift test` (not just `swift build`) before committing; the test target is -not built by `make install`. +For code changes, run the full offline suite (§6) before committing; +`make install` does not build the test target. ## 4. Concurrency model — `@unchecked Sendable` invariants @@ -191,67 +288,8 @@ type is used, re-derive the reason or the annotation becomes a lie. - **`AppleSpeechBackend`** — `sessionGen` behind a `Lock`, because three threads touch it: main, the audio-frame path, and Speech framework callbacks. Codes 203/216/1110/301 are benign and intentionally dropped. -- **`MLXStreamingASR` / `MLXSessionInput`** — lifecycle state is actor-isolated; - `ingest` is a nonisolated, lock-only PCM append for the real-time audio path: - no conversion, MLX calls, tasks, logging or actor hops on the tap thread. - The same lock protects both the drain buffer and an append-only full-session - PCM copy; prepare, cancel and completion clear both. Only append happens on - the tap thread. A final-pass request may retain its frozen PCM after delivery - or cancellation until the non-cancellable batch call returns. - The detached feeder wakes every 100ms and schedules PCM conversion + - `feedAudio` on a private serial queue, off both audio and main threads. - Final drain, native `stop` and native `cancel` use that same queue, so they - cannot overtake an in-progress feed. Odd PCM bytes carry into the next feed; - only a malformed final odd byte is discarded. -- **MLX generations and callbacks** — every recording owns fresh setup, - final and event-finished channels; stale setup/cleanup retains its captured - objects, never a newer session's lease. `.ended` signals its final channel - before the generation guard. Only cumulative `.displayUpdate` events replace - the transcript; confirmed/provisional events must not be appended again. - `MLXTranscriptState` strips complete Qwen protocol headers from partials and - finals, hiding incomplete tails only in auto mode and keeping a safe confirmed prefix. - The preview and streaming fallback also shorten five-or-more consecutive - repeats of a 1–4 character unit (with optional punctuation) to four repeats; - ordinary reduplication such as 哈哈哈 / 很多很多 stays untouched. This guard - never transforms batch text. Batch output is still passed through - `QwenTranscriptFilter`, although upstream `parseGeneratedChunk` already strips - auto-language headers. - Partials, errors and the one stop completion dispatch on main and recheck - the generation there; cancel immediately quarantines input and callbacks. -- **`MLXModelStore` / `MLXLoadedModel` (`@unchecked Sendable`)** — the store - actor caches weights and coalesces loading/warmup. A single process-wide - lease gate, shared even by separately constructed stores, covers model load, - replacement, the complete streaming session and the subsequent batch pass. - Streaming cleanup precedes batch generation; never run both at once. - The loaded model is never - mutated by the store after publication and is only used by the lease holder; - model construction/loading runs off main. Cancelled callers do not cancel a - shared download, and setup retains the lease until that load settles. -- **MLX final-pass ownership and deadline** — stop waits up to 30s for setup, - then up to 10s for `.ended` (saving the latest display text on timeout), then - up to 3s for streaming cleanup and up to `finalPassTimeout` (default 30s) - for batch. The batch runs off main with temperature 0, repetition penalty - 1.15 and upstream's 32-token repetition context: a modest discouragement of - loops without aggressively suppressing ordinary repeated words. The full PCM - is passed in one `generate` call (upstream only splits very long audio at its - default 1200s low-energy boundaries). `finalPass: false` disables the pass for - A/B smoke checks; `prepareSession` context defaults to empty for compatibility. - Deadline expiry delivers streaming fallback exactly once; the detached owner - keeps the lease until batch returns, discarding its late text. Cancel during - batch suppresses completion and likewise does not release the lease early. - Final-pass metrics log lengths and timings, never transcript or context text. -- **MLX streaming shutdown** — cleanup waits 3s for queued native cancel; if a - synchronous feed is stuck, it retains the lease until cancel actually returns. - It then waits up to 3s for event-stream closure before cancelling the consumer - and releasing the lease. Upstream `cancel()` has **no native decoder completion - barrier**: event-stream closure is its strongest exposed signal, not proof - that all detached GPU work has completed. Therefore the final-pass owner - waits for natural `.ended` before invoking cancel/cleanup and batch, including - after the initial 10s wait timed out. If natural stop outlasts the cleanup - allowance, deliver fallback, abandon batch and retain the lease until natural - stop actually ends; a stuck upstream decoder can consequently block subsequent - sessions. Do not substitute cancel's event closure for this barrier or weaken - the lease to permit concurrent inference. +- **MLX types** — §2.1 defines actor/lock isolation, generation guards, serial + feed/teardown ordering and the process-wide lease protecting `QwenLoadedModel`. - **`AudioTapHub` (actor)** — the tap closure runs on the audio thread and captures only local snapshots; it never hops back into the actor (real-time safety). The `nonisolated(unsafe) var fed` is safe because the converter @@ -284,10 +322,15 @@ without new evidence. - **`TextInjector` does not switch input sources** (QUA-132, Chromium desync) **and does not restore the clipboard** (matches Superwhisper/Vistaflow UX). Both are decisions, not omissions. -- **`SettingsView` is one struct; panes are not child views.** API keys and - test results live only in `@State` until 保存并测试; `NavigationSplitView` +- **`SettingsView` is one struct; panes are not child views.** All state, + including sheets, lives here. API keys and test results stay in `@State` + until 保存并测试; `NavigationSplitView` destroys child views on pane switch, so child-view panes would silently drop - unsaved edits. + unsaved edits. Sheets follow the same rule: present them from `SettingsView` + and keep their editable state there across dismissal; helpers accept only + values, bindings, and closures, without owning draft state. Use native + `Form` with `.formStyle(.grouped)` and standard controls, an engine list with + option sheets, and row subtitles / ⓘ help instead of footers. No chips/cards. - **`HotkeyMonitor` and `CancelKeyMonitor` stay separate classes.** The shared part (tap lifecycle) is already extracted into `EventTap`; merging the rest couples two independently-fragile event paths. @@ -296,8 +339,10 @@ without new evidence. agents: locality beats file count, and splitting an actor across extension files separates state from the code that guards it. Don't split files for size alone. -- **`MultiEngineBackend`'s helpers stay nested** — they are private to its - orchestration and their doc comments reference its callback ordering. +- **Keep multi-engine routing** even when only one engine is selected; parallel + capture, language selection and partial/error fallback are deliberate features. + `MultiEngineBackend` helpers stay nested: they are private to its orchestration + and their comments reference its callback ordering. ## 6. Directory map & verification discipline @@ -305,75 +350,19 @@ Directories group targets by role: ``` Sources/ - Common/ bottom of the graph — libraries everything imports - ConcurrencySupport/ concurrency utils (no deps) - ASRSupport/ shared ASR domain types + TranscriptFormatter - Engines/ one streaming ASR client per provider - DoubaoASR/ Doubao WS client (protobuf, opus, reconnect, credentials) - SonioxASR/ Soniox WS + async-batch client - MLXASR/ local Qwen3 streaming preview + batch final, model store and session lease - Apps/ - Dousha/ macOS app: AppDelegate wiring, RecordingController, - MultiEngineBackend + AudioTapHub, backends (adapters), - HUD (FloatingHUD*), Settings, hotkey monitors, Preferences - Tools/ developer-facing executables, never shipped - SmokeCLI/ smoke harness — real Doubao servers or real local MLX - weights; never Tests/ (suite must stay offline/deterministic) - SmokeCLISupport/ Doubao file-transcription helper -Tests/DoushaTests/ all unit tests (one target, @testable imports everything; - macOS-only because it imports the app target) -``` - -Builds use Xcode (`xcodebuild`) so MLX's Metal shaders are compiled and the -`mlx-swift_Cmlx.bundle` resource is placed beside the executable (and copied -into the app by the Makefile). `swift build` / `swift test` remain useful for -compilation and offline logic tests, but the SwiftPM build path does not supply -the metallib needed to run MLX inference. Use full Xcode, not CommandLineTools: - -```sh -export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer -swift build -swift test -xcodebuild -list # smoke-cli is the headless executable scheme -xcodebuild -scheme smoke-cli -configuration Release \ - -destination 'platform=macOS,arch=arm64' -derivedDataPath .build/xcode \ - CODE_SIGNING_ALLOWED=NO build -.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ - ~/Library/Caches/Dousha/last_recording.wav --language zh-CN -.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ - ~/Library/Caches/Dousha/last_recording.wav --language auto --realtime -.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ - ~/Library/Caches/Dousha/last_recording.wav --language auto --final-pass off -.build/xcode/Build/Products/Release/smoke-cli mlx-transcribe \ - ~/Library/Caches/Dousha/last_recording.wav --language zh-CN --context 'Vocabulary: Dousha, Qwen3' + Common/ ConcurrencySupport (no deps), ASRSupport + Engines/ DoubaoASR, SonioxASR, MLXASR + Apps/Dousha/ app wiring, capture, orchestration, UI, preferences + Tools/SmokeCLI/ real-server / real-model checks; never shipped + Tools/SmokeCLISupport/ Doubao file-transcription helper +Tests/DoushaTests/ one target, @testable imports everything; macOS-only + (imports app); must stay offline/deterministic ``` -`mlx-transcribe` uses `WavReader` (16kHz mono s16le only) and the actual engine -lifecycle, with 100ms PCM chunks, 10ms inter-chunk sleeps by default, or audio -rate with `--realtime`. It downloads the default model on first use; `--model` -accepts a compatible Hugging Face repo. Model load time includes any download -(open→store ready, sampled every 20ms); first-partial latency starts at first -ingest, and stop latency ends at completion. The command prints partial count, -final length, streaming preview and final transcript; keep private transcripts -out of shared reports. `--final-pass on|off` defaults to on; `--context` supplies -only the final batch system message. `stop_to_final_ms` includes streaming drain, -cleanup and batch; `batch_ms` measures batch only (elapsed time at fallback -delivery on a deadline), and `final_source` is `batch` or `stream`. -It services main-queue callbacks through async waits, with 600s model-load and -80s stop watchdogs. Inspect engine logs for `finalTimedOut` even when a nonempty -fallback makes the command succeed. - -`Package.resolved` is committed and the mlx-audio-swift revision is pinned in -`Package.swift`. Upgrade mlx-audio-swift, mlx-swift, mlx-swift-lm and their -resolved dependencies as one compatibility set; rerun offline tests and both -language smoke checks after an upgrade. - Verification for any pipeline change: -1. `swift test` — full suite, green. -2. `make install` (quit the running app first) + relaunch — see CLAUDE.md for - the TCC/signing rules; a worktree needs `Makefile.local` or signing falls - back to ad-hoc and resets TCC grants. +1. `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test` — full suite. +2. Installed-bundle verification: follow AGENTS.md (quit → install → relaunch). 3. Real-device regression set for engine/network changes: - short dictation round-trip; - 3+ minute dictation with mid-sentence pauses (ping keepalive); @@ -381,26 +370,19 @@ Verification for any pipeline change: log for reconnect outcome); - rapid back-to-back recordings (detached close / concurrent-quota path). -For local-engine inference, also run the Xcode-built `mlx-transcribe` checks -above. They validate file-based inference without touching the installed app; -they do not replace installed-bundle testing of audio, hotkeys, HUD or TCC. +For local inference, also run §2.1’s Xcode-built smoke checks; they do not +replace installed-app testing of audio, hotkeys, HUD or TCC. ## 7. Removed design assets (QUA-263 / QUA-265) `ASRSegmentModel` (QUA-265) and `StreamingTextReconciler` (QUA-263) were -pure, fully-tested logic written ahead of features that never materialized. -They sat unwired with zero production evidence for their differentiators -(`nonstream_result=true`: **0 of 2935** streamed results across two log -files; `pauseBoundary`: no consumer) and were deleted in 2026-07. Resurrect -from git history — or rewrite against the notes below — only when a trigger -fires: - -- Segment model: Doubao `nonstream_result` late revisions observed in - production, a third engine needs a shared segmentation layer, or +unwired speculative logic, deleted in 2026-07. Restore from history only when: + +- Segment model: production Doubao `nonstream_result` late revisions appear + (previously 0/2935 results), a third engine needs shared segmentation, or pause-aware utterance boundaries become a product feature. -- Reconciler: text injection becomes incremental (per-key backspace + - retype). Today `TextInjector` is clipboard+⌘V by design (§5) and the HUD - reveal already keeps the stable prefix stable. +- Reconciler: injection becomes incremental (backspace/retype); today it is + clipboard+⌘V (§5), and HUD reveal already preserves the stable prefix. Decisions worth not re-deriving: @@ -420,6 +402,5 @@ Decisions worth not re-deriving: utterance's VAD final → revision; Soniox growing token tail → partial, utterance text at `` → final. -`TranscriptCorrector` (QUA-264) is unrelated to the above and IS wired — -`RecordingController.handleFinal` applies it once per dictation. The other -two are not, and should not be wired without the triggers above. +`TranscriptCorrector` (QUA-264) remains wired once per dictation in +`RecordingController.handleFinal`; the removed designs require the triggers above. diff --git a/README.md b/README.md index 591bfc5..57704f3 100644 --- a/README.md +++ b/README.md @@ -4,31 +4,39 @@

豆沙 (Dousha)

-A tiny macOS menu-bar app that gives you Doubao's voice dictation **without installing the Doubao IME**. Doubao (豆包) minus its wrapper = Dousha (豆沙) — just the filling, no bun. +A menu-bar dictation app for **macOS 14+ on Apple Silicon**. Speech becomes text wherever you're typing. -Hold a key, talk, release — the transcript pastes into whatever you're typing in. +- Choose **豆包 (Doubao)**, **Soniox**, **Apple Speech**, or **local Qwen3 via MLX**. Qwen3 runs offline after a one-time model download. +- Optionally run engines in parallel and route Chinese, English, and mixed-language results to your preferred engine. +- Refine transcripts with an OpenAI-compatible LLM and a custom prompt. +- Add names and terms to your glossary, and re-transcribe saved recordings from history. Spiritual fork of [SpeechMore](https://github.com/gfreezy/SpeechMore) + [DoubaoASR](https://github.com/gfreezy/DoubaoASR) by [@gfreezy](https://github.com/gfreezy). ## Install -Via [Homebrew](https://brew.sh) (easiest — `brew upgrade` keeps it current): +Via [Homebrew](https://brew.sh): ```sh brew install --cask giraphant/tap/dousha ``` -Or download the DMG from the [latest release](https://github.com/giraphant/dousha/releases/latest), drag to `/Applications`, and grant Microphone + Accessibility when asked. +Or download the DMG from the [latest release](https://github.com/giraphant/dousha/releases/latest) and drag Dousha to `/Applications`. -Or build it yourself (macOS 14+): +To build from source, install full Xcode, quit any running Dousha, then run: ```sh make install +open /Applications/Dousha.app ``` +See [AGENTS.md](AGENTS.md) for signing setup. Grant Microphone and Accessibility when prompted; Apple Speech also needs Speech Recognition permission. + ## Use -Hold **Right Shift**, speak, release. That's it. Open Settings from the menu bar to change the key, switch to toggle mode, or pick a language / engine. +Hold **Right Shift**, speak, release (push-to-talk). In Settings, you can change the key or use toggle mode: press once to start, again to stop. + +Open Settings from the menu bar to choose engines and language routing, download a local model, or configure Soniox and LLM credentials. For fully offline dictation, select only Qwen3 and leave LLM refinement off. Re-transcription results are copied to the clipboard. ## License diff --git a/Sources/Apps/Dousha/AppDelegate.swift b/Sources/Apps/Dousha/AppDelegate.swift index 23c1d79..8345a09 100644 --- a/Sources/Apps/Dousha/AppDelegate.swift +++ b/Sources/Apps/Dousha/AppDelegate.swift @@ -8,6 +8,7 @@ import AVFoundation import ServiceManagement import DoubaoASR import SonioxASR +import MLXASR import ASRSupport import ConcurrencySupport @@ -124,6 +125,10 @@ final class AppDelegate: NSObject, NSApplicationDelegate { _ = refiner.refineLater(text, traceId: traceId) { refined in let pb = NSPasteboard.general let changedWhileRefining = pb.changeCount != originalChangeCount + guard !changedWhileRefining else { + doushaLog("[Dousha] deferred refine skipped reason=clipboard_changed traceId=\(traceId)") + return + } pb.clearContents() let setOK = pb.setString(refined, forType: .string) if setOK { @@ -197,6 +202,7 @@ final class AppDelegate: NSObject, NSApplicationDelegate { if prefs.activeEngines.contains(.doubao) { DoubaoCredentialStore.shared.warmup() } + if prefs.activeEngines.contains(.qwen3) { MLXModelStore.shared.warmup(prefs.qwen3Model) } } /// If the app is relaunched while already running (Finder/Spotlight), bring @@ -389,8 +395,9 @@ final class AppDelegate: NSObject, NSApplicationDelegate { @objc private func handleEngineRoutingChanged() { // Routing slots / primary language changed in Settings. Refresh the menu - // (engine summary, checkmarks, language) and warm Doubao if now active. + // (engine summary, checkmarks, language) and warm active engines. if prefs.activeEngines.contains(.doubao) { DoubaoCredentialStore.shared.warmup() } + if prefs.activeEngines.contains(.qwen3) { MLXModelStore.shared.warmup(prefs.qwen3Model) } menuBar.rebuildMenu() } diff --git a/Sources/Apps/Dousha/SettingsWindow.swift b/Sources/Apps/Dousha/SettingsWindow.swift index 5d2adfd..2feda09 100644 --- a/Sources/Apps/Dousha/SettingsWindow.swift +++ b/Sources/Apps/Dousha/SettingsWindow.swift @@ -62,6 +62,11 @@ private enum SettingsPane: String, CaseIterable, Identifiable, Hashable { } } +private struct SettingsEngineSheet: Identifiable { + let engine: Engine + var id: Engine { engine } +} + struct SettingsView: View { let actions: SettingsActions @@ -106,9 +111,10 @@ struct SettingsView: View { @State private var sonioxStatusIsError: Bool = false @State private var isTestingSoniox: Bool = false - // Card expansion and model-management state belong to this view, so switching - // panes or collapsing a card never discards unsaved credentials/test results. - @State private var expandedEngines: Set = [.qwen3] + // Sheets and panes share this owner: dismissal/navigation must not discard + // unsaved credentials, test results, or model-management state. + @State private var selectedEngineSheet: SettingsEngineSheet? + @State private var presentedHelp: String? @State private var activeEngines: [Engine] = Preferences.shared.activeEngines @State private var qwen3ModelRepoID: String = Preferences.shared.qwen3ModelRepoID @State private var modelInventory: [String: MLXModelInventory.Entry] = [:] @@ -171,6 +177,9 @@ struct SettingsView: View { detail .navigationTitle((selectedPane ?? .general).title) } + .sheet(item: $selectedEngineSheet) { selection in + engineSheet(selection.engine) + } } @ViewBuilder @@ -189,67 +198,46 @@ struct SettingsView: View { private var generalPane: some View { Form { - Section("启动行为") { + Section("启动与显示") { Toggle("开机时启动豆沙", isOn: $launchAtLogin) .onChange(of: launchAtLogin) { _, newValue in applyLaunchAtLogin(newValue) } if !launchAtLoginError.isEmpty { - Text(launchAtLoginError) - .font(.callout) - .foregroundStyle(.red) - .fixedSize(horizontal: false, vertical: true) + statusLabel(launchAtLoginError, isError: true) } - } - - Section("显示") { Toggle("显示程序坞图标", isOn: $showDockIcon) .onChange(of: showDockIcon) { _, newValue in actions.setDockIconVisible(newValue) } - Toggle("显示状态栏图标", isOn: $showMenuBarIcon) - .onChange(of: showMenuBarIcon) { _, newValue in - actions.setMenuBarIconVisible(newValue) - } - Text("两者都隐藏后,可从 Finder 或聚焦搜索重新打开豆沙以再次显示本窗口。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + Toggle(isOn: $showMenuBarIcon) { + Text("显示状态栏图标") + Text("均隐藏后,从 Finder 或聚焦搜索重开豆沙") + } + .onChange(of: showMenuBarIcon) { _, newValue in + actions.setMenuBarIconVisible(newValue) + } } Section("热键") { LabeledContent("触发键") { HStack(spacing: 8) { Text(HotkeyMonitor.displayName(forKeyCode: hotkeyKeyCode)) - .padding(.horizontal, 8) - .padding(.vertical, 4) - .background(RoundedRectangle(cornerRadius: 6).fill(Color.gray.opacity(0.12))) Button(isRecordingHotkey ? "按下修饰键…" : "录制") { toggleRecording() } } } Picker("模式", selection: $hotkeyMode) { - Text("长按说话").tag(HotkeyMode.pushToTalk) - Text("切换").tag(HotkeyMode.toggle) + Text("长按说话(按住录音)").tag(HotkeyMode.pushToTalk) + Text("切换(按一下开始/停止)").tag(HotkeyMode.toggle) } - .pickerStyle(.segmented) .onChange(of: hotkeyMode) { _, newMode in let cfg = HotkeyConfig(keyCode: Preferences.shared.hotkey.keyCode, mode: newMode) Preferences.shared.hotkey = cfg NotificationCenter.default.post(name: .doushaHotkeyConfigChanged, object: nil) } - Text("长按说话:按住时录音;切换:每次按下开始/停止。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) - } - - Section("取消") { - LabeledContent("取消键") { + LabeledContent { HStack(spacing: 8) { Text(cancelHotkey.displayName) - .padding(.horizontal, 8) - .padding(.vertical, 4) - .background(RoundedRectangle(cornerRadius: 6).fill(Color.gray.opacity(0.12))) Button(isRecordingCancelHotkey ? "按任意键…" : "录制") { toggleCancelHotkeyRecording() } Button("关闭") { cancelHotkey = .disabled @@ -257,12 +245,13 @@ struct SettingsView: View { NotificationCenter.default.post(name: .doushaCancelHotkeyConfigChanged, object: nil) } .disabled(!cancelHotkey.isEnabled) + settingsHelp("取消键仅在录音时丢弃当前录音且不转写。", + details: "取消会丢弃当前录音且不转写。仅在录音时生效,其余情况下按键正常透传。") } + } label: { + Text("取消键") + Text("仅在录音时丢弃,不转写") } - Text("取消会丢弃当前录音且不转写。仅在录音时生效,其余情况下按键正常透传。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) } } .formStyle(.grouped) @@ -339,12 +328,11 @@ struct SettingsView: View { @ViewBuilder private func microphoneRow(_ device: AudioInputDevice) -> some View { HStack(spacing: 8) { - Circle() - .fill(device.isDefaultInput ? Color.green : Color.gray.opacity(0.3)) - .frame(width: 8, height: 8) + Image(systemName: "mic") + .foregroundStyle(.secondary) VStack(alignment: .leading, spacing: 2) { Text(device.name) - Text("\(device.transportDescription) · \(device.inputChannelCount) 通道") + Text("\(device.transportDescription) · \(device.inputChannelCount) 通道\(device.isDefaultInput ? " · 系统默认" : "")") .font(.caption) .foregroundStyle(.secondary) } @@ -354,19 +342,24 @@ struct SettingsView: View { } .buttonStyle(.borderless) .help("上移") + .accessibilityLabel("上移 \(device.name)") Button(action: { moveMicrophone(device, delta: 1) }) { Image(systemName: "chevron.down") } .buttonStyle(.borderless) .help("下移") + .accessibilityLabel("下移 \(device.name)") } } private var audioPane: some View { Form { - Section("麦克风") { - Toggle("使用系统默认", isOn: $microphoneUseSystemDefault) - .onChange(of: microphoneUseSystemDefault) { _, _ in persistMicrophoneSelection() } + Section { + Toggle(isOn: $microphoneUseSystemDefault) { + Text("使用系统默认") + Text("关闭后按下方顺序选择,下次录音生效") + } + .onChange(of: microphoneUseSystemDefault) { _, _ in persistMicrophoneSelection() } if orderedMicrophones.isEmpty { Text("未找到可用麦克风。") @@ -378,34 +371,39 @@ struct SettingsView: View { .disabled(microphoneUseSystemDefault) } + if let active = activeMicrophoneName { + LabeledContent("当前", value: active) + } + Button("刷新设备") { refreshMicrophones() } + } header: { HStack { - Button("刷新设备") { refreshMicrophones() } - Spacer() - if let active = activeMicrophoneName { - Text("当前:\(active)") - .font(.callout) - .foregroundStyle(.secondary) - } + Text("麦克风") + settingsHelp("关闭系统默认后,按上方顺序选择麦克风,下一次录音生效。", + details: "关闭「使用系统默认」后,豆沙会按列表顺序尝试麦克风;内建或外接麦克风会优先尝试。下一次录音起生效。") } - Text("关闭「使用系统默认」后,豆沙会按上方顺序尝试麦克风;内建或外接麦克风会优先尝试。下一次录音起生效。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) } - Section("音频和反馈") { - Toggle("录音时静音", isOn: $muteSystemAudioDuringRecording) - .onChange(of: muteSystemAudioDuringRecording) { _, newValue in - Preferences.shared.muteSystemAudioDuringRecording = newValue - } - Toggle("录音时暂停媒体", isOn: $pauseMediaDuringRecording) - .onChange(of: pauseMediaDuringRecording) { _, newValue in - Preferences.shared.pauseMediaDuringRecording = newValue - } - Text("录音时静音会临时静音系统输出并在结束后恢复;暂停媒体会在系统 Now Playing 显示正在播放时暂停,并在录音结束后恢复,未播放时不会启动媒体。少数不发布 Now Playing 状态的应用或网页可能不会被暂停,可同时开启录音时静音作为兜底。系统语音隔离当前不作为可用选项暴露,避免再次切到系统播放/多通道输入。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + Section { + Toggle(isOn: $muteSystemAudioDuringRecording) { + Text("录音时静音") + Text("临时静音系统输出,结束后恢复") + } + .onChange(of: muteSystemAudioDuringRecording) { _, newValue in + Preferences.shared.muteSystemAudioDuringRecording = newValue + } + Toggle(isOn: $pauseMediaDuringRecording) { + Text("录音时暂停媒体") + Text("暂停正在播放的媒体,结束后恢复") + } + .onChange(of: pauseMediaDuringRecording) { _, newValue in + Preferences.shared.pauseMediaDuringRecording = newValue + } + } header: { + HStack { + Text("音频和反馈") + settingsHelp("录音时静音或暂停正在播放的媒体,结束后恢复。", + details: "录音时静音会临时静音系统输出并在结束后恢复;暂停媒体会在系统 Now Playing 显示正在播放时暂停,并在录音结束后恢复,未播放时不会启动媒体。少数不发布 Now Playing 状态的应用或网页可能不会被暂停,可同时开启录音时静音作为兜底。系统语音隔离当前不作为可用选项暴露,避免再次切到系统播放/多通道输入。") + } } } .formStyle(.grouped) @@ -413,11 +411,40 @@ struct SettingsView: View { } private var modelPane: some View { - ScrollView { - modelPaneContent - .padding(20) + Form { + Section { + Picker(selection: $primaryLanguage) { + Text("中文").tag(Language.zh_CN) + Text("英文").tag(Language.en_US) + } label: { + Text("主要语言") + Text("对应引擎提供悬浮窗实时字幕") + } + .onChange(of: primaryLanguage) { _, newValue in + Preferences.shared.language = newValue.rawValue + NotificationCenter.default.post(name: .doushaEngineRoutingChanged, object: nil) + } + Picker("中文", selection: $chineseEngine) { engineOptions() } + .onChange(of: chineseEngine) { _, _ in persistRouting() } + Picker("英文", selection: $englishEngine) { engineOptions() } + .onChange(of: englishEngine) { _, _ in persistRouting() } + Picker("中英混合", selection: $mixedEngine) { engineOptions() } + .onChange(of: mixedEngine) { _, _ in persistRouting() } + } header: { + HStack { + Text("引擎路由") + settingsHelp("用到的引擎会在录音时并行运行,停录后按语言挑选结果。", + details: "按语言把识别结果路由到对应引擎。用到的引擎会在录音时并行运行——例如「中文→豆包、英文→Soniox」就会同时跑两家,停录后按整段语言占比挑一家的结果。三个都选同一个引擎即单引擎、零额外开销。主要语言(菜单「语言」)对应的引擎为主引擎,其实时字幕显示在悬浮窗。") + } + } + Section("引擎") { + engineRow(.qwen3) + engineRow(.soniox) + engineRow(.doubao) + engineRow(.apple) + } } - .background(Color(nsColor: .windowBackgroundColor)) + .formStyle(.grouped) .task { refreshEnginePreferences() await observeModelInventory() @@ -427,133 +454,75 @@ struct SettingsView: View { } } - // Computed fragments only: all bindings and tasks remain owned by SettingsView. - private var modelPaneContent: some View { - VStack(alignment: .leading, spacing: 14) { - routingCard - engineCards + private func engineSummary(_ engine: Engine) -> String { + let summary: String + switch engine { + case .qwen3: + summary = "本地 · \(selectedQwen3Model.displayName)" + case .soniox: + summary = "云端 · \(sonioxMode == .realtime ? "实时" : "高精度(异步)") · \(Preferences.shared.sonioxAPIKey.isEmpty ? "未配置密钥" : "密钥已配置")" + case .doubao: + summary = "云端 · 凭据自动管理" + case .apple: + summary = "本地 · 系统内置 · 语言在菜单中选择" } - .frame(maxWidth: .infinity, alignment: .leading) + return summary + (activeEngines.contains(engine) ? " · 使用中" : "") } - private var engineCards: some View { - VStack(alignment: .leading, spacing: 14) { - engineCard(.qwen3, summary: qwen3Summary) { qwen3Settings } - engineCard(.soniox, summary: "\(sonioxMode == .realtime ? "实时" : "高精度(异步)") · \(Preferences.shared.sonioxAPIKey.isEmpty ? "未配置密钥" : "密钥已配置")") { - sonioxSettings - } - engineCard(.doubao, summary: "实时识别 · 凭据自动管理") { - VStack(alignment: .leading, spacing: 10) { - Button("重置豆包凭据…") { actions.resetDoubaoCredentials() } - Text("删除缓存的 device_id 与 token,下次听写时重新注册。遇到 exceedconcurrentquota 等错误时使用。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) - } - } - engineCard(.apple, summary: "系统内置 · 无需配置") { - Text("使用系统内置语音识别,无需额外配置。语言在菜单「语言」中选择。") - .font(.callout) + private func engineRow(_ engine: Engine) -> some View { + HStack(spacing: 10) { + Image(systemName: engine.symbol) + .foregroundStyle(.secondary) + .accessibilityHidden(true) + VStack(alignment: .leading, spacing: 2) { + Text(engine == .apple ? "Apple 听写" : engine.displayName) + Text(engineSummary(engine)) + .font(.caption) .foregroundStyle(.secondary) .fixedSize(horizontal: false, vertical: true) } - } - .frame(maxWidth: .infinity, alignment: .leading) - } - - private var routingCard: some View { - VStack(alignment: .leading, spacing: 12) { - Text("引擎路由").font(.headline) - Picker("主要语言", selection: $primaryLanguage) { - Text("中文").tag(Language.zh_CN) - Text("英文").tag(Language.en_US) - } - .pickerStyle(.segmented) - .onChange(of: primaryLanguage) { _, newValue in - Preferences.shared.language = newValue.rawValue - NotificationCenter.default.post(name: .doushaEngineRoutingChanged, object: nil) + Spacer() + if engine != .apple { + Button("选项…") { selectedEngineSheet = SettingsEngineSheet(engine: engine) } + .accessibilityLabel("\(engine.displayName),选项") + .accessibilityValue(engineSummary(engine)) } - Picker("中文", selection: $chineseEngine) { engineOptions() } - .onChange(of: chineseEngine) { _, _ in persistRouting() } - Picker("英文", selection: $englishEngine) { engineOptions() } - .onChange(of: englishEngine) { _, _ in persistRouting() } - Picker("中英混合", selection: $mixedEngine) { engineOptions() } - .onChange(of: mixedEngine) { _, _ in persistRouting() } - Text("按语言把识别结果路由到对应引擎。用到的引擎会在录音时并行运行——例如「中文→豆包、英文→Soniox」就会同时跑两家,停录后按整段语言占比挑一家的结果。三个都选同一个引擎即单引擎、零额外开销。主要语言(菜单「语言」)对应的引擎为主引擎,其实时字幕显示在悬浮窗。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) } - .padding(16) - .background(Color(nsColor: .controlBackgroundColor), in: RoundedRectangle(cornerRadius: 12)) - .overlay(RoundedRectangle(cornerRadius: 12).strokeBorder(.quaternary, lineWidth: 1)) + .help(engine == .apple ? "使用系统内置语音识别,无需额外配置。语言在菜单「语言」中选择。" : engineSummary(engine)) } - private func engineCard(_ engine: Engine, summary: String, - @ViewBuilder content: () -> Content) -> some View { - let isExpanded = expandedEngines.contains(engine) - return VStack(alignment: .leading, spacing: 0) { - Button { - if isExpanded { expandedEngines.remove(engine) } - else { expandedEngines.insert(engine) } - } label: { - VStack(alignment: .leading, spacing: 10) { - HStack(spacing: 10) { - Image(systemName: engine.symbol) - .font(.system(size: 19, weight: .medium)) - .foregroundStyle(engine == .qwen3 ? Color.accentColor : .secondary) - .frame(width: 36, height: 36) - .background(.quaternary.opacity(0.5), in: RoundedRectangle(cornerRadius: 9)) - Text(engine.displayName).font(.headline) - capsule(engine.kind) - Spacer(minLength: 8) - Image(systemName: isExpanded ? "chevron.up" : "chevron.down") - .font(.caption.weight(.semibold)) - .foregroundStyle(.secondary) - } - Text(summary) - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) - HStack(spacing: 6) { - ForEach(engine.tags, id: \.self) { capsule($0) } - if activeEngines.contains(engine) { capsule("使用中", active: true) } - } + // Computed fragments only: sheet presentation and every editable binding + // belong to SettingsView, just like the six navigation panes. + private func engineSheet(_ engine: Engine) -> some View { + VStack(spacing: 0) { + Form { + switch engine { + case .qwen3: qwen3Settings + case .soniox: sonioxSettings + case .doubao: doubaoSettings + case .apple: EmptyView() } - .frame(maxWidth: .infinity, alignment: .leading) - .contentShape(Rectangle()) - .padding(16) } - .buttonStyle(.plain) - .accessibilityLabel("\(engine.displayName),\(isExpanded ? "收起" : "展开")设置") - .accessibilityValue(summary) - if isExpanded { - Divider().padding(.horizontal, 16) - content() - .frame(maxWidth: .infinity, alignment: .leading) - .padding(16) + .formStyle(.grouped) + HStack { + Spacer() + Button("完成") { selectedEngineSheet = nil } + .keyboardShortcut(.defaultAction) } + .padding() } - .background(Color(nsColor: .controlBackgroundColor), in: RoundedRectangle(cornerRadius: 12)) - .overlay(RoundedRectangle(cornerRadius: 12).strokeBorder(.quaternary, lineWidth: 1)) - } - - private func capsule(_ title: String, active: Bool = false) -> some View { - Text(title) - .font(.caption) - .foregroundStyle(active ? Color.accentColor : .secondary) - .padding(.horizontal, 8) - .padding(.vertical, 3) - .background(active ? Color.accentColor.opacity(0.1) : Color.secondary.opacity(0.08), in: Capsule()) + .frame(width: 560, height: engine == .doubao ? 240 : 420) } private var sonioxSettings: some View { - VStack(alignment: .leading, spacing: 12) { - Picker("模式", selection: $sonioxMode) { + Section { + Picker(selection: $sonioxMode) { Text("实时").tag(SonioxMode.realtime) Text("高精度(异步)").tag(SonioxMode.async) + } label: { + Text("模式") + Text("自动检测语言,无需手动选择") } - .pickerStyle(.segmented) .onChange(of: sonioxMode) { _, newValue in Preferences.shared.sonioxMode = newValue } @@ -561,33 +530,42 @@ struct SettingsView: View { HStack(spacing: 6) { SecureField("", text: $sonioxAPIKey) .disableAutocorrection(true) - Button("清除") { sonioxAPIKey = "" } + .accessibilityLabel("Soniox API 密钥") + Button("清除", role: .destructive) { sonioxAPIKey = "" } .help("删除已保存的 Soniox API 密钥") } } HStack { Spacer() - testStatusLabel(isTesting: isTestingSoniox, - status: sonioxStatus, - isError: sonioxStatusIsError) Button("保存并测试") { saveAndTestSoniox() } .disabled(isTestingSoniox || sonioxAPIKey.isEmpty) } - Text("Soniox 实时语音转写引擎。自动检测语言,无需选择。在菜单「引擎」中切换到 Soniox 后生效。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + if isTestingSoniox || !sonioxStatus.isEmpty { + statusLabel(isTestingSoniox ? "测试中…" : sonioxStatus, + isError: sonioxStatusIsError, isProgress: isTestingSoniox) + } + } header: { + HStack { + Text("Soniox") + settingsHelp("Soniox 自动检测语言,无需手动选择。", + details: "Soniox 实时语音转写引擎。自动检测语言,无需选择。在菜单「引擎」中切换到 Soniox 后生效。") + } } - .textFieldStyle(.roundedBorder) } - private var selectedQwen3Model: MLXModelDescriptor { - MLXModelCatalog.all.first { $0.repoID == qwen3ModelRepoID } ?? MLXModelCatalog.defaultModel + private var doubaoSettings: some View { + Section("豆包") { + HStack { + Button("重置豆包凭据…", role: .destructive) { actions.resetDoubaoCredentials() } + Spacer() + settingsHelp("重置后,下次听写时自动重新注册凭据。", + details: "删除缓存的 device_id 与 token,下次听写时自动重新注册。遇到 exceedconcurrentquota 等错误时使用。") + } + } } - private var qwen3Summary: String { - let installed = modelInventory.values.filter(\.isInstalled).count - return "模型 \(MLXModelCatalog.all.count) · 已安装 \(installed)/\(MLXModelCatalog.all.count) · 使用中 \(selectedQwen3Model.displayName)" + private var selectedQwen3Model: MLXModelDescriptor { + MLXModelCatalog.all.first { $0.repoID == qwen3ModelRepoID } ?? MLXModelCatalog.defaultModel } private var isModelOperationPending: Bool { @@ -596,24 +574,18 @@ struct SettingsView: View { } private var qwen3Settings: some View { - VStack(alignment: .leading, spacing: 12) { + Section { ForEach(MLXModelCatalog.all, id: \.repoID) { descriptor in - if descriptor != MLXModelCatalog.all.first { Divider() } qwen3ModelRow(descriptor) } - Text("安装会下载并加载模型,首次可能需要几分钟;下载完成后可离线听写。切换模型在下一次录音生效。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) - Text("已加载的模型暂不可卸载。请先使用另一个模型并等待加载完成,或重新打开豆沙后卸载。") - .font(.caption) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) if !modelManagementError.isEmpty { - Text(modelManagementError) - .font(.callout) - .foregroundStyle(.red) - .fixedSize(horizontal: false, vertical: true) + statusLabel(modelManagementError, isError: true) + } + } header: { + HStack { + Text("Qwen3 模型") + settingsHelp("下载后可离线听写,切换模型在下一次录音生效。", + details: "安装会下载并加载模型,首次可能需要几分钟;下载完成后可离线听写。切换模型在下一次录音生效。\n\n已加载的模型暂不可卸载。请先使用另一个模型并等待加载完成,或重新打开豆沙后卸载。") } } } @@ -624,43 +596,49 @@ struct SettingsView: View { let selected = qwen3ModelRepoID == descriptor.repoID let loading = modelStoreState == .loading(repoID: descriptor.repoID) let loaded = modelStoreState == .ready(repoID: descriptor.repoID) + let pending = loading || requestedModelRepoID == descriptor.repoID || uninstallingModelRepoID == descriptor.repoID return VStack(alignment: .leading, spacing: 8) { - HStack(spacing: 8) { - VStack(alignment: .leading, spacing: 5) { - Text(descriptor.displayName).font(.body.weight(.medium)) - Text(modelSizeLabel(descriptor, entry: entry)) - .font(.caption) - .foregroundStyle(.secondary) + HStack(spacing: 12) { + Button { selectModel(descriptor) } label: { + HStack(spacing: 8) { + Image(systemName: selected ? "checkmark.circle" : "circle") + .foregroundStyle(.secondary) + VStack(alignment: .leading, spacing: 2) { + Text(descriptor.displayName) + .foregroundStyle(.primary) + Text(modelSizeLabel(descriptor, entry: entry) + (selected ? " · 使用中" : "")) + .font(.caption) + .foregroundStyle(.secondary) + } + } } + .buttonStyle(.borderless) + .disabled(!installed || selected || isModelOperationPending) + .accessibilityLabel("使用 \(descriptor.displayName)") + .accessibilityValue(selected ? "使用中" : (installed ? "已安装" : "尚未安装")) Spacer(minLength: 4) - if selected { capsule("使用中", active: true) } + if pending { + statusLabel(uninstallingModelRepoID == descriptor.repoID ? "正在删除…" : (loading ? "下载并加载中…" : "等待模型空闲…"), + isProgress: true) + } else if !installed { + Button { installModel(descriptor) } label: { + Image(systemName: "arrow.down.circle") + } + .buttonStyle(.borderless) + .disabled(isModelOperationPending) + .help("下载 \(descriptor.displayName)") + .accessibilityLabel("下载 \(descriptor.displayName)") + } if installed { - Button("卸载") { uninstallModel(descriptor) } + Button("删除", role: .destructive) { uninstallModel(descriptor) } .disabled(isModelOperationPending || loaded) .help(loaded ? "模型仍已加载,请先切换模型或重新打开豆沙" : "删除此模型的本地缓存") - if !selected { - Button("使用") { selectModel(descriptor) } - .disabled(isModelOperationPending) - } - } else { - Button("安装") { installModel(descriptor) } - .disabled(isModelOperationPending) - } - } - if loading || requestedModelRepoID == descriptor.repoID || uninstallingModelRepoID == descriptor.repoID { - HStack(spacing: 8) { - ProgressView().controlSize(.small) - Text(uninstallingModelRepoID == descriptor.repoID ? "正在卸载…" : (loading ? "下载并加载中…" : "等待模型空闲…")) - .font(.caption) - .foregroundStyle(.secondary) + .accessibilityLabel("删除 \(descriptor.displayName)") } } if case .failed(let repoID, let message) = modelStoreState, repoID == descriptor.repoID { HStack(alignment: .top) { - Text("加载失败:\(message)") - .font(.caption) - .foregroundStyle(.red) - .fixedSize(horizontal: false, vertical: true) + statusLabel("加载失败:\(message)", isError: true) Spacer(minLength: 0) Button("重试") { installModel(descriptor) }.disabled(isModelOperationPending) } @@ -742,48 +720,28 @@ struct SettingsView: View { private var glossaryPane: some View { Form { - Section("个性词库") { - Toggle("启用个性词库", isOn: $glossaryEnabled) - .onChange(of: glossaryEnabled) { _, newValue in - Preferences.shared.glossaryEnabled = newValue - } - - // Free-text editor: paste / edit a batch, separated by 顿号 / 逗号 / - // 换行. Parsed into terms on change; we deliberately don't rewrite - // the text while editing, so the cursor never jumps. The editor is - // gated by the toggle; the toggle itself stays enabled. - VStack(alignment: .leading, spacing: 6) { - ZStack(alignment: .topLeading) { - TextEditor(text: $glossaryText) - .font(.body) - .frame(height: 200) - .scrollContentBackground(.hidden) - .padding(4) - .overlay( - RoundedRectangle(cornerRadius: 6) - .stroke(Color(nsColor: .separatorColor)) - ) - .onChange(of: glossaryText) { _, newValue in - Preferences.shared.glossaryTerms = GlossaryText.parse(newValue) - } - if glossaryText.isEmpty { - Text("输入关键词、专业术语…\n示例:布迪厄、哈贝马斯、福柯") - .foregroundStyle(.tertiary) - .padding(.top, 12) - .padding(.leading, 9) - .allowsHitTesting(false) - } + Section { + Toggle(isOn: $glossaryEnabled) { + Text("启用个性词库") + Text("顿号、逗号或换行分隔,下次录音生效") + } + .onChange(of: glossaryEnabled) { _, newValue in + Preferences.shared.glossaryEnabled = newValue + } + // Parse without rewriting the draft so the insertion point stays put. + settingsEditor("个性词库", text: $glossaryText, + placeholder: "输入关键词、专业术语…\n示例:布迪厄、哈贝马斯、福柯") + .onChange(of: glossaryText) { _, newValue in + Preferences.shared.glossaryTerms = GlossaryText.parse(newValue) } - Text("共 \(GlossaryText.parse(glossaryText).count) 个术语") - .font(.caption) - .foregroundStyle(.secondary) + .disabled(!glossaryEnabled) + LabeledContent("术语数量", value: "共 \(GlossaryText.parse(glossaryText).count) 个术语") + } header: { + HStack { + Text("个性词库") + settingsHelp("用顿号、逗号或换行分隔术语,下一次录音生效。", + details: "把常用术语、专有名词加进词库,用顿号、逗号或换行分隔;识别时会优先往这些词上靠,提高准确率。对豆包和 Soniox 引擎都生效,下一次录音起作用。Soniox 用词表(terms)偏置,效果通常更明显。") } - .disabled(!glossaryEnabled) - - Text("把常用术语、专有名词加进词库,用顿号、逗号或换行分隔;识别时会优先往这些词上靠,提高准确率。对豆包和 Soniox 引擎都生效,下一次录音起作用。Soniox 用词表(terms)偏置,效果通常更明显。") - .font(.callout) - .foregroundColor(.secondary) - .fixedSize(horizontal: false, vertical: true) } } .formStyle(.grouped) @@ -793,28 +751,36 @@ struct SettingsView: View { private var enhancePane: some View { Form { - Section("文本格式") { - Picker("引号", selection: $quoteStyle) { + Section { + Picker(selection: $quoteStyle) { Text("中文用「」").tag(TranscriptCorrector.QuoteStyle.corner) Text("中文用“”").tag(TranscriptCorrector.QuoteStyle.curly) Text("不处理").tag(TranscriptCorrector.QuoteStyle.off) + } label: { + Text("引号") + Text("将听写中的直引号配对成规范引号") } .pickerStyle(.menu) .onChange(of: quoteStyle) { _, newValue in Preferences.shared.quoteStyle = newValue } - Text("把听写出来的直引号配对成规范引号。英文始终用 “ ”,撇号写成 don’t;上面这项只决定中文语境用哪套。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + } header: { + HStack { + Text("文本格式") + settingsHelp("将听写中的直引号配对成规范引号。", + details: "把听写出来的直引号配对成规范引号。英文始终用 “ ”,撇号写成 don’t;这项只决定中文语境用哪套。") + } } - Section("润色") { - Toggle("启用润色", isOn: $llmEnabled) - .onChange(of: llmEnabled) { _, newValue in - Preferences.shared.llmEnabled = newValue - NotificationCenter.default.post(name: .doushaLLMEnabledChanged, object: nil) - } + Section { + Toggle(isOn: $llmEnabled) { + Text("启用润色") + Text("通过兼容 OpenAI 的接口清理听写结果") + } + .onChange(of: llmEnabled) { _, newValue in + Preferences.shared.llmEnabled = newValue + NotificationCenter.default.post(name: .doushaLLMEnabledChanged, object: nil) + } TextField("API 地址", text: $baseURL, prompt: Text("https://api.openai.com/v1")) .disableAutocorrection(true) @@ -822,7 +788,8 @@ struct SettingsView: View { HStack(spacing: 6) { SecureField("", text: $apiKey, prompt: Text("sk-…")) .disableAutocorrection(true) - Button("清除") { apiKey = "" } + .accessibilityLabel("润色 API 密钥") + Button("清除", role: .destructive) { apiKey = "" } .help("删除已保存的 API 密钥") } } @@ -839,41 +806,44 @@ struct SettingsView: View { } HStack { Spacer() - testStatusLabel(isTesting: isTesting, - status: status, - isError: statusIsError) Button("保存并测试") { saveAndTestLLM() } .disabled(isTesting || apiKey.isEmpty || baseURL.isEmpty || model.isEmpty) } - Text("使用任意兼容 OpenAI 的对话补全接口来清理听写结果。默认仅修正明显的识别错误;可在下方自定义润色规则。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + if isTesting || !status.isEmpty { + statusLabel(isTesting ? "测试中…" : status, + isError: statusIsError, isProgress: isTesting) + } + } header: { + HStack { + Text("润色") + settingsHelp("使用兼容 OpenAI 的接口,按润色规则清理听写结果。", + details: "使用任意兼容 OpenAI 的对话补全接口来清理听写结果。默认仅修正明显的识别错误;可在下方自定义润色规则。") + } } - Section("润色提示词") { - TextEditor(text: $llmSystemPrompt) - .font(.body) + Section { + settingsEditor("润色提示词", text: $llmSystemPrompt, + placeholder: "输入润色规则…\n留空则使用默认规则。") .disableAutocorrection(true) - .frame(height: 220) - .padding(6) - .background(.background, in: RoundedRectangle(cornerRadius: 6)) - .overlay(RoundedRectangle(cornerRadius: 6).stroke(.quaternary)) - .accessibilityLabel("润色提示词") + LabeledContent { + Text(llmSystemPrompt == savedLLMSystemPrompt ? "已保存" : "未保存") + } label: { + Text("保存状态") + Text("保存后生效;留空使用默认规则") + } HStack { - Button("恢复默认") { llmSystemPrompt = TextRefiner.defaultSystemPrompt } + Button("恢复默认", role: .destructive) { llmSystemPrompt = TextRefiner.defaultSystemPrompt } .disabled(llmSystemPrompt == TextRefiner.defaultSystemPrompt) Spacer() - Text(llmSystemPrompt == savedLLMSystemPrompt ? "已保存" : "未保存") - .font(.callout) - .foregroundStyle(.secondary) Button("保存提示词") { saveLLMPrompt() } .disabled(llmSystemPrompt == savedLLMSystemPrompt) } - Text("自定义提示词会完整替换默认规则,可允许清理重复、修复断句或轻度润色。建议要求模型只输出处理后的正文。修改后点击「保存提示词」,后续润色请求生效;留空则使用默认规则。") - .font(.callout) - .foregroundStyle(.secondary) - .fixedSize(horizontal: false, vertical: true) + } header: { + HStack { + Text("润色提示词") + settingsHelp("保存后用于后续润色,留空则使用默认规则。", + details: "自定义提示词会完整替换默认规则,可允许清理重复、修复断句或轻度润色。建议要求模型只输出处理后的正文。修改后点击「保存提示词」,后续润色请求生效;留空则使用默认规则。") + } } } .formStyle(.grouped) @@ -884,18 +854,20 @@ struct SettingsView: View { private var historyPane: some View { Form { Section { - Stepper("保留最近 \(historyLimit) 条录音", value: $historyLimit, in: 1...50) - .onChange(of: historyLimit) { _, newValue in - Preferences.shared.historyLimit = newValue - history.prune(limit: newValue) - } - Text("每次听写的原始录音和转录文本保存在本机缓存目录,超出条数自动删除最旧的。") - .font(.caption) - .foregroundColor(.secondary) + Stepper(value: $historyLimit, in: 1...50) { + Text("保留最近 \(historyLimit) 条录音") + Text("本机缓存原始录音与文本,超出后删最旧的") + } + .onChange(of: historyLimit) { _, newValue in + Preferences.shared.historyLimit = newValue + history.prune(limit: newValue) + } + } header: { + Text("存储") } Section("最近录音") { if historyEntries.isEmpty { - Text("暂无录音").foregroundColor(.secondary) + Text("暂无录音").foregroundStyle(.secondary) } ForEach(historyEntries) { entry in historyRow(entry) @@ -926,20 +898,17 @@ struct SettingsView: View { HStack(spacing: 8) { VStack(alignment: .leading, spacing: 2) { HStack(spacing: 6) { - Text(entry.date, format: .dateTime.month().day().hour().minute()) - Text(Self.durationLabel(entry.duration)).foregroundColor(.secondary) - if !entry.engine.isEmpty { - Text(entry.engine).foregroundColor(.secondary) - } - if let err = entry.error { - Text("转录失败:\(err)").foregroundColor(.red).lineLimit(1) + Text(entry.transcript.isEmpty ? "(无文本)" : entry.transcript) + .lineLimit(1) + .truncationMode(.tail) + .foregroundStyle(entry.transcript.isEmpty ? .secondary : .primary) + if let error = entry.error { + statusLabel("转录失败:\(error)", isError: true, symbolOnly: true) } } - .font(.caption) - Text(entry.transcript.isEmpty ? "(无文本)" : entry.transcript) - .lineLimit(1) - .truncationMode(.tail) - .foregroundColor(entry.transcript.isEmpty ? .secondary : .primary) + Text(historyMetadata(entry)) + .font(.caption) + .foregroundStyle(.secondary) } Spacer() Button("重新转录") { actions.retranscribe(entry.id) } @@ -951,17 +920,28 @@ struct SettingsView: View { } label: { Image(systemName: "doc.on.doc") } + .buttonStyle(.borderless) .help("复制文本") + .accessibilityLabel("复制文本") .disabled(entry.transcript.isEmpty) Button { NSWorkspace.shared.activateFileViewerSelecting([history.wavURL(id: entry.id)]) } label: { Image(systemName: "folder") } + .buttonStyle(.borderless) .help("在 Finder 中显示") + .accessibilityLabel("在 Finder 中显示") } } + private func historyMetadata(_ entry: RecordingHistoryEntry) -> String { + var components = [entry.date.formatted(.dateTime.month().day().hour().minute()), + Self.durationLabel(entry.duration)] + if !entry.engine.isEmpty { components.append(entry.engine) } + return components.joined(separator: " · ") + } + private static func durationLabel(_ d: TimeInterval) -> String { let s = Int(d.rounded()) return String(format: "%d:%02d", s / 60, s % 60) @@ -988,24 +968,71 @@ struct SettingsView: View { // MARK: - 润色 / Soniox 操作 - /// Persistent, single-line test result shown beside the 「保存并测试」 button: - /// a spinner while testing, then a coloured icon + short text. - @ViewBuilder - private func testStatusLabel(isTesting: Bool, status: String, isError: Bool) -> some View { - if isTesting { - HStack(spacing: 4) { + // MARK: - Shared native presentation + + /// All result/progress text stays secondary; only the status symbol is coloured. + private func statusLabel(_ message: String, isError: Bool = false, + isProgress: Bool = false, symbolOnly: Bool = false) -> some View { + HStack(alignment: .firstTextBaseline, spacing: 6) { + if isProgress { ProgressView().controlSize(.small) - Text("测试中…").foregroundStyle(.secondary) + } else { + Image(systemName: isError ? "exclamationmark.triangle.fill" : "checkmark.circle.fill") + .foregroundStyle(isError ? .orange : .green) + } + if !symbolOnly { + Text(message) + .foregroundStyle(.secondary) + .fixedSize(horizontal: false, vertical: true) } - .font(.callout) - } else if !status.isEmpty { - HStack(spacing: 4) { - Image(systemName: isError ? "xmark.circle.fill" : "checkmark.circle.fill") - Text(status).lineLimit(1) + } + .help(message) + .accessibilityElement(children: .ignore) + .accessibilityLabel(message) + } + + /// Both editors share the form row's surface and an appearance-aware hairline. + private func settingsEditor(_ title: String, text: Binding, placeholder: String) -> some View { + ZStack(alignment: .topLeading) { + TextEditor(text: text) + .font(.body) + .scrollContentBackground(.hidden) + .frame(height: 200) + .accessibilityLabel(title) + if text.wrappedValue.isEmpty { + Text(placeholder) + .foregroundStyle(.tertiary) + .padding(.top, 8) + .padding(.leading, 5) + .allowsHitTesting(false) + .accessibilityHidden(true) } - .font(.callout) - .foregroundStyle(isError ? .red : .green) - .help(status) + } + .overlay { + RoundedRectangle(cornerRadius: 4) + .strokeBorder(Color(nsColor: .separatorColor), lineWidth: 0.5) + .allowsHitTesting(false) + .accessibilityHidden(true) + } + } + + /// The popover's presentation lives here too, so helper fragments have no state. + private func settingsHelp(_ summary: String, details: String) -> some View { + Button { presentedHelp = details } label: { + Image(systemName: "info.circle") + } + .buttonStyle(.borderless) + .accessibilityLabel("更多说明:\(summary)") + .help(details) + .popover(isPresented: Binding( + get: { presentedHelp == details }, + set: { if !$0 { presentedHelp = nil } } + )) { + Text(details) + .font(.body) + .padding() + .frame(width: 340) + .fixedSize(horizontal: false, vertical: true) } } diff --git a/Sources/Apps/Dousha/TextInjector.swift b/Sources/Apps/Dousha/TextInjector.swift index 5b2ef97..79fc451 100644 --- a/Sources/Apps/Dousha/TextInjector.swift +++ b/Sources/Apps/Dousha/TextInjector.swift @@ -30,7 +30,7 @@ final class TextInjector { doushaLog("[Dousha] TextInjector.inject: SKIPPED (empty text)") return } - doushaLog("[Dousha] TextInjector.inject: text=\"\(text.prefix(60))\" len=\(text.count)") + doushaLog("[Dousha] TextInjector.inject: len=\(text.count)") let pasteboard = NSPasteboard.general pasteboard.clearContents() diff --git a/Sources/Common/ConcurrencySupport/Logging.swift b/Sources/Common/ConcurrencySupport/Logging.swift index 765d79b..4c85317 100644 --- a/Sources/Common/ConcurrencySupport/Logging.swift +++ b/Sources/Common/ConcurrencySupport/Logging.swift @@ -17,14 +17,14 @@ private let _doushaLogger = Logger(subsystem: "com.dousha.app", category: "gener /// it never captured a failure either. This file is the durable record. public func doushaLog(_ message: String) { _doushaLogger.log("\(message, privacy: .public)") - DoushaFileLog.shared.append(message) + DoushaFileLog.shared?.append(message) } /// Append-only persistent log at `~/Library/Logs/Dousha/dousha.log`, mirroring /// every `doushaLog` line with a timestamp. Thread-safe via a private serial /// queue; rotates once at ~5 MB so it can't grow unbounded. final class DoushaFileLog: @unchecked Sendable { - static let shared = DoushaFileLog() + static let shared = NSClassFromString("XCTestCase") == nil ? DoushaFileLog() : nil private let queue = DispatchQueue(label: "com.dousha.app.filelog") private let fileURL: URL? diff --git a/Sources/Engines/DoubaoASR/DoubaoASR.swift b/Sources/Engines/DoubaoASR/DoubaoASR.swift index c65c159..b73590a 100644 --- a/Sources/Engines/DoubaoASR/DoubaoASR.swift +++ b/Sources/Engines/DoubaoASR/DoubaoASR.swift @@ -674,8 +674,7 @@ public actor DoubaoASR { if contextHint.isEmpty { doushaLog("[DoubaoASR] StartSession context: (none)") } else { - let preview = contextHint.count > 60 ? String(contextHint.prefix(60)) + "…" : contextHint - doushaLog("[DoubaoASR] StartSession context.len=\(contextHint.count) preview=\(preview)") + doushaLog("[DoubaoASR] StartSession context.len=\(contextHint.count)") } try await sendData(AsrMessageBuilder.startSession(requestId: requestId, token: token, configJSON: configJSON)) let resp2 = try await waitForResponse(timeout: 5.0) { @@ -827,7 +826,7 @@ public actor DoubaoASR { signalFinished() return case "TaskFailed", "SessionFailed": - doushaLog("[DoubaoASR] \(resp.messageType) statusCode=\(resp.statusCode) statusMessage=\(resp.statusMessage) resultJson=\(resp.resultJson)") + doushaLog("[DoubaoASR] \(resp.messageType) statusCode=\(resp.statusCode) statusMessage=\(resp.statusMessage) resultJson.len=\(resp.resultJson.count)") let msg = resp.statusMessage.isEmpty ? "ASR failed (\(resp.statusCode))" : "\(resp.statusMessage) (\(resp.statusCode))" deliverError(NSError(domain: "DoubaoASR", code: Int(resp.statusCode), userInfo: [NSLocalizedDescriptionKey: msg])) @@ -840,9 +839,8 @@ public actor DoubaoASR { guard let update = resultState.ingest(resultJson: resp.resultJson) else { return } self.lastTranscriptAt = Date() - let preview = update.text.prefix(40).replacingOccurrences(of: "\n", with: " ") let stopDelta = stopStartedAt.map { " stop+\(elapsedMs(since: $0))ms" } ?? "" - doushaLog("[DoubaoASR] traceId=\(requestId) t=\(traceElapsedMs())ms\(stopDelta) result isInterim=\(update.isInterim) vadFinished=\(update.vadFinished) nonstream=\(update.nonstreamResult) textLen=\(update.text.count) currentInterimLen=\(update.previousInterimLength) newUtterance=\(update.looksLikeNewUtterance) preview=\(preview)") + doushaLog("[DoubaoASR] traceId=\(requestId) t=\(traceElapsedMs())ms\(stopDelta) result isInterim=\(update.isInterim) vadFinished=\(update.vadFinished) nonstream=\(update.nonstreamResult) textLen=\(update.text.count) currentInterimLen=\(update.previousInterimLength) newUtterance=\(update.looksLikeNewUtterance)") switch update.commit { case .rescued(let rescued): diff --git a/Sources/Engines/MLXASR/MLXModelStore.swift b/Sources/Engines/MLXASR/MLXModelStore.swift index 8ab00b9..3e5b85f 100644 --- a/Sources/Engines/MLXASR/MLXModelStore.swift +++ b/Sources/Engines/MLXASR/MLXModelStore.swift @@ -1,15 +1,43 @@ import Foundation import MLXAudioSTT +import MLX import ConcurrencySupport -/// Safe to transfer after loading: the store never mutates the published model, -/// and only the holder of the process-wide lease creates a streaming session -/// or runs its subsequent full-audio batch pass using it. MLXAudioSTT owns -/// synchronization inside the streaming session. Do not expose -/// the model to callers that bypass the lease (ARCHITECTURE §4). -internal final class MLXLoadedModel: @unchecked Sendable { +/// Offline seam for successful loads and prewarm without allocating MLX arrays. +/// Both operations require the process-wide lease and run off main. +internal protocol MLXLoadedModel: Sendable { + func prepareSession(language: String?) -> MLXPreparedSession + func prewarm() async throws +} + +/// Safe to transfer after loading: only the process-wide lease holder accesses +/// the native model, including prewarm, streaming and the final full-audio pass. +/// MLXAudioSTT owns synchronization inside the streaming session (§4). +private final class QwenLoadedModel: MLXLoadedModel, @unchecked Sendable { let model: Qwen3ASRModel init(_ model: Qwen3ASRModel) { self.model = model } + + func prepareSession(language: String?) -> MLXPreparedSession { + MLXPreparedSession( + stream: StreamingInferenceSession(model: model, config: StreamingConfig(language: language)), + finalPass: { request in + let samples = MLXASRSupport.floatSamples(fromPCM16LE: request.pcm) + guard !samples.isEmpty else { return "" } + return self.model.generate(audio: MLXArray(samples), maxTokens: 8192, + temperature: 0, context: request.context, + language: request.language, chunkDuration: 1200, + minChunkDuration: 1, repetitionPenalty: 1.15, + repetitionContextSize: 32).text + }) + } + + func prewarm() async throws { + // One second of silence exercises encoder + decoder kernels. Bound the + // token count so a silent-input hallucination cannot prolong startup. + _ = model.generate(audio: MLXArray(Array(repeating: Float(0), count: 16_000)), + maxTokens: 8, temperature: 0, language: "Chinese", + repetitionPenalty: 1.15, repetitionContextSize: 32) + } } /// A token can be released repeatedly, including after a cancelled acquire: @@ -75,21 +103,21 @@ public actor MLXModelStore { } public private(set) var state: State = .idle - private var cached: (repoID: String, model: MLXLoadedModel)? - private var inFlight: (repoID: String, task: Task)? + private var cached: (repoID: String, model: any MLXLoadedModel)? + private var inFlight: (repoID: String, task: Task)? private var warming: Set = [] - private let loadModel: @Sendable (String) async throws -> MLXLoadedModel + private let loadModel: @Sendable (String) async throws -> any MLXLoadedModel public init() { loadModel = { repoID in - MLXLoadedModel(try await Qwen3ASRModel.fromPretrained(repoID)) + QwenLoadedModel(try await Qwen3ASRModel.fromPretrained(repoID)) } doushaLog("[MLXModelStore] state=idle") } - /// Offline failure/cancellation tests inject a loader that never touches - /// Hugging Face or Metal. The production API always uses fromPretrained. - internal init(loadModel: @escaping @Sendable (String) async throws -> MLXLoadedModel) { + /// Offline lifecycle/prewarm tests inject a loader and model operations + /// that never touch Hugging Face or Metal. Production uses fromPretrained. + internal init(loadModel: @escaping @Sendable (String) async throws -> any MLXLoadedModel) { self.loadModel = loadModel doushaLog("[MLXModelStore] state=idle") } @@ -103,7 +131,8 @@ public actor MLXModelStore { try await MLXSessionLeaseGate.shared.acquire() } - private func preload(_ descriptor: MLXModelDescriptor) async { + /// Awaitable warmup for the smoke harness; app call sites use fire-and-forget warmup. + public func preload(_ descriptor: MLXModelDescriptor) async { if cached?.repoID == descriptor.repoID { return } // Coalesce warmups even while they are waiting for another session's // lease, including a failed load (no queue of identical retry loads). @@ -118,7 +147,22 @@ public actor MLXModelStore { do { let lease = try await acquireSessionLease() do { - _ = try await load(descriptor, lease: lease) + // Recheck after waiting: a recording may have loaded this model + // first. Never add prewarm to a recording-triggered load. + if cached?.repoID != descriptor.repoID { + let model = try await load(descriptor, lease: lease) + await Task.detached(priority: .userInitiated) { + let started = ProcessInfo.processInfo.systemUptime + do { + try await model.prewarm() + } catch { + // Weights remain usable; prewarm is only an optimization. + doushaLog("[MLXModelStore] prewarmFailed repoID=\(descriptor.repoID) error=\(error.localizedDescription)") + } + let milliseconds = Int((ProcessInfo.processInfo.systemUptime - started) * 1_000) + doushaLog("[MLXModelStore] prewarm repoID=\(descriptor.repoID) ms=\(milliseconds)") + }.value + } } catch { // load() publishes and logs the error; warmup never emits an // engine error or changes an active recording's callbacks. @@ -133,7 +177,7 @@ public actor MLXModelStore { /// Keeping the lease throughout loading prevents a model switch while the /// old session is still draining or running its final batch pass. A cancelled caller does not cancel the /// shared download; its setup path releases the lease after the load settles. - internal func load(_ descriptor: MLXModelDescriptor, lease: MLXSessionLease) async throws -> MLXLoadedModel { + internal func load(_ descriptor: MLXModelDescriptor, lease: MLXSessionLease) async throws -> any MLXLoadedModel { if let cached, cached.repoID == descriptor.repoID { return cached.model } if let inFlight, inFlight.repoID == descriptor.repoID { return try await inFlight.task.value diff --git a/Sources/Engines/MLXASR/MLXStreamingASR.swift b/Sources/Engines/MLXASR/MLXStreamingASR.swift index 007fdda..7731dd1 100644 --- a/Sources/Engines/MLXASR/MLXStreamingASR.swift +++ b/Sources/Engines/MLXASR/MLXStreamingASR.swift @@ -2,7 +2,6 @@ import Foundation import ASRSupport import ConcurrencySupport import MLXAudioSTT -import MLX /// The tap only appends bytes under this lock. Swapping the buffer on drain /// keeps conversion, inference, actor hops, tasks and logging off the audio thread. @@ -145,11 +144,27 @@ internal struct MLXPreparedSession: Sendable { let finalPass: @Sendable (MLXFinalPassRequest) async throws -> String } +/// Batch-only deadline; streaming shutdown retains its separate bounded waits. +public struct MLXFinalPassDeadline: Sendable { + public let baseSeconds: TimeInterval + public let audioRatio: Double + + public init(baseSeconds: TimeInterval = 5, audioRatio: Double = 0.10) { + self.baseSeconds = max(0, baseSeconds) + self.audioRatio = max(0, audioRatio) + } + + public func seconds(forAudioSeconds audioSeconds: Double) -> TimeInterval { + max(baseSeconds, audioRatio * audioSeconds) + } +} + public struct MLXFinalPassMetrics: Sendable { public let source: String /// Elapsed batch time at delivery; on a deadline this is not its eventual runtime. public let batchMilliseconds: Double public let audioSeconds: Double + public let deadlineSeconds: TimeInterval public let streamLength: Int public let finalLength: Int } @@ -254,7 +269,7 @@ public actor MLXStreamingASR { private let model: MLXModelDescriptor private let store: MLXModelStore private let finalPassEnabled: Bool - private let finalPassTimeout: TimeInterval + private let finalPassDeadline: MLXFinalPassDeadline private let streamEndTimeout: TimeInterval private let streamCleanupTimeout: TimeInterval internal typealias SessionFactory = @Sendable (MLXModelDescriptor, MLXModelStore, MLXSessionLease, String?) async throws -> MLXPreparedSession @@ -303,23 +318,23 @@ public actor MLXStreamingASR { private var recording: Recording? public init(model: MLXModelDescriptor = MLXModelCatalog.defaultModel, store: MLXModelStore = .shared, - finalPass: Bool = true, finalPassTimeout: TimeInterval = 30) { + finalPass: Bool = true, finalPassDeadline: MLXFinalPassDeadline = MLXFinalPassDeadline()) { self.model = model self.store = store self.finalPassEnabled = finalPass - self.finalPassTimeout = max(0, finalPassTimeout) + self.finalPassDeadline = finalPassDeadline self.streamEndTimeout = 10 self.streamCleanupTimeout = 3 self.makeSession = Self.makeNativeSession } internal init(store: MLXModelStore = .shared, finalPass: Bool = true, - finalPassTimeout: TimeInterval = 30, streamEndTimeout: TimeInterval = 10, + finalPassDeadline: MLXFinalPassDeadline = MLXFinalPassDeadline(), streamEndTimeout: TimeInterval = 10, streamCleanupTimeout: TimeInterval = 3, makeSession: @escaping SessionFactory) { self.model = MLXModelCatalog.defaultModel self.store = store self.finalPassEnabled = finalPass - self.finalPassTimeout = max(0, finalPassTimeout) + self.finalPassDeadline = finalPassDeadline self.streamEndTimeout = streamEndTimeout self.streamCleanupTimeout = streamCleanupTimeout self.makeSession = makeSession @@ -331,17 +346,7 @@ public actor MLXStreamingASR { try Task.checkCancellation() // Construction and generation both allocate MLX arrays; run off main. return await Task.detached(priority: .userInitiated) { - MLXPreparedSession( - stream: StreamingInferenceSession(model: loaded.model, config: StreamingConfig(language: language)), - finalPass: { request in - let samples = MLXASRSupport.floatSamples(fromPCM16LE: request.pcm) - guard !samples.isEmpty else { return "" } - return loaded.model.generate(audio: MLXArray(samples), maxTokens: 8192, - temperature: 0, context: request.context, - language: request.language, chunkDuration: 1200, - minChunkDuration: 1, repetitionPenalty: 1.15, - repetitionContextSize: 32).text - }) + loaded.prepareSession(language: language) }.value } @@ -505,13 +510,14 @@ public actor MLXStreamingASR { } let pcm = input.fullAudio(generation) let audioSeconds = Double(pcm.count / 2) / 16_000 + let deadlineSeconds = finalPassDeadline.seconds(forAudioSeconds: audioSeconds) let work = startFinalPass(current, pcm: pcm) // Includes up to 3s for streaming cleanup. After a streaming timeout, // natural .ended is still required before batch inference can start. // A detached owner retains the lease if either stage exceeds this wait. let output: WaitOutcome if case .signaled = await work.ready.wait(timeout: streamCleanupTimeout) { - output = await work.result.wait(timeout: max(0, finalPassTimeout - work.elapsedMilliseconds / 1_000)) + output = await work.result.wait(timeout: max(0, deadlineSeconds - work.elapsedMilliseconds / 1_000)) } else { output = .timeout } @@ -519,7 +525,7 @@ public actor MLXStreamingASR { guard isCurrent(current) else { return } let batch: String let milliseconds: Double - if case .signaled(let value) = output, value.milliseconds <= finalPassTimeout * 1_000 { + if case .signaled(let value) = output, value.milliseconds <= deadlineSeconds * 1_000 { batch = value.text milliseconds = value.milliseconds } else { @@ -579,10 +585,11 @@ public actor MLXStreamingASR { private func recordFinalPass(_ current: Recording, text: String, fallback: String, batchMilliseconds: Double, audioSeconds: Double, source: String = "stream") { + let deadlineSeconds = finalPassDeadline.seconds(forAudioSeconds: audioSeconds) finalPassMetrics = MLXFinalPassMetrics(source: source, batchMilliseconds: batchMilliseconds, - audioSeconds: audioSeconds, streamLength: fallback.count, + audioSeconds: audioSeconds, deadlineSeconds: deadlineSeconds, streamLength: fallback.count, finalLength: text.count) - log(current, "finalPass source=\(source) batchMs=\(Int(batchMilliseconds)) audioSeconds=\(audioSeconds) streamLength=\(fallback.count) finalLength=\(text.count)") + log(current, "finalPass source=\(source) batchMs=\(Int(batchMilliseconds)) audioSeconds=\(audioSeconds) deadlineSeconds=\(deadlineSeconds) streamLength=\(fallback.count) finalLength=\(text.count)") } public nonisolated func cancel() { diff --git a/Sources/Engines/SonioxASR/SonioxASR.swift b/Sources/Engines/SonioxASR/SonioxASR.swift index 80238c4..30da1a1 100644 --- a/Sources/Engines/SonioxASR/SonioxASR.swift +++ b/Sources/Engines/SonioxASR/SonioxASR.swift @@ -381,8 +381,7 @@ public actor SonioxASR { if contextTerms.isEmpty { doushaLog("[SonioxASR] traceId=\(requestId) config context: (none)") } else { - let preview = contextTerms.prefix(8).joined(separator: "、") - doushaLog("[SonioxASR] traceId=\(requestId) config context.terms=\(contextTerms.count) preview=\(preview)") + doushaLog("[SonioxASR] traceId=\(requestId) config context.terms=\(contextTerms.count)") } try await ws.send(.string(json)) } diff --git a/Sources/Tools/SmokeCLI/MLXSmokeTranscriber.swift b/Sources/Tools/SmokeCLI/MLXSmokeTranscriber.swift index 86774d0..2ae93e0 100644 --- a/Sources/Tools/SmokeCLI/MLXSmokeTranscriber.swift +++ b/Sources/Tools/SmokeCLI/MLXSmokeTranscriber.swift @@ -7,6 +7,7 @@ struct MLXSmokeOptions { var model = MLXModelCatalog.defaultModel.repoID var language = "auto" var realtime = false + var prewarm = false var finalPass = true var context = "" @@ -36,6 +37,8 @@ struct MLXSmokeOptions { } finalPass = arguments[index] == "on" } + case "--prewarm": + prewarm = true case "--realtime": realtime = true default: @@ -75,10 +78,15 @@ final class MLXSmokeTranscriber { return false } print("wav: \(pcm.count) bytes, \(String(format: "%.3f", Double(pcm.count) / 32_000))s @16kHz mono s16le") - print("mlx: model=\(options.model) language=\(options.language) realtime=\(options.realtime) final_pass=\(options.finalPass ? "on" : "off")") + print("mlx: model=\(options.model) language=\(options.language) realtime=\(options.realtime) prewarm=\(options.prewarm) final_pass=\(options.finalPass ? "on" : "off")") let model = MLXModelDescriptor(repoID: options.model, displayName: options.model) let store = MLXModelStore.shared + if options.prewarm { + let started = now + await store.preload(model) + print("preload_ms: \(milliseconds(now - started)) (load + kernel prewarm)") + } let engine = MLXStreamingASR(model: model, store: store, finalPass: options.finalPass) await engine.prepareSession(onPartial: { partial in MainActor.assumeIsolated { @@ -136,11 +144,11 @@ final class MLXSmokeTranscriber { print("audio: fed \(offset) bytes in \(milliseconds(now - audioStarted))ms") let stopStarted = now - let result = await finish(engine) + let result = await finish(engine, audioSeconds: Double(pcm.count) / 32_000) let stopSeconds = now - stopStarted printMetrics(load: loadSeconds, stop: stopSeconds, finalLength: result?.text.count ?? 0) if let metrics = await engine.finalPassMetrics { - print("final_pass: batch_ms=\(String(format: "%.1f", metrics.batchMilliseconds)) final_source=\(metrics.source)") + print("final_pass: batch_ms=\(String(format: "%.1f", metrics.batchMilliseconds)) final_source=\(metrics.source) deadline_seconds=\(metrics.deadlineSeconds)") } print("streaming_preview_text: \(previewText)") if let result { @@ -150,15 +158,16 @@ final class MLXSmokeTranscriber { return failure == nil && result?.text.isEmpty == false } - private func finish(_ engine: MLXStreamingASR) async -> TranscriptionResult? { + private func finish(_ engine: MLXStreamingASR, audioSeconds: Double) async -> TranscriptionResult? { await withCheckedContinuation { continuation in completion = continuation - // The engine allows 30s setup + 10s stream + 3s cleanup + 30s batch. A separate - // CLI watchdog makes a missing callback visible instead of hanging. + // Cover 30s setup + 10s stream + 3s cleanup + the scaled batch + // deadline, with margin, so long WAVs don't hit the watchdog early. + let timeout = max(80, 48 + MLXFinalPassDeadline().seconds(forAudioSeconds: audioSeconds)) watchdog = Task { - do { try await Task.sleep(for: .seconds(80)) } + do { try await Task.sleep(for: .seconds(timeout)) } catch { return } - failure = "stop completion timed out after 80s" + failure = "stop completion timed out after \(timeout)s" print("mlx: ERROR — \(failure!)") engine.cancel() resolve(nil) diff --git a/Sources/Tools/SmokeCLI/SmokeCLI.swift b/Sources/Tools/SmokeCLI/SmokeCLI.swift index 69bd707..2cb9136 100644 --- a/Sources/Tools/SmokeCLI/SmokeCLI.swift +++ b/Sources/Tools/SmokeCLI/SmokeCLI.swift @@ -42,11 +42,12 @@ struct SmokeCLI { --format defaults to speech_opus (needs an encoder, macOS only); pcm probes whether the server accepts raw audio. --profile defaults to official; also supports fast and minimal. - smoke-cli mlx-transcribe [--model ] [--language ] [--realtime] [--final-pass on|off] [--context ] + smoke-cli mlx-transcribe [--model ] [--language ] [--realtime] [--prewarm] [--final-pass on|off] [--context ] Local MLX preview + final batch. WAV must be 16kHz mono s16le. Model defaults to mlx-community/Qwen3-ASR-0.6B-8bit; language defaults to auto (also accepts zh-CN / en-US). --realtime feeds at audio rate; default feeds with 10ms pauses. + --prewarm loads and warms kernels before streaming. --final-pass defaults to on; context reaches only the final batch. Build with xcodebuild for the Metal shader bundle. """) diff --git a/Tests/DoushaTests/MLXFinalPassTests.swift b/Tests/DoushaTests/MLXFinalPassTests.swift index ab8cc3f..7cfeaff 100644 --- a/Tests/DoushaTests/MLXFinalPassTests.swift +++ b/Tests/DoushaTests/MLXFinalPassTests.swift @@ -5,7 +5,7 @@ import ConcurrencySupport import MLXAudioSTT @testable import MLXASR -private final class FakeMLXStream: MLXStreamingSession { +final class FakeMLXStream: MLXStreamingSession { let events: AsyncStream let continuation: AsyncStream.Continuation let stopped = OneShotChannel() @@ -54,6 +54,57 @@ final class MLXFinalPassTests: XCTestCase { return result } + func testShortAudioUsesBaseDeadline() { + XCTAssertEqual(MLXFinalPassDeadline().seconds(forAudioSeconds: 1), 5) + XCTAssertEqual(MLXFinalPassDeadline().seconds(forAudioSeconds: 0), 5) + } + + func testLongAudioUsesScaledDeadline() { + XCTAssertEqual(MLXFinalPassDeadline().seconds(forAudioSeconds: 300), 30) + } + + func testDeadlineBoundaryAndInjectedPolicy() { + let deadline = MLXFinalPassDeadline() + XCTAssertEqual(deadline.seconds(forAudioSeconds: 49), 5) + XCTAssertEqual(deadline.seconds(forAudioSeconds: 50), 5) + XCTAssertEqual(deadline.seconds(forAudioSeconds: 51), 5.1, accuracy: 0.0001) + XCTAssertEqual(MLXFinalPassDeadline(baseSeconds: 2, audioRatio: 0.5).seconds(forAudioSeconds: 10), 5) + } + + func testTimeoutUsesAudioScaledDeadlineRatherThanBase() async throws { + let started = OneShotChannel() + let releaseBatch = OneShotChannel() + let engine = MLXStreamingASR(finalPassDeadline: MLXFinalPassDeadline(baseSeconds: 0.01, audioRatio: 0.1)) { _, _, _, _ in + MLXPreparedSession(stream: FakeMLXStream()) { _ in + started.finish(()) + try await releaseBatch.wait() + return "too late" + } + } + await engine.prepareSession(onPartial: { _ in }, onError: { _ in XCTFail("Unexpected error") }, language: "auto") + engine.ingest(Data(repeating: 0, count: 64_000)) // Exactly two seconds → 0.2s deadline. + engine.openStream() + let result = stop(engine) + _ = try await started.wait() + if case .signaled = await result.wait(timeout: 0.05) { + XCTFail("The base deadline was used instead of the audio-scaled deadline") + } + let outcome = await result.wait(timeout: 1) + releaseBatch.finish(()) + guard case .signaled(let final) = outcome else { + XCTFail("Computed deadline did not deliver fallback") + engine.cancel() + return + } + XCTAssertEqual(final.text, "preview") + let metrics = await engine.finalPassMetrics + XCTAssertEqual(metrics?.deadlineSeconds, 0.2) + XCTAssertEqual(metrics?.source, "stream") + XCTAssertGreaterThanOrEqual(metrics?.batchMilliseconds ?? 0, 190) + let lease = try await MLXModelStore().acquireSessionLease() + await lease.release() + } + func testBatchWinsAfterStreamingCleanupAndReceivesFullPCMContextAndLanguage() async throws { let stream = FakeMLXStream() let request = Lock(nil) @@ -135,7 +186,7 @@ final class MLXFinalPassTests: XCTestCase { let stream = FakeMLXStream() let started = OneShotChannel() let releaseBatch = OneShotChannel() - let engine = MLXStreamingASR(finalPassTimeout: 0.03) { _, _, _, _ in + let engine = MLXStreamingASR(finalPassDeadline: MLXFinalPassDeadline(baseSeconds: 0.03, audioRatio: 0)) { _, _, _, _ in MLXPreparedSession(stream: stream) { _ in started.finish(()) try await releaseBatch.wait() @@ -144,13 +195,17 @@ final class MLXFinalPassTests: XCTestCase { } await prepare(engine) let completed = expectation(description: "one fallback completion") + completed.expectedFulfillmentCount = 1 + completed.assertForOverFulfill = true let count = Lock(0) - engine.stop { result in + let completion: @Sendable (TranscriptionResult) -> Void = { result in XCTAssertEqual(result.text, "preview") count.withLock { $0 += 1 } completed.fulfill() } - engine.stop { _ in XCTFail("Duplicate stop emitted a completion") } + // Nonisolated stop calls schedule unordered tasks; either callback may win. + engine.stop(completion: completion) + engine.stop(completion: completion) _ = try await started.wait() await fulfillment(of: [completed], timeout: 1) let store = MLXModelStore() @@ -196,7 +251,7 @@ final class MLXFinalPassTests: XCTestCase { func testStreamingTimeoutNeverStartsConcurrentBatchAndRetainsLeaseUntilNaturalEnd() async throws { let stream = FakeMLXStream(endOnStop: false) - let engine = MLXStreamingASR(finalPassTimeout: 0.03, streamEndTimeout: 0.03, streamCleanupTimeout: 0.03) { _, _, _, _ in + let engine = MLXStreamingASR(finalPassDeadline: MLXFinalPassDeadline(baseSeconds: 0.03, audioRatio: 0), streamEndTimeout: 0.03, streamCleanupTimeout: 0.03) { _, _, _, _ in MLXPreparedSession(stream: stream) { _ in XCTFail("An abandoned final pass must not start late") return "batch" diff --git a/Tests/DoushaTests/MLXModelPrewarmTests.swift b/Tests/DoushaTests/MLXModelPrewarmTests.swift new file mode 100644 index 0000000..17690d4 --- /dev/null +++ b/Tests/DoushaTests/MLXModelPrewarmTests.swift @@ -0,0 +1,91 @@ +import XCTest +import Foundation +import ConcurrencySupport +@testable import MLXASR + +private struct FakeLoadedMLXModel: MLXLoadedModel { + let runPrewarm: @Sendable () async throws -> Void + + func prewarm() async throws { try await runPrewarm() } + func prepareSession(language: String?) -> MLXPreparedSession { + MLXPreparedSession(stream: FakeMLXStream()) { _ in "batch" } + } +} + +@MainActor +final class MLXModelPrewarmTests: XCTestCase { + func testWarmupPrewarmsOffMainOnceAndKeepsCachedModel() async throws { + let loads = Lock(0) + let prewarms = Lock(0) + let store = MLXModelStore { _ in + loads.withLock { $0 += 1 } + return FakeLoadedMLXModel { + XCTAssertFalse(Thread.isMainThread) + prewarms.withLock { $0 += 1 } + } + } + await store.preload(MLXModelCatalog.defaultModel) + await store.preload(MLXModelCatalog.defaultModel) + let lease = try await store.acquireSessionLease() + _ = try await store.load(MLXModelCatalog.defaultModel, lease: lease) + await lease.release() + XCTAssertEqual(loads.value(), 1) + XCTAssertEqual(prewarms.value(), 1) + } + + func testRecordingLoadAndWarmupJoiningThatLoadNeverPrewarm() async throws { + let started = OneShotChannel() + let finishLoad = OneShotChannel() + let store = MLXModelStore { _ in + started.finish(()) + try await finishLoad.wait() + return FakeLoadedMLXModel { XCTFail("A recording-triggered load must never prewarm") } + } + let engine = MLXStreamingASR(store: store) + await engine.prepareSession(onPartial: { _ in }, onError: { _ in XCTFail("Unexpected error") }, language: "zh-CN") + engine.openStream() + _ = try await started.wait() + store.warmup(MLXModelCatalog.defaultModel) + finishLoad.finish(()) + let completed = expectation(description: "recording finishes") + engine.stop { _ in completed.fulfill() } + await fulfillment(of: [completed], timeout: 2) + await store.preload(MLXModelCatalog.defaultModel) + let state = await store.state + XCTAssertEqual(state, .ready(repoID: MLXModelCatalog.defaultModel.repoID)) + } + + func testPrewarmHoldsProcessLeaseUntilItReturns() async throws { + let started = OneShotChannel() + let finishPrewarm = OneShotChannel() + let store = MLXModelStore { _ in + FakeLoadedMLXModel { + started.finish(()) + try await finishPrewarm.wait() + } + } + store.warmup(MLXModelCatalog.defaultModel) + _ = try await started.wait() + let acquired = OneShotChannel(MLXSessionLease.self) + let waiter = Task { acquired.finish(try await MLXModelStore().acquireSessionLease()) } + if case .signaled = await acquired.wait(timeout: 0.03) { + XCTFail("Prewarm released the process-wide lease before returning") + } + finishPrewarm.finish(()) + let lease = try await acquired.wait() + await lease.release() + _ = await waiter.result + } + + func testPrewarmFailureLeavesReadyAndReleasesLease() async throws { + let store = MLXModelStore { _ in + FakeLoadedMLXModel { throw CancellationError() } + } + await store.preload(MLXModelCatalog.defaultModel) + let state = await store.state + XCTAssertEqual(state, .ready(repoID: MLXModelCatalog.defaultModel.repoID)) + let lease = try await store.acquireSessionLease() + _ = try await store.load(MLXModelCatalog.defaultModel, lease: lease) + await lease.release() + } +}