diff --git a/README.md b/README.md index 2daad1f8..95db6448 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,28 @@ Whisper supports up to 99 languages, depending on the model size you choose. --- +## Local API (for on-device agents) + +FluidVoice can expose a **loopback-only** HTTP API so other apps and AI agents running **on the same Mac** can reuse its transcription and post-processing — instead of bundling and loading their own ASR model. One model stays warm in FluidVoice; agents get FluidVoice-quality transcription with no extra memory footprint. + +**Enable it:** `Settings → Local API (on-device agents)`. It is **off by default** and **refuses all non-loopback connections** — any non-local peer is rejected at accept time, before it can send a request — so the API is usable only from processes on this Mac. + +- Base URL: `http://127.0.0.1:47733` (port configurable via the `LocalAPIPort` preference) +- `GET /v1/health` — liveness check +- `POST /v1/transcribe` — transcribe an audio file → `{ text, confidence, sampleCount, provider }` +- `POST /v1/postprocess` — run text post-processing/enhancement + +```bash +curl -s http://127.0.0.1:47733/v1/transcribe \ + -H 'Content-Type: application/json' \ + -d '{"path": "/absolute/path/to/audio.ogg"}' +# → {"text":"…","confidence":0.97,"sampleCount":…,"provider":"Parakeet TDT v3 (Multilingual)"} +``` + +Audio can also be sent inline as `{"audioBase64": "…", "filename": "clip.wav"}`. Limits: **25 MB** request body and **300 s (5 min)** of audio per transcription (independent caps). Note: `/v1/transcribe` runs on-device, but `/v1/postprocess` uses your configured AI provider (Settings → AI Enhancement), which may be remote. Full reference: [docs/local-api.md](docs/local-api.md). + +--- + ## Requirements - macOS 15.0 (Sequoia) or later diff --git a/Sources/Fluid/Persistence/SettingsStore.swift b/Sources/Fluid/Persistence/SettingsStore.swift index e786893f..91ebfa65 100644 --- a/Sources/Fluid/Persistence/SettingsStore.swift +++ b/Sources/Fluid/Persistence/SettingsStore.swift @@ -1799,6 +1799,15 @@ final class SettingsStore: ObservableObject { set { self.defaults.set(newValue, forKey: Keys.copyTranscriptionToClipboard) } } + /// When enabled, the loopback-only Local API server (`LocalAPIServer`) is started so + /// on-device agents can reuse FluidVoice's transcription (POST /v1/transcribe) instead + /// of shipping their own ASR model. Off by default. Uses the same "LocalAPIEnabled" + /// key that `LocalAPI.Configuration` reads. + var localAPIEnabled: Bool { + get { self.defaults.bool(forKey: Keys.localAPIEnabled) } + set { self.defaults.set(newValue, forKey: Keys.localAPIEnabled) } + } + var preferredInputDeviceUID: String? { get { self.defaults.string(forKey: Keys.preferredInputDeviceUID) } set { self.defaults.set(newValue, forKey: Keys.preferredInputDeviceUID) } @@ -4915,6 +4924,7 @@ private extension SettingsStore { static let experimentalDirectAudioCaptureForcedOff = "ExperimentalDirectAudioCaptureForcedOff" static let directAudioCaptureConsecutiveFailures = "DirectAudioCaptureConsecutiveFailures" static let copyTranscriptionToClipboard = "CopyTranscriptionToClipboard" + static let localAPIEnabled = "LocalAPIEnabled" static let textInsertionMode = "TextInsertionMode" static let autoUpdateCheckEnabled = "AutoUpdateCheckEnabled" static let betaReleasesEnabled = "BetaReleasesEnabled" diff --git a/Sources/Fluid/Services/LocalAPI/LocalAPIServer.swift b/Sources/Fluid/Services/LocalAPI/LocalAPIServer.swift index 2da5eb1a..d6af7a10 100644 --- a/Sources/Fluid/Services/LocalAPI/LocalAPIServer.swift +++ b/Sources/Fluid/Services/LocalAPI/LocalAPIServer.swift @@ -40,7 +40,10 @@ final class LocalAPIServer { } Task { @MainActor [weak self] in - guard let self else { + // Drop connections accepted just before the server was disabled: stop() + // clears `listener`, so a task still queued here must not spin up a handler + // after the API was turned off (e.g. a live toggle-off via refresh()). + guard let self, self.listener != nil else { connection.cancel() return } @@ -56,8 +59,13 @@ final class LocalAPIServer { handler.start(on: self.queue) } } - listener.stateUpdateHandler = { state in + listener.stateUpdateHandler = { [weak self, weak listener] state in Task { @MainActor in + // Ignore callbacks from a superseded listener. A rapid off→on toggle + // (now user-reachable via refresh()) can install a new listener before + // an old one's async .cancelled/.failed callback reaches the main actor; + // without this guard that stale callback would clear/cancel the new one. + guard let self, let listener, self.listener === listener else { return } self.handleState(state) } } @@ -79,6 +87,17 @@ final class LocalAPIServer { self.listener = nil } + /// Start or stop the server to match the current `LocalAPIEnabled` setting. + /// Call this after the user toggles the Local API preference so the change takes + /// effect immediately, without restarting the app. + func refresh() { + if LocalAPI.Configuration.current.enabled { + self.start() + } else { + self.stop() + } + } + private func handleState(_ state: NWListener.State) { switch state { case .ready: diff --git a/Sources/Fluid/UI/SettingsView.swift b/Sources/Fluid/UI/SettingsView.swift index 115ebc21..e2941687 100644 --- a/Sources/Fluid/UI/SettingsView.swift +++ b/Sources/Fluid/UI/SettingsView.swift @@ -277,6 +277,21 @@ struct SettingsView: View { ) Divider().opacity(0.2) + // Local API (on-device agents) + self.settingsToggleRow( + title: "Local API (on-device agents)", + description: "Loopback-only HTTP API for on-device agents to reuse transcription (POST /v1/transcribe).", + footnote: "127.0.0.1:47733, off by default. /v1/postprocess may use a remote AI provider.", + isOn: Binding( + get: { SettingsStore.shared.localAPIEnabled }, + set: { newValue in + SettingsStore.shared.localAPIEnabled = newValue + LocalAPIServer.shared.refresh() + } + ) + ) + Divider().opacity(0.2) + // Accent Color VStack(alignment: .leading, spacing: 6) { HStack(alignment: .center) { diff --git a/docs/local-api.md b/docs/local-api.md new file mode 100644 index 00000000..550a297e --- /dev/null +++ b/docs/local-api.md @@ -0,0 +1,109 @@ +# Local API + +FluidVoice ships a small, **loopback-only** HTTP API that lets other apps and AI agents +running **on the same Mac** reuse its speech-to-text and text post-processing. This avoids +each agent bundling and loading its own ASR model — FluidVoice keeps one model warm and +serves transcription requests locally. + +## Enabling + +The API is **off by default**. Turn it on in `Settings → Local API (on-device agents)`. + +You can also toggle it via `defaults`: + +```bash +defaults write com.FluidApp.app LocalAPIEnabled -bool true +# optional custom port (default 47733): +defaults write com.FluidApp.app LocalAPIPort -int 47733 +``` + +The Settings toggle starts/stops the server immediately. Setting the preference with `defaults` +while FluidVoice is running only writes the value — **relaunch FluidVoice** (or use the toggle) for +it to take effect, since the server is otherwise started only at app launch. + +## Security + +- The listener **refuses all non-loopback connections** — any non-local peer is rejected at + accept time, before it can send a request — so the API is usable only from processes on this Mac. +- It is opt-in and off by default. +- Limits: request body **25 MB**; transcription audio **300 s (5 min)** per request. For file-`path` + input, audio longer than the limit is **rejected**; for inline `audioBase64` input it is currently + **truncated to the first 300 s**. These caps are independent — compressed audio can satisfy the byte + limit yet still hit the duration limit. +- **Locality caveat:** `/v1/transcribe` runs fully on-device. `/v1/postprocess` runs your configured + post-processing provider (Settings → AI Enhancement); if that provider is a remote API + (OpenAI/Groq/Anthropic/…), the submitted text is sent there. Only the HTTP listener and transcription + are guaranteed local. + +## Endpoints + +Base URL: `http://127.0.0.1:47733` + +### `GET /v1/health` + +Liveness probe. Returns `200 OK` when the server is running. + +### `POST /v1/transcribe` + +Transcribe an audio file with the currently selected speech model (reuses the already-loaded +model — no second instance). + +Request body (JSON), one of: + +```jsonc +{ "path": "/absolute/path/to/audio.ogg" } // transcribe a file by path +{ "audioBase64": "", "filename": "clip.wav" } // or inline audio (extension hint) +``` + +Supported inputs include WAV, MP3, M4A, OGG and other common formats (decoded internally). + +Response: + +```json +{ + "text": "the transcribed text", + "confidence": 0.97, + "sampleCount": 1402136, + "provider": "Parakeet TDT v3 (Multilingual)" +} +``` + +### `POST /v1/postprocess` + +Run FluidVoice's text post-processing/enhancement on a string. + +```json +{ "text": "raw text to clean up" } +``` + +Response: + +```json +{ "text": "cleaned text", "provider": "…", "model": "…" } +``` + +## Example integrations + +Shell: + +```bash +curl -s http://127.0.0.1:47733/v1/transcribe \ + -H 'Content-Type: application/json' \ + -d '{"path": "'"$PWD"'/memo.ogg"}' | jq -r .text +``` + +Python: + +```python +import requests +r = requests.post("http://127.0.0.1:47733/v1/transcribe", + json={"path": "/tmp/memo.ogg"}, timeout=90) +print(r.json()["text"]) +``` + +## Notes + +- If the selected model is not resident yet, the first request loads it; subsequent requests + are fast while it stays warm. +- Other endpoints exist for history and the custom dictionary (`GET /v1/history`, + `GET|POST /v1/dictionary/*`); this document focuses on the ones most useful to agents.