feat: pluggable brain architecture supporting local OpenAI-compatible LLMs (llama-server, Ollama, vLLM)

README.md

@@ -54,6 +54,22 @@ Two engines, and the setup wizard offers you both instead of quietly defaulting.
 
 **ElevenLabs, the natural one.** The human-sounding voice most people actually want, on your own API key. The free tier is enough to audition it; day-to-day talking runs on the paid starter plan. The wizard walks the whole thing with you: account, key into the keychain, then an audition of real voices through backtalk's own mouth until one fits. Want the exact voice from my videos? It's called **Tarquin** in the ElevenLabs voice library: search it by name and you're done hunting. Under the hood it is: set `elevenlabs.enabled` and your `voice_id` in the config, and have `ffmpeg` installed. **The key never goes in a file.** On macOS, seed it into the Keychain once with `security add-generic-password -a "$USER" -s backtalk-elevenlabs -T /usr/bin/security -w` (it prompts for the secret) and backtalk reads it from there. Linux: `secret-tool store --label backtalk service backtalk-elevenlabs`. The `ELEVENLABS_API_KEY` environment variable works as a last resort, but an export in a shell profile is a plaintext key on disk; the keychain is the grown-up path. Kokoro stays wired in as the automatic fallback, so if the cloud fails the voice degrades instead of going mute, and `logs/backtalk.log` records why.
 
+## Local AI Brain (Optional)
+
+While backtalk defaults to Claude Code via the Agent SDK, it also supports local OpenAI-compatible inference servers (such as `llama-server`, Ollama, vLLM, or Aphrodite) for a 100% offline, private voice assistant.
+
+To switch to a local model, set `"brain": "local"` in your `backtalk.json`:
+
+```json
+{
+  "brain": "local",
+  "api_base": "http://127.0.0.1:8080/v1",
+  "model": "default"
+}
+```
+
+The local brain maintains multi-step tool execution (recursive directory inspection, file reading, command execution, and web browsing) and streams partial sentences directly to the voice pipeline with sentence boundary chunking. See `backtalk.local.json.example`.
+
 ## Give it a face (optional)
 
 backtalk writes tiny state files while it listens, thinks, and speaks, so anything can watch them and react in real time.

backtalk.local.json.example

@@ -0,0 +1,13 @@
+{
+  "brain": "local",
+  "api_base": "http://127.0.0.1:8080/v1",
+  "model": "default",
+  "agent_dir": "~",
+  "name": "Assistant",
+  "ptt_key": "home",
+  "voice": "bm_lewis",
+  "tts_device": "cpu",
+  "stt_model": "small.en",
+  "stt_device": "auto",
+  "extra_dirs": []
+}

backtalk/config.py

@@ -42,7 +42,13 @@
     # Display name, used in logs and to build the quit phrases
     # ("goodbye <name>" hangs up). Match your agent's actual name.
     "name": "Assistant",
-    # The brain. Full model id ON PURPOSE — never a bare alias like
+    # The brain backend: "claude" (default, uses Claude Agent SDK) or
+    # "local" (connects to an OpenAI-compatible endpoint such as llama-server,
+    # Ollama, vLLM, Aphrodite, etc.).
+    "brain": "claude",
+    # Endpoint base URL for local OpenAI-compatible brain.
+    "api_base": "http://127.0.0.1:8080/v1",
+    # The brain model. When using "claude", specify full model id on purpose
     # "sonnet": the SDK resolves aliases through its own bundled CLI and
     # can silently land on an older model. The fast tier is most of the
     # speed difference people ask about; a deep-work model makes every
@@ -131,6 +137,8 @@
     # language pipeline (a=American, b=British, e/f/h/i/j/p/z = other
     # languages), so keep voice and accent matched.
     "voice": "bm_lewis",
+    # Device for local Kokoro TTS pipeline: "cpu" (default) or "cuda"
+    "tts_device": "cpu",
     # Speech recognition (faster-whisper, local, free).
     # Models: tiny.en / base.en / small.en / medium.en — small.en is the
     # accuracy/speed sweet spot on a normal machine.

backtalk/ears.py

@@ -289,12 +289,13 @@ def warm():
                                        verbose=None)
                 _model, _backend = repo, "mlx"
             else:
-                from faster_whisper import WhisperModel
                 want = CFG["stt_device"]
+                compute = CFG["stt_compute"]
                 log(f"[ears] loading {CFG['stt_model']} "
-                    f"({want}/{CFG['stt_compute']})...")
+                    f"({want}/{compute})...")
+                from faster_whisper import WhisperModel
                 _model = WhisperModel(CFG["stt_model"], device=want,
-                                      compute_type=CFG["stt_compute"])
+                                      compute_type=compute)
                 # PROVE the device before the greeting, not at the first
                 # spoken sentence. WhisperModel CONSTRUCTS perfectly well
                 # against a GPU it cannot actually use: "auto" picks CUDA
@@ -418,6 +419,12 @@ def record_held(is_held, max_s: float = 60.0, min_s: float = 0.25) -> str | None
             frames.append(block[:, 0].copy())
     if len(frames) * FRAME_MS / 1000 < min_s:
         return None
+    try:
+        from backtalk.signals import set_state
+        set_state("thinking")
+    except Exception:
+        pass
+    print("[ptt] transcribing...", flush=True)
     return transcribe(np.concatenate(frames))
 
 

backtalk/main.py

@@ -679,7 +679,7 @@ async def amain():
     loop = asyncio.get_event_loop()
     # Warm the engines while the greeting plays: the STT model load and
     # the brain's prompt-cache toll both hide behind the spoken line.
-    loop.run_in_executor(None, warm_ears)
+    warm_ears_fut = loop.run_in_executor(None, warm_ears)
     # THE BRAIN CONNECT, guarded. This is the one startup step that
     # needs a signed-in Claude Code, internet, and available usage.
     # When it fails or hangs, the mouth still works, so SAY SO instead
@@ -699,14 +699,24 @@ async def _warmup():
         kind = ("timed out" if isinstance(e, asyncio.TimeoutError)
                 else f"failed: {e!r}"[:220])
         log(f"[backtalk] BRAIN CONNECT {kind}")
-        mouth.say("Bad news. The voice and the face are fine, but I "
-                  "couldn't reach my brain, the Claude Code session. "
-                  "Check this window for the error. The usual causes: "
-                  "Claude Code isn't signed in, the internet is down, "
-                  "or the plan is out of usage.")
+        if str(CFG.get("brain", "claude")).lower() in ("local", "openai", "llama", "vllm", "ollama"):
+            mouth.say("Bad news. The voice and the face are fine, but I "
+                      f"couldn't reach my local brain at {CFG.get('api_base')}. "
+                      "Check if your local inference server is running.")
+        else:
+            mouth.say("Bad news. The voice and the face are fine, but I "
+                      "couldn't reach my brain, the Claude Code session. "
+                      "Check this window for the error. The usual causes: "
+                      "Claude Code isn't signed in, the internet is down, "
+                      "or the plan is out of usage.")
         mouth.wait_done(timeout=30)
         raise SystemExit(1)
     log("[backtalk] brain warm")
+    try:
+        await asyncio.wait_for(asyncio.shield(warm_ears_fut), 60)
+        log("[backtalk] ears warm and ready")
+    except Exception as e:
+        log(f"[ears] warmup error: {e!r}")
     # the hidden warmup ping is plumbing, not conversation
     brain.session.update(turns=0, out_tokens=0, in_tokens=0, cost=0.0)
     # a configured effort level applies at launch (saved by the spoken

backtalk/mouth.py

@@ -171,8 +171,9 @@ def warm():
             lang = (CFG["voice"] or "bm_lewis")[0]
             log(f"[mouth] loading kokoro (lang '{lang}', "
                 f"voice {CFG['voice']})...")
-            _pipe = KPipeline(lang_code=lang)
-            log("[mouth] voice ready")
+            device = CFG.get("tts_device", "cpu")
+            _pipe = KPipeline(lang_code=lang, device=device)
+            log(f"[mouth] voice ready ({device})")
     return _pipe