Browser RAG UI, GPU/CPU tuning, tests, CI, and LAN serving

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    # The device-side code is plain standard library, so the whole RAG path is
+    # exercised here against stand-in model servers — no phone, no GGUF weights.
+    name: tests (python ${{ matrix.python }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: python -m pip install --upgrade pip pytest
+      - run: python -m pytest tests/ -q
+
+  lint:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install --upgrade pip ruff
+      - name: ruff
+        run: ruff check .
+
+  shell:
+    name: shellcheck
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: shellcheck
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y shellcheck
+          # Termux scripts use a Termux shebang that shellcheck cannot resolve,
+          # so the shell dialect is named explicitly.
+          find . -name '*.sh' -not -path './.git/*' -print0 \
+            | xargs -0 -r shellcheck --shell=bash --external-sources

.gitignore

@@ -9,3 +9,11 @@ bench/raw/
 rag/corpus/
 rag/index/
 *.jsonl
+
+# Local Claude Code worktrees
+.claude/worktrees/
+__pycache__/
+*.pyc
+
+# Deployment-specific / explanatory docs — never publish (contains local network detail)
+private/

README.md

@@ -16,7 +16,6 @@ OpenAI-compatible chat API over the LAN. No root, no custom ROM, no cloud.
         └──── USB (adb forward) ── or ── Wi-Fi ────────┘
 ```
 
-> **New here? Read [GUIDE.md](GUIDE.md) first — plain English, no jargon.**
 
 ## Why this is interesting
 
@@ -48,43 +47,61 @@ echo "<the API key install.sh printed>" > ~/.config/s25-llm-key
 The client auto-selects USB when a device is attached and falls back to `$LLM_HOST`
 over Wi-Fi otherwise.
 
+**In a browser** — the notes assistant serves a chat page on :8083 that applies
+retrieval before answering (unlike :8081, which is the raw model):
+
+```sh
+adb forward tcp:8083 tcp:8083 && xdg-open http://localhost:8083
+```
+
 ## What's in here
 
 | Path | Runs on | Purpose |
 |---|---|---|
 | `install.sh` | phone | One-shot setup: packages, dirs, API key |
 | `bin/fetch-model.sh` | phone | Resumable GGUF download |
-| `bin/llm-server.sh` | phone | Launches `llama-server` with a wake lock |
-| `boot/start-lab.sh` | phone | Termux:Boot autostart — sshd, tmux, LLM |
+| `bin/llm-server.sh` | phone | Launches `llama-server` (GPU prompt eval, pinned CPU cores) |
+| `rag/bin/rag-web.py` | phone | Browser UI + OpenAI-compatible RAG endpoint on :8083 |
+| `boot/start-lab.sh` | phone | Termux:Boot autostart — sshd, tmux, LLM, embed, RAG web |
 | `client/llm` | laptop | CLI client, USB-or-Wi-Fi transport selection |
+| `tests/` | laptop / CI | Full test suite against stand-in model servers |
 
 ## Documentation
 
-- **[GUIDE.md](GUIDE.md)** — start here. Plain-English: what it is, how to use it, how to fix it.
-- **[rag/](rag/README.md)** — the CPTS study assistant (RAG over your own notes).
+- **[rag/](rag/README.md)** — the CPTS study assistant (RAG over your own notes), and
+  the browser UI.
 
 - **[Architecture](docs/ARCHITECTURE.md)** — how every layer works, from the Android
   sandbox up through quantization and the request path. Written to be readable with
   no prior systems background.
 - **[Setup log](docs/SETUP.md)** — the actual build, in order, including what broke.
 - **[Networking](docs/NETWORKING.md)** — why remote access is the hard part: NAT,
-  private addressing, and a DPI-filtered campus network.
+  private addressing, and a DPI-filtered network.
+- **[Benchmarks](bench/RESULTS.md)** — measured throughput, the GPU story, and the
+  core-pinning numbers.
 
 
 ## Measured performance
 
-On the Galaxy S25 (Snapdragon 8 Elite, 6 of 8 cores, Qwen3-4B-Instruct Q4_K_M):
+On the Galaxy S25 (Snapdragon 8 Elite, Qwen3-4B-Instruct Q4_K_M). Prompt
+processing runs on the Adreno GPU, generation on six pinned CPU cores:
 
 | Metric | Value |
 |---|---|
-| Generation | ~15 tokens/sec (flash-attn, 8 threads) |
-| Prompt processing | ~33 tokens/sec |
+| Prompt processing | 70 tokens/sec (Adreno 830, Vulkan) |
+| Generation | 12 tokens/sec (6 pinned CPU cores) |
 | Model load time | ~2.6 s |
 | Idle RAM headroom | ~4 GB free with model resident |
 | Context window | 8192 tokens (q8_0 KV cache) |
-| First-token latency | sub-second over USB |
 
-Fast enough to read along with. Not instant, but usable for real work.
+Prompt processing is what you wait on before an answer starts, so it is the
+number that matters: a 605-token retrieval prompt begins answering after 8.9
+seconds rather than 36. Using the GPU is worth 4x there, and it is *slower* at
+generating tokens, which is why the two halves run on different hardware.
+
+[bench/RESULTS.md](bench/RESULTS.md) has the full matrix, the core-pinning
+measurements, and the OpenCL dead end. `bench/probe.py` reproduces the headline
+numbers against a running server.
 
 ## Endpoint authentication
 
@@ -103,12 +120,16 @@ without the key.
 ## Security
 
 `llama-server` binds `0.0.0.0`, so it is reachable by anything that can route to the
-phone. On the network this was built on, client isolation is **off** — any device on
-the same `/20` can reach it. So the API key is mandatory, not decorative:
-
-- key generated with `openssl rand -hex 24`, stored `chmod 600` at `~/.config/llm-api-key`
-- `.gitignore` excludes the key and all `*.gguf` weights
-- the key never appears in this repository
+phone. Where the local network has client isolation off, other devices on it can reach
+the port, so the API key is mandatory, not decorative:
+
+- key generated with `python3 -c "import secrets; print(secrets.token_hex(24))"`,
+  stored `chmod 600` at `~/.config/llm-api-key`
+- `.gitignore` excludes the key, the notes corpus, the built index, and all `*.gguf`
+  weights
+- the bearer token rides plaintext HTTP, so on an untrusted network it is protection
+  against casual use, not against someone able to watch the traffic — prefer loopback
+  plus `adb forward`, or a WireGuard tunnel, there
 
 ## Status
 

bench/RESULTS.md

@@ -0,0 +1,95 @@
+# Measured performance
+
+Galaxy S25 (`SM-S931B`), Snapdragon 8 Elite, 12 GB RAM, unrooted Termux.
+Model: Qwen3-4B-Instruct-2507, Q4_K_M, 2.32 GiB.
+llama.cpp build b10516 (Termux `llama-cpp` package).
+Numbers from `llama-bench -p 256 -n 32`, and from `bench/probe.py` against the
+live server. `pp` is prompt processing, `tg` is token generation.
+
+## Summary
+
+| | prompt eval | generation |
+|---|---:|---:|
+| before (CPU only, 8 threads) | 17.9 tok/s | 10.4 tok/s |
+| after (GPU + 6 pinned cores) | **70.2 tok/s** | **12.0 tok/s** |
+
+Prompt processing is what you wait on before an answer starts. On a real RAG
+question (605-token prompt) that is **36 s → 8.9 s**.
+
+## Where the time goes
+
+Retrieval is not the bottleneck and never was: embedding the question, scoring
+all 1560 chunks in pure Python and picking the top 5 takes **161 ms**. The rest
+is llama-server.
+
+## The CPU is heterogeneous
+
+`/sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_max_freq`:
+
+| cores | clock | role |
+|---|---|---|
+| cpu0–5 | 3.53 GHz | performance |
+| cpu6–7 | 4.47 GHz | prime |
+
+Splitting work evenly across all 8 is *slower* than using the 6 matched cores.
+Threads on the prime cores finish their share early and idle, and leaving those
+two cores free lets the OS and the GPU driver run without preempting a worker.
+
+| config | pp256 | tg32 |
+|---|---:|---:|
+| `-t 8` unpinned | 75.12 | 9.14 |
+| `-t 8 -C 0xff --cpu-strict 1` | 75.16 | 7.80 |
+| **`-t 6 -C 0x3f --cpu-strict 1`** | **75.35** | **12.16** |
+| `-t 2 -C 0xc0 --cpu-strict 1` | 74.94 | 8.00 |
+
+So the chat server takes cpu0–5 and the embedding server takes cpu6–7. Before
+this split the two asked for 12 threads between them on an 8-core phone.
+
+## The GPU was never being used
+
+The Adreno 830 is reachable from unrooted Termux, but not the way it first
+appears:
+
+- The vendor OpenCL driver cannot be loaded. `/vendor/lib64/libOpenCL.so` is
+  listed in `/vendor/etc/public.libraries.txt`, but the Android linker refuses
+  an absolute path into `/vendor/lib64` from an app namespace, and the bare
+  soname resolves to Termux's own ICD loader instead. `ggml_opencl: platform
+  IDs not available` is that dead end.
+- Vulkan works. `pkg install llama-cpp-backend-vulkan mesa-vulkan-icd-freedreno`
+  gives Mesa's **turnip** driver, entirely in userspace, and `vulkaninfo`
+  then reports `Adreno (TM) 830`.
+
+CPU-only prompt eval, for comparison — note it scales with thread count, so the
+CPU is genuinely compute-starved here in a way the GPU fixes:
+
+| threads | flash-attn | pp256 | tg32 |
+|---|---|---:|---:|
+| 4 | off | 11.89 | 11.04 |
+| 6 | off | 15.78 | 12.92 |
+| 8 | off | 18.66 | 12.82 |
+| 4 | on | 12.10 | 11.52 |
+| 6 | on | 15.99 | 13.43 |
+| 8 | on | 17.57 | 14.34 |
+
+Flash-attention makes no meaningful difference to prompt eval on this CPU.
+
+### Do not set `GGML_BACKEND_PATH`
+
+ggml finds its backend libraries by looking next to the `llama-server` binary.
+Setting `GGML_BACKEND_PATH` to a single `.so` **restricts** it to that one
+backend — pointing it at `libggml-cpu.so` silently disables the GPU and costs
+4x on prompt eval. Setting it to a directory, or to a colon-separated list,
+fails outright. Leave it unset and start the server from a Termux shell (which
+is what `boot/start-lab.sh` does via tmux); started from a bare `ssh` command
+the binary path resolves to `/apex/...` and no backend is found at all.
+
+## Reproducing
+
+```bash
+# On the phone
+llama-bench -m ~/models/qwen3-4b.gguf -p 256 -n 32 -t 6 -C 0x3f --cpu-strict 1
+
+# From the laptop, against the running server
+adb forward tcp:8081 tcp:8081
+python3 bench/probe.py
+```

bench/probe.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+"""Report prompt-eval and generation speed straight from llama-server's timings.
+
+    python3 bench/probe.py [URL]
+
+Run it against the chat server (default http://localhost:8081) after an
+`adb forward tcp:8081 tcp:8081`. Prompt-eval speed is the number that decides
+how long you wait before an answer starts, and it is the one that changes when
+the GPU backend is or is not in play.
+"""
+import json
+import os
+import sys
+import urllib.request
+
+URL = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081/v1/chat/completions"
+KEYFILE = os.path.expanduser(os.environ.get("LLM_KEYFILE", "~/.config/llm-api-key"))
+KEY = os.environ.get("LLM_API_KEY") or (
+    open(KEYFILE).read().strip() if os.path.exists(KEYFILE) else "")
+
+payload = {
+    # Long enough that prompt processing dominates and is measured accurately.
+    "messages": [{"role": "user", "content": "Explain SMB enumeration. " * 60}],
+    "max_tokens": 16,
+    "temperature": 0,
+}
+req = urllib.request.Request(
+    URL, data=json.dumps(payload).encode(),
+    headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"})
+with urllib.request.urlopen(req, timeout=600) as resp:
+    t = json.load(resp)["timings"]
+
+print(f"prompt: {t['prompt_n']:>5} tok at {t['prompt_per_second']:>6.1f} tok/s")
+print(f"gen:    {t['predicted_n']:>5} tok at {t['predicted_per_second']:>6.1f} tok/s")

bin/llm-server.sh

@@ -6,16 +6,42 @@ set -euo pipefail
 MODEL="${LLM_MODEL:-$HOME/models/qwen3-4b.gguf}"
 KEYFILE="${LLM_KEYFILE:-$HOME/.config/llm-api-key}"
 PORT="${LLM_PORT:-8081}"
-THREADS="${LLM_THREADS:-8}"
 CTX="${LLM_CTX:-8192}"
 
+# Snapdragon 8 Elite is 6 performance cores (cpu0-5, 3.53 GHz) plus 2 prime
+# cores (cpu6-7, 4.47 GHz). Splitting work evenly across all 8 is slower than
+# using the 6 matched cores: the threads on the prime cores finish their share
+# early and then idle, and the two cores left free absorb the OS and the GPU
+# driver. Measured on this device, generation goes 9.1 -> 12.2 tokens/sec by
+# dropping from 8 unpinned threads to 6 pinned ones.
+THREADS="${LLM_THREADS:-6}"
+CPU_MASK="${LLM_CPU_MASK:-0x3f}"   # cpu0-5
+
+# The GPU (Adreno 830, reached through Mesa's turnip Vulkan driver) is about
+# 4x faster than the CPU at prompt processing -- 75 vs 19 tokens/sec -- and
+# prompt processing is what decides how long you wait before an answer starts.
+# It is slower at generating tokens, so the split below is deliberate: GPU for
+# the prompt, the pinned CPU cores for generation.
+#
+# ggml discovers its backend libraries by looking next to the llama-server
+# binary, which only resolves correctly when the server is started from a
+# Termux shell (as boot/start-lab.sh does via tmux). Do NOT set
+# GGML_BACKEND_PATH here: pointing it at a single .so restricts ggml to that
+# one backend, which silently drops the GPU and costs 4x on prompt eval.
+NGL="${LLM_NGL:-99}"
+
 [ -r "$MODEL" ]   || { echo "no model at $MODEL — run bin/fetch-model.sh first" >&2; exit 1; }
 [ -r "$KEYFILE" ] || { echo "no API key at $KEYFILE — run install.sh first" >&2; exit 1; }
 
 # Android suspends the CPU when idle; without this the server stalls mid-request.
 termux-wake-lock
 
-# Flags tuned for a phone CPU:
+# Flags tuned for this phone:
+#   -ngl 99                offload to the Adreno GPU for prompt processing
+#   --cpu-strict           keep the worker threads on the cores chosen above
+#   --poll 100             spin rather than sleep between batches
+# (--prio is deliberately absent: raising thread priority needs root, and an
+# unrooted Termux just logs "Operation not permitted" once per thread.)
 #   --flash-attn on        faster attention, lower memory
 #   --cache-type-k/v q8_0  quantized KV cache — fits an 8192 context in RAM
 #   --host 0.0.0.0         LAN-reachable, which is why --api-key is mandatory
@@ -25,7 +51,11 @@ exec llama-server \
   --port      "$PORT" \
   --api-key   "$(cat "$KEYFILE")" \
   --ctx-size  "$CTX" \
+  --n-gpu-layers "$NGL" \
   --threads   "$THREADS" \
+  --cpu-mask  "$CPU_MASK" \
+  --cpu-strict 1 \
+  --poll      100 \
   --flash-attn on \
   --cache-type-k q8_0 \
   --cache-type-v q8_0 \

boot/start-lab.sh

@@ -22,3 +22,18 @@ if [ -r "$HOME/models/nomic-embed.gguf" ] && [ -x "$HOME/rag/bin/rag-embed-serve
   tmux has-session -t embsrv 2>/dev/null || \
     tmux new-session -d -s embsrv "$HOME/rag/bin/rag-embed-server.sh"
 fi
+
+# Browser front end for the notes assistant on :8083. Unlike :8081 this applies
+# retrieval before answering, so it needs the index.
+#
+# It is bound LAN-wide so other devices on the Wi-Fi can use it, which means
+# every request except /health must carry the bearer token. Note what that
+# exposes: this endpoint reads out of the private notes corpus and the token is
+# sent in clear text over HTTP. Set RAG_WEB_HOST=127.0.0.1 to go back to
+# loopback-only (reachable via adb forward tcp:8083 tcp:8083), which is the
+# safer default on a network you do not trust.
+if [ -r "$HOME/rag/index.jsonl" ] && [ -r "$HOME/rag/bin/rag-web.py" ]; then
+  tmux has-session -t ragweb 2>/dev/null || \
+    tmux new-session -d -s ragweb \
+      "RAG_WEB_HOST=${RAG_WEB_HOST:-0.0.0.0} python3 $HOME/rag/bin/rag-web.py"
+fi