iOS glasses WebRTC client + Mac-mic agent input

.gitignore

@@ -21,3 +21,10 @@ Thumbs.db
 
 # Locks (keep pyproject.toml, exclude uv.lock if desired)
 # uv.lock
+
+# Xcode / iOS (ios/)
+xcuserdata/
+*.xcuserstate
+DerivedData/
+.swiftpm/
+ios/**/.build/

ios/PROTOCOL.md

@@ -0,0 +1,77 @@
+# Glasses Streaming Protocol
+
+The semantic contract between the iOS glasses app and its server. This is
+transport-independent: it describes *what* flows and *what* control messages
+exist. Section 1 is the current (proven) WebSocket transport. Section 2 maps
+the same contract onto WebRTC, the target transport for the template.
+
+The host is currently hardcoded at
+`RayBan/ViewModels/StreamSessionViewModel.swift:16`
+(`ws://10.10.10.121:57308`) and must become configurable in the template.
+
+---
+
+## 1. Current transport — raw WebSocket + HTTP (what exists today)
+
+### 1.1 Video uplink — `ws /publish`
+- iOS captures `VideoFrame` from DAT (`VideoCodec.raw`, 24fps), converts to
+  `UIImage`, JPEG-encodes at quality 0.5, sends each frame as a **binary**
+  WebSocket message.
+- Only sent while `videoEnabled == true`.
+
+**Control messages on this same socket (JSON text frames):**
+- iOS → server: `{"type":"pause"}`, `{"type":"resume"}`
+- server → iOS: `{"type":"video_off"}`, `{"type":"video_on"}`,
+  `{"type":"capture_photo","request_id":"<id>"}`
+
+### 1.2 Audio uplink — `ws /publish-audio?agent=1`
+- First message: JSON header `{"sampleRate":48000,"channels":1}` (rate is the
+  glasses' actual input rate, sent dynamically).
+- Subsequent messages: raw **Float32 LE mono PCM** binary frames.
+- The `agent=1` query flag tells the server to bridge this audio into the
+  voice agent.
+
+### 1.3 Audio downlink — `ws /agent-audio`
+- Server sends JSON header `{"sampleRate":24000}`, then **Int16 LE mono PCM @
+  24kHz** binary frames (the voice agent talking back).
+- iOS plays these through the glasses speaker over HFP.
+
+### 1.4 Photo — `POST /publish/photo`
+- Full-resolution JPEG body, `Content-Type: image/jpeg`.
+- `X-Request-Id` header echoes the `request_id` from a `capture_photo` control
+  message (when the capture was remote-triggered).
+
+---
+
+## 2. Target transport — WebRTC (template)
+
+One peer connection per session. Recommended: a LiveKit (or comparable SFU)
+"room" rather than hand-rolled signaling + TURN.
+
+### 2.1 Video → outbound video track
+- Custom video source: `VideoFrame` → `CVPixelBuffer` → `RTCVideoFrame` (or
+  LiveKit buffer capturer). **No JPEG re-encode** — WebRTC encodes VP8/H.264.
+- The `videoEnabled` toggle becomes muting/unmuting the track (or a DataChannel
+  message — see 2.4).
+
+### 2.2 Audio uplink → outbound audio track
+- WebRTC's audio device module captures the system input route, which is the
+  glasses HFP mic. The current AVAudioEngine tap is likely removed.
+- No manual sample-rate header — SDP negotiates Opus.
+
+### 2.3 Audio downlink → inbound audio track
+- The agent's audio arrives as a remote track and auto-plays to the output
+  route (glasses speaker). The `AgentAudioPlayer` class is likely removed.
+
+### 2.4 Control → DataChannel
+- Same messages as 1.1, sent as JSON over a reliable DataChannel:
+  `pause` / `resume` / `video_off` / `video_on` / `capture_photo`.
+
+### 2.5 Photo → keep HTTP `POST /publish/photo`
+- Stills are not real-time media; leave this on HTTP exactly as today.
+
+### 2.6 Open risk to de-risk first
+- WebRTC wants to own `AVAudioSession`. DAT's HFP routing is delicate (today's
+  code adds the camera stream *before* activating HFP). Spike a single bidi
+  WebRTC audio track over HFP alongside a live DAT camera stream before
+  building the rest.

ios/README.md

@@ -0,0 +1,76 @@
+# CourtLine Glass — iOS app + WebRTC audio integration
+
+iOS app that streams from **Meta Ray-Ban Display glasses** and connects to the
+CourtLine agent (`../yc-voice-agents-hackathon/server`) over **WebRTC**.
+
+## The audio architecture (why it's shaped this way)
+
+We tried to send the **glasses microphone** to the agent over WebRTC and hit a
+hard iOS limitation: WebRTC's iOS audio module couldn't reliably capture the
+Bluetooth-HFP glasses mic while the DAT camera stream was live (it negotiates
+and routes to HFP, but no mic samples reach the server). WebRTC on iOS also
+offers no supported way to inject externally-captured PCM into an audio track.
+
+So the working split is:
+
+```
+  Human speaks
+      │
+      ▼
+  MacBook mic ─► agent (bot-courtline.py, MAC_MIC_INPUT=1)
+                   │  NVidia STT → vLLM → Gradium TTS
+                   ▼
+             SmallWebRTCTransport.output()
+                   │  WebRTC (SRTP, peer-to-peer)
+                   ▼
+             iOS app (receive-only, enableMic = false)
+                   │  Bluetooth HFP
+                   ▼
+             🕶️  Glasses speaker  ← agent's voice
+```
+
+- **Agent listens via the Mac mic** (`MAC_MIC_INPUT=1`) — see the server change
+  in `bot-courtline.py`.
+- **Agent speaks through the glasses** — the iOS app connects **receive-only**
+  (`WebRTCAudioSpike.swift`, `enableMic = false`) and plays the agent's audio
+  out through the glasses' HFP speaker. This leg is verified working.
+
+## Setup
+
+**Agent (server):**
+```bash
+cd ../yc-voice-agents-hackathon/server
+uv sync                       # picks up the new sounddevice dep
+export MAC_MIC_INPUT=1        # capture this Mac's mic into the pipeline
+uv run main.py                # agent on http://0.0.0.0:7860 (serves /api/offer)
+# macOS will prompt for microphone access for the terminal — allow it.
+```
+
+**iOS app:**
+1. Open `RayBan.xcodeproj`, set your signing team.
+2. Set `webRTCServerURL` in `RayBan/ViewModels/WebRTCAudioSpike.swift` to the
+   Mac's LAN IP + port, e.g. `http://192.168.1.42:7860`.
+3. Build & run on a real device with the glasses paired (same Wi-Fi/LAN — see
+   the network note below). Start a stream, tap the **waveform** button.
+
+## What's verified vs. pending
+
+- ✅ iOS app compiles against the real SDKs (PipecatClientIOS / SmallWebRTC /
+  WebRTC) and connects to a SmallWebRTC `/api/offer`; the agent's audio plays
+  through the glasses (downlink proven end-to-end).
+- ✅ Mac-mic capture (`sounddevice`) streams audio into a Pipecat pipeline
+  (verified standalone; wired here behind `MAC_MIC_INPUT`).
+- ⏳ The full Mac-mic → STT → LLM → TTS → glasses loop needs an on-stack run with
+  the agent's API keys (Gradium / NVIDIA / OpenAI) — not runnable in CI.
+- ⏳ Glasses **video → agent vision** (`vision.py` / `check_camera`) is not wired
+  yet; the legacy WebSocket video path (`streamPublishHost`) is a placeholder.
+
+## Network note
+
+Peer-to-peer WebRTC needs a routable media path. On the same Wi-Fi/LAN this
+works via host ICE candidates. It will **not** traverse a Cloudflare HTTP tunnel
+(HTTP only, not the UDP media). For client-isolation networks (e.g. some phone
+hotspots) you need a TURN server.
+
+> The Xcode target is named `RayBan` internally; the display name is CourtLine.
+> Derived from Meta's DAT sample app — source headers retain Meta's copyright.