diff --git a/.gitignore b/.gitignore index 5c4385e..b53b80f 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,10 @@ Thumbs.db # Locks (keep pyproject.toml, exclude uv.lock if desired) # uv.lock + +# Xcode / iOS (ios/) +xcuserdata/ +*.xcuserstate +DerivedData/ +.swiftpm/ +ios/**/.build/ diff --git a/ios/PROTOCOL.md b/ios/PROTOCOL.md new file mode 100644 index 0000000..4e8d58e --- /dev/null +++ b/ios/PROTOCOL.md @@ -0,0 +1,77 @@ +# Glasses Streaming Protocol + +The semantic contract between the iOS glasses app and its server. This is +transport-independent: it describes *what* flows and *what* control messages +exist. Section 1 is the current (proven) WebSocket transport. Section 2 maps +the same contract onto WebRTC, the target transport for the template. + +The host is currently hardcoded at +`RayBan/ViewModels/StreamSessionViewModel.swift:16` +(`ws://10.10.10.121:57308`) and must become configurable in the template. + +--- + +## 1. Current transport — raw WebSocket + HTTP (what exists today) + +### 1.1 Video uplink — `ws /publish` +- iOS captures `VideoFrame` from DAT (`VideoCodec.raw`, 24fps), converts to + `UIImage`, JPEG-encodes at quality 0.5, sends each frame as a **binary** + WebSocket message. +- Only sent while `videoEnabled == true`. + +**Control messages on this same socket (JSON text frames):** +- iOS → server: `{"type":"pause"}`, `{"type":"resume"}` +- server → iOS: `{"type":"video_off"}`, `{"type":"video_on"}`, + `{"type":"capture_photo","request_id":""}` + +### 1.2 Audio uplink — `ws /publish-audio?agent=1` +- First message: JSON header `{"sampleRate":48000,"channels":1}` (rate is the + glasses' actual input rate, sent dynamically). +- Subsequent messages: raw **Float32 LE mono PCM** binary frames. +- The `agent=1` query flag tells the server to bridge this audio into the + voice agent. + +### 1.3 Audio downlink — `ws /agent-audio` +- Server sends JSON header `{"sampleRate":24000}`, then **Int16 LE mono PCM @ + 24kHz** binary frames (the voice agent talking back). +- iOS plays these through the glasses speaker over HFP. + +### 1.4 Photo — `POST /publish/photo` +- Full-resolution JPEG body, `Content-Type: image/jpeg`. +- `X-Request-Id` header echoes the `request_id` from a `capture_photo` control + message (when the capture was remote-triggered). + +--- + +## 2. Target transport — WebRTC (template) + +One peer connection per session. Recommended: a LiveKit (or comparable SFU) +"room" rather than hand-rolled signaling + TURN. + +### 2.1 Video → outbound video track +- Custom video source: `VideoFrame` → `CVPixelBuffer` → `RTCVideoFrame` (or + LiveKit buffer capturer). **No JPEG re-encode** — WebRTC encodes VP8/H.264. +- The `videoEnabled` toggle becomes muting/unmuting the track (or a DataChannel + message — see 2.4). + +### 2.2 Audio uplink → outbound audio track +- WebRTC's audio device module captures the system input route, which is the + glasses HFP mic. The current AVAudioEngine tap is likely removed. +- No manual sample-rate header — SDP negotiates Opus. + +### 2.3 Audio downlink → inbound audio track +- The agent's audio arrives as a remote track and auto-plays to the output + route (glasses speaker). The `AgentAudioPlayer` class is likely removed. + +### 2.4 Control → DataChannel +- Same messages as 1.1, sent as JSON over a reliable DataChannel: + `pause` / `resume` / `video_off` / `video_on` / `capture_photo`. + +### 2.5 Photo → keep HTTP `POST /publish/photo` +- Stills are not real-time media; leave this on HTTP exactly as today. + +### 2.6 Open risk to de-risk first +- WebRTC wants to own `AVAudioSession`. DAT's HFP routing is delicate (today's + code adds the camera stream *before* activating HFP). Spike a single bidi + WebRTC audio track over HFP alongside a live DAT camera stream before + building the rest. diff --git a/ios/README.md b/ios/README.md new file mode 100644 index 0000000..d9cd174 --- /dev/null +++ b/ios/README.md @@ -0,0 +1,76 @@ +# CourtLine Glass — iOS app + WebRTC audio integration + +iOS app that streams from **Meta Ray-Ban Display glasses** and connects to the +CourtLine agent (`../yc-voice-agents-hackathon/server`) over **WebRTC**. + +## The audio architecture (why it's shaped this way) + +We tried to send the **glasses microphone** to the agent over WebRTC and hit a +hard iOS limitation: WebRTC's iOS audio module couldn't reliably capture the +Bluetooth-HFP glasses mic while the DAT camera stream was live (it negotiates +and routes to HFP, but no mic samples reach the server). WebRTC on iOS also +offers no supported way to inject externally-captured PCM into an audio track. + +So the working split is: + +``` + Human speaks + │ + ▼ + MacBook mic ─► agent (bot-courtline.py, MAC_MIC_INPUT=1) + │ NVidia STT → vLLM → Gradium TTS + ▼ + SmallWebRTCTransport.output() + │ WebRTC (SRTP, peer-to-peer) + ▼ + iOS app (receive-only, enableMic = false) + │ Bluetooth HFP + ▼ + 🕶️ Glasses speaker ← agent's voice +``` + +- **Agent listens via the Mac mic** (`MAC_MIC_INPUT=1`) — see the server change + in `bot-courtline.py`. +- **Agent speaks through the glasses** — the iOS app connects **receive-only** + (`WebRTCAudioSpike.swift`, `enableMic = false`) and plays the agent's audio + out through the glasses' HFP speaker. This leg is verified working. + +## Setup + +**Agent (server):** +```bash +cd ../yc-voice-agents-hackathon/server +uv sync # picks up the new sounddevice dep +export MAC_MIC_INPUT=1 # capture this Mac's mic into the pipeline +uv run main.py # agent on http://0.0.0.0:7860 (serves /api/offer) +# macOS will prompt for microphone access for the terminal — allow it. +``` + +**iOS app:** +1. Open `RayBan.xcodeproj`, set your signing team. +2. Set `webRTCServerURL` in `RayBan/ViewModels/WebRTCAudioSpike.swift` to the + Mac's LAN IP + port, e.g. `http://192.168.1.42:7860`. +3. Build & run on a real device with the glasses paired (same Wi-Fi/LAN — see + the network note below). Start a stream, tap the **waveform** button. + +## What's verified vs. pending + +- ✅ iOS app compiles against the real SDKs (PipecatClientIOS / SmallWebRTC / + WebRTC) and connects to a SmallWebRTC `/api/offer`; the agent's audio plays + through the glasses (downlink proven end-to-end). +- ✅ Mac-mic capture (`sounddevice`) streams audio into a Pipecat pipeline + (verified standalone; wired here behind `MAC_MIC_INPUT`). +- ⏳ The full Mac-mic → STT → LLM → TTS → glasses loop needs an on-stack run with + the agent's API keys (Gradium / NVIDIA / OpenAI) — not runnable in CI. +- ⏳ Glasses **video → agent vision** (`vision.py` / `check_camera`) is not wired + yet; the legacy WebSocket video path (`streamPublishHost`) is a placeholder. + +## Network note + +Peer-to-peer WebRTC needs a routable media path. On the same Wi-Fi/LAN this +works via host ICE candidates. It will **not** traverse a Cloudflare HTTP tunnel +(HTTP only, not the UDP media). For client-isolation networks (e.g. some phone +hotspots) you need a TURN server. + +> The Xcode target is named `RayBan` internally; the display name is CourtLine. +> Derived from Meta's DAT sample app — source headers retain Meta's copyright. diff --git a/ios/RayBan.xcodeproj/project.pbxproj b/ios/RayBan.xcodeproj/project.pbxproj new file mode 100644 index 0000000..5ce887d --- /dev/null +++ b/ios/RayBan.xcodeproj/project.pbxproj @@ -0,0 +1,824 @@ +// !$*UTF8*$! +{ + archiveVersion = 1; + classes = { + }; + objectVersion = 70; + objects = { + +/* Begin PBXBuildFile section */ + 666087EB27F1DA0E8B6CFE6F /* PipecatClientIOSSmallWebrtc in Frameworks */ = {isa = PBXBuildFile; productRef = 2AD8DB5BA32EA7072395DCF4 /* PipecatClientIOSSmallWebrtc */; }; + 8F2D23802E856711002D0588 /* DebugMenuViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8F2D237F2E856711002D0588 /* DebugMenuViewModel.swift */; }; + 8F8F00782E8ACB4600A4BDAF /* WearablesViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8F8F00772E8ACB4500A4BDAF /* WearablesViewModel.swift */; }; + 8FD96B7F2E6F0A9800F56AB1 /* RayBanApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FD96B792E6F0A9800F56AB1 /* RayBanApp.swift */; }; + 8FD96B812E6F0A9800F56AB1 /* HomeScreenView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FD96B722E6F0A9800F56AB1 /* HomeScreenView.swift */; }; + 8FD96B872E6F0A9800F56AB1 /* StreamSessionViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FD96B6F2E6F0A9800F56AB1 /* StreamSessionViewModel.swift */; }; + 8FD96B882E6F0A9800F56AB1 /* StreamSessionView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FD96B752E6F0A9800F56AB1 /* StreamSessionView.swift */; }; + 8FD96B8A2E6F0A9800F56AB1 /* PhotoPreviewView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FD96B742E6F0A9800F56AB1 /* PhotoPreviewView.swift */; }; + 8FD96B8D2E6F0A9800F56AB1 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 8FD96B772E6F0A9800F56AB1 /* Assets.xcassets */; }; + 8FD96B952E6F0F7D00F56AB1 /* MWDATCore in Frameworks */ = {isa = PBXBuildFile; productRef = 8FD96B942E6F0F7D00F56AB1 /* MWDATCore */; }; + 8FD96B952E6F0F7D00F56AB2 /* MWDATCamera in Frameworks */ = {isa = PBXBuildFile; productRef = 8FD96B942E6F0F7D00F56AB2 /* MWDATCamera */; }; + 8FD96B952E6F0F7D00F56AB3 /* MWDATMockDevice in Frameworks */ = {isa = PBXBuildFile; productRef = 8FD96B942E6F0F7D00F56AB3 /* MWDATMockDevice */; }; + 8FFD5FF52E8422580035E446 /* CircleButton.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD5FF22E8422580035E446 /* CircleButton.swift */; }; + 8FFD5FF62E8422580035E446 /* CustomButton.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD5FF32E8422580035E446 /* CustomButton.swift */; }; + 8FFD602C2E8433E20035E446 /* MockDeviceKitViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD60292E8433E20035E446 /* MockDeviceKitViewModel.swift */; }; + 8FFD602D2E8433E20035E446 /* MockDeviceViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD602A2E8433E20035E446 /* MockDeviceViewModel.swift */; }; + 8FFD60342E8434070035E446 /* MediaPickerView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD60322E8434070035E446 /* MediaPickerView.swift */; }; + 8FFD60352E8434070035E446 /* StatusText.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD60332E8434070035E446 /* StatusText.swift */; }; + 8FFD60542E849D0D0035E446 /* RegistrationView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD60532E849D0D0035E446 /* RegistrationView.swift */; }; + 8FFD60602E84A2F70035E446 /* MainAppView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD605F2E84A2F70035E446 /* MainAppView.swift */; }; + 8FFD60612E84A2F70035E446 /* DebugMenuView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8FFD605E2E84A2F70035E446 /* DebugMenuView.swift */; }; + E66CC01B2F3E1A9300B07379 /* plant.png in Resources */ = {isa = PBXBuildFile; fileRef = E66CC0192F3E1A9300B07379 /* plant.png */; }; + E66CC01C2F3E1A9300B07379 /* plant.mp4 in Resources */ = {isa = PBXBuildFile; fileRef = E66CC0182F3E1A9300B07379 /* plant.mp4 */; }; + E66CC01D2F3E1A9300B07380 /* MWDATMockDeviceTestClient in Frameworks */ = {isa = PBXBuildFile; productRef = E66CC01E2F3E1A9300B07380 /* MWDATMockDeviceTestClient */; }; + E66D30242E7DA71900470B48 /* MockDeviceKitButton.swift in Sources */ = {isa = PBXBuildFile; fileRef = E66D30232E7DA71900470B48 /* MockDeviceKitButton.swift */; }; + E6A188482EB918740097D0E1 /* StreamView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E6A188472EB918740097D0E1 /* StreamView.swift */; }; + E6DA451D2E79A63100E3F688 /* MockDeviceCardView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E6DA45182E79A63100E3F688 /* MockDeviceCardView.swift */; }; + E6DA451E2E79A63100E3F688 /* CardView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E6DA45172E79A63100E3F688 /* CardView.swift */; }; + E6DA451F2E79A63100E3F688 /* MockDeviceKitView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E6DA451A2E79A63100E3F688 /* MockDeviceKitView.swift */; }; + E6FD3BCE2EB4D53A00E7FE5D /* NonStreamView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E6FD3BCD2EB4D53A00E7FE5D /* NonStreamView.swift */; }; + FEEE00012F95000100000001 /* DeviceSessionManager.swift in Sources */ = {isa = PBXBuildFile; fileRef = FEEE00002F95000100000001 /* DeviceSessionManager.swift */; }; + FEEE00112F95000100000011 /* WebRTCAudioSpike.swift in Sources */ = {isa = PBXBuildFile; fileRef = FEEE00102F95000100000010 /* WebRTCAudioSpike.swift */; }; +/* End PBXBuildFile section */ + +/* Begin PBXContainerItemProxy section */ + E66CC0132F3E05D900B07379 /* PBXContainerItemProxy */ = { + isa = PBXContainerItemProxy; + containerPortal = CCCCCCCCCCCCCCCCCCCCCC /* Project object */; + proxyType = 1; + remoteGlobalIDString = 8A8A8A8A8A8A8A8A8A8A8A8A; + remoteInfo = RayBan; + }; + E699CC992E8150670052C240 /* PBXContainerItemProxy */ = { + isa = PBXContainerItemProxy; + containerPortal = CCCCCCCCCCCCCCCCCCCCCC /* Project object */; + proxyType = 1; + remoteGlobalIDString = 8A8A8A8A8A8A8A8A8A8A8A8A; + remoteInfo = RayBan; + }; +/* End PBXContainerItemProxy section */ + +/* Begin PBXCopyFilesBuildPhase section */ + 8FD96B932E6F0C6E00F56AB1 /* Embed Frameworks */ = { + isa = PBXCopyFilesBuildPhase; + buildActionMask = 2147483647; + dstPath = ""; + dstSubfolderSpec = 10; + files = ( + ); + name = "Embed Frameworks"; + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXCopyFilesBuildPhase section */ + +/* Begin PBXFileReference section */ + 3A3A3A3A3A3A3A3A3A3A3A3A /* RayBan.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = RayBan.app; sourceTree = BUILT_PRODUCTS_DIR; }; + 8F2D237F2E856711002D0588 /* DebugMenuViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DebugMenuViewModel.swift; sourceTree = ""; }; + 8F8F00772E8ACB4500A4BDAF /* WearablesViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WearablesViewModel.swift; sourceTree = ""; }; + 8FD96B6F2E6F0A9800F56AB1 /* StreamSessionViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = StreamSessionViewModel.swift; sourceTree = ""; }; + 8FD96B722E6F0A9800F56AB1 /* HomeScreenView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HomeScreenView.swift; sourceTree = ""; }; + 8FD96B742E6F0A9800F56AB1 /* PhotoPreviewView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PhotoPreviewView.swift; sourceTree = ""; }; + 8FD96B752E6F0A9800F56AB1 /* StreamSessionView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = StreamSessionView.swift; sourceTree = ""; }; + 8FD96B772E6F0A9800F56AB1 /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = ""; }; + 8FD96B782E6F0A9800F56AB1 /* RayBan.entitlements */ = {isa = PBXFileReference; lastKnownFileType = text.plist.entitlements; path = RayBan.entitlements; sourceTree = ""; }; + 8FD96B792E6F0A9800F56AB1 /* RayBanApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RayBanApp.swift; sourceTree = ""; }; + 8FD96B7B2E6F0A9800F56AB1 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; + 8FFD5FF22E8422580035E446 /* CircleButton.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CircleButton.swift; sourceTree = ""; }; + 8FFD5FF32E8422580035E446 /* CustomButton.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CustomButton.swift; sourceTree = ""; }; + 8FFD60292E8433E20035E446 /* MockDeviceKitViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MockDeviceKitViewModel.swift; sourceTree = ""; }; + 8FFD602A2E8433E20035E446 /* MockDeviceViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MockDeviceViewModel.swift; sourceTree = ""; }; + 8FFD60322E8434070035E446 /* MediaPickerView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPickerView.swift; sourceTree = ""; }; + 8FFD60332E8434070035E446 /* StatusText.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = StatusText.swift; sourceTree = ""; }; + 8FFD60532E849D0D0035E446 /* RegistrationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RegistrationView.swift; sourceTree = ""; }; + 8FFD605E2E84A2F70035E446 /* DebugMenuView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DebugMenuView.swift; sourceTree = ""; }; + 8FFD605F2E84A2F70035E446 /* MainAppView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MainAppView.swift; sourceTree = ""; }; + E66CC00D2F3E05D900B07379 /* RayBanUITests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = RayBanUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; + E66CC0182F3E1A9300B07379 /* plant.mp4 */ = {isa = PBXFileReference; lastKnownFileType = file; path = plant.mp4; sourceTree = ""; }; + E66CC0192F3E1A9300B07379 /* plant.png */ = {isa = PBXFileReference; lastKnownFileType = image.png; path = plant.png; sourceTree = ""; }; + E66D30232E7DA71900470B48 /* MockDeviceKitButton.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MockDeviceKitButton.swift; sourceTree = ""; }; + E699CC952E8150670052C240 /* RayBanTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = RayBanTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; + E6A188472EB918740097D0E1 /* StreamView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = StreamView.swift; sourceTree = ""; }; + E6DA45172E79A63100E3F688 /* CardView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CardView.swift; sourceTree = ""; }; + E6DA45182E79A63100E3F688 /* MockDeviceCardView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MockDeviceCardView.swift; sourceTree = ""; }; + E6DA451A2E79A63100E3F688 /* MockDeviceKitView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MockDeviceKitView.swift; sourceTree = ""; }; + E6FD3BCD2EB4D53A00E7FE5D /* NonStreamView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NonStreamView.swift; sourceTree = ""; }; + FEEE00002F95000100000001 /* DeviceSessionManager.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DeviceSessionManager.swift; sourceTree = ""; }; + FEEE00102F95000100000010 /* WebRTCAudioSpike.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebRTCAudioSpike.swift; sourceTree = ""; }; +/* End PBXFileReference section */ + +/* Begin PBXFileSystemSynchronizedRootGroup section */ + E66CC00E2F3E05D900B07379 /* RayBanUITests */ = {isa = PBXFileSystemSynchronizedRootGroup; explicitFileTypes = {}; explicitFolders = (); path = RayBanUITests; sourceTree = ""; }; + E699CC962E8150670052C240 /* RayBanTests */ = {isa = PBXFileSystemSynchronizedRootGroup; explicitFileTypes = {}; explicitFolders = (); path = RayBanTests; sourceTree = ""; }; +/* End PBXFileSystemSynchronizedRootGroup section */ + +/* Begin PBXFrameworksBuildPhase section */ + 4A4A4A4A4A4A4A4A4A4A4A4A /* Frameworks */ = { + isa = PBXFrameworksBuildPhase; + buildActionMask = 2147483647; + files = ( + 8FD96B952E6F0F7D00F56AB1 /* MWDATCore in Frameworks */, + 8FD96B952E6F0F7D00F56AB2 /* MWDATCamera in Frameworks */, + 8FD96B952E6F0F7D00F56AB3 /* MWDATMockDevice in Frameworks */, + 666087EB27F1DA0E8B6CFE6F /* PipecatClientIOSSmallWebrtc in Frameworks */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; + E66CC00A2F3E05D900B07379 /* Frameworks */ = { + isa = PBXFrameworksBuildPhase; + buildActionMask = 2147483647; + files = ( + E66CC01D2F3E1A9300B07380 /* MWDATMockDeviceTestClient in Frameworks */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXFrameworksBuildPhase section */ + +/* Begin PBXGroup section */ + 5A5A5A5A5A5A5A5A5A5A5A5A = { + isa = PBXGroup; + children = ( + 8FD96B7D2E6F0A9800F56AB1 /* RayBan */, + E66CC00E2F3E05D900B07379 /* RayBanUITests */, + 8F2A14F12DDBCEB900D4E5F2 /* Frameworks */, + 7A7A7A7A7A7A7A7A7A7A7A7A /* Products */, + E699CC962E8150670052C240 /* RayBanTests */, + ); + sourceTree = ""; + }; + 7A7A7A7A7A7A7A7A7A7A7A7A /* Products */ = { + isa = PBXGroup; + children = ( + 3A3A3A3A3A3A3A3A3A3A3A3A /* RayBan.app */, + E699CC952E8150670052C240 /* RayBanTests.xctest */, + E66CC00D2F3E05D900B07379 /* RayBanUITests.xctest */, + ); + name = Products; + sourceTree = ""; + }; + 8F2A14F12DDBCEB900D4E5F2 /* Frameworks */ = { + isa = PBXGroup; + children = ( + ); + name = Frameworks; + sourceTree = ""; + }; + 8FD96B702E6F0A9800F56AB1 /* ViewModels */ = { + isa = PBXGroup; + children = ( + 8FFD602B2E8433E20035E446 /* MockDeviceKit */, + 8F2D237F2E856711002D0588 /* DebugMenuViewModel.swift */, + FEEE00002F95000100000001 /* DeviceSessionManager.swift */, + 8FD96B6F2E6F0A9800F56AB1 /* StreamSessionViewModel.swift */, + FEEE00102F95000100000010 /* WebRTCAudioSpike.swift */, + 8F8F00772E8ACB4500A4BDAF /* WearablesViewModel.swift */, + ); + path = ViewModels; + sourceTree = ""; + }; + 8FD96B762E6F0A9800F56AB1 /* Views */ = { + isa = PBXGroup; + children = ( + 8FFD5FF42E8422580035E446 /* Components */, + E6DA451B2E79A63100E3F688 /* MockDeviceKit */, + 8FFD605E2E84A2F70035E446 /* DebugMenuView.swift */, + 8FD96B722E6F0A9800F56AB1 /* HomeScreenView.swift */, + 8FFD605F2E84A2F70035E446 /* MainAppView.swift */, + 8FD96B742E6F0A9800F56AB1 /* PhotoPreviewView.swift */, + 8FFD60532E849D0D0035E446 /* RegistrationView.swift */, + 8FD96B752E6F0A9800F56AB1 /* StreamSessionView.swift */, + E6A188472EB918740097D0E1 /* StreamView.swift */, + E6FD3BCD2EB4D53A00E7FE5D /* NonStreamView.swift */, + ); + path = Views; + sourceTree = ""; + }; + 8FD96B7D2E6F0A9800F56AB1 /* RayBan */ = { + isa = PBXGroup; + children = ( + E66CC01A2F3E1A9300B07379 /* TestResources */, + 8FD96B702E6F0A9800F56AB1 /* ViewModels */, + 8FD96B762E6F0A9800F56AB1 /* Views */, + 8FD96B772E6F0A9800F56AB1 /* Assets.xcassets */, + 8FD96B782E6F0A9800F56AB1 /* RayBan.entitlements */, + 8FD96B792E6F0A9800F56AB1 /* RayBanApp.swift */, + 8FD96B7B2E6F0A9800F56AB1 /* Info.plist */, + ); + path = RayBan; + sourceTree = ""; + }; + 8FFD5FF42E8422580035E446 /* Components */ = { + isa = PBXGroup; + children = ( + 8FFD60322E8434070035E446 /* MediaPickerView.swift */, + 8FFD60332E8434070035E446 /* StatusText.swift */, + 8FFD5FF22E8422580035E446 /* CircleButton.swift */, + 8FFD5FF32E8422580035E446 /* CustomButton.swift */, + E6DA45172E79A63100E3F688 /* CardView.swift */, + ); + path = Components; + sourceTree = ""; + }; + 8FFD602B2E8433E20035E446 /* MockDeviceKit */ = { + isa = PBXGroup; + children = ( + 8FFD60292E8433E20035E446 /* MockDeviceKitViewModel.swift */, + 8FFD602A2E8433E20035E446 /* MockDeviceViewModel.swift */, + ); + path = MockDeviceKit; + sourceTree = ""; + }; + E66CC01A2F3E1A9300B07379 /* TestResources */ = { + isa = PBXGroup; + children = ( + E66CC0182F3E1A9300B07379 /* plant.mp4 */, + E66CC0192F3E1A9300B07379 /* plant.png */, + ); + path = TestResources; + sourceTree = ""; + }; + E6DA451B2E79A63100E3F688 /* MockDeviceKit */ = { + isa = PBXGroup; + children = ( + E6DA45182E79A63100E3F688 /* MockDeviceCardView.swift */, + E6DA451A2E79A63100E3F688 /* MockDeviceKitView.swift */, + E66D30232E7DA71900470B48 /* MockDeviceKitButton.swift */, + ); + path = MockDeviceKit; + sourceTree = ""; + }; +/* End PBXGroup section */ + +/* Begin PBXNativeTarget section */ + 8A8A8A8A8A8A8A8A8A8A8A8A /* RayBan */ = { + isa = PBXNativeTarget; + buildConfigurationList = 9A9A9A9A9A9A9A9A9A9A9A9A /* Build configuration list for PBXNativeTarget "RayBan" */; + buildPhases = ( + AAAAAAAAAAAAAAAAAAAAAA /* Sources */, + 4A4A4A4A4A4A4A4A4A4A4A4A /* Frameworks */, + BBBBBBBBBBBBBBBBBBBBBB /* Resources */, + 8FD96B932E6F0C6E00F56AB1 /* Embed Frameworks */, + ); + buildRules = ( + ); + dependencies = ( + ); + name = RayBan; + packageProductDependencies = ( + 2AD8DB5BA32EA7072395DCF4 /* PipecatClientIOSSmallWebrtc */, + ); + productName = RayBan; + productReference = 3A3A3A3A3A3A3A3A3A3A3A3A /* RayBan.app */; + productType = "com.apple.product-type.application"; + }; + E66CC00C2F3E05D900B07379 /* RayBanUITests */ = { + isa = PBXNativeTarget; + buildConfigurationList = E66CC0172F3E05D900B07379 /* Build configuration list for PBXNativeTarget "RayBanUITests" */; + buildPhases = ( + E66CC0092F3E05D900B07379 /* Sources */, + E66CC00A2F3E05D900B07379 /* Frameworks */, + E66CC00B2F3E05D900B07379 /* Resources */, + ); + buildRules = ( + ); + dependencies = ( + E66CC0142F3E05D900B07379 /* PBXTargetDependency */, + ); + fileSystemSynchronizedGroups = ( + E66CC00E2F3E05D900B07379 /* RayBanUITests */, + ); + name = RayBanUITests; + packageProductDependencies = ( + E66CC01E2F3E1A9300B07380 /* MWDATMockDeviceTestClient */, + ); + productName = RayBanUITests; + productReference = E66CC00D2F3E05D900B07379 /* RayBanUITests.xctest */; + productType = "com.apple.product-type.bundle.ui-testing"; + }; + E699CC942E8150670052C240 /* RayBanTests */ = { + isa = PBXNativeTarget; + buildConfigurationList = E699CC9D2E8150670052C240 /* Build configuration list for PBXNativeTarget "RayBanTests" */; + buildPhases = ( + E699CC912E8150670052C240 /* Sources */, + E699CC932E8150670052C240 /* Resources */, + ); + buildRules = ( + ); + dependencies = ( + E699CC9A2E8150670052C240 /* PBXTargetDependency */, + ); + fileSystemSynchronizedGroups = ( + E699CC962E8150670052C240 /* RayBanTests */, + ); + name = RayBanTests; + packageProductDependencies = ( + ); + productName = RayBanTests; + productReference = E699CC952E8150670052C240 /* RayBanTests.xctest */; + productType = "com.apple.product-type.bundle.unit-test"; + }; +/* End PBXNativeTarget section */ + +/* Begin PBXProject section */ + CCCCCCCCCCCCCCCCCCCCCC /* Project object */ = { + isa = PBXProject; + attributes = { + LastSwiftUpdateCheck = 2620; + LastUpgradeCheck = 1250; + TargetAttributes = { + 8A8A8A8A8A8A8A8A8A8A8A8A = { + CreatedOnToolsVersion = 12.5; + }; + E66CC00C2F3E05D900B07379 = { + CreatedOnToolsVersion = 26.2; + TestTargetID = 8A8A8A8A8A8A8A8A8A8A8A8A; + }; + E699CC942E8150670052C240 = { + CreatedOnToolsVersion = 26.0; + TestTargetID = 8A8A8A8A8A8A8A8A8A8A8A8A; + }; + }; + }; + buildConfigurationList = DDDDDDDDDDDDDDDDDDDDDD /* Build configuration list for PBXProject "RayBan" */; + compatibilityVersion = "Xcode 9.3"; + developmentRegion = en; + hasScannedForEncodings = 0; + knownRegions = ( + en, + Base, + ); + mainGroup = 5A5A5A5A5A5A5A5A5A5A5A5A; + packageReferences = ( + 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */, + 1B659E20A56E28E4E597FDDA /* XCRemoteSwiftPackageReference "pipecat-client-ios-small-webrtc" */, + ); + productRefGroup = 7A7A7A7A7A7A7A7A7A7A7A7A /* Products */; + projectDirPath = ""; + projectRoot = ""; + targets = ( + 8A8A8A8A8A8A8A8A8A8A8A8A /* RayBan */, + E699CC942E8150670052C240 /* RayBanTests */, + E66CC00C2F3E05D900B07379 /* RayBanUITests */, + ); + }; +/* End PBXProject section */ + +/* Begin PBXResourcesBuildPhase section */ + BBBBBBBBBBBBBBBBBBBBBB /* Resources */ = { + isa = PBXResourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + E66CC01B2F3E1A9300B07379 /* plant.png in Resources */, + E66CC01C2F3E1A9300B07379 /* plant.mp4 in Resources */, + 8FD96B8D2E6F0A9800F56AB1 /* Assets.xcassets in Resources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; + E66CC00B2F3E05D900B07379 /* Resources */ = { + isa = PBXResourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + runOnlyForDeploymentPostprocessing = 0; + }; + E699CC932E8150670052C240 /* Resources */ = { + isa = PBXResourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXResourcesBuildPhase section */ + +/* Begin PBXSourcesBuildPhase section */ + AAAAAAAAAAAAAAAAAAAAAA /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + 8FD96B7F2E6F0A9800F56AB1 /* RayBanApp.swift in Sources */, + 8FD96B812E6F0A9800F56AB1 /* HomeScreenView.swift in Sources */, + 8F2D23802E856711002D0588 /* DebugMenuViewModel.swift in Sources */, + FEEE00012F95000100000001 /* DeviceSessionManager.swift in Sources */, + FEEE00112F95000100000011 /* WebRTCAudioSpike.swift in Sources */, + 8FFD60542E849D0D0035E446 /* RegistrationView.swift in Sources */, + 8FFD60342E8434070035E446 /* MediaPickerView.swift in Sources */, + 8FFD60352E8434070035E446 /* StatusText.swift in Sources */, + 8FFD602C2E8433E20035E446 /* MockDeviceKitViewModel.swift in Sources */, + 8FFD602D2E8433E20035E446 /* MockDeviceViewModel.swift in Sources */, + E6DA451D2E79A63100E3F688 /* MockDeviceCardView.swift in Sources */, + E6DA451E2E79A63100E3F688 /* CardView.swift in Sources */, + E6DA451F2E79A63100E3F688 /* MockDeviceKitView.swift in Sources */, + 8FD96B872E6F0A9800F56AB1 /* StreamSessionViewModel.swift in Sources */, + 8FD96B882E6F0A9800F56AB1 /* StreamSessionView.swift in Sources */, + 8FFD60602E84A2F70035E446 /* MainAppView.swift in Sources */, + 8FFD60612E84A2F70035E446 /* DebugMenuView.swift in Sources */, + 8FD96B8A2E6F0A9800F56AB1 /* PhotoPreviewView.swift in Sources */, + E66D30242E7DA71900470B48 /* MockDeviceKitButton.swift in Sources */, + 8F8F00782E8ACB4600A4BDAF /* WearablesViewModel.swift in Sources */, + E6A188482EB918740097D0E1 /* StreamView.swift in Sources */, + 8FFD5FF52E8422580035E446 /* CircleButton.swift in Sources */, + 8FFD5FF62E8422580035E446 /* CustomButton.swift in Sources */, + E6FD3BCE2EB4D53A00E7FE5D /* NonStreamView.swift in Sources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; + E66CC0092F3E05D900B07379 /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + runOnlyForDeploymentPostprocessing = 0; + }; + E699CC912E8150670052C240 /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXSourcesBuildPhase section */ + +/* Begin PBXTargetDependency section */ + E66CC0142F3E05D900B07379 /* PBXTargetDependency */ = { + isa = PBXTargetDependency; + target = 8A8A8A8A8A8A8A8A8A8A8A8A /* RayBan */; + targetProxy = E66CC0132F3E05D900B07379 /* PBXContainerItemProxy */; + }; + E699CC9A2E8150670052C240 /* PBXTargetDependency */ = { + isa = PBXTargetDependency; + target = 8A8A8A8A8A8A8A8A8A8A8A8A /* RayBan */; + targetProxy = E699CC992E8150670052C240 /* PBXContainerItemProxy */; + }; +/* End PBXTargetDependency section */ + +/* Begin XCBuildConfiguration section */ + 0B0B0B0B0B0B0B0B0B0B0B0B /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor; + CODE_SIGN_ENTITLEMENTS = RayBan/RayBan.entitlements; + CODE_SIGN_IDENTITY = "Apple Development"; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_ASSET_PATHS = ""; + DEVELOPMENT_TEAM = 8HJ6LVW8JZ; + ENABLE_PREVIEWS = YES; + FRAMEWORK_SEARCH_PATHS = ""; + INFOPLIST_FILE = RayBan/Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + ); + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.joshua.CourtLine; + PRODUCT_NAME = "$(TARGET_NAME)"; + PROVISIONING_PROFILE_SPECIFIER = ""; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + }; + name = Debug; + }; + 0C0C0C0C0C0C0C0C0C0C0C0C /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor; + CODE_SIGN_ENTITLEMENTS = RayBan/RayBan.entitlements; + CODE_SIGN_IDENTITY = "Apple Development"; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_ASSET_PATHS = ""; + DEVELOPMENT_TEAM = 8HJ6LVW8JZ; + ENABLE_PREVIEWS = YES; + INFOPLIST_FILE = RayBan/Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + ); + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.joshua.CourtLine; + PRODUCT_NAME = "$(TARGET_NAME)"; + PROVISIONING_PROFILE_SPECIFIER = ""; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + }; + name = Release; + }; + E66CC0152F3E05D900B07379 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++20"; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu17; + GENERATE_INFOPLIST_FILE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LOCALIZATION_PREFERS_STRING_CATALOGS = YES; + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.meta.wearables.external.RayBanUITests; + PRODUCT_NAME = "$(TARGET_NAME)"; + STRING_CATALOG_GENERATE_SYMBOLS = NO; + SWIFT_ACTIVE_COMPILATION_CONDITIONS = "DEBUG $(inherited)"; + SWIFT_APPROACHABLE_CONCURRENCY = YES; + SWIFT_EMIT_LOC_STRINGS = NO; + SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + TEST_TARGET_NAME = RayBan; + }; + name = Debug; + }; + E66CC0162F3E05D900B07379 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++20"; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu17; + GENERATE_INFOPLIST_FILE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LOCALIZATION_PREFERS_STRING_CATALOGS = YES; + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.meta.wearables.external.RayBanUITests; + PRODUCT_NAME = "$(TARGET_NAME)"; + STRING_CATALOG_GENERATE_SYMBOLS = NO; + SWIFT_APPROACHABLE_CONCURRENCY = YES; + SWIFT_EMIT_LOC_STRINGS = NO; + SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + TEST_TARGET_NAME = RayBan; + }; + name = Release; + }; + E699CC9B2E8150670052C240 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + BUNDLE_LOADER = "$(TEST_HOST)"; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++20"; + CODE_SIGN_STYLE = Manual; + CURRENT_PROJECT_VERSION = 1; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu17; + GENERATE_INFOPLIST_FILE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LOCALIZATION_PREFERS_STRING_CATALOGS = YES; + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.meta.wearables.external.RayBanTests; + PRODUCT_NAME = "$(TARGET_NAME)"; + STRING_CATALOG_GENERATE_SYMBOLS = NO; + SUPPORTED_PLATFORMS = "iphoneos iphonesimulator"; + SUPPORTS_MACCATALYST = NO; + SWIFT_ACTIVE_COMPILATION_CONDITIONS = "DEBUG $(inherited)"; + SWIFT_APPROACHABLE_CONCURRENCY = YES; + SWIFT_EMIT_LOC_STRINGS = NO; + SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + TEST_HOST = "$(BUILT_PRODUCTS_DIR)/RayBan.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/RayBan"; + }; + name = Debug; + }; + E699CC9C2E8150670052C240 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES; + BUNDLE_LOADER = "$(TEST_HOST)"; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++20"; + CODE_SIGN_STYLE = Manual; + CURRENT_PROJECT_VERSION = 1; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu17; + GENERATE_INFOPLIST_FILE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + LOCALIZATION_PREFERS_STRING_CATALOGS = YES; + MARKETING_VERSION = 1.0; + PRODUCT_BUNDLE_IDENTIFIER = com.meta.wearables.external.RayBanTests; + PRODUCT_NAME = "$(TARGET_NAME)"; + STRING_CATALOG_GENERATE_SYMBOLS = NO; + SUPPORTED_PLATFORMS = "iphoneos iphonesimulator"; + SUPPORTS_MACCATALYST = NO; + SWIFT_APPROACHABLE_CONCURRENCY = YES; + SWIFT_EMIT_LOC_STRINGS = NO; + SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES; + SWIFT_VERSION = 5.0; + TARGETED_DEVICE_FAMILY = "1,2"; + TEST_HOST = "$(BUILT_PRODUCTS_DIR)/RayBan.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/RayBan"; + }; + name = Release; + }; + EEEEEEEEEEEEEEEEEEEEEE /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_ENABLE_OBJC_WEAK = YES; + CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_COMMA = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INFINITE_RECURSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; + CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; + CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES; + CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; + CLANG_WARN_STRICT_PROTOTYPES = YES; + CLANG_WARN_SUSPICIOUS_MOVE = YES; + CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = dwarf; + ENABLE_STRICT_OBJC_MSGSEND = YES; + ENABLE_TESTABILITY = YES; + GCC_C_LANGUAGE_STANDARD = gnu11; + GCC_DYNAMIC_NO_PIC = NO; + GCC_NO_COMMON_BLOCKS = YES; + GCC_OPTIMIZATION_LEVEL = 0; + GCC_PREPROCESSOR_DEFINITIONS = ( + "DEBUG=1", + "$(inherited)", + ); + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 17.0; + MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE; + MTL_FAST_MATH = YES; + ONLY_ACTIVE_ARCH = YES; + SDKROOT = iphoneos; + SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG; + SWIFT_OPTIMIZATION_LEVEL = "-Onone"; + }; + name = Debug; + }; + FFFFFFFFFFFFFFFFFFFFFFFF /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_ENABLE_OBJC_WEAK = YES; + CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_COMMA = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INFINITE_RECURSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; + CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; + CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES; + CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; + CLANG_WARN_STRICT_PROTOTYPES = YES; + CLANG_WARN_SUSPICIOUS_MOVE = YES; + CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; + ENABLE_NS_ASSERTIONS = NO; + ENABLE_STRICT_OBJC_MSGSEND = YES; + GCC_C_LANGUAGE_STANDARD = gnu11; + GCC_NO_COMMON_BLOCKS = YES; + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 14.5; + MTL_ENABLE_DEBUG_INFO = NO; + MTL_FAST_MATH = YES; + SDKROOT = iphoneos; + SWIFT_COMPILATION_MODE = wholemodule; + SWIFT_OPTIMIZATION_LEVEL = "-O"; + VALIDATE_PRODUCT = YES; + }; + name = Release; + }; +/* End XCBuildConfiguration section */ + +/* Begin XCConfigurationList section */ + 9A9A9A9A9A9A9A9A9A9A9A9A /* Build configuration list for PBXNativeTarget "RayBan" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + 0B0B0B0B0B0B0B0B0B0B0B0B /* Debug */, + 0C0C0C0C0C0C0C0C0C0C0C0C /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; + DDDDDDDDDDDDDDDDDDDDDD /* Build configuration list for PBXProject "RayBan" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + EEEEEEEEEEEEEEEEEEEEEE /* Debug */, + FFFFFFFFFFFFFFFFFFFFFFFF /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; + E66CC0172F3E05D900B07379 /* Build configuration list for PBXNativeTarget "RayBanUITests" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + E66CC0152F3E05D900B07379 /* Debug */, + E66CC0162F3E05D900B07379 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; + E699CC9D2E8150670052C240 /* Build configuration list for PBXNativeTarget "RayBanTests" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + E699CC9B2E8150670052C240 /* Debug */, + E699CC9C2E8150670052C240 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; +/* End XCConfigurationList section */ + +/* Begin XCRemoteSwiftPackageReference section */ + 1B659E20A56E28E4E597FDDA /* XCRemoteSwiftPackageReference "pipecat-client-ios-small-webrtc" */ = { + isa = XCRemoteSwiftPackageReference; + repositoryURL = "https://github.com/pipecat-ai/pipecat-client-ios-small-webrtc.git"; + requirement = { + kind = upToNextMajorVersion; + minimumVersion = 1.2.0; + }; + }; + 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */ = { + isa = XCRemoteSwiftPackageReference; + repositoryURL = "https://github.com/facebook/meta-wearables-dat-ios"; + requirement = { + kind = exactVersion; + version = 0.7.0; + }; + }; +/* End XCRemoteSwiftPackageReference section */ + +/* Begin XCSwiftPackageProductDependency section */ + 2AD8DB5BA32EA7072395DCF4 /* PipecatClientIOSSmallWebrtc */ = { + isa = XCSwiftPackageProductDependency; + package = 1B659E20A56E28E4E597FDDA /* XCRemoteSwiftPackageReference "pipecat-client-ios-small-webrtc" */; + productName = PipecatClientIOSSmallWebrtc; + }; + 8FD96B942E6F0F7D00F56AB1 /* MWDATCore */ = { + isa = XCSwiftPackageProductDependency; + package = 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */; + productName = MWDATCore; + }; + 8FD96B942E6F0F7D00F56AB2 /* MWDATCamera */ = { + isa = XCSwiftPackageProductDependency; + package = 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */; + productName = MWDATCamera; + }; + 8FD96B942E6F0F7D00F56AB3 /* MWDATMockDevice */ = { + isa = XCSwiftPackageProductDependency; + package = 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */; + productName = MWDATMockDevice; + }; + E66CC01E2F3E1A9300B07380 /* MWDATMockDeviceTestClient */ = { + isa = XCSwiftPackageProductDependency; + package = 95F7F3332DEF0D91006D1C1A /* XCRemoteSwiftPackageReference "meta-wearables-dat-ios" */; + productName = MWDATMockDeviceTestClient; + }; +/* End XCSwiftPackageProductDependency section */ + }; + rootObject = CCCCCCCCCCCCCCCCCCCCCC /* Project object */; +} diff --git a/ios/RayBan.xcodeproj/project.xcworkspace/contents.xcworkspacedata b/ios/RayBan.xcodeproj/project.xcworkspace/contents.xcworkspacedata new file mode 100644 index 0000000..919434a --- /dev/null +++ b/ios/RayBan.xcodeproj/project.xcworkspace/contents.xcworkspacedata @@ -0,0 +1,7 @@ + + + + + diff --git a/ios/RayBan.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved b/ios/RayBan.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved new file mode 100644 index 0000000..cf8844f --- /dev/null +++ b/ios/RayBan.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved @@ -0,0 +1,42 @@ +{ + "originHash" : "c7e9632b29b00e0dbe8f26843ce884bb8be8664360cfc339b4710c378606ca03", + "pins" : [ + { + "identity" : "meta-wearables-dat-ios", + "kind" : "remoteSourceControl", + "location" : "https://github.com/facebook/meta-wearables-dat-ios", + "state" : { + "revision" : "c40db3978a76b97504b85ed9f082be5549c5a486", + "version" : "0.7.0" + } + }, + { + "identity" : "pipecat-client-ios", + "kind" : "remoteSourceControl", + "location" : "https://github.com/pipecat-ai/pipecat-client-ios.git", + "state" : { + "revision" : "3d19581822fcc5462f9bf7532e20ed4e23b0b338", + "version" : "1.2.0" + } + }, + { + "identity" : "pipecat-client-ios-small-webrtc", + "kind" : "remoteSourceControl", + "location" : "https://github.com/pipecat-ai/pipecat-client-ios-small-webrtc.git", + "state" : { + "revision" : "4288581799b4626b00b86f209293de80a16acd43", + "version" : "1.2.0" + } + }, + { + "identity" : "webrtc", + "kind" : "remoteSourceControl", + "location" : "https://github.com/stasel/WebRTC", + "state" : { + "revision" : "5b2eb61cace7d62726b29a38b768b07d6bc55c45", + "version" : "134.0.0" + } + } + ], + "version" : 3 +} diff --git a/ios/RayBan.xcodeproj/xcshareddata/xcschemes/RayBan.xcscheme b/ios/RayBan.xcodeproj/xcshareddata/xcschemes/RayBan.xcscheme new file mode 100644 index 0000000..7e67398 --- /dev/null +++ b/ios/RayBan.xcodeproj/xcshareddata/xcschemes/RayBan.xcscheme @@ -0,0 +1,102 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ios/RayBan/Assets.xcassets/AccentColor.colorset/Contents.json b/ios/RayBan/Assets.xcassets/AccentColor.colorset/Contents.json new file mode 100644 index 0000000..4bf76f7 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/AccentColor.colorset/Contents.json @@ -0,0 +1,20 @@ +{ + "colors" : [ + { + "color" : { + "color-space" : "srgb", + "components" : { + "alpha" : "1.000", + "blue" : "1.000", + "green" : "0.475", + "red" : "0.000" + } + }, + "idiom" : "universal" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/AppIcon.appiconset/Contents.json b/ios/RayBan/Assets.xcassets/AppIcon.appiconset/Contents.json new file mode 100644 index 0000000..f7757d4 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/AppIcon.appiconset/Contents.json @@ -0,0 +1,14 @@ +{ + "images" : [ + { + "filename" : "imagine_a_film_camera_in_the_style.jpeg", + "idiom" : "universal", + "platform" : "ios", + "size" : "1024x1024" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/AppIcon.appiconset/imagine_a_film_camera_in_the_style.jpeg b/ios/RayBan/Assets.xcassets/AppIcon.appiconset/imagine_a_film_camera_in_the_style.jpeg new file mode 100644 index 0000000..b91090c Binary files /dev/null and b/ios/RayBan/Assets.xcassets/AppIcon.appiconset/imagine_a_film_camera_in_the_style.jpeg differ diff --git a/ios/RayBan/Assets.xcassets/Contents.json b/ios/RayBan/Assets.xcassets/Contents.json new file mode 100644 index 0000000..73c0059 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/Contents.json @@ -0,0 +1,6 @@ +{ + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/appPrimaryColor.colorset/Contents.json b/ios/RayBan/Assets.xcassets/appPrimaryColor.colorset/Contents.json new file mode 100644 index 0000000..9bce167 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/appPrimaryColor.colorset/Contents.json @@ -0,0 +1,29 @@ +{ + "colors" : [ + { + "color" : { + "color-space" : "srgb", + "components" : { + "alpha" : "1.000", + "blue" : "0xE0", + "green" : "0x64", + "red" : "0x00" + } + }, + "idiom" : "universal" + }, + { + "appearances" : [ + { + "appearance" : "luminosity", + "value" : "dark" + } + ], + "idiom" : "universal" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/Contents.json new file mode 100644 index 0000000..fc786d2 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "filename" : "cameraAccessIcon.png", + "idiom" : "universal", + "scale" : "2x" + }, + { + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/cameraAccessIcon.png b/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/cameraAccessIcon.png new file mode 100644 index 0000000..7f919e3 Binary files /dev/null and b/ios/RayBan/Assets.xcassets/cameraAccessIcon.imageset/cameraAccessIcon.png differ diff --git a/ios/RayBan/Assets.xcassets/destructiveBackground.colorset/Contents.json b/ios/RayBan/Assets.xcassets/destructiveBackground.colorset/Contents.json new file mode 100644 index 0000000..aa78b91 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/destructiveBackground.colorset/Contents.json @@ -0,0 +1,20 @@ +{ + "colors" : [ + { + "color" : { + "color-space" : "srgb", + "components" : { + "alpha" : "1.000", + "blue" : "0xDB", + "green" : "0xD8", + "red" : "0xFF" + } + }, + "idiom" : "universal" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/destructiveForeground.colorset/Contents.json b/ios/RayBan/Assets.xcassets/destructiveForeground.colorset/Contents.json new file mode 100644 index 0000000..31dafdb --- /dev/null +++ b/ios/RayBan/Assets.xcassets/destructiveForeground.colorset/Contents.json @@ -0,0 +1,20 @@ +{ + "colors" : [ + { + "color" : { + "color-space" : "srgb", + "components" : { + "alpha" : "1.000", + "blue" : "0x1E", + "green" : "0x07", + "red" : "0xAA" + } + }, + "idiom" : "universal" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/Contents.json new file mode 100644 index 0000000..ae0a59a --- /dev/null +++ b/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "filename" : "smartGlassesIcon.png", + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/smartGlassesIcon.png b/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/smartGlassesIcon.png new file mode 100644 index 0000000..25da5d9 Binary files /dev/null and b/ios/RayBan/Assets.xcassets/smartGlassesIcon.imageset/smartGlassesIcon.png differ diff --git a/ios/RayBan/Assets.xcassets/soundIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/soundIcon.imageset/Contents.json new file mode 100644 index 0000000..faa7379 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/soundIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "filename" : "soundIcon.png", + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/soundIcon.imageset/soundIcon.png b/ios/RayBan/Assets.xcassets/soundIcon.imageset/soundIcon.png new file mode 100644 index 0000000..4f3a2ca Binary files /dev/null and b/ios/RayBan/Assets.xcassets/soundIcon.imageset/soundIcon.png differ diff --git a/ios/RayBan/Assets.xcassets/tapIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/tapIcon.imageset/Contents.json new file mode 100644 index 0000000..6b2b21b --- /dev/null +++ b/ios/RayBan/Assets.xcassets/tapIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "filename" : "tapIcon.png", + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/tapIcon.imageset/tapIcon.png b/ios/RayBan/Assets.xcassets/tapIcon.imageset/tapIcon.png new file mode 100644 index 0000000..92a3805 Binary files /dev/null and b/ios/RayBan/Assets.xcassets/tapIcon.imageset/tapIcon.png differ diff --git a/ios/RayBan/Assets.xcassets/videoIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/videoIcon.imageset/Contents.json new file mode 100644 index 0000000..b432781 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/videoIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "filename" : "videoIcon.png", + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/videoIcon.imageset/videoIcon.png b/ios/RayBan/Assets.xcassets/videoIcon.imageset/videoIcon.png new file mode 100644 index 0000000..c7757a9 Binary files /dev/null and b/ios/RayBan/Assets.xcassets/videoIcon.imageset/videoIcon.png differ diff --git a/ios/RayBan/Assets.xcassets/walkingIcon.imageset/Contents.json b/ios/RayBan/Assets.xcassets/walkingIcon.imageset/Contents.json new file mode 100644 index 0000000..fefdf38 --- /dev/null +++ b/ios/RayBan/Assets.xcassets/walkingIcon.imageset/Contents.json @@ -0,0 +1,21 @@ +{ + "images" : [ + { + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "filename" : "walkingIcon.png", + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + } +} diff --git a/ios/RayBan/Assets.xcassets/walkingIcon.imageset/walkingIcon.png b/ios/RayBan/Assets.xcassets/walkingIcon.imageset/walkingIcon.png new file mode 100644 index 0000000..a64ab02 Binary files /dev/null and b/ios/RayBan/Assets.xcassets/walkingIcon.imageset/walkingIcon.png differ diff --git a/ios/RayBan/Info.plist b/ios/RayBan/Info.plist new file mode 100644 index 0000000..3a5c270 --- /dev/null +++ b/ios/RayBan/Info.plist @@ -0,0 +1,103 @@ + + + + + CFBundleDevelopmentRegion + $(DEVELOPMENT_LANGUAGE) + CFBundleDisplayName + CourtLine + CFBundleExecutable + $(EXECUTABLE_NAME) + CFBundleIdentifier + $(PRODUCT_BUNDLE_IDENTIFIER) + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + RayBan + CFBundlePackageType + $(PRODUCT_BUNDLE_PACKAGE_TYPE) + CFBundleShortVersionString + $(MARKETING_VERSION) + CFBundleURLTypes + + + CFBundleTypeRole + Editor + CFBundleURLName + $(PRODUCT_BUNDLE_IDENTIFIER) + CFBundleURLSchemes + + rayban + + + + CFBundleVersion + $(CURRENT_PROJECT_VERSION) + MWDAT + + AppLinkURLScheme + rayban:// + ClientToken + $(CLIENT_TOKEN) + MetaAppID + $(META_APP_ID) + TeamID + $(DEVELOPMENT_TEAM) + + NSAppTransportSecurity + + NSAllowsLocalNetworking + + + NSBluetoothAlwaysUsageDescription + Needed to connect to Meta AI Glasses + NSBonjourServices + + _bonjour._tcp + + NSCameraUsageDescription + This app needs camera access to stream from your phone's camera as a mock device feed. + NSLocalNetworkUsageDescription + This allows your phone to find and connect to your glasses over Wi-Fi. + NSMicrophoneUsageDescription + This app needs microphone access to stream audio from your glasses. + NSPhotoLibraryAddUsageDescription + This app needs access to save photos captured from your glasses. + UIApplicationSceneManifest + + UIApplicationSupportsMultipleScenes + + + UIApplicationSupportsIndirectInputEvents + + UIBackgroundModes + + audio + processing + bluetooth-central + bluetooth-peripheral + external-accessory + + UILaunchScreen + + UIRequiredDeviceCapabilities + + armv7 + + UISupportedExternalAccessoryProtocols + + com.meta.ar.wearable + + UISupportedInterfaceOrientations + + UIInterfaceOrientationPortrait + + UISupportedInterfaceOrientations~ipad + + UIInterfaceOrientationPortrait + UIInterfaceOrientationPortraitUpsideDown + UIInterfaceOrientationLandscapeLeft + UIInterfaceOrientationLandscapeRight + + + diff --git a/ios/RayBan/RayBan.entitlements b/ios/RayBan/RayBan.entitlements new file mode 100644 index 0000000..44c3622 --- /dev/null +++ b/ios/RayBan/RayBan.entitlements @@ -0,0 +1,10 @@ + + + + + keychain-access-groups + + $(AppIdentifierPrefix)$(CFBundleIdentifier) + + + diff --git a/ios/RayBan/RayBanApp.swift b/ios/RayBan/RayBanApp.swift new file mode 100644 index 0000000..f8af56d --- /dev/null +++ b/ios/RayBan/RayBanApp.swift @@ -0,0 +1,109 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// RayBanApp.swift +// +// Main entry point for the RayBan sample app demonstrating the Meta Wearables DAT SDK. +// This app shows how to connect to wearable devices (like Ray-Ban Meta smart glasses), +// stream live video from their cameras, and capture photos. It provides a complete example +// of DAT SDK integration including device registration, permissions, and media streaming. +// + +import Foundation +import MWDATCore +import Network +import SwiftUI + +#if DEBUG +import MWDATMockDevice +#endif + +@main +struct RayBanApp: App { + #if DEBUG + // Debug menu for simulating device connections during development + @State private var debugMenuViewModel = DebugMenuViewModel(mockDeviceKit: MockDeviceKit.shared) + #endif + private let wearables: WearablesInterface + @State private var wearablesViewModel: WearablesViewModel + + init() { + do { + try Wearables.configure() + } catch { + #if DEBUG + NSLog("[RayBan] Failed to configure Wearables SDK: \(error)") + #endif + } + + Self.triggerLocalNetworkPrompt() + + #if DEBUG + // Start the test server when launched by XCUITests so tests can control + // mock device setup via HTTP commands from the test process. + if ProcessInfo.processInfo.arguments.contains("--ui-testing") { + MockDeviceKit.shared.enable(config: MockDeviceKitConfig(initiallyRegistered: false)) + + let portFilePath = ProcessInfo.processInfo.environment["MWDAT_TEST_SERVER_PORT_FILE"] + Task { + try await MockDeviceKit.shared.startTestServer(portFilePath: portFilePath) + } + } + #endif + + let wearables = Wearables.shared + self.wearables = wearables + self._wearablesViewModel = State(wrappedValue: WearablesViewModel(wearables: wearables)) + } + + /// iOS only prompts for Local Network permission when the app first touches + /// something the OS classifies as a local-network operation. URLSession to + /// a LAN IP literal *should* qualify, but in practice the prompt often + /// doesn't fire — the connect is silently cancelled (NSURLErrorCancelled). + /// Starting a short-lived NWBrowser for our declared Bonjour service trips + /// the permission flow reliably. Keep the browser alive long enough for + /// the prompt to actually appear, then cancel — we don't need any results. + private static func triggerLocalNetworkPrompt() { + let browser = NWBrowser( + for: .bonjour(type: "_bonjour._tcp", domain: nil), + using: .init() + ) + browser.start(queue: .main) + DispatchQueue.main.asyncAfter(deadline: .now() + 5) { + browser.cancel() + } + } + + var body: some Scene { + WindowGroup { + // Main app view with access to the shared Wearables SDK instance + // The Wearables.shared singleton provides the core DAT API + MainAppView(wearables: Wearables.shared, viewModel: wearablesViewModel) + // Show error alerts for view model failures + .alert("Error", isPresented: $wearablesViewModel.showError) { + Button("OK") { + wearablesViewModel.dismissError() + } + } message: { + Text(wearablesViewModel.errorMessage) + } + #if DEBUG + .sheet(isPresented: $debugMenuViewModel.showDebugMenu) { + MockDeviceKitView(viewModel: debugMenuViewModel.mockDeviceKitViewModel) + } + .overlay { + DebugMenuView(debugMenuViewModel: debugMenuViewModel) + } + #endif + + // Registration view handles the flow for connecting to the glasses via Meta AI + RegistrationView(viewModel: wearablesViewModel) + } + } +} diff --git a/ios/RayBan/TestResources/plant.mp4 b/ios/RayBan/TestResources/plant.mp4 new file mode 100644 index 0000000..e838dfa Binary files /dev/null and b/ios/RayBan/TestResources/plant.mp4 differ diff --git a/ios/RayBan/TestResources/plant.png b/ios/RayBan/TestResources/plant.png new file mode 100644 index 0000000..96798c3 Binary files /dev/null and b/ios/RayBan/TestResources/plant.png differ diff --git a/ios/RayBan/ViewModels/DebugMenuViewModel.swift b/ios/RayBan/ViewModels/DebugMenuViewModel.swift new file mode 100644 index 0000000..74f0028 --- /dev/null +++ b/ios/RayBan/ViewModels/DebugMenuViewModel.swift @@ -0,0 +1,36 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// DebugMenuViewModel.swift +// +// Debug-only view model that provides access to mock devices for development and testing. +// This enables developers to test DAT SDK streaming functionality without physical Meta +// wearable devices. Mock devices simulate the behavior of real devices, allowing for +// comprehensive testing of streaming, photo capture, and error handling workflows. +// + +#if DEBUG + +import MWDATMockDevice +import Observation +import SwiftUI + +@Observable +@MainActor +class DebugMenuViewModel { + public var showDebugMenu: Bool + public var mockDeviceKitViewModel: MockDeviceKitView.ViewModel + + init(mockDeviceKit: MockDeviceKitInterface) { + self.mockDeviceKitViewModel = MockDeviceKitView.ViewModel(mockDeviceKit: mockDeviceKit) + self.showDebugMenu = false + } +} + +#endif diff --git a/ios/RayBan/ViewModels/DeviceSessionManager.swift b/ios/RayBan/ViewModels/DeviceSessionManager.swift new file mode 100644 index 0000000..f290ed0 --- /dev/null +++ b/ios/RayBan/ViewModels/DeviceSessionManager.swift @@ -0,0 +1,208 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +import MWDATCore +import Observation +import SwiftUI + +/// Manages DeviceSession lifecycle with 1:1 device-to-session mapping. +/// Monitors device availability and creates sessions on demand via `getSession()`. +@Observable +@MainActor +final class DeviceSessionManager { + private(set) var isReady: Bool = false + private(set) var hasActiveDevice: Bool = false + + private let wearables: WearablesInterface + private let deviceSelector: AutoDeviceSelector + private var deviceSession: DeviceSession? + @ObservationIgnored private var deviceMonitorTask: Task? + @ObservationIgnored private var stateObserverTask: Task? + + init(wearables: WearablesInterface) { + self.wearables = wearables + self.deviceSelector = AutoDeviceSelector(wearables: wearables) + startDeviceMonitoring() + } + + isolated deinit { + deviceMonitorTask?.cancel() + stateObserverTask?.cancel() + deviceSession?.stop() + } + + /// Stops the device session and cancels monitoring. Call before releasing. + /// The stateObserverTask handles cleanup when .stopped arrives. + func cleanup() { + deviceMonitorTask?.cancel() + deviceMonitorTask = nil + deviceSession?.stop() + } + + /// Stops only the device session, leaving device-availability monitoring + /// active so the next getSession() can build a fresh session. MWDAT 0.7 + /// requires a new DeviceSession per streaming run (no removeStream API). + /// + /// Awaits the .stopped transition before returning. Without this, a quick + /// Start→Stop→Start cycle hits ActivityManagerError 11 on the second start + /// because the glasses-side activity manager hasn't released the previous + /// session by the time wearables.createSession() runs. + func stopDeviceSession() async { + guard let session = deviceSession else { return } + + // Take over teardown from the state observer so we own the .stopped wait. + stateObserverTask?.cancel() + stateObserverTask = nil + + if session.state == .stopped { + deviceSession = nil + isReady = false + return + } + + // Subscribe BEFORE calling stop — stateStream doesn't buffer past events. + let stateStream = session.stateStream() + session.stop() + for await state in stateStream where state == .stopped { break } + deviceSession = nil + isReady = false + } + + /// Returns a ready DeviceSession, creating one if needed. + /// Waits for the session to reach .started state before returning. + func getSession() async throws(DeviceSessionError) -> DeviceSession { + if let session = deviceSession, session.state == .started { + isReady = true + return session + } + + if deviceSession?.state == .stopped { + deviceSession = nil + } + + // Wait for an in-progress session to finish starting + if let session = deviceSession { + // The session may have already transitioned to .started before the + // for-await loop begins iterating (stateStream doesn't buffer past events). + if session.state == .started { + isReady = true + startStateObserver(for: session) + return session + } + + try await waitForSessionStart( + stateStream: session.stateStream(), + errorStream: session.errorStream() + ) + isReady = true + startStateObserver(for: session) + return session + } + + // Create a new session + do { + let session = try wearables.createSession(deviceSelector: deviceSelector) + deviceSession = session + + let stateStream = session.stateStream() + let errorStream = session.errorStream() + try session.start() + + // The session may have already transitioned to .started before the + // for-await loop begins iterating (the state change is delivered on + // another thread and the stream does not buffer past events). + if session.state == .started { + isReady = true + startStateObserver(for: session) + return session + } + + try await waitForSessionStart(stateStream: stateStream, errorStream: errorStream) + isReady = true + startStateObserver(for: session) + return session + } catch { + isReady = false + deviceSession = nil + throw error + } + } + + // MARK: - Private + + private func waitForSessionStart( + stateStream: AsyncStream, + errorStream: AsyncStream + ) async throws(DeviceSessionError) { + do { + try await withThrowingTaskGroup(of: Void.self) { group in + group.addTask { + for await state in stateStream { + if state == .started { + return + } + if state == .stopped { + throw DeviceSessionError.unexpectedError(description: "The session failed to start") + } + } + guard !Task.isCancelled else { + return + } + throw DeviceSessionError.unexpectedError(description: "The session failed to start") + } + + group.addTask { + for await error in errorStream { + throw error + } + guard !Task.isCancelled else { + return + } + throw DeviceSessionError.unexpectedError(description: "The session failed to start") + } + + guard try await group.next() != nil else { + throw DeviceSessionError.unexpectedError(description: "The session failed to start") + } + group.cancelAll() + } + } catch let error as DeviceSessionError { + throw error + } catch { + throw .unexpectedError(description: error.localizedDescription) + } + } + + /// Monitors device availability only — does NOT create sessions. + /// Session creation is deferred to `getSession()` to avoid races. + private func startDeviceMonitoring() { + deviceMonitorTask = Task { [weak self] in + guard let self else { return } + for await device in deviceSelector.activeDeviceStream() { + hasActiveDevice = device != nil + } + } + } + + private func startStateObserver(for session: DeviceSession) { + stateObserverTask?.cancel() + stateObserverTask = Task { [weak self] in + for await state in session.stateStream() { + guard let self else { return } + if state == .started { + isReady = true + } else if state == .stopped { + isReady = false + deviceSession = nil + stateObserverTask = nil + return + } + } + } + } +} diff --git a/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceKitViewModel.swift b/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceKitViewModel.swift new file mode 100644 index 0000000..2a46edf --- /dev/null +++ b/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceKitViewModel.swift @@ -0,0 +1,63 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MockDeviceKitViewModel.swift +// +// View model for managing mock devices during development and testing of DAT SDK features. +// Mock devices simulate real Meta wearable device behavior, allowing developers to test +// streaming, photo capture, and device management workflows without physical hardware. +// + +#if DEBUG + +import Foundation +import MWDATMockDevice +import Observation + +extension MockDeviceKitView { + @Observable + @MainActor + class ViewModel { + private let mockDeviceKit: MockDeviceKitInterface + var cardViewModels: [MockDeviceCardView.ViewModel] = [] + var isEnabled: Bool + + init(mockDeviceKit: MockDeviceKitInterface) { + self.mockDeviceKit = mockDeviceKit + self.isEnabled = mockDeviceKit.isEnabled + self.cardViewModels = mockDeviceKit.pairedDevices.map { MockDeviceCardView.ViewModel(device: $0) } + } + + func enable() { + mockDeviceKit.enable() + isEnabled = true + } + + func disable() { + mockDeviceKit.disable() + cardViewModels = [] + isEnabled = false + } + + // Add a new mock Ray-Ban Meta device + func pairRaybanMeta() { + let mockDevice = mockDeviceKit.pairRaybanMeta() + cardViewModels.append(MockDeviceCardView.ViewModel(device: mockDevice)) + } + + func unpairDevice(_ device: MockDevice) { + if let idx = cardViewModels.firstIndex(where: { $0.id == device.deviceIdentifier }) { + cardViewModels.remove(at: idx) + mockDeviceKit.unpairDevice(device) + } + } + } +} + +#endif diff --git a/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceViewModel.swift b/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceViewModel.swift new file mode 100644 index 0000000..4571b18 --- /dev/null +++ b/ios/RayBan/ViewModels/MockDeviceKit/MockDeviceViewModel.swift @@ -0,0 +1,145 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MockDeviceViewModel.swift +// +// View model for individual mock devices used in development and testing of DAT SDK features. +// This controls mock device behaviors like power states, physical states (folded/unfolded), +// and media content (camera feeds and captured images). +// + +#if DEBUG + +import AVFoundation +import Foundation +import MWDATMockDevice +import Observation +import UIKit + +extension MockDeviceCardView { + @Observable + @MainActor + final class ViewModel { + let device: MockDevice + var hasCameraFeed: Bool = false + var hasCapturedImage: Bool = false + var cameraSource: CameraFacing? + var isPoweredOn: Bool = false + var isDonned: Bool = false + var isUnfolded: Bool = false + var showCameraPermissionAlert: Bool = false + + init(device: MockDevice, hasCameraFeed: Bool = false, hasCapturedImage: Bool = false) { + self.device = device + self.hasCameraFeed = hasCameraFeed + self.hasCapturedImage = hasCapturedImage + } + + var id: String { device.deviceIdentifier } + + // Display name for the mock device in the UI + var deviceName: String { + if device is MockRaybanMeta { + return "RayBan Meta Glasses" + } + return "Device" + } + + func powerOn() { + device.powerOn() + isPoweredOn = true + } + + func powerOff() { + device.powerOff() + isPoweredOn = false + isDonned = false + isUnfolded = false + } + + func don() { + device.don() + isDonned = true + isUnfolded = true + } + + func doff() { + device.doff() + isDonned = false + } + + func unfold() { + if let rayBanDevice = device as? MockDisplaylessGlasses { + rayBanDevice.unfold() + isUnfolded = true + } + } + + func fold() { + if let rayBanDevice = device as? MockDisplaylessGlasses { + rayBanDevice.fold() + isUnfolded = false + isDonned = false + } + } + + func captouchTap() { + (device as? MockDisplaylessGlasses)?.services.captouch.tap() + } + + func captouchTapAndHold() { + (device as? MockDisplaylessGlasses)?.services.captouch.tapAndHold() + } + + func setCameraFeed(_ facing: CameraFacing) { + if let cameraKit = (device as? MockDisplaylessGlasses)?.services.camera { + Task { + let status = AVCaptureDevice.authorizationStatus(for: .video) + if status == .denied || status == .restricted { + self.showCameraPermissionAlert = true + return + } + let granted = await AVCaptureDevice.requestAccess(for: .video) + guard granted else { + self.showCameraPermissionAlert = true + return + } + await cameraKit.setCameraFeed(cameraFacing: facing) + self.cameraSource = facing + self.hasCameraFeed = false + } + } + } + + func openSettings() { + if let url = URL(string: UIApplication.openSettingsURLString) { + UIApplication.shared.open(url) + } + } + + // Load mock video content + func selectVideo(from url: URL) { + if let cameraKit = (device as? MockDisplaylessGlasses)?.services.camera { + cameraKit.setCameraFeed(fileURL: url) + hasCameraFeed = true + cameraSource = nil + } + } + + // Load mock image content + func selectImage(from url: URL) { + if let cameraKit = (device as? MockDisplaylessGlasses)?.services.camera { + cameraKit.setCapturedImage(fileURL: url) + hasCapturedImage = true + } + } + } +} + +#endif diff --git a/ios/RayBan/ViewModels/StreamSessionViewModel.swift b/ios/RayBan/ViewModels/StreamSessionViewModel.swift new file mode 100644 index 0000000..3139cae --- /dev/null +++ b/ios/RayBan/ViewModels/StreamSessionViewModel.swift @@ -0,0 +1,923 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +import AVFoundation +import MWDATCamera +import MWDATCore +import Observation +import SwiftUI + +// ⚙️ CONFIG — legacy WebSocket video/photo path (the glasses-streaming template's +// transport). The CourtLine agent integration uses WebRTC for audio (see +// WebRTCAudioSpike.swift); video-over-WebRTC / wiring frames into the agent's +// vision endpoint is future work. Left as a placeholder for now. +private let streamPublishHost = "ws://localhost:8000" + +private func publishURL(path: String) -> URL { + URL(string: "\(streamPublishHost)\(path)")! +} + +private func publishHTTPURL(path: String) -> URL { + let httpHost = streamPublishHost + .replacingOccurrences(of: "wss://", with: "https://") + .replacingOccurrences(of: "ws://", with: "http://") + return URL(string: "\(httpHost)\(path)")! +} + +enum StreamingStatus { + case streaming + case waiting + case stopped +} + +private final class StreamPublisher { + private let queue = DispatchQueue(label: "stream.publisher") + private var task: URLSessionWebSocketTask? + private var sentCount = 0 + private var paused = false + + /// Invoked on the main thread when the server pushes a JSON control text + /// frame (e.g. `{"type":"video_on"}`). Only the `type` field is forwarded. + var onControl: (@MainActor (String) -> Void)? + + /// Invoked when the server asks for a high-res still via + /// `{"type":"capture_photo","request_id":"..."}`. The request_id is + /// echoed back as an `X-Request-Id` header on the upload so the server's + /// awaiting request resolves. + var onCapturePhoto: (@MainActor (String) -> Void)? + + func start(url: URL) { + queue.async { + print("[StreamPublisher] start \(url)") + let t = URLSession.shared.webSocketTask(with: url) + t.resume() + self.task = t + self.sentCount = 0 + self.paused = false + self.drain(t) + } + } + + func stop() { + queue.async { + print("[StreamPublisher] stop (sent=\(self.sentCount))") + self.task?.cancel(with: .normalClosure, reason: nil) + self.task = nil + self.paused = false + } + } + + func pause() { + queue.async { + guard !self.paused else { return } + self.paused = true + self.sendControl("pause") + } + } + + func resume() { + queue.async { + guard self.paused else { return } + self.paused = false + self.sendControl("resume") + } + } + + private func sendControl(_ type: String) { + guard let task = self.task else { return } + let payload = "{\"type\":\"\(type)\"}" + task.send(.string(payload)) { error in + if let error { print("[StreamPublisher] control \(type) send error: \(error)") } + } + print("[StreamPublisher] control \(type)") + } + + func send(_ data: Data) { + queue.async { + guard let task = self.task, !self.paused else { return } + task.send(.data(data)) { error in + if let error { + print("[StreamPublisher] send error: \(error)") + } + } + self.sentCount += 1 + if self.sentCount == 1 || self.sentCount % 30 == 0 { + print("[StreamPublisher] sent #\(self.sentCount) bytes=\(data.count)") + } + } + } + + private func drain(_ task: URLSessionWebSocketTask) { + task.receive { [weak self] result in + guard let self else { return } + switch result { + case .success(let message): + if case .string(let text) = message { + self.handleControl(text) + } + self.drain(task) + case .failure(let err): + print("[StreamPublisher] receive failure: \(err)") + self.queue.async { if self.task === task { self.task = nil } } + } + } + } + + private func handleControl(_ text: String) { + guard let data = text.data(using: .utf8), + let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let type = obj["type"] as? String else { + print("[StreamPublisher] non-JSON server text: \(text)") + return + } + print("[StreamPublisher] control from server: \(type)") + if type == "capture_photo", let requestId = obj["request_id"] as? String { + Task { @MainActor in self.onCapturePhoto?(requestId) } + return + } + Task { @MainActor in self.onControl?(type) } + } +} + +/// Captures audio from the current input route (Bluetooth HFP when glasses are +/// the active audio source) and ships Float32 mono PCM over a WebSocket. +private final class AudioPublisher { + private let queue = DispatchQueue(label: "audio.publisher") + private var task: URLSessionWebSocketTask? + private let engine = AVAudioEngine() + private var converter: AVAudioConverter? + private var targetFormat: AVAudioFormat? + private var headerSent = false + private var sentChunks = 0 + private var configChangeObserver: NSObjectProtocol? + private var routeChangeObserver: NSObjectProtocol? + private var interruptionObserver: NSObjectProtocol? + private var mediaServicesLostObserver: NSObjectProtocol? + private var mediaServicesResetObserver: NSObjectProtocol? + private var paused = false + + /// Configure the HFP audio session. Per Meta DAT docs, do this BEFORE + /// starting the video stream and allow HFP time to negotiate. + /// + /// `.defaultToSpeaker` overrides `.playAndRecord`'s built-in-receiver + /// default so any playback (agent voice) is audible. It only + /// applies when no BT output route is connected, so it's harmless when + /// the glasses are paired and HFP/A2DP is active. + static func configureAudioSession() { + let session = AVAudioSession.sharedInstance() + do { + try session.setCategory( + .playAndRecord, + mode: .default, + options: [.allowBluetooth, .allowBluetoothA2DP, .defaultToSpeaker] + ) + try session.setActive(true, options: .notifyOthersOnDeactivation) + if let hfp = session.availableInputs?.first(where: { $0.portType == .bluetoothHFP }) { + try session.setPreferredInput(hfp) + print("[AudioPublisher] pinned input to \(hfp.portName)") + } else { + let available = session.availableInputs?.map { "\($0.portName)(\($0.portType.rawValue))" } ?? [] + print("[AudioPublisher] no BluetoothHFP input yet; available=\(available)") + } + let outputs = session.currentRoute.outputs.map { "\($0.portName)(\($0.portType.rawValue))" } + print("[AudioPublisher] audio session configured outputs=\(outputs)") + } catch { + print("[AudioPublisher] session error: \(error)") + } + } + + func start(url: URL) { + queue.async { + print("[AudioPublisher] start \(url)") + + let t = URLSession.shared.webSocketTask(with: url) + t.resume() + self.task = t + self.headerSent = false + self.sentChunks = 0 + self.drain(t) + + self.installTapAndStartEngine() + self.installSystemObservers() + } + } + + private func installTapAndStartEngine() { + let input = engine.inputNode + let inFormat = input.outputFormat(forBus: 0) + guard inFormat.sampleRate > 0 else { + print("[AudioPublisher] input format not ready, sampleRate=\(inFormat.sampleRate)") + return + } + guard let outFormat = AVAudioFormat( + commonFormat: .pcmFormatFloat32, + sampleRate: inFormat.sampleRate, + channels: 1, + interleaved: false + ) else { + print("[AudioPublisher] could not create output format") + return + } + targetFormat = outFormat + converter = AVAudioConverter(from: inFormat, to: outFormat) + headerSent = false + + input.removeTap(onBus: 0) + input.installTap(onBus: 0, bufferSize: 1024, format: inFormat) { [weak self] buffer, _ in + self?.handle(buffer) + } + + do { + try engine.start() + print("[AudioPublisher] engine started, sampleRate=\(inFormat.sampleRate) channels=\(inFormat.channelCount)") + } catch { + print("[AudioPublisher] engine start error: \(error)") + } + } + + private func installSystemObservers() { + let nc = NotificationCenter.default + configChangeObserver = nc.addObserver( + forName: .AVAudioEngineConfigurationChange, + object: engine, + queue: nil + ) { [weak self] _ in + guard let self else { return } + print("[AudioPublisher] engine configuration changed — restarting tap") + self.queue.async { + self.engine.inputNode.removeTap(onBus: 0) + if self.engine.isRunning { self.engine.stop() } + self.installTapAndStartEngine() + } + } + routeChangeObserver = nc.addObserver( + forName: AVAudioSession.routeChangeNotification, + object: nil, + queue: nil + ) { [weak self] note in + let reason = (note.userInfo?[AVAudioSessionRouteChangeReasonKey] as? UInt).flatMap(AVAudioSession.RouteChangeReason.init(rawValue:)) + let inputs = AVAudioSession.sharedInstance().currentRoute.inputs.map { "\($0.portName)(\($0.portType.rawValue))" } + print("[AudioPublisher] route changed reason=\(reason.map(String.init(describing:)) ?? "?") inputs=\(inputs)") + // Re-pin HFP if it's available but isn't the current input. + let session = AVAudioSession.sharedInstance() + if let hfp = session.availableInputs?.first(where: { $0.portType == .bluetoothHFP }), + session.currentRoute.inputs.contains(where: { $0.portType == .bluetoothHFP }) == false { + try? session.setPreferredInput(hfp) + print("[AudioPublisher] re-pinned input to \(hfp.portName)") + self?.queue.async { + guard let self else { return } + self.engine.inputNode.removeTap(onBus: 0) + if self.engine.isRunning { self.engine.stop() } + self.installTapAndStartEngine() + } + } + } + interruptionObserver = nc.addObserver( + forName: AVAudioSession.interruptionNotification, + object: AVAudioSession.sharedInstance(), + queue: nil + ) { [weak self] note in + guard let type = (note.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt) + .flatMap(AVAudioSession.InterruptionType.init(rawValue:)) else { return } + print("[AudioPublisher] interruption \(type == .began ? "began" : "ended")") + guard let self else { return } + self.queue.async { + switch type { + case .began: + if self.engine.isRunning { self.engine.pause() } + case .ended: + let opts = (note.userInfo?[AVAudioSessionInterruptionOptionKey] as? UInt).map(AVAudioSession.InterruptionOptions.init(rawValue:)) + if opts?.contains(.shouldResume) == true { + try? self.engine.start() + } + @unknown default: break + } + } + } + mediaServicesLostObserver = nc.addObserver( + forName: AVAudioSession.mediaServicesWereLostNotification, + object: AVAudioSession.sharedInstance(), + queue: nil + ) { _ in + print("[AudioPublisher] media services were LOST") + } + mediaServicesResetObserver = nc.addObserver( + forName: AVAudioSession.mediaServicesWereResetNotification, + object: AVAudioSession.sharedInstance(), + queue: nil + ) { [weak self] _ in + print("[AudioPublisher] media services were reset — rebuilding") + guard let self else { return } + self.queue.async { + AudioPublisher.configureAudioSession() + self.installTapAndStartEngine() + } + } + } + + private func removeSystemObservers() { + let nc = NotificationCenter.default + if let o = configChangeObserver { nc.removeObserver(o) } + if let o = routeChangeObserver { nc.removeObserver(o) } + if let o = interruptionObserver { nc.removeObserver(o) } + if let o = mediaServicesLostObserver { nc.removeObserver(o) } + if let o = mediaServicesResetObserver { nc.removeObserver(o) } + configChangeObserver = nil + routeChangeObserver = nil + interruptionObserver = nil + mediaServicesLostObserver = nil + mediaServicesResetObserver = nil + } + + func stop() { + queue.async { + print("[AudioPublisher] stop (chunks=\(self.sentChunks))") + self.removeSystemObservers() + self.engine.inputNode.removeTap(onBus: 0) + if self.engine.isRunning { self.engine.stop() } + self.task?.cancel(with: .normalClosure, reason: nil) + self.task = nil + self.paused = false + try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation) + } + } + + func pause() { + queue.async { + guard !self.paused else { return } + self.paused = true + print("[AudioPublisher] pause") + } + } + + func resume() { + queue.async { + guard self.paused else { return } + self.paused = false + print("[AudioPublisher] resume") + } + } + + private func handle(_ buffer: AVAudioPCMBuffer) { + queue.async { + guard let task = self.task, let target = self.targetFormat else { return } + + if !self.headerSent { + let header = "{\"sampleRate\":\(Int(target.sampleRate)),\"channels\":1}" + task.send(.string(header)) { error in + if let error { print("[AudioPublisher] header send error: \(error)") } + } + self.headerSent = true + } + + if self.paused { return } + + // Fast path: input already matches our target (Float32 mono, same SR) — + // just ship the channel data directly. + let inFormat = buffer.format + let sameFormat = inFormat.commonFormat == .pcmFormatFloat32 + && inFormat.channelCount == 1 + && inFormat.sampleRate == target.sampleRate + if sameFormat, let channelData = buffer.floatChannelData?[0] { + let byteCount = Int(buffer.frameLength) * MemoryLayout.size + let data = Data(bytes: channelData, count: byteCount) + task.send(.data(data)) { error in + if let error { print("[AudioPublisher] send error: \(error)") } + } + self.sentChunks += 1 + if self.sentChunks == 1 || self.sentChunks % 50 == 0 { + print("[AudioPublisher] sent #\(self.sentChunks) bytes=\(byteCount)") + } + return + } + + // Slow path: format mismatch (multi-channel, different SR, Int16, etc.) — convert. + guard let converter = self.converter else { return } + let capacity = AVAudioFrameCount(Double(buffer.frameLength) * target.sampleRate / inFormat.sampleRate) + 64 + guard let out = AVAudioPCMBuffer(pcmFormat: target, frameCapacity: capacity) else { return } + var error: NSError? + var supplied = false + let status = converter.convert(to: out, error: &error) { _, status in + if supplied { + // Do NOT signal .endOfStream — that permanently retires the converter. + status.pointee = .noDataNow + return nil + } + supplied = true + status.pointee = .haveData + return buffer + } + if status == .error || out.frameLength == 0 { + print("[AudioPublisher] convert produced no data status=\(status.rawValue) outFrames=\(out.frameLength) inFrames=\(buffer.frameLength) error=\(error?.localizedDescription ?? "nil")") + return + } + guard let channelData = out.floatChannelData?[0] else { return } + let byteCount = Int(out.frameLength) * MemoryLayout.size + let data = Data(bytes: channelData, count: byteCount) + task.send(.data(data)) { error in + if let error { print("[AudioPublisher] send error: \(error)") } + } + self.sentChunks += 1 + if self.sentChunks == 1 || self.sentChunks % 50 == 0 { + print("[AudioPublisher] sent #\(self.sentChunks) bytes=\(byteCount)") + } + } + } + + private func drain(_ task: URLSessionWebSocketTask) { + task.receive { [weak self] result in + guard let self else { return } + switch result { + case .success: + self.drain(task) + case .failure(let err): + print("[AudioPublisher] receive failure: \(err)") + self.queue.async { if self.task === task { self.task = nil } } + } + } + } +} + +/// Plays PCM frames received from the server-side voice agent. The +/// server pushes Int16 LE PCM (24 kHz mono) after a JSON header. We schedule +/// them on an AVAudioPlayerNode whose output routes via the shared +/// `.playAndRecord` HFP session — i.e. out through the glasses' speakers. +private final class AgentAudioPlayer { + private let queue = DispatchQueue(label: "agent.audio.player") + private var task: URLSessionWebSocketTask? + private let engine = AVAudioEngine() + private let player = AVAudioPlayerNode() + private var playerFormat: AVAudioFormat? + private var receivedChunks = 0 + + func start(url: URL) { + queue.async { + self.teardownLocked() + print("[AgentAudioPlayer] start \(url)") + let t = URLSession.shared.webSocketTask(with: url) + t.resume() + self.task = t + self.receivedChunks = 0 + self.receive(t) + } + } + + func stop() { + queue.async { + print("[AgentAudioPlayer] stop (chunks=\(self.receivedChunks))") + self.teardownLocked() + } + } + + /// Caller MUST already be running on `queue`. + private func teardownLocked() { + task?.cancel(with: .normalClosure, reason: nil) + task = nil + if player.isPlaying { player.stop() } + if engine.isRunning { engine.stop() } + if playerFormat != nil { engine.detach(player) } + playerFormat = nil + } + + private func configure(sampleRate: Double) { + guard let format = AVAudioFormat( + commonFormat: .pcmFormatFloat32, + sampleRate: sampleRate, + channels: 1, + interleaved: false + ) else { + print("[AgentAudioPlayer] could not create player format at \(sampleRate)") + return + } + playerFormat = format + engine.attach(player) + engine.connect(player, to: engine.mainMixerNode, format: format) + do { + try engine.start() + player.play() + let outputs = AVAudioSession.sharedInstance().currentRoute.outputs + .map { "\($0.portName)(\($0.portType.rawValue))" } + print("[AgentAudioPlayer] engine started, sampleRate=\(sampleRate) outputs=\(outputs)") + } catch { + print("[AgentAudioPlayer] engine start error: \(error)") + } + } + + private func handleHeader(_ text: String) { + guard let data = text.data(using: .utf8), + let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + print("[AgentAudioPlayer] bad header text=\(text)") + return + } + let rate = (obj["sampleRate"] as? Double) + ?? (obj["sampleRate"] as? Int).map(Double.init) + ?? 24000 + print("[AgentAudioPlayer] header sampleRate=\(rate)") + configure(sampleRate: rate) + } + + private func handleAudio(_ data: Data) { + guard let format = playerFormat else { return } + let frameCount = data.count / 2 + guard frameCount > 0, + let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(frameCount)), + let channel = buffer.floatChannelData?[0] else { return } + buffer.frameLength = AVAudioFrameCount(frameCount) + data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in + let int16Ptr = raw.bindMemory(to: Int16.self) + for i in 0..? + @ObservationIgnored private var deviceStreamTask: Task? + @ObservationIgnored private var setupDeviceStreamTask: Task? + private let wearables: WearablesInterface + private var deviceCompatibility: [DeviceIdentifier: Compatibility] = [:] + private var compatibilityListenerTokens: [DeviceIdentifier: AnyListenerToken] = [:] + + init(wearables: WearablesInterface) { + self.wearables = wearables + self.devices = wearables.devices + self.registrationState = wearables.registrationState + + // Set up device stream immediately to handle MockDevice events + setupDeviceStreamTask = Task { + await setupDeviceStream() + } + + registrationTask = Task { + for await registrationState in wearables.registrationStateStream() { + let previousState = self.registrationState + self.registrationState = registrationState + if self.showGettingStartedSheet == false && registrationState == .registered && previousState == .registering { + self.showGettingStartedSheet = true + } + } + } + } + + isolated deinit { + registrationTask?.cancel() + deviceStreamTask?.cancel() + setupDeviceStreamTask?.cancel() + } + + private func setupDeviceStream() async { + if let task = deviceStreamTask, !task.isCancelled { + task.cancel() + } + + deviceStreamTask = Task { + for await devices in wearables.devicesStream() { + self.devices = devices + // Monitor compatibility for each device + monitorDeviceCompatibility(devices: devices) + } + } + } + + private func monitorDeviceCompatibility(devices: [DeviceIdentifier]) { + // Remove listeners for devices that are no longer present + let deviceSet = Set(devices) + compatibilityListenerTokens = compatibilityListenerTokens.filter { deviceSet.contains($0.key) } + deviceCompatibility = deviceCompatibility.filter { deviceSet.contains($0.key) } + updateFirmwareUpdateRequired() + + // Add listeners for new devices + for deviceId in devices { + guard compatibilityListenerTokens[deviceId] == nil else { continue } + guard let device = wearables.deviceForIdentifier(deviceId) else { continue } + deviceCompatibility[deviceId] = device.compatibility() + updateFirmwareUpdateRequired() + + // Capture device name before the closure to avoid Sendable issues + let deviceName = device.nameOrId() + let token = device.addCompatibilityListener { [weak self] compatibility in + Task { [weak self] in + await self?.handleCompatibilityChange( + compatibility, + deviceId: deviceId, + deviceName: deviceName + ) + } + } + compatibilityListenerTokens[deviceId] = token + } + } + + func connectGlasses() { + guard registrationState != .registering else { return } + Task { @MainActor in + do { + try await wearables.startRegistration() + } catch let error as RegistrationError { + showError(error.description) + } catch { + showError(error.localizedDescription) + } + } + } + + func disconnectGlasses() { + Task { @MainActor in + do { + try await wearables.startUnregistration() + } catch let error as UnregistrationError { + showError(error.description) + } catch { + showError(error.localizedDescription) + } + } + } + + func openFirmwareUpdate() async { + do { + try await wearables.openFirmwareUpdate() + } catch { + showError(error.description) + } + } + + func openDATGlassesAppUpdate() async { + do { + try await wearables.openDATGlassesAppUpdate() + } catch { + showError(error.description) + } + } + + func showError(_ error: String) { + errorMessage = error + showError = true + } + + func dismissError() { + showError = false + } + + private func updateFirmwareUpdateRequired() { + requiresFirmwareUpdate = deviceCompatibility.values.contains(.deviceUpdateRequired) + } + + private func handleCompatibilityChange( + _ compatibility: Compatibility, + deviceId: DeviceIdentifier, + deviceName: String + ) { + deviceCompatibility[deviceId] = compatibility + updateFirmwareUpdateRequired() + if compatibility == .deviceUpdateRequired { + showError("Device '\(deviceName)' requires an update to work with this app") + } + } +} diff --git a/ios/RayBan/ViewModels/WebRTCAudioSpike.swift b/ios/RayBan/ViewModels/WebRTCAudioSpike.swift new file mode 100644 index 0000000..b312629 --- /dev/null +++ b/ios/RayBan/ViewModels/WebRTCAudioSpike.swift @@ -0,0 +1,191 @@ +/* + * CourtLine — WebRTC audio coexistence spike (PROTOCOL.md §2.6). + * + * This is NOT the production transport. It is the de-risking spike the + * protocol doc calls out: stand up a single bidirectional WebRTC audio leg + * over the glasses' Bluetooth HFP route *while a DAT camera stream is live*, + * and confirm the two coexist. The open risk is that WebRTC's audio device + * module wants to own `AVAudioSession`, whereas the existing code configures + * HFP by hand (see StreamSessionViewModel's AudioPublisher). + * + * Transport: Pipecat's SmallWebRTC client, which peers **directly with our own + * server** (`server/webrtc_server.py`) — no SFU, no third party. Media (SRTP) + * flows glasses -> our server over a peer connection; signaling is a plain HTTP + * SDP exchange at `/api/offer`. The server beeps periodically + * (downlink) and logs the mic audio it receives (uplink). + * + * Validation signals (watch the Xcode console while wearing the glasses): + * - `onLocalAudioLevel > 0` -> glasses MIC reaching WebRTC (uplink OK) + * - `onRemoteAudioLevel > 0` -> server beep arriving (downlink OK) + * - audio route logs show `bluetoothHFP` -> routed to the glasses, not phone + * - the DAT camera preview keeps updating -> coexistence holds + * + * NETWORK NOTE: peer-to-peer WebRTC needs a routable media path. Use the Mac's + * LAN IP here; this does NOT traverse the Cloudflare HTTP tunnel (that forwards + * HTTP only, not the UDP media). + */ + +import AVFoundation +import Foundation +import Observation +import PipecatClientIOS +import PipecatClientIOSSmallWebrtc + +// ⚙️ CONFIG — base URL of the CourtLine agent (yc-voice-agents-hackathon/server, +// Pipecat runner, SmallWebRTC). Point this at the host running the agent, e.g. +// the Mac's LAN IP + port 7860. The client POSTs its SDP offer to +// `/api/offer` (the runner serves that route). Compile-time constant — +// rebuild to change. NOTE: peer-to-peer WebRTC needs a routable media path; use +// a LAN IP on the same network (a Cloudflare HTTP tunnel won't carry the media). +let webRTCServerURL = "http://YOUR-AGENT-HOST:7860" + +@MainActor +@Observable +final class WebRTCAudioSpike: NSObject { + + enum SpikeState: String { + case idle, joining, joined, failed, left + } + + private(set) var state: SpikeState = .idle + /// Human-readable last event, surfaced in the debug UI. + private(set) var lastEvent: String = "not started" + + private var client: PipecatClient? + private var localLevelLogCount = 0 + private var remoteLevelLogCount = 0 + + /// Connect to the server, publish the mic over HFP, and play the server's + /// audio back. Deliberately does NOT touch `AVAudioSession` directly — the + /// SmallWebRTC transport's AudioManager owns it. Call this *after* the DAT + /// camera stream is running so we exercise the real coexistence case. + func start() async { + guard client == nil else { + log("already running (state=\(state))") + return + } + guard let offerURL = URL(string: "\(webRTCServerURL)/api/offer"), + !webRTCServerURL.contains("YOUR-MAC-LAN-IP") else { + state = .failed + log("set `webRTCServerURL` to your Mac's LAN IP, e.g. http://192.168.1.42:7860") + return + } + + state = .joining + logAudioRoute(tag: "before connect") + + // Receive-only: we do NOT capture the glasses mic (WebRTC iOS can't reliably + // pull the HFP mic). Audio is captured on the Mac server side; this app just + // plays the agent's voice out through the glasses speaker. + let options = PipecatClientOptions( + transport: SmallWebRTCTransport(iceConfig: nil), + enableMic: false, + enableCam: false + ) + let pipecat = PipecatClient(options: options) + pipecat.delegate = self + client = pipecat + + // Connect DIRECTLY to our single `/api/offer` endpoint — no RTVI `/start` + // session indirection. The transport POSTs its SDP offer there and gets the + // answer back (matches server/webrtc_server.py and webrtc_client.html). + let params = SmallWebRTCTransportConnectionParams( + webrtcRequestParams: APIRequest(endpoint: offerURL), + iceConfig: nil + ) + pipecat.connect(transportParams: params) { [weak self] result in + Task { @MainActor in + guard let self else { return } + switch result { + case .success: + self.state = .joined + self.log("connected to \(webRTCServerURL)") + self.logAudioRoute(tag: "after connect") + case .failure(let error): + self.state = .failed + self.log("connect failed: \(error)") + self.client = nil + } + } + } + } + + func stop() async { + guard let pipecat = client else { return } + client = nil + do { + try await pipecat.disconnect() + log("disconnected") + } catch { + log("disconnect error: \(error)") + } + state = .left + logAudioRoute(tag: "after disconnect") + } + + // MARK: - Logging + + private func log(_ message: String) { + lastEvent = message + print("[WebRTCAudioSpike] \(message)") + } + + private func logAudioRoute(tag: String) { + let session = AVAudioSession.sharedInstance() + let route = session.currentRoute + let inputs = route.inputs.map { "\($0.portName)(\($0.portType.rawValue))" } + let outputs = route.outputs.map { "\($0.portName)(\($0.portType.rawValue))" } + print("[WebRTCAudioSpike] route \(tag): category=\(session.category.rawValue) " + + "mode=\(session.mode.rawValue) in=\(inputs) out=\(outputs)") + } +} + +// MARK: - PipecatClientDelegate +// The protocol ships default no-op implementations, so we override only the +// handful that prove the two audio legs and surface coexistence problems. +// Callbacks may arrive off the main actor, so hop back before touching state. +extension WebRTCAudioSpike: PipecatClientDelegate { + + nonisolated func onConnected() { + Task { @MainActor in self.log("transport connected") } + } + + nonisolated func onDisconnected() { + Task { @MainActor in self.log("transport disconnected") } + } + + nonisolated func onTransportStateChanged(state: TransportState) { + Task { @MainActor in self.log("transport state=\(state)") } + } + + nonisolated func onBotReady(botReadyData: BotReadyData) { + Task { @MainActor in self.log("server bot ready") } + } + + /// Glasses mic → WebRTC. Non-zero means the uplink leg is alive and HFP + /// capture survived the SmallWebRTC transport taking over the audio session. + nonisolated func onLocalAudioLevel(level: Float) { + guard level > 0.01 else { return } + Task { @MainActor in + self.localLevelLogCount += 1 + if self.localLevelLogCount == 1 || self.localLevelLogCount % 25 == 0 { + self.log(String(format: "uplink mic level=%.3f (#%d)", level, self.localLevelLogCount)) + } + } + } + + /// Server audio → glasses speaker. Non-zero means the downlink leg is alive. + nonisolated func onRemoteAudioLevel(level: Float, participant: Participant) { + guard level > 0.01 else { return } + Task { @MainActor in + self.remoteLevelLogCount += 1 + if self.remoteLevelLogCount == 1 || self.remoteLevelLogCount % 25 == 0 { + self.log(String(format: "downlink server level=%.3f (#%d)", level, self.remoteLevelLogCount)) + } + } + } + + nonisolated func onBotStartedSpeaking() { + Task { @MainActor in self.log("server started sending audio") } + } +} diff --git a/ios/RayBan/Views/Components/CardView.swift b/ios/RayBan/Views/Components/CardView.swift new file mode 100644 index 0000000..4d58ab7 --- /dev/null +++ b/ios/RayBan/Views/Components/CardView.swift @@ -0,0 +1,37 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// CardView.swift +// +// Reusable container component that provides consistent card styling throughout the app. +// + +import SwiftUI + +struct CardView: View { + let content: Content + + init(@ViewBuilder content: () -> Content) { + self.content = content() + } + + var body: some View { + VStack(spacing: 0) { + content + } + .background(Color(.secondarySystemGroupedBackground)) + .cornerRadius(12) + .shadow( + color: Color.black.opacity(0.15), + radius: 4, + x: 0, + y: 2 + ) + } +} diff --git a/ios/RayBan/Views/Components/CircleButton.swift b/ios/RayBan/Views/Components/CircleButton.swift new file mode 100644 index 0000000..079ed84 --- /dev/null +++ b/ios/RayBan/Views/Components/CircleButton.swift @@ -0,0 +1,41 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// CircleButton.swift +// +// Reusable circular button component used in streaming controls and UI actions. +// + +import SwiftUI + +struct CircleButton: View { + let icon: String + let text: String? + let action: () -> Void + + var body: some View { + Button(action: action) { + if let text { + VStack(spacing: 2) { + Image(systemName: icon) + .font(.system(size: 14)) + Text(text) + .font(.system(size: 10, weight: .medium)) + } + } else { + Image(systemName: icon) + .font(.system(size: 16)) + } + } + .foregroundStyle(.black) + .frame(width: 56, height: 56) + .background(.white) + .clipShape(Circle()) + } +} diff --git a/ios/RayBan/Views/Components/CustomButton.swift b/ios/RayBan/Views/Components/CustomButton.swift new file mode 100644 index 0000000..a1eceef --- /dev/null +++ b/ios/RayBan/Views/Components/CustomButton.swift @@ -0,0 +1,58 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// CustomButton.swift +// +// Reusable button component used throughout the RayBan app for consistent styling. +// + +import SwiftUI + +struct CustomButton: View { + let title: String + let style: ButtonStyle + let isDisabled: Bool + let action: () -> Void + + enum ButtonStyle { + case primary, destructive + + var backgroundColor: Color { + switch self { + case .primary: + return .appPrimary + case .destructive: + return .destructiveBackground + } + } + + var foregroundColor: Color { + switch self { + case .primary: + return .white + case .destructive: + return .destructiveForeground + } + } + } + + var body: some View { + Button(action: action) { + Text(title) + .font(.system(size: 15, weight: .semibold)) + .foregroundStyle(style.foregroundColor) + .frame(maxWidth: .infinity) + .frame(height: 56) + .background(style.backgroundColor) + .cornerRadius(30) + } + .disabled(isDisabled) + .opacity(isDisabled ? 0.6 : 1.0) + } +} diff --git a/ios/RayBan/Views/Components/MediaPickerView.swift b/ios/RayBan/Views/Components/MediaPickerView.swift new file mode 100644 index 0000000..e9a59ec --- /dev/null +++ b/ios/RayBan/Views/Components/MediaPickerView.swift @@ -0,0 +1,91 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MediaPickerView.swift +// +// UIKit-SwiftUI bridge component for selecting media from the device photo library. +// + +import AVFoundation +import SwiftUI +import UIKit + +struct MediaPickerView: UIViewControllerRepresentable { + enum MediaType { + case video + case image + } + + let mode: MediaType + let onMediaSelected: (URL, MediaType) -> Void + + func makeUIViewController(context: Context) -> UIImagePickerController { + let picker = UIImagePickerController() + picker.delegate = context.coordinator + picker.sourceType = .photoLibrary + switch mode { + case .video: + picker.mediaTypes = ["public.movie"] + picker.videoExportPreset = AVAssetExportPresetHEVCHighestQuality + case .image: + picker.mediaTypes = ["public.image"] + } + picker.allowsEditing = false + return picker + } + + func updateUIViewController(_ uiViewController: UIImagePickerController, context: Context) {} + + func makeCoordinator() -> Coordinator { + Coordinator(self) + } + + final class Coordinator: NSObject, UIImagePickerControllerDelegate, UINavigationControllerDelegate { + let parent: MediaPickerView + + init(_ parent: MediaPickerView) { + self.parent = parent + } + + func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any]) { + if let mediaType = info[.mediaType] as? String { + if mediaType == "public.movie", let videoURL = info[.mediaURL] as? URL { + parent.onMediaSelected(videoURL, .video) + } else if mediaType == "public.image" { + let image = info[.editedImage] as? UIImage ?? info[.originalImage] as? UIImage + if let image, let imageURL = saveImageToTemporaryFile(image: image) { + parent.onMediaSelected(imageURL, .image) + } + } + } + picker.dismiss(animated: true) + } + + private func saveImageToTemporaryFile(image: UIImage) -> URL? { + let tempDirectory = FileManager.default.temporaryDirectory + let fileName = UUID().uuidString + ".jpg" + let fileURL = tempDirectory.appendingPathComponent(fileName) + + guard let imageData = image.jpegData(compressionQuality: 0.8) else { + return nil + } + + do { + try imageData.write(to: fileURL) + return fileURL + } catch { + return nil + } + } + + func imagePickerControllerDidCancel(_ picker: UIImagePickerController) { + picker.dismiss(animated: true) + } + } +} diff --git a/ios/RayBan/Views/Components/StatusText.swift b/ios/RayBan/Views/Components/StatusText.swift new file mode 100644 index 0000000..34908cc --- /dev/null +++ b/ios/RayBan/Views/Components/StatusText.swift @@ -0,0 +1,43 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// StatusText.swift +// +// Reusable UI component for displaying conditional status text throughout the RayBan app. +// + +import SwiftUI + +struct StatusText: View { + let isActive: Bool + let activeText: String + let inactiveText: String + let activeColor: Color + let inactiveColor: Color + + init( + isActive: Bool, + activeText: String, + inactiveText: String, + activeColor: Color = .green, + inactiveColor: Color = .secondary + ) { + self.isActive = isActive + self.activeText = activeText + self.inactiveText = inactiveText + self.activeColor = activeColor + self.inactiveColor = inactiveColor + } + + var body: some View { + Text(isActive ? activeText : inactiveText) + .foregroundStyle(isActive ? activeColor : inactiveColor) + .frame(maxWidth: .infinity, alignment: .leading) + } +} diff --git a/ios/RayBan/Views/DebugMenuView.swift b/ios/RayBan/Views/DebugMenuView.swift new file mode 100644 index 0000000..16bac09 --- /dev/null +++ b/ios/RayBan/Views/DebugMenuView.swift @@ -0,0 +1,46 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// DebugMenuView.swift +// +// Debug-only overlay that provides access to mock device functionality during development. +// This view demonstrates how to integrate mock devices for testing DAT SDK features +// without requiring physical Meta wearable devices. +// + +#if DEBUG + +import SwiftUI + +struct DebugMenuView: View { + var debugMenuViewModel: DebugMenuViewModel + + var body: some View { + HStack { + Spacer() + VStack { + Spacer() + Button(action: { + debugMenuViewModel.showDebugMenu = true + }) { + Image(systemName: "ladybug.fill") + .foregroundStyle(.white) + .padding() + .background(.secondary) + .clipShape(Circle()) + .shadow(radius: 4) + }.accessibilityIdentifier("debug_menu_button") + Spacer() + } + .padding(.trailing) + } + } +} + +#endif diff --git a/ios/RayBan/Views/HomeScreenView.swift b/ios/RayBan/Views/HomeScreenView.swift new file mode 100644 index 0000000..f194050 --- /dev/null +++ b/ios/RayBan/Views/HomeScreenView.swift @@ -0,0 +1,105 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// HomeScreenView.swift +// +// Welcome screen that guides users through the DAT SDK registration process. +// This view is displayed when the app is not yet registered. +// + +import MWDATCore +import SwiftUI + +struct HomeScreenView: View { + var viewModel: WearablesViewModel + + var body: some View { + ZStack { + Color.white.edgesIgnoringSafeArea(.all) + + VStack(spacing: 12) { + Spacer() + + Image(.cameraAccessIcon) + .resizable() + .aspectRatio(contentMode: .fit) + .frame(width: 120) + + VStack(spacing: 12) { + HomeTipItemView( + resource: .smartGlassesIcon, + title: "Video Capture", + text: "Record videos directly from your glasses, from your point of view." + ) + HomeTipItemView( + resource: .soundIcon, + title: "Open-Ear Audio", + text: "Hear notifications while keeping your ears open to the world around you." + ) + HomeTipItemView( + resource: .walkingIcon, + title: "Enjoy On-the-Go", + text: "Stay hands-free while you move through your day. Move freely, stay connected." + ) + } + + Spacer() + + VStack(spacing: 20) { + Text("You'll be redirected to the Meta AI app to confirm your connection.") + .font(.system(size: 14)) + .foregroundStyle(.gray) + .multilineTextAlignment(.center) + .fixedSize(horizontal: false, vertical: true) + .padding(.horizontal, 12) + + CustomButton( + title: viewModel.registrationState == .registering ? "Connecting..." : "Connect my glasses", + style: .primary, + isDisabled: viewModel.registrationState == .registering + ) { + viewModel.connectGlasses() + } + } + } + .padding(.all, 24) + } + } + +} + +struct HomeTipItemView: View { + let resource: ImageResource + let title: String + let text: String + + var body: some View { + HStack(alignment: .top, spacing: 12) { + Image(resource) + .resizable() + .renderingMode(.template) + .foregroundStyle(.black) + .aspectRatio(contentMode: .fit) + .frame(width: 24) + .padding(.leading, 4) + .padding(.top, 4) + + VStack(alignment: .leading, spacing: 6) { + Text(title) + .font(.system(size: 18, weight: .semibold)) + .foregroundStyle(.black) + + Text(text) + .font(.system(size: 15)) + .foregroundStyle(.gray) + } + Spacer() + } + } +} diff --git a/ios/RayBan/Views/MainAppView.swift b/ios/RayBan/Views/MainAppView.swift new file mode 100644 index 0000000..f1727f1 --- /dev/null +++ b/ios/RayBan/Views/MainAppView.swift @@ -0,0 +1,32 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MainAppView.swift +// +// Central navigation hub that displays different views based on DAT SDK registration and device states. +// When unregistered, shows the registration flow. When registered, shows the device selection screen +// for choosing which Meta wearable device to stream from. +// + +import MWDATCore +import SwiftUI + +struct MainAppView: View { + let wearables: WearablesInterface + var viewModel: WearablesViewModel + + var body: some View { + if viewModel.registrationState == .registered { + StreamSessionView(wearables: wearables, wearablesVM: viewModel) + } else { + // User not registered - show registration/onboarding flow + HomeScreenView(viewModel: viewModel) + } + } +} diff --git a/ios/RayBan/Views/MockDeviceKit/MockDeviceCardView.swift b/ios/RayBan/Views/MockDeviceKit/MockDeviceCardView.swift new file mode 100644 index 0000000..1da83e3 --- /dev/null +++ b/ios/RayBan/Views/MockDeviceKit/MockDeviceCardView.swift @@ -0,0 +1,216 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MockDeviceCardView.swift +// +// UI component for managing individual mock Meta wearable devices during development. +// This card provides controls for simulating device states (power, wearing, folding) +// and loading mock media content for testing DAT SDK streaming and photo capture features. +// Useful for testing without requiring physical Meta hardware. +// + +#if DEBUG + +import MWDATMockDevice +import SwiftUI + +struct MockDeviceCardView: View { + @Bindable var viewModel: ViewModel + let onUnpairDevice: () -> Void + + @State private var showingVideoPicker = false + @State private var showingImagePicker = false + @State private var expanded = true + + private var isCameraSourceSelected: Bool { + viewModel.cameraSource == .front || viewModel.cameraSource == .back + } + + var body: some View { + CardView { + VStack(spacing: 0) { + // Header: device name + unpair, tappable to expand/collapse + HStack { + VStack(alignment: .leading, spacing: 4) { + Text(viewModel.deviceName) + .font(.headline) + .fontWeight(.semibold) + .foregroundStyle(.primary) + .lineLimit(1) + .truncationMode(.tail) + Text(viewModel.id) + .font(.caption) + .foregroundStyle(.secondary) + .lineLimit(1) + .truncationMode(.middle) + } + + Spacer() + + MockDeviceKitButton("Unpair", style: .destructive, expandsHorizontally: false) { + onUnpairDevice() + } + } + .contentShape(Rectangle()) + .onTapGesture { + withAnimation { + expanded.toggle() + } + } + + // Collapsible content + if expanded { + Divider() + .padding(.vertical, 4) + + VStack(spacing: 8) { + // Toggle switches + VStack(spacing: 0) { + Toggle( + "Power", + isOn: Binding( + get: { viewModel.isPoweredOn }, + set: { newValue in + if newValue { viewModel.powerOn() } else { viewModel.powerOff() } + } + ) + ) + .frame(height: 36) + + Toggle( + "Donned", + isOn: Binding( + get: { viewModel.isDonned }, + set: { newValue in + if newValue { viewModel.don() } else { viewModel.doff() } + } + ) + ) + .frame(height: 36) + + Toggle( + "Unfolded", + isOn: Binding( + get: { viewModel.isUnfolded }, + set: { newValue in + if newValue { viewModel.unfold() } else { viewModel.fold() } + } + ) + ) + .frame(height: 36) + } + + // Captouch gesture buttons + Text("Capacitive Touch Events") + .font(.subheadline) + .fontWeight(.medium) + .foregroundStyle(.secondary) + .frame(maxWidth: .infinity, alignment: .leading) + + HStack(spacing: 8) { + MockDeviceKitButton("Tap") { + viewModel.captouchTap() + } + MockDeviceKitButton("Tap & Hold") { + viewModel.captouchTapAndHold() + } + } + + // Camera source picker + CameraSourcePicker( + cameraSource: viewModel.cameraSource, + hasCameraFeed: viewModel.hasCameraFeed, + onFrontCamera: { viewModel.setCameraFeed(.front) }, + onBackCamera: { viewModel.setCameraFeed(.back) }, + onVideoFile: { showingVideoPicker = true } + ) + .sheet(isPresented: $showingVideoPicker) { + MediaPickerView(mode: .video) { url, _ in + viewModel.selectVideo(from: url) + } + } + + // Captured image control — hidden when a camera source (front/back) is selected + if !isCameraSourceSelected { + if viewModel.hasCapturedImage { + Text("Has captured image") + .font(.caption) + .foregroundStyle(.green) + .frame(maxWidth: .infinity, alignment: .leading) + } + + MockDeviceKitButton("Select image") { + showingImagePicker = true + } + .sheet(isPresented: $showingImagePicker) { + MediaPickerView(mode: .image) { url, _ in + viewModel.selectImage(from: url) + } + } + } + } + } + } + .padding() + } + .alert("Camera Access Required", isPresented: $viewModel.showCameraPermissionAlert) { + Button("Open Settings") { + viewModel.openSettings() + } + Button("Cancel", role: .cancel) {} + } message: { + Text("Camera access was denied. Please enable it in Settings to use the camera source.") + } + } +} + +private struct CameraSourcePicker: View { + let cameraSource: CameraFacing? + let hasCameraFeed: Bool + let onFrontCamera: () -> Void + let onBackCamera: () -> Void + let onVideoFile: () -> Void + + private var currentSourceLabel: String { + if let source = cameraSource { + return source == .front ? "Front Camera" : "Back Camera" + } else if hasCameraFeed { + return "Video File" + } + return "None" + } + + var body: some View { + Menu { + Button("Front Camera") { onFrontCamera() } + Button("Back Camera") { onBackCamera() } + Button("Video File") { onVideoFile() } + } label: { + HStack { + Text("Camera Source: \(currentSourceLabel)") + .font(.body) + .foregroundStyle(.primary) + Spacer() + Image(systemName: "chevron.down") + .font(.caption) + .foregroundStyle(.secondary) + } + .frame(maxWidth: .infinity, minHeight: 44) + .padding(.horizontal, 12) + .overlay( + RoundedRectangle(cornerRadius: 16) + .stroke(Color.secondary.opacity(0.3), lineWidth: 1) + ) + } + } +} + +// Replace this with PhotosPicker once we're on iOS 16 or newer + +#endif diff --git a/ios/RayBan/Views/MockDeviceKit/MockDeviceKitButton.swift b/ios/RayBan/Views/MockDeviceKit/MockDeviceKitButton.swift new file mode 100644 index 0000000..4162ec6 --- /dev/null +++ b/ios/RayBan/Views/MockDeviceKit/MockDeviceKitButton.swift @@ -0,0 +1,80 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MockDeviceKitButton.swift +// +// Specialized button component for mock device controls in the debug interface. +// + +#if DEBUG + +import SwiftUI + +struct MockDeviceKitButtonStyle: ButtonStyle { + @Environment(\.isEnabled) private var isEnabled + + var backgroundColor: Color + var foregroundColor: Color = .white + var isFullWidth: Bool = true + + func makeBody(configuration: Configuration) -> some View { + configuration.label + .foregroundStyle(foregroundColor.opacity(isEnabled ? 1.0 : 0.6)) + .padding(.horizontal) + .frame(maxWidth: isFullWidth ? .infinity : nil, minHeight: 44) + .background(backgroundColor.opacity(isEnabled ? 1.0 : 0.4)) + .clipShape(RoundedRectangle(cornerRadius: 16)) + .opacity(configuration.isPressed ? 0.8 : 1.0) + } +} + +struct MockDeviceKitButton: View { + enum Style { + case primary + case destructive + + var backgroundColor: Color { + switch self { + case .primary: + return .appPrimary + case .destructive: + return .red + } + } + } + + let title: String + let style: Style + let expandsHorizontally: Bool + let disabled: Bool + let action: () -> Void + + init(_ title: String, style: Style = .primary, expandsHorizontally: Bool = true, disabled: Bool = false, action: @escaping () -> Void) { + self.title = title + self.style = style + self.expandsHorizontally = expandsHorizontally + self.disabled = disabled + self.action = action + } + + var body: some View { + Button(title) { + action() + } + .buttonStyle( + MockDeviceKitButtonStyle( + backgroundColor: style.backgroundColor, + isFullWidth: expandsHorizontally + ) + ) + .disabled(disabled) + } +} + +#endif diff --git a/ios/RayBan/Views/MockDeviceKit/MockDeviceKitView.swift b/ios/RayBan/Views/MockDeviceKit/MockDeviceKitView.swift new file mode 100644 index 0000000..3e71d44 --- /dev/null +++ b/ios/RayBan/Views/MockDeviceKit/MockDeviceKitView.swift @@ -0,0 +1,89 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// MockDeviceKitView.swift +// +// Debug-only interface for managing mock Meta wearable devices during development. +// This view allows developers to create, configure, and test with simulated devices +// without requiring physical Meta hardware. +// + +#if DEBUG + +import Foundation +import SwiftUI + +struct MockDeviceKitView: View { + var viewModel: ViewModel + + var body: some View { + NavigationView { + ScrollView { + VStack(spacing: 12) { + CardView { + VStack(spacing: 6) { + HStack { + Text("Mock Device Kit") + .font(.headline) + .fontWeight(.bold) + .foregroundStyle(.primary) + Spacer() + + if viewModel.isEnabled { + Text("\(viewModel.cardViewModels.count) device(s) paired") + .font(.subheadline) + .foregroundStyle(.green) + } + } + + Text("This screen handles simulating devices, mocking capabilities, and states") + .font(.caption) + .foregroundStyle(.secondary) + .frame(maxWidth: .infinity, alignment: .leading) + + Divider() + + if viewModel.isEnabled { + MockDeviceKitButton("Disable MockDeviceKit", style: .destructive) { + viewModel.disable() + } + + MockDeviceKitButton("Pair RayBan Meta", disabled: viewModel.cardViewModels.count >= 3) { + viewModel.pairRaybanMeta() + } + } else { + MockDeviceKitButton("Enable MockDeviceKit") { + viewModel.enable() + } + } + } + .padding(12) + } + + if viewModel.isEnabled { + ForEach(viewModel.cardViewModels, id: \.id) { cardViewModel in + MockDeviceCardView( + viewModel: cardViewModel, + onUnpairDevice: { + viewModel.unpairDevice(cardViewModel.device) + } + ) + } + } + + Spacer() + } + .padding() + } + .background(Color(.systemGroupedBackground)) + } + } +} + +#endif diff --git a/ios/RayBan/Views/NonStreamView.swift b/ios/RayBan/Views/NonStreamView.swift new file mode 100644 index 0000000..d0afd6b --- /dev/null +++ b/ios/RayBan/Views/NonStreamView.swift @@ -0,0 +1,282 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// NonStreamView.swift +// +// Default screen to show getting started tips after app connection +// Initiates streaming +// + +import MWDATCore +import SwiftUI + +private let updateRequiredBackgroundColor = Color(red: 1.0, green: 0.957, blue: 0.839) +private let updateRequiredForegroundColor = Color(red: 0.541, green: 0.294, blue: 0.0) +private let updateRequiredTitle = "Update required" +private let waitingForActiveDeviceText = "Waiting for an active device" + +struct NonStreamView: View { + @Bindable var viewModel: StreamSessionViewModel + @Bindable var wearablesVM: WearablesViewModel + @State private var sheetHeight: CGFloat = 300 + @State private var showSettingsMenu: Bool = false + + private var isUpdateRequired: Bool { + wearablesVM.requiresFirmwareUpdate || viewModel.requiresDATAppUpdate + } + + var body: some View { + ZStack { + Color.black.edgesIgnoringSafeArea(.all) + + // Dismiss overlay when tapping outside the settings menu (placed first so it's behind content) + if showSettingsMenu { + Color.clear + .contentShape(Rectangle()) + .onTapGesture { + withAnimation(.spring(response: 0.3, dampingFraction: 0.7)) { + showSettingsMenu = false + } + } + .edgesIgnoringSafeArea(.all) + } + + VStack { + HStack { + Spacer() + Button { + withAnimation(.spring(response: 0.3, dampingFraction: 0.7)) { + showSettingsMenu.toggle() + } + } label: { + Image(systemName: "gearshape") + .resizable() + .aspectRatio(contentMode: .fit) + .foregroundStyle(.white) + .frame(width: 24, height: 24) + } + .overlay(alignment: .trailing) { + if showSettingsMenu { + CustomButton( + title: "Disconnect", + style: .destructive, + isDisabled: wearablesVM.registrationState != .registered + ) { + wearablesVM.disconnectGlasses() + withAnimation(.spring(response: 0.3, dampingFraction: 0.7)) { + showSettingsMenu = false + } + } + .frame(width: 120) + .transition(.scale(scale: 0.01, anchor: .trailing).combined(with: .opacity)) + } + } + } + + Spacer() + + VStack(spacing: 12) { + Image(.cameraAccessIcon) + .resizable() + .renderingMode(.template) + .foregroundStyle(.white) + .aspectRatio(contentMode: .fit) + .frame(width: 120) + + Text("Stream Your Glasses Camera") + .font(.system(size: 20, weight: .semibold)) + .foregroundStyle(.white) + + Text("Tap Start Streaming to stream audio and video from your glasses to the server, or use the camera button to take a photo.") + .font(.system(size: 15)) + .multilineTextAlignment(.center) + .foregroundStyle(.white) + } + .padding(.horizontal, 12) + + Spacer() + + VStack(spacing: 12) { + HStack(spacing: 8) { + Image(systemName: "hourglass") + .resizable() + .aspectRatio(contentMode: .fit) + .foregroundStyle(Color.white.opacity(0.7)) + .frame(width: 16, height: 16) + + Text(waitingForActiveDeviceText) + .font(.system(size: 14)) + .foregroundStyle(Color.white.opacity(0.7)) + } + .opacity(viewModel.hasActiveDevice ? 0 : 1) + + if isUpdateRequired { + UpdateRequiredMessage( + showFirmwareUpdate: wearablesVM.requiresFirmwareUpdate, + showDATAppUpdate: viewModel.requiresDATAppUpdate + ) + } + + if wearablesVM.requiresFirmwareUpdate { + CustomButton( + title: "Update firmware", + style: .primary, + isDisabled: false + ) { + Task { + await wearablesVM.openFirmwareUpdate() + } + } + } + + if viewModel.requiresDATAppUpdate { + CustomButton( + title: "Update app on glasses", + style: .primary, + isDisabled: false + ) { + Task { + await wearablesVM.openDATGlassesAppUpdate() + } + } + } + + CustomButton( + title: "Start Streaming", + style: .primary, + isDisabled: !viewModel.hasActiveDevice || isUpdateRequired + ) { + Task { + await viewModel.handleStartStreaming() + } + } + } + } + .padding(.all, 24) + } + .sheet(isPresented: $wearablesVM.showGettingStartedSheet) { + GettingStartedSheetView(height: $sheetHeight) + .presentationDetents([.height(sheetHeight)]) + .presentationDragIndicator(.visible) + } + } +} + +struct UpdateRequiredMessage: View { + let showFirmwareUpdate: Bool + let showDATAppUpdate: Bool + + private var message: String { + if showFirmwareUpdate && showDATAppUpdate { + return "Your glasses firmware and app need updates before Camera Access can start." + } + if showFirmwareUpdate { + return "Your glasses firmware needs an update before Camera Access can start." + } + return "The app on your glasses needs an update before Camera Access can start." + } + + var body: some View { + HStack(alignment: .top, spacing: 12) { + Image(systemName: "exclamationmark.triangle.fill") + .resizable() + .aspectRatio(contentMode: .fit) + .foregroundStyle(updateRequiredForegroundColor) + .frame(width: 24, height: 24) + .accessibilityHidden(true) + + VStack(alignment: .leading, spacing: 4) { + Text(updateRequiredTitle) + .font(.system(size: 16, weight: .semibold)) + .foregroundStyle(updateRequiredForegroundColor) + + Text(message) + .font(.system(size: 15)) + .foregroundStyle(updateRequiredForegroundColor) + .fixedSize(horizontal: false, vertical: true) + } + + Spacer(minLength: 0) + } + .padding(.all, 16) + .frame(maxWidth: .infinity, alignment: .leading) + .background(updateRequiredBackgroundColor) + .clipShape(RoundedRectangle(cornerRadius: 20, style: .continuous)) + } +} + +struct GettingStartedSheetView: View { + @Environment(\.dismiss) var dismiss + @Binding var height: CGFloat + + var body: some View { + VStack(spacing: 24) { + Text("Getting started") + .font(.system(size: 18, weight: .semibold)) + .foregroundStyle(.primary) + + VStack(spacing: 12) { + TipItemView( + resource: .videoIcon, + text: "First, Camera Access needs permission to use your glasses camera." + ) + TipItemView( + resource: .tapIcon, + text: "Capture photos by tapping the camera button." + ) + TipItemView( + resource: .smartGlassesIcon, + text: "The capture LED lets others know when you're capturing content or going live." + ) + } + .padding(.bottom, 16) + + CustomButton( + title: "Continue", + style: .primary, + isDisabled: false + ) { + dismiss() + } + } + .padding(.all, 24) + .background( + GeometryReader { geo -> Color in + DispatchQueue.main.async { + height = geo.size.height + } + return Color.clear + } + ) + } +} + +struct TipItemView: View { + let resource: ImageResource + let text: String + + var body: some View { + HStack(alignment: .top, spacing: 12) { + Image(resource) + .resizable() + .renderingMode(.template) + .foregroundStyle(.primary) + .aspectRatio(contentMode: .fit) + .frame(width: 24) + .padding(.leading, 4) + .padding(.top, 4) + + Text(text) + .font(.system(size: 15)) + .foregroundStyle(.primary) + .fixedSize(horizontal: false, vertical: true) + } + .frame(maxWidth: .infinity, alignment: .leading) + } +} diff --git a/ios/RayBan/Views/PhotoPreviewView.swift b/ios/RayBan/Views/PhotoPreviewView.swift new file mode 100644 index 0000000..c93e305 --- /dev/null +++ b/ios/RayBan/Views/PhotoPreviewView.swift @@ -0,0 +1,124 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// PhotoPreviewView.swift +// +// UI for previewing and sharing photos captured from Meta wearable devices via the DAT SDK. +// This view displays photos captured using Stream.capturePhoto() and provides sharing +// functionality. +// + +import SwiftUI + +struct PhotoPreviewView: View { + let photo: UIImage + let onDismiss: () -> Void + + @State private var showShareSheet = false + @State private var dragOffset = CGSize.zero + + var body: some View { + ZStack { + // Semi-transparent background overlay + Color.black.opacity(0.8) + .ignoresSafeArea() + .onTapGesture { + dismissWithAnimation() + } + + VStack(spacing: 20) { + photoDisplayView + + CircleButton(icon: "square.and.arrow.up", text: nil) { + showShareSheet = true + } + } + .padding() + .offset(dragOffset) + .animation(.spring(response: 0.6, dampingFraction: 0.8), value: dragOffset) + + // Close button in top right + VStack { + HStack { + Spacer() + CircleButton(icon: "xmark", text: nil) { + dismissWithAnimation() + } + .accessibilityIdentifier("close_preview_button") + .padding(.trailing, 20) + .padding(.top, 50) + } + Spacer() + } + } + .sheet(isPresented: $showShareSheet) { + ShareSheet(photo: photo) + } + } + + private var photoDisplayView: some View { + GeometryReader { geometry in + Image(uiImage: photo) + .resizable() + .aspectRatio(contentMode: .fit) + .frame(maxWidth: geometry.size.width, maxHeight: geometry.size.height * 0.6) + .cornerRadius(12) + .shadow(color: .black.opacity(0.3), radius: 10, x: 0, y: 5) + .frame(width: geometry.size.width, height: geometry.size.height) + .gesture( + DragGesture() + .onChanged { value in + dragOffset = value.translation + } + .onEnded { value in + if abs(value.translation.height) > 100 { + dismissWithAnimation() + } else { + withAnimation(.spring()) { + dragOffset = .zero + } + } + } + ) + } + } + + private func dismissWithAnimation() { + withAnimation(.easeInOut(duration: 0.3)) { + dragOffset = CGSize(width: 0, height: UIScreen.main.bounds.height) + } + Task { + try? await Task.sleep(nanoseconds: 300_000_000) + onDismiss() + } + } +} + +struct ShareSheet: UIViewControllerRepresentable { + let photo: UIImage + + func makeUIViewController(context: Context) -> UIActivityViewController { + let activityViewController = UIActivityViewController( + activityItems: [photo], + applicationActivities: nil + ) + + // Exclude certain activity types if needed + activityViewController.excludedActivityTypes = [ + .assignToContact, + .addToReadingList, + ] + + return activityViewController + } + + func updateUIViewController(_ uiViewController: UIActivityViewController, context: Context) { + // No updates needed + } +} diff --git a/ios/RayBan/Views/RegistrationView.swift b/ios/RayBan/Views/RegistrationView.swift new file mode 100644 index 0000000..8200f05 --- /dev/null +++ b/ios/RayBan/Views/RegistrationView.swift @@ -0,0 +1,48 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// RegistrationView.swift +// +// Background view that handles callbacks from the Meta AI mobile app during +// DAT SDK registration and permission flows. This invisible view processes deep links +// that complete the OAuth authorization process initiated by the DAT SDK. +// + +import MWDATCore +import SwiftUI + +struct RegistrationView: View { + var viewModel: WearablesViewModel + + var body: some View { + EmptyView() + // Handle callback URLs from the Meta mobile app + // This is essential for completing DAT SDK registration and permission flows + .onOpenURL { url in + guard + let components = URLComponents(url: url, resolvingAgainstBaseURL: false), + // Check if this URL is related to DAT SDK workflows (contains metaWearablesAction query param) + components.queryItems?.contains(where: { $0.name == "metaWearablesAction" }) == true + else { + return // URL is not related to DAT SDK - ignore it + } + Task { + do { + // Pass the callback URL to the DAT SDK for processing + // This handles registration completion and permission grant responses + _ = try await Wearables.shared.handleUrl(url) + } catch let error as RegistrationError { + viewModel.showError(error.description) + } catch { + viewModel.showError("Unknown error: \(error.localizedDescription)") + } + } + } + } +} diff --git a/ios/RayBan/Views/StreamSessionView.swift b/ios/RayBan/Views/StreamSessionView.swift new file mode 100644 index 0000000..bef73be --- /dev/null +++ b/ios/RayBan/Views/StreamSessionView.swift @@ -0,0 +1,56 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// StreamSessionView.swift +// +// + +import MWDATCore +import SwiftUI + +struct StreamSessionView: View { + let wearables: WearablesInterface + var wearablesViewModel: WearablesViewModel + @State private var viewModel: StreamSessionViewModel + + init(wearables: WearablesInterface, wearablesVM: WearablesViewModel) { + self.wearables = wearables + self.wearablesViewModel = wearablesVM + self._viewModel = State(wrappedValue: StreamSessionViewModel(wearables: wearables)) + } + + var body: some View { + ZStack { + if viewModel.isStreaming { + // Full-screen video view with streaming controls + StreamView(viewModel: viewModel, wearablesVM: wearablesViewModel) + } else { + // Pre-streaming setup view with permissions and start button + NonStreamView(viewModel: viewModel, wearablesVM: wearablesViewModel) + } + } + .onDisappear { + viewModel.endSession() + } + .alert("Error", isPresented: $viewModel.showError) { + Button("OK") { + viewModel.dismissError() + } + } message: { + Text(viewModel.errorMessage) + } + .alert("Photo capture failed", isPresented: $viewModel.showPhotoCaptureError) { + Button("OK") { + viewModel.dismissPhotoCaptureError() + } + } message: { + Text("Unable to capture photo. This may be due to low storage on device or another capture already in progress. Please try again in a few moments.") + } + } +} diff --git a/ios/RayBan/Views/StreamView.swift b/ios/RayBan/Views/StreamView.swift new file mode 100644 index 0000000..88cc8f9 --- /dev/null +++ b/ios/RayBan/Views/StreamView.swift @@ -0,0 +1,136 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +// +// StreamView.swift +// +// Main UI for video streaming from Meta wearable devices using the DAT SDK. +// This view demonstrates the complete streaming API: video streaming with real-time display, photo capture, +// and error handling. +// + +import MWDATCore +import SwiftUI + +struct StreamView: View { + @Bindable var viewModel: StreamSessionViewModel + var wearablesVM: WearablesViewModel + + var body: some View { + ZStack { + // Black background for letterboxing/pillarboxing + Color.black + .edgesIgnoringSafeArea(.all) + + // Video backdrop + if let videoFrame = viewModel.currentVideoFrame, viewModel.hasReceivedFirstFrame { + GeometryReader { geometry in + Image(uiImage: videoFrame) + .resizable() + .aspectRatio(contentMode: .fill) + .frame(width: geometry.size.width, height: geometry.size.height) + .clipped() + } + .edgesIgnoringSafeArea(.all) + } else { + ProgressView() + .scaleEffect(1.5) + .foregroundStyle(.white) + } + + // Bottom controls layer + + VStack { + Spacer() + ControlsView(viewModel: viewModel) + } + .padding(.all, 24) + } + .onDisappear { + Task { + if viewModel.streamingStatus != .stopped { + await viewModel.stopSession() + } + } + } + // Show captured photos from DAT SDK in a preview sheet + .sheet(isPresented: $viewModel.showPhotoPreview) { + if let photo = viewModel.capturedPhoto { + PhotoPreviewView( + photo: photo, + onDismiss: { + viewModel.dismissPhotoPreview() + } + ) + } + } + } +} + +// Extracted controls for clarity +struct ControlsView: View { + var viewModel: StreamSessionViewModel + + var body: some View { + // Controls row + HStack(spacing: 8) { + CustomButton( + title: "Stop streaming", + style: .destructive, + isDisabled: false + ) { + Task { + await viewModel.stopSession() + } + } + + CustomButton( + title: viewModel.isPaused ? "Resume" : "Pause", + style: .primary, + isDisabled: viewModel.streamingStatus != .streaming + ) { + if viewModel.isPaused { + viewModel.resumeSession() + } else { + viewModel.pauseSession() + } + } + .accessibilityIdentifier("pause_resume_button") + + // Video feed toggle: enables / disables publishing JPEG frames to the + // server. Audio publishes regardless. The server can also flip this + // back on via a `{"type":"video_on"}` control message. + CircleButton( + icon: viewModel.videoEnabled ? "video.fill" : "video.slash.fill", + text: nil + ) { + viewModel.toggleVideoFeed() + } + .accessibilityIdentifier("video_toggle_button") + + #if DEBUG + // PROTOCOL.md §2.6 spike: toggle the direct WebRTC audio leg over HFP + // while the camera stream keeps running. Watch the Xcode console for + // up/downlink audio levels and the audio route. + CircleButton( + icon: viewModel.isWebRTCSpikeActive ? "waveform.circle.fill" : "waveform.circle", + text: nil + ) { + Task { + if viewModel.isWebRTCSpikeActive { + await viewModel.stopWebRTCAudioSpike() + } else { + await viewModel.startWebRTCAudioSpike() + } + } + } + .accessibilityIdentifier("webrtc_spike_button") + #endif + } + } +} diff --git a/ios/RayBanTests/RayBanTests.swift b/ios/RayBanTests/RayBanTests.swift new file mode 100644 index 0000000..1c68a28 --- /dev/null +++ b/ios/RayBanTests/RayBanTests.swift @@ -0,0 +1,224 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +@testable import RayBan +import Foundation +import MWDATCore +import MWDATMockDevice +import Observation +import SwiftUI +import XCTest + +@MainActor +final class ViewModelIntegrationTests: XCTestCase { + + private var mockDevice: MockRaybanMeta? + private var cameraKit: MockCameraKit? + private var viewModel: StreamSessionViewModel? + + override func setUp() async throws { + try await super.setUp() + try? Wearables.configure() + + MockDeviceKit.shared.enable() + + // Pair mock device and set up camera kit + let pairedMockDevice = MockDeviceKit.shared.pairRaybanMeta() + mockDevice = pairedMockDevice + cameraKit = pairedMockDevice.services.camera + + // Power on and unfold the device to make it available + pairedMockDevice.powerOn() + pairedMockDevice.unfold() + + // Wait for device to be available in Wearables + try await Task.sleep(nanoseconds: 1_000_000_000) + } + + override func tearDown() async throws { + viewModel?.endSession() + viewModel = nil + MockDeviceKit.shared.disable() + mockDevice = nil + cameraKit = nil + try await super.tearDown() + } + + // MARK: - Video Streaming Flow Tests + + func testVideoStreamingFlow() async throws { + guard let camera = cameraKit else { + XCTFail("Mock device and camera should be available") + return + } + + guard let videoURL = Bundle.main.url(forResource: "plant", withExtension: "mp4") + else { + XCTFail("Test resources not found") + return + } + + // Setup camera feed + camera.setCameraFeed(fileURL: videoURL) + + let viewModel = StreamSessionViewModel(wearables: Wearables.shared) + self.viewModel = viewModel + + // Wait for the mock device to be detected + await observeUntil(timeout: 5) { viewModel.hasActiveDevice } + + // Initially not streaming + XCTAssertEqual(viewModel.streamingStatus, .stopped) + XCTAssertFalse(viewModel.isStreaming) + XCTAssertFalse(viewModel.hasReceivedFirstFrame) + XCTAssertNil(viewModel.currentVideoFrame) + + // Start streaming session + await viewModel.handleStartStreaming() + + // Wait for streaming to establish + await observeUntil(timeout: 10) { + viewModel.isStreaming && viewModel.hasReceivedFirstFrame && viewModel.currentVideoFrame != nil + } + + // Verify streaming is active and receiving frames + XCTAssertTrue(viewModel.isStreaming) + XCTAssertTrue(viewModel.hasReceivedFirstFrame) + XCTAssertNotNil(viewModel.currentVideoFrame) + XCTAssertTrue([.streaming, .waiting].contains(viewModel.streamingStatus)) + + // Stop streaming + await viewModel.stopSession() + + // Wait for session to stop + await observeUntil(timeout: 5) { !viewModel.isStreaming } + + // Verify streaming stopped (allow for final states to be stopped or waiting) + XCTAssertFalse(viewModel.isStreaming) + XCTAssertTrue([.stopped, .waiting].contains(viewModel.streamingStatus)) + } + + // MARK: - Photo Capture Flow Tests + + func testStreamingAndPhotoCaptureFlow() async throws { + guard let camera = cameraKit else { + XCTFail("Mock device and camera should be available") + return + } + + guard let videoURL = Bundle.main.url(forResource: "plant", withExtension: "mp4"), + let imageURL = Bundle.main.url(forResource: "plant", withExtension: "png") + else { + XCTFail("Test resources not found") + return + } + + // Setup camera feed + camera.setCameraFeed(fileURL: videoURL) + camera.setCapturedImage(fileURL: imageURL) + + let viewModel = StreamSessionViewModel(wearables: Wearables.shared) + self.viewModel = viewModel + + // Wait for the mock device to be detected + await observeUntil(timeout: 5) { viewModel.hasActiveDevice } + + // Initially not streaming + XCTAssertEqual(viewModel.streamingStatus, .stopped) + XCTAssertFalse(viewModel.isStreaming) + XCTAssertFalse(viewModel.hasReceivedFirstFrame) + XCTAssertNil(viewModel.currentVideoFrame) + + // Start streaming session + await viewModel.handleStartStreaming() + + // Wait for streaming to establish + await observeUntil(timeout: 10) { + viewModel.isStreaming && viewModel.hasReceivedFirstFrame && viewModel.currentVideoFrame != nil + } + + // Verify streaming is active and receiving frames + XCTAssertTrue(viewModel.isStreaming) + XCTAssertTrue(viewModel.hasReceivedFirstFrame) + XCTAssertNotNil(viewModel.currentVideoFrame) + XCTAssertTrue([.streaming, .waiting].contains(viewModel.streamingStatus)) + + // Capture photo while streaming + viewModel.capturePhoto() + await observeUntil(timeout: 10) { viewModel.capturedPhoto != nil } + + // Verify photo captured while maintaining stream (allow for some timing flexibility) + XCTAssertTrue(viewModel.capturedPhoto != nil) + XCTAssertTrue(viewModel.showPhotoPreview) + XCTAssertTrue(viewModel.isStreaming) + + // Dismiss photo and stop streaming + viewModel.dismissPhotoPreview() + XCTAssertFalse(viewModel.showPhotoPreview) + XCTAssertNil(viewModel.capturedPhoto) + + await viewModel.stopSession() + await observeUntil(timeout: 5) { !viewModel.isStreaming } + + XCTAssertFalse(viewModel.isStreaming) + XCTAssertTrue([.stopped, .waiting].contains(viewModel.streamingStatus)) + } +} + +// MARK: - Test Helpers + +/// Thread-safe one-shot flag for protecting continuation resumption. +private final class ResumeOnce: @unchecked Sendable { + private let lock = NSLock() + private var resumed = false + func tryResume() -> Bool { + lock.lock() + defer { lock.unlock() } + guard !resumed else { return false } + resumed = true + return true + } +} + +/// Reactively waits for a condition on @Observable objects to become true. +/// Uses `withObservationTracking` to wake up immediately on property changes +/// instead of polling on a fixed interval. +@MainActor +private func observeUntil( + timeout: TimeInterval, + file: StaticString = #filePath, + line: UInt = #line, + condition: @escaping () -> Bool +) async { + guard !condition() else { return } + + let deadline = ContinuousClock.now + .seconds(timeout) + + while !condition() { + guard ContinuousClock.now < deadline else { + XCTFail("Condition not met within \(timeout) seconds", file: file, line: line) + return + } + + await withUnsafeContinuation { cont in + let once = ResumeOnce() + + withObservationTracking { + _ = condition() + } onChange: { + if once.tryResume() { cont.resume() } + } + + // Periodic fallback so we can re-evaluate the deadline + Task { + try? await Task.sleep(for: .milliseconds(100)) + if once.tryResume() { cont.resume() } + } + } + } +} diff --git a/ios/RayBanUITests/RayBanUITests.swift b/ios/RayBanUITests/RayBanUITests.swift new file mode 100644 index 0000000..96f9249 --- /dev/null +++ b/ios/RayBanUITests/RayBanUITests.swift @@ -0,0 +1,262 @@ +/* + * Copyright (c) Meta Platforms, Inc. and affiliates. + * All rights reserved. + * + * This source code is licensed under the license found in the + * LICENSE file in the root directory of this source tree. + */ + +import MWDATMockDeviceTestClient +import XCTest + +final class RayBanUITests: XCTestCase { + var portFilePath: String { + NSTemporaryDirectory() + "mwdat_test_server_port.txt" + } + private let app = XCUIApplication() + // swiftlint:disable implicitly_unwrapped_optional + private var mockClient: MockDeviceTestClient! + private var pairedDeviceId: String! + // swiftlint:enable implicitly_unwrapped_optional + + override func setUpWithError() throws { + continueAfterFailure = false + + // Remove any stale port file from a previous run so readPort() waits + // for the new server to write its port instead of returning the old one. + try? FileManager.default.removeItem(atPath: portFilePath) + + app.launchArguments = ["--ui-testing"] + app.launchEnvironment["MWDAT_TEST_SERVER_PORT_FILE"] = portFilePath + app.launch() + + // Initialize the client *after* launch so the server has time to write the port file. + mockClient = MockDeviceTestClient(portFilePath: portFilePath) + XCTAssertTrue(mockClient.waitForServer(timeout: 10), "Test server should be running") + } + + override func tearDownWithError() throws { + if pairedDeviceId != nil { + mockClient.unpairDevice(deviceId: pairedDeviceId) + pairedDeviceId = nil + } + } + + // MARK: - Helpers + + /// Taps "Connect my glasses" to trigger registration via the fake handler, + /// dismisses the getting-started sheet, and waits for the streaming screen + /// to be fully ready for device connections. + private func registerViaUI() { + let connectButton = app.buttons["Connect my glasses"] + XCTAssertTrue(connectButton.waitForExistence(timeout: 10), "Should start on HomeScreenView") + connectButton.tap() + + // Dismiss the getting-started sheet if it appears after registration. + // In some environments (e.g. RE) the sheet may be skipped and the app + // transitions directly to the stream screen. + let continueButton = app.buttons["Continue"] + if continueButton.waitForExistence(timeout: 5) { + continueButton.tap() + } + + // Wait for the NonStreamView to be fully rendered with device monitoring active + let waitingText = app.staticTexts["Waiting for an active device"] + XCTAssertTrue(waitingText.waitForExistence(timeout: 15), "Should show waiting state before a device is paired") + } + + /// Pairs a device with default camera resources via the test server. + private func pairDeviceWithCameraResources() { + registerViaUI() + + let deviceId = mockClient.pairDevice() + XCTAssertNotNil(deviceId, "pairDevice should return a deviceId") + pairedDeviceId = deviceId + + mockClient.setCameraFeed(deviceId: pairedDeviceId, resourceName: "plant", ext: "mp4") + mockClient.setCapturedImage(deviceId: pairedDeviceId, resourceName: "plant", ext: "png") + } + + /// Waits for the "Start streaming" button to exist and be enabled (mock device active). + @discardableResult + private func waitForStartStreamingEnabled(timeout: TimeInterval = 15) -> XCUIElement { + let startButton = app.buttons["Start streaming"] + XCTAssertTrue(startButton.waitForExistence(timeout: timeout), "Start streaming button should appear") + + let predicate = NSPredicate(format: "isEnabled == true") + expectation(for: predicate, evaluatedWith: startButton) + waitForExpectations(timeout: timeout) + + return startButton + } + + /// Waits for the "Start streaming" button to exist and be disabled (device inactive). + @discardableResult + private func waitForStartStreamingDisabled(timeout: TimeInterval = 15) -> XCUIElement { + let startButton = app.buttons["Start streaming"] + XCTAssertTrue(startButton.waitForExistence(timeout: timeout), "Start streaming button should appear") + + let predicate = NSPredicate(format: "isEnabled == false") + expectation(for: predicate, evaluatedWith: startButton) + waitForExpectations(timeout: timeout) + + return startButton + } + + /// Starts streaming and waits for the StreamView to appear. + private func startStreaming(timeout: TimeInterval = 15) { + let startButton = waitForStartStreamingEnabled(timeout: timeout) + startButton.tap() + + let stopButton = app.buttons["Stop streaming"] + XCTAssertTrue(stopButton.waitForExistence(timeout: timeout), "Stop streaming button should appear after starting") + } + + // MARK: - Device Pairing & Navigation Tests + + /// Verifies that launching without pairing a device shows the home screen. + @MainActor + func testLaunchWithoutDeviceShowsHomeScreen() { + let connectButton = app.buttons["Connect my glasses"] + XCTAssertTrue( + connectButton.waitForExistence(timeout: 10), + "HomeScreenView should show 'Connect my glasses' when no device is paired" + ) + } + + /// Verifies that registering and pairing a device transitions the UI from the home screen + /// to the stream screen with an active device. + @MainActor + func testRegisterAndPairTransitionsToStreamScreen() { + pairDeviceWithCameraResources() + waitForStartStreamingEnabled() + } + + /// Verifies that the device state query reflects the correct number of paired devices. + @MainActor + func testDeviceStateReflectsPairedDevices() { + // Initially no devices paired + let state0 = mockClient.getDeviceState() + XCTAssertNotNil(state0, "getDeviceState should return a response") + XCTAssertEqual(state0?["pairedDeviceCount"] as? Int, 0, "Should have 0 paired devices initially") + + // Register and pair a device + pairDeviceWithCameraResources() + + let state1 = mockClient.getDeviceState() + XCTAssertNotNil(state1, "getDeviceState should return a response after pairing") + XCTAssertEqual(state1?["pairedDeviceCount"] as? Int, 1, "Should have 1 paired device") + + // Unpair + mockClient.unpairDevice(deviceId: pairedDeviceId) + pairedDeviceId = nil + + let state2 = mockClient.getDeviceState() + XCTAssertNotNil(state2, "getDeviceState should return a response after unpairing") + XCTAssertEqual(state2?["pairedDeviceCount"] as? Int, 0, "Should have 0 paired devices after unpairing") + } + + // MARK: - Device Activity Tests + + /// Verifies that doff makes the device inactive (disables streaming button) + /// and don reactivates it. + @MainActor + func testDoffMakesDeviceInactiveAndDonReactivates() { + pairDeviceWithCameraResources() + waitForStartStreamingEnabled() + + // Doff the device → should become inactive + mockClient.doff(deviceId: pairedDeviceId) + waitForStartStreamingDisabled() + + // Don the device → should become active again + mockClient.don(deviceId: pairedDeviceId) + waitForStartStreamingEnabled() + } + + /// Verifies that powering off makes the device inactive and powering on + /// with don reactivates it. + @MainActor + func testPowerCycleAffectsDeviceActivity() { + pairDeviceWithCameraResources() + waitForStartStreamingEnabled() + + // Power off → device becomes inactive + XCTAssertTrue(mockClient.powerOff(deviceId: pairedDeviceId), "Power off should succeed") + waitForStartStreamingDisabled() + + // Power on + don → device becomes active again + XCTAssertTrue(mockClient.powerOn(deviceId: pairedDeviceId), "Power on should succeed") + XCTAssertTrue(mockClient.don(deviceId: pairedDeviceId), "Don should succeed") + waitForStartStreamingEnabled() + } + + // MARK: - Streaming Tests + + /// Verifies the complete start → stop streaming flow. + // TestRail: C1599889064, C1602923640, C1602923646 + @MainActor + func testStartAndStopStreaming() { + pairDeviceWithCameraResources() + startStreaming() + + // Stop streaming + let stopButton = app.buttons["Stop streaming"] + stopButton.tap() + + // Should return to NonStreamView + let startButton = app.buttons["Start streaming"] + XCTAssertTrue(startButton.waitForExistence(timeout: 10), "Should return to NonStreamView after stopping") + XCTAssertTrue(app.staticTexts["Stream Your Glasses Camera"].exists, "NonStreamView title should reappear") + } + + /// Verifies photo capture shows a preview and can be dismissed while continuing to stream. + // TestRail: C1619609872, C1619610952 + @MainActor + func testPhotoCaptureAndDismiss() { + pairDeviceWithCameraResources() + startStreaming() + + // Tap the capture button + let captureButton = app.buttons["capture_photo_button"] + XCTAssertTrue(captureButton.waitForExistence(timeout: 10), "Capture button should be visible during streaming") + captureButton.tap() + + // Photo preview should appear + let closeButton = app.buttons["close_preview_button"] + XCTAssertTrue(closeButton.waitForExistence(timeout: 15), "Photo preview close button should appear after capture") + + // Dismiss the preview + closeButton.tap() + + // Should still be streaming after dismissing preview + let stopButton = app.buttons["Stop streaming"] + XCTAssertTrue(stopButton.waitForExistence(timeout: 10), "Should still be streaming after dismissing photo preview") + + // Stop streaming + stopButton.tap() + + // Should return to NonStreamView + let startButton = app.buttons["Start streaming"] + XCTAssertTrue(startButton.waitForExistence(timeout: 10), "Should return to NonStreamView after stopping") + } + + /// Verifies that folding the glasses while streaming causes streaming to stop. + @MainActor + func testFoldDuringStreamingStopsStream() { + pairDeviceWithCameraResources() + startStreaming() + + // Fold the glasses → streaming should stop (hinges closed) + XCTAssertTrue(mockClient.fold(deviceId: pairedDeviceId), "Fold command should succeed") + + // Fold triggers a hingesClosed error alert — dismiss it so the view hierarchy settles. + let alertOK = app.alerts.buttons["OK"] + if alertOK.waitForExistence(timeout: 15) { + alertOK.tap() + } + + // Should return to NonStreamView with the button disabled (device is folded). + waitForStartStreamingDisabled() + } +} diff --git a/yc-voice-agents-hackathon/server/bot-courtline.py b/yc-voice-agents-hackathon/server/bot-courtline.py index 9d96628..a760281 100644 --- a/yc-voice-agents-hackathon/server/bot-courtline.py +++ b/yc-voice-agents-hackathon/server/bot-courtline.py @@ -14,13 +14,14 @@ import asyncio import json import os +import queue as pyqueue from datetime import date from dotenv import load_dotenv from loguru import logger from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.audio.vad.silero import SileroVADAnalyzer -from pipecat.frames.frames import LLMRunFrame +from pipecat.frames.frames import InputAudioRawFrame, LLMRunFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.processors.aggregators.llm_context import LLMContext @@ -302,9 +303,70 @@ async def on_assistant_turn_stopped(agg, message): ), ) + # ── Mac-microphone audio input (optional) ───────────────────────────────── + # When MAC_MIC_INPUT=1, capture THIS machine's microphone and feed it into + # the pipeline as the user-audio source. This is the glasses use case: the + # iOS app connects receive-only (WebRTC on iOS can't reliably capture the + # Bluetooth-HFP glasses mic), so the agent listens via the Mac while it + # speaks back over WebRTC to the glasses. No-op unless the flag is set, so + # the normal browser/WebRTC audio-in path is unaffected. PortAudio delivers + # blocks on its own thread; we hand them to the loop via a thread-safe queue. + _mic_enabled = os.getenv("MAC_MIC_INPUT") == "1" + _mic_queue: "pyqueue.Queue[bytes | None]" = pyqueue.Queue() + _mic_stream = None + _mic_task = None + + def _mic_callback(indata, frames, time_info, status): + if status: + logger.warning(f"[mac-mic] stream status: {status}") + _mic_queue.put(bytes(indata)) + + async def _mic_pump(): + loop = asyncio.get_event_loop() + n = 0 + while True: + data = await loop.run_in_executor(None, _mic_queue.get) + if data is None: # sentinel on disconnect + break + await worker.queue_frames( + [InputAudioRawFrame(audio=data, sample_rate=16000, num_channels=1)] + ) + n += 1 + if n == 1 or n % 100 == 0: + logger.info(f"[mac-mic] fed {n} chunks into the pipeline") + + def _start_mic(): + nonlocal _mic_stream, _mic_task + if not _mic_enabled or _mic_stream is not None: + return + try: + import sounddevice as sd # lazy: only needed when MAC_MIC_INPUT=1 + + _mic_stream = sd.RawInputStream( + samplerate=16000, channels=1, dtype="int16", + blocksize=320, callback=_mic_callback, # 20 ms @ 16 kHz + ) + _mic_stream.start() + _mic_task = asyncio.create_task(_mic_pump()) + logger.info("[mac-mic] capturing local microphone -> agent") + except Exception as e: + logger.error(f"[mac-mic] could not open microphone: {e}") + + def _stop_mic(): + nonlocal _mic_stream, _mic_task + if _mic_stream is not None: + _mic_stream.stop() + _mic_stream.close() + _mic_stream = None + _mic_queue.put(None) # unblock _mic_pump + if _mic_task is not None: + _mic_task.cancel() + _mic_task = None + @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): logger.info("CourtLine client connected") + _start_mic() context.add_message( { "role": "user", @@ -319,6 +381,7 @@ async def on_client_connected(transport, client): @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): logger.info("CourtLine client disconnected") + _stop_mic() await worker.cancel() runner = WorkerRunner(handle_sigint=False) diff --git a/yc-voice-agents-hackathon/server/pyproject.toml b/yc-voice-agents-hackathon/server/pyproject.toml index 942b941..5b2f26e 100644 --- a/yc-voice-agents-hackathon/server/pyproject.toml +++ b/yc-voice-agents-hackathon/server/pyproject.toml @@ -12,6 +12,7 @@ dependencies = [ "python-multipart>=0.0.9", "fer>=22.5.0", "opencv-python-headless>=4.8.0", + "sounddevice>=0.5", # optional Mac-microphone input (MAC_MIC_INPUT=1) ] [dependency-groups] diff --git a/yc-voice-agents-hackathon/server/uv.lock b/yc-voice-agents-hackathon/server/uv.lock index 8e0aa26..e311f1c 100644 --- a/yc-voice-agents-hackathon/server/uv.lock +++ b/yc-voice-agents-hackathon/server/uv.lock @@ -3774,6 +3774,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] +[[package]] +name = "sounddevice" +version = "0.5.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2a/f9/2592608737553638fca98e21e54bfec40bf577bb98a61b2770c912aab25e/sounddevice-0.5.5.tar.gz", hash = "sha256:22487b65198cb5bf2208755105b524f78ad173e5ab6b445bdab1c989f6698df3", size = 143191, upload-time = "2026-01-23T18:36:43.529Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/0a/478e441fd049002cf308520c0d62dd8333e7c6cc8d997f0dda07b9fbcc46/sounddevice-0.5.5-py3-none-any.whl", hash = "sha256:30ff99f6c107f49d25ad16a45cacd8d91c25a1bcdd3e81a206b921a3a6405b1f", size = 32807, upload-time = "2026-01-23T18:36:35.649Z" }, + { url = "https://files.pythonhosted.org/packages/56/f9/c037c35f6d0b6bc3bc7bfb314f1d6f1f9a341328ef47cd63fc4f850a7b27/sounddevice-0.5.5-py3-none-macosx_10_6_x86_64.macosx_10_6_universal2.whl", hash = "sha256:05eb9fd6c54c38d67741441c19164c0dae8ce80453af2d8c4ad2e7823d15b722", size = 108557, upload-time = "2026-01-23T18:36:37.41Z" }, + { url = "https://files.pythonhosted.org/packages/88/a1/d19dd9889cd4bce2e233c4fac007cd8daaf5b9fe6e6a5d432cf17be0b807/sounddevice-0.5.5-py3-none-win32.whl", hash = "sha256:1234cc9b4c9df97b6cbe748146ae0ec64dd7d6e44739e8e42eaa5b595313a103", size = 317765, upload-time = "2026-01-23T18:36:39.047Z" }, + { url = "https://files.pythonhosted.org/packages/c3/0e/002ed7c4c1c2ab69031f78989d3b789fee3a7fba9e586eb2b81688bf4961/sounddevice-0.5.5-py3-none-win_amd64.whl", hash = "sha256:cfc6b2c49fb7f555591c78cb8ecf48d6a637fd5b6e1db5fec6ed9365d64b3519", size = 365324, upload-time = "2026-01-23T18:36:40.496Z" }, + { url = "https://files.pythonhosted.org/packages/4e/39/a61d4b83a7746b70d23d9173be688c0c6bfc7173772344b7442c2c155497/sounddevice-0.5.5-py3-none-win_arm64.whl", hash = "sha256:3861901ddd8230d2e0e8ae62ac320cdd4c688d81df89da036dcb812f757bb3e6", size = 317115, upload-time = "2026-01-23T18:36:42.235Z" }, +] + [[package]] name = "soxr" version = "1.0.0" @@ -4515,6 +4531,7 @@ dependencies = [ { name = "pipecat-ai", extra = ["gradium", "openai", "runner", "webrtc", "websocket"] }, { name = "pipecatcloud" }, { name = "python-multipart" }, + { name = "sounddevice" }, { name = "uvicorn", extra = ["standard"] }, ] @@ -4533,6 +4550,7 @@ requires-dist = [ { name = "pipecat-ai", extras = ["gradium", "openai", "runner", "silero", "webrtc", "websocket"], specifier = ">=1.3.0" }, { name = "pipecatcloud", specifier = ">=0.7.1" }, { name = "python-multipart", specifier = ">=0.0.9" }, + { name = "sounddevice", specifier = ">=0.5" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.30.0" }, ]