🐱 Kitty Captions

Real-time "translation" for your cats — shown right on your Meta Ray-Ban Display glasses.

Teach the app a few examples of what your cat's behaviors mean ("by the food bowl = wants snacks"), and whenever your kitty looks that way again, a caption pops up on the glasses HUD in real time — so friends and cat sitters always know how they feel and what they want.

Built on the Meta Wearables Device Access Toolkit (DAT 0.7) for iOS.

---

🎥 Demo

Coming soon — a glasses-POV clip of captions popping up over Chocobo & Rufus. Sample caption cards (styled like the HUD, transparent PNGs for compositing over footage) live in overlays/.

---

✨ What it does

• Live mood captions on the lens. The glasses camera watches your cat; matched captions render on the in-lens display.
• Teach by example. Snap a frame, label what it means. Add a few angles per caption for reliability.
• Fully on-device. No cloud, no per-frame API calls, no training step.
• Per-pet captions. e.g. Chocobo: I want cuddles ❤️, Rufus: Snacks! 🍟.

🧠 How the matching works

Captions uses few-shot, example-based image matching with Apple's Vision framework, entirely on-device:

1. Each saved reference photo is run through VNGenerateImageFeaturePrintRequest, producing a feature print — a compact perceptual embedding of what the image looks like.
2. Every sampled video frame from the glasses gets its own feature print.
3. A nearest-neighbor search (VNFeaturePrintObservation.computeDistance) finds the closest saved example. If that distance is under a tunable threshold, its label becomes the caption.
4. To avoid flicker, a caption only appears after it's the top match for several consecutive frames, then lingers briefly before clearing.

No training, no cloud — just embeddings + a distance metric, computed locally and streamed to the heads-up display.

🏗️ Architecture notes

A couple of things that were non-obvious to get working with the DAT SDK:

• Camera + HUD share one session. A device is 1:1 with a DeviceSession; the camera Stream and the Display capability are both added to that same session so live video and HUD captions run concurrently.
• DAMEnabled is required. The Display capability silently never reaches .started unless MWDAT.DAMEnabled = true is set in Info.plist. (This one cost a few hours.)
• Matching runs ~3×/sec on a background queue; captions are pushed to the HUD only when they actually change.

📋 Requirements

• Meta Ray-Ban Display glasses (display-capable) paired in the Meta AI app
• iOS 17+, Xcode 26+
• A device — the DAT SDK has no simulator support
• Developer Mode enabled in the Meta AI app

🚀 Setup

1. Open CameraAccess.xcodeproj in Xcode.
2. Select the CameraAccess target → Signing & Capabilities → set your Team (automatic signing). The bundle ID is com.fasai.captions — change it if needed.
3. In the Meta AI app, turn on Developer Mode (so no Wearables Developer Center registration is needed — Info.plist uses MetaAppID = 0).
4. Make sure Info.plist → MWDAT → DAMEnabled is true (it is, by default here).
5. Build & run on your iPhone, tap Connect, then Start Session.

🎮 Using it

• Start Session — begins watching through the glasses.
• ＋ Teach — freezes the current frame, then you name the mood (reuse-existing chips keep multi-angle rules tidy).
• ♥ Captions — manage saved rules (rename / delete / add angles).
• ⚙️ Settings — match sensitivity (default 0.75), caption linger, and a Session Debugger (live feed + distance + matched-frame count) for tuning.

📁 Project structure

CameraAccess/
├── ViewModels/
│   ├── CaptionEngine.swift            # Vision feature-print matching, debounce, persistence
│   ├── CaptionDisplayPresenter.swift  # renders captions to the glasses HUD
│   ├── StreamSessionViewModel.swift   # camera stream + session lifecycle
│   └── DeviceSessionManager.swift
└── Views/
    ├── Theme.swift                    # pastel "kitty diary" design system
    ├── NonStreamView.swift            # home
    ├── StreamView.swift               # active session (caption-as-hero)
    ├── SettingsView.swift             # settings
    ├── SessionDebuggerView.swift      # live feed + calibration
    ├── ReferencesView.swift           # manage caption rules
    └── CaptureCaptionSheet.swift      # capture-then-label flow
overlays/                              # transparent caption PNGs for demos

🎬 Demo overlays

overlays/ contains transparent PNG caption cards (styled like the HUD) for compositing over glasses footage, plus preview.html to view them.

📜 License & attribution

Built on top of Meta's Wearables Device Access Toolkit sample code. Use of the toolkit is governed by the Meta Wearables Developer Terms — see LICENSE. This project is a personal experiment and is not affiliated with or endorsed by Meta.

---

Made with 🩷 for Chocobo & Rufus.