watchme

Turn one narrated screen recording into a draft automation spec a developer can build an agent from.

The bottleneck in automating office routines isn't capture — it's getting the why (the branches, exceptions, judgment calls) out of the person who does the work. So the worker performs the routine once while narrating their decisions out loud. The narration is the product; capture is plumbing.

watchme is the plumbing. It takes the recording and produces a folder that local Claude Code reads and synthesizes into the spec. The tool itself makes no API calls — that keeps v0 free of API keys and lets the synthesis prompt be tuned conversationally.

v0 scope. Personal experiment. Capture uses macOS's own recorder — watchme record drives it for you, or record manually (QuickTime works too) and pass the file. No recorder daemon, no keylogger, no Accessibility tree, no redaction layer. Those earn their place only after synthesis is proven to work. See routine-capture-v0-brief.md.

Install

brew install ffmpeg whisper-cpp          # ffmpeg + local transcription (Metal-accelerated)

# a whisper.cpp model. small (multilingual) is the safe default — it auto-detects the
# narration language. Use small.en only if you know the narration is English.
mkdir -p ~/.cache/whisper-cpp
curl -L -o ~/.cache/whisper-cpp/ggml-small.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin

watchme auto-discovers models in ~/.cache/whisper-cpp/ and picks a multilingual one unless you pass --language en.

Then either run it directly (no Python deps):

python3 watchme.py process ~/Desktop/my-recording.mov

…or install the watchme command:

pipx install .        # or: uv tool install .
watchme process ~/Desktop/my-recording.mov

openai-whisper works too — if whisper is on your PATH and no whisper.cpp is found, watchme uses it automatically.

Guided setup

Before the first recording, run:

watchme setup

It checks ffmpeg/ffprobe and local Whisper/model first, then walks permissions in strict sequence: Screen & System Audio Recording, restart, and real read access to the folder where macOS saves recordings. It emits a status before macOS can show consent UI and never stacks the Screen Recording and Desktop-folder prompts. Use --no-open for a diagnostic-only check, or --open screen|microphone|files|full-disk to open a specific pane.

After automated checks pass, run a short watchme record --no-process smoke test. Microphone access is requested by macOS the first time you choose a mic in the capture toolbar; watchme setup --open microphone opens the pane for review. If Files & Folders is enabled but an agent host still cannot read Desktop, add its command helper under Full Disk Access too, restart the host, and rerun setup.

Record (skip QuickTime)

watchme record        # record → transcribe → timeline, in one go

A watchme popup explains the flow and then hands over to Apple's native capture UI (the ⌘⇧5 toolbar): pick full screen or an area, choose your mic under Options, click Record, narrate, and stop with the ⏹ button in the menu bar. The toolbar hands the file to the system Screenshot service, so watchme watches the service's save location (Options ▸ Save to, Desktop by default) and adopts the newest recording automatically; then the pipeline below runs and the .mov is kept in the run folder as recording.mov.

Why Apple's UI and not our own stop button: anything watchme draws during recording would be captured — macOS 15+ has no public API to exempt a window — so watchme's popup closes before capture starts. Apple's toolbar dismisses before recording begins too. The one trace left in a full-screen recording is the menu bar's tiny stop icon and your final click on it (the menu bar is ordinary screen content); recording a selected portion below the menu bar keeps even that out of frame.

The "Ready to record?" popup is a styled SwiftUI panel (frosted glass over an ambient backdrop) that walks you through the capture UI: numbered steps with screenshots showing which toolbar button to click (Record Entire Screen) and the Stop Recording button to end with. watchme compiles it on first use from assets/watchme-dialog.swift into ~/.cache/watchme/. It needs Apple's command-line Swift tools (swiftc, present with the Xcode Command Line Tools); if they're missing — or on failure, or off macOS — watchme falls back to a plain system dialog automatically. The images live in ~/.cache/watchme/ (dialog-bg.png, step-record.png, step-stop.png — bundled defaults; the backdrop was generated with the Higgsfield CLI); drop your own PNGs there to reskin or to match your own menu bar.

One-time setup is handled by watchme setup; watchme record also refuses to start if Screen Recording or access to the configured save folder is missing, so it cannot wait indefinitely for a recording it is not allowed to retrieve.

Options: --plain records the full screen with no native UI (stop with any key in the terminal — useful over SSH/scripts); --no-process just saves the .mov; --save PATH keeps the recording at PATH instead of inside the run folder. All process options (below) work on record too.

Use it from Claude Code (plugin)

watchme ships as a Claude Code plugin so you can run the whole loop from inside the repo you want to automate — capture → spec → implementation — without leaving the session.

Prerequisites: ffmpeg + a local whisper + a model — see Install above. The commands also check for these and tell you what's missing.

Install the plugin (recommended — from GitHub)

Run these /plugin commands inside Claude Code (needs access to the private wearevolt/watchme repo; auth via gh or an SSH key):

/plugin marketplace add wearevolt/watchme
/plugin install watchme@watchme

The commands are then available in every repo. Update later with /plugin update watchme@watchme; refresh the catalog with /plugin marketplace update watchme.

Alternatives (local checkout)

# quick, one session only (loads in place, always reflects your working copy):
claude --plugin-dir /path/to/watchme

# persistent, from a local path (copies the folder into the plugin cache):
#   /plugin marketplace add /path/to/watchme
#   /plugin install watchme@watchme

Use --plugin-dir while hacking on the plugin itself; use the GitHub install for daily use (a git source pulls only tracked files, leaving your .watchme/ runs behind).

Use it

In any repo, the simplest way is the single entry point:

/watchme:watchme                          # no args → record now; then spec; then offer to build
/watchme:watchme setup                    # guided prerequisites + macOS permissions
/watchme:watchme /path/to/recording.mov   # a recording → process it
/watchme:watchme .watchme/<run>/spec.md   # a spec → implement it

/watchme:watchme routes to the right step by what you pass (it's the first match when you type /watchme — Claude Code namespaces every plugin command, so there is no bare /watchme). The granular commands stay available if you prefer to drive each step:

/watchme:record                          # record right now, then process + spec
/watchme:setup                           # diagnose/setup before the first recording
/watchme:process /path/to/recording.mov  # process an existing recording
/watchme:implement                       # newest spec, or pass .watchme/<run>/spec.md

/watchme:setup checks dependencies and guides macOS permission grants. /watchme:record starts the native macOS recorder (stop with ⏹ in the menu bar); /watchme:process takes a file you already have. Either way the pipeline runs into a scratch .watchme/<date>-<slug>/ folder in the current repo, then Claude reads the timeline, views the frames, writes spec.md — and offers to build it. /watchme:implement resolves the spec's open questions with you and implements it in the current repo, following its conventions. Spec and implementation are separate on purpose: you review the spec before any code is written.

Requirements are the same as the CLI (ffmpeg + whisper + a model); the command checks them and tells you what's missing.

Use it from Codex (skill)

watchme also ships as a Codex CLI skill at codex/watchme/ — a self-contained skill directory (its own SKILL.md + a bundled copy of watchme.py and assets/), so it works with no separate CLI install. Codex uses the same Agent Skills format, so the same setup → record → spec → implement flow is available there.

Install it from this repo with Codex's built-in skill-installer (ask Codex to "install the watchme skill from wearevolt/watchme", or run its helper directly):

install-skill-from-github.py --repo wearevolt/watchme --path codex/watchme

That drops the skill into ~/.codex/skills/watchme/ (works for private repos via your git creds / GITHUB_TOKEN); it's available on Codex's next turn. You can also just copy codex/watchme/ into ~/.codex/skills/ (global) or .codex/skills/ (per-project, shared via git). Then tell Codex to set up WatchMe, record a routine, process a .mov, or implement a spec.md — it routes accordingly. Recording needs macOS + the same prerequisites. Frame-aware synthesis works — the current Codex CLI views the local frame images directly (verified on Codex 0.144.5), so the spec gets the same on-screen context as the Claude path; a Codex build without local-image support would degrade to transcript-only.

The Codex skill's watchme.py/assets/ are a copy — the repo root is canonical. After changing either, run scripts/sync-codex-skill.sh to refresh the copy before committing.

Usage (CLI)

watchme setup                   # check deps + guide macOS privacy permissions
  --no-open              # report only; do not open System Settings
  --open PANE            # screen, microphone, files, or full-disk

watchme record [options]        # record now (native macOS recorder), then process
  --plain                # no native UI: record full screen, stop with any key here
  --no-process           # just save the .mov, skip the pipeline
  --save PATH            # keep the .mov at PATH (default: run folder/recording.mov)
  # …plus all `process` options below

watchme process RECORDING [options]

  --runs-dir DIR         base dir holding per-run folders (default: runs)
  --name SLUG            explicit run-folder slug (skips claude auto-naming)
  --out DIR              exact output dir (overrides --runs-dir and auto-naming)
  --scene-threshold F    starting ffmpeg scene-change threshold (default 0.2, self-adapts)
  --min-frames N         min frames to aim for (default 14)
  --max-frames N         max frames to aim for (default 40)
  --frame-width N        max frame width in px (default 1600; Retina gets downscaled)
  --model PATH|NAME      ggml model path (whisper.cpp) or model name (openai-whisper)
  --whisper-bin PATH     override whisper binary
  --language CODE        narration language, or 'auto' (default: config value, else auto)
  --translate            translate non-English narration to English in the transcript
  --verbose              stream whisper's backend/model logs (hidden by default)

Language

The narration language is auto-detected up front (first ~45s) before transcription, so you normally don't set anything — watchme picks the right multilingual model and tells you what it found. You can also:

watchme detect RECORDING            # just report the detected language + confidence
watchme config                      # show current settings
watchme config language ru          # pin a language (accepts codes or names; validated)
watchme config language             # interactive set, with suggestions on bad input
watchme config languages russian    # search the supported languages

Set values are validated against the languages whisper accepts — an invalid value is rejected with "did you mean …?" suggestions rather than saved. Precedence: --language flag → pinned config → auto-detect.

Run folders

Each run gets its own folder named runs/<recording-date>-<slug>/. The slug is generated by the local claude CLI from the transcript (e.g. 2026-07-01-triage-disliked-llm-answers), so runs are self-describing and never clobber each other. If claude isn't installed or times out, watchme falls back to a slug built from the transcript's own words. Override with --name, or force an exact path with --out.

What it does

1. Transcribe (local) — extracts 16 kHz mono audio and runs whisper, keeping segment timestamps ({start, end, text}). Timestamps are what make alignment work.
2. Sample frames — extracts frames on scene change (not fixed-rate) via ffmpeg, adapting the threshold to land in the ~20–40 meaningful-frame band. Each frame keeps its timestamp in the filename (frame_0036.png).
3. Interleave — merges frames and narration into one chronological timeline.md, so what was on screen and what was said about it sit together in order. That alignment is what recovers intent instead of just actions.

Output

runs/2026-07-01-triage-disliked-llm-answers/
  transcript.json      # {start, end, text} narration segments
  frames/              # frame_0000.png, frame_0032.png, …  (scene-change screenshots)
  timeline.md          # interleaved narration + frame refs — what Claude Code reads
  synthesis-prompt.md  # the task you give Claude Code (trivially editable — tune this)
  spec.md              # written by Claude Code in the synthesis step (below)

Synthesize (the actual payoff)

cd runs/<the-run-folder>
claude          # then: "follow synthesis-prompt.md"

Claude Code reads timeline.md, views the frames, and writes spec.md — a SKILL.md-style scaffold with: Goal & trigger · Inputs · Systems touched (mapped to candidate MCPs/integrations) · Process (happy path) · Decision branches & exceptions (the high-value part) · Open questions (what to clarify on a re-record).

Success criterion

Run it once on one real narrated routine. If the spec captures the branches and exceptions correctly and a developer could start building from it, the core loop is validated. If it's mush, that's been learned for the cost of one recording.