flipoff

Sings the message out loud the instant it sees the gesture, then sends 𝗙𝗨𝗖𝗞 𝗬𝗢𝗨 — bold, uppercase — to whichever iMessage conversation you have open.

One take, filmed on a phone. Sharper copy in docs/demo.mp4.

git clone https://github.com/angusbuilds/flipoff && cd flipoff

./flipoff                        # run it
./flipoff --tune                 # camera window + live score, sends nothing
./flipoff --to "Jane Doe"        # pin one person instead
./flipoff -m "hey" --effect none # different message, no animation
./test_gesture.py                # the suite

Needs macOS and uv (brew install uv). flipoff.py declares mediapipe, opencv and numpy inline as a PEP 723 script, so uv builds a throwaway environment on first run and your own Python install is left alone. That run also fetches hand_landmarker.task (7.8 MB) beside the script.

How it decides

MediaPipe Tasks HandLandmarker scored off hand_world_landmarks — metric 3D, not the normalized 2D ones. That matters because the natural way to flip off a webcam points the finger straight down the camera axis, where a 2D projection foreshortens it into something that looks curled.

Each finger is measured by extension: straight-line knuckle-to-tip divided by the length of the bone path. 1.0 is straight, ~0.35 is a clenched fist. It needs no reference frame, so hand orientation and metacarpal geometry can't skew it.

Every number below came off a real webcam, not a model:

finger	in a real flip-off
middle	0.93 – 0.97
ring / pinky	0.59 – 0.74
index	0.33 – 0.59

The score is min() over four constraints — middle extended, and index / ring / pinky each folded — so one straight finger sinks it. Each is a smooth ramp, not a cutoff, because real flip-offs are sloppy.

Three deliberate choices:

• The fold bands are loose. Index and pinky ramp from 0.88 down to 0.76; the ring finger gets 0.90 down to 0.78. Tight bands (0.80 down to 0.70) read better on paper and threw away real gestures — one live flip-off scored 0.33 against them. Loosening them fixed that and let part of the hand-to-face family back in, which is the cost written down in Known limitations.
• The ring finger gets slack. Its tendons share a sheath with the middle finger's, so many people physically can't fold it while the middle stands up. Safe only because the index and pinky still have to fold properly.
• The thumb is ignored. People tuck it, splay it, or wrap it over the index, and none of that changes what the gesture means.

A hand running off the frame edge, or one the tracker reports at low confidence, is dropped before it gets a vote — its world coordinates are extrapolation.

Detector turns per-frame scores into one trigger: EMA smoothing to ride out dropped frames, a release threshold below the trigger threshold so the hold timer doesn't restart on wobble, a 0.2s hold, and a re-arm requirement so holding the pose only fires once. About a quarter second, gesture to sent.

Measured

Real recorded frames, replayed through the classifier. This is the result the thresholds were actually calibrated against:

real flip-off:       10/10 fire
real ordinary hand:   0/14 fire

Under synthetic landmark noise, holding a pose 2s at 30fps (MediaPipe's error is ~2–3mm on a well-lit hand, so 4mm is pessimistic):

pose	fires
textbook flip-off @ 3mm	100%	wanted
hyperextended middle @ 3mm	100%	wanted
open palm @ 4mm	0%	wanted
nose scratch @ 4mm	35%	false positive
push glasses up @ 4mm	100%	false positive

Those last two are real failures, and ./test_gesture.py reports them as failures rather than quietly passing. They are what the loose fold bands cost. The bands stay loose because the recorded-frame numbers above outrank synthetic ones: these poses come from a hand model that has already been wrong three times, and the real negatives that would settle them aren't recorded yet. Until they are, assume anything that parks your middle finger near your face can send a text.

Cost of leaving it on

It watches the camera continuously, so the bill is CPU rather than correctness. Measured with CPU-time deltas on an 18-core M5 Max:

	one core
hand in frame, 30 fps	~20%
nothing happening, 2 fps	~13%*
memory, steady	363 MB

* That idle figure is honest but pessimistic — hands kept drifting into frame while measuring, so some of it is active work. Before the throttle existed it ran the model at 30 fps regardless, which is a full core spent watching an empty room.

Idle drops to a slow poll after three handless seconds and snaps back to full speed the moment a hand appears, so the throttle costs at most one idle interval before the gesture timer even starts. The log self-truncates at 4 MB, a camera that stops returning frames backs off instead of spinning, and a malformed frame is reported and skipped rather than ending a week-long run.

Known limitations

The first two are printed by the test suite on every run rather than buried here:

• A hand resting under your chin with three fingers folded and the middle extended is the gesture, geometrically. Sit like that and it fires.
• A middle finger straight but folded 90° at the knuckle reads as extended. Real gestures run 68–82° at the knuckle and ordinary hands 48–127° — the ranges overlap, so every threshold that rejects this also rejects real gestures.
• A lazy flip-off counts. Middle finger out with the others only half-curled scores 1.00 on the current bands, which is the same reason pushing your glasses up can send a text. Raise --threshold and you start losing real gestures instead; the two failures are the same knob.
• Nothing here is a face detector. It scores one hand in isolation, so it cannot tell a gesture aimed at the camera from the identical shape aimed at your own nose. Fixing that needs the hand's position relative to your face, which needs a second model.

Sending

By default it texts whichever conversation you have selected. The Messages window title is the contact's name, and that name resolves to a handle, so the message is addressed to the person directly — no dependence on the message box holding keyboard focus.

	how	needs
default	selected conversation → name → handle → send ... to participant	nothing
--to "Name"	same, but pinned to one person whatever is open	nothing
fallback	pastes into the front window and hits return	Accessibility

The fallback only runs when a title matches no single participant, which mostly means group chats. It pastes rather than types, because keystroke cannot produce astral-plane characters — the bold capitals live at U+1D5D4 and up — and typing them emits a run of junk instead of the message.

Fireworks ride on iMessage's trigger-phrase detection, which runs on the recipient's device: the message carries Happy New Year, and their phone plays the animation. macOS has no send-with-effect picker — composing effects is iOS-only — so this is the only route from a Mac. --effect none drops it.

Permissions

System Settings → Privacy & Security, granted to whichever terminal launches it:

• Camera — see the gesture
• Automation — read which conversation is open
• Accessibility — only for the default paste-and-send path; --to skips it

Accessibility is checked at startup and exits with instructions if missing. A denied Automation prompt is reported the first time it bites, rather than silently turning the whole thing into a no-op that looks like it's working.

Menu bar

swiftc -O menubar.swift -o flipoff-menu && ./flipoff-menu

A middle finger in the menu bar: dimmed when off, solid when armed, click to toggle. Native AppKit, so it adds no dependencies — swiftc ships with the Command Line Tools. Flags live in an args file beside it, one per line.

Launching the detector from here is also the cleanest fix for permissions. TCC attributes Camera and Accessibility to the responsible process, and a menu-bar app is a real GUI app that can show the prompt — which is exactly what the LaunchAgent below cannot do.

Running in the background

nohup ./flipoff >> flipoff.log 2>&1 &

Survives closing the terminal; dies on logout. com.angus.flipoff.plist is a LaunchAgent for login persistence — its paths are placeholders, since launchd resolves nothing relative — but be warned: TCC grants attach to the responsible process, and a launchd job is a different identity with no Camera or Accessibility of its own. Loading it produces a crash loop. Getting it working needs those permissions granted to the launchd binary, or an .app wrapper to give TCC a stable identity. The menu-bar app above is the easier route.

Options

-m, --message TEXT     what to send (default: "fuck you")
    --to NAME          pin a recipient; skips Accessibility entirely
    --effect NAME      fireworks | confetti | balloons | lasers | none
    --bold/--no-bold   bold uppercase (default: on)
    --say/--no-say     sing it out loud (default: on)
    --voice NAME       Cellos | Good News | Bells | Organ | Bad News | Boing
    --hand right|left|any
    --threshold 0..1   confidence to fire (default: 0.5)
    --hold SECONDS     how long to hold it (default: 0.2)
    --cooldown SECONDS between sends (default: 8)
    --tune             camera window with a live score bar; never sends
    --test             log detections only; never sends
    --record FILE      append every scored frame as JSON

Tuning it to your hand

./flipoff --tune --record mine.jsonl

Watch the bar; green past the tick means it would fire. The recording is real landmarks from your hand, which is what fixtures.json was built from — and what every threshold in here was calibrated against after a synthetic model got them badly wrong.

./docs/chart.py and ./docs/tuneshot.py redraw both figures from fixtures.json, reading the bands out of flipoff.py, so add your own frames and the pictures update with them. The data figure used to be a hand-built image, which is how it spent a while advertising thresholds the code had already moved off.