Skip to content

About

macOS skill and CoreGraphics controller for reliable iPhone Mirroring control.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

iPhone Control for Codex

iphone-control is a personal Codex skill for controlling a paired iPhone through Apple's iPhone Mirroring app on macOS. It provides a short, state-aware operating procedure plus a small Swift controller for the gestures that are unreliable through ordinary UI automation: press-hold drags, waypoint routing, folder drops, and hover-before-release.

It is intentionally a low-level control foundation. It does not decide how to organize apps, run unattended, bypass Apple security, or authorize protected device actions.

What it does

  • Locates the visible iPhone Mirroring window dynamically by the com.apple.ScreenContinuity process. All action coordinates are local to that window, so the controller does not rely on hard-coded desktop coordinates.
  • Supports read-only status checks, taps, right-clicks, directional swipes, press-hold drags with optional waypoints, folder drops, and Home/App Switcher/Spotlight shortcuts.
  • Uses a fast execution contract: inspect once, plan a small batch, perform deterministic actions, then verify only meaningful checkpoints.
  • Stops and requires fresh state after device disconnects, window changes, page changes, folder/layout reflows, or user interruption.
  • Uses --dry-run and a fixture window for safe parsing and geometry checks without touching a connected phone.

Requirements

  • A Mac that supports Apple's iPhone Mirroring feature, with a paired, connected iPhone.
  • The iPhone Mirroring app open and its phone window visible.
  • Swift toolchain available at /usr/bin/swiftc (included with Xcode or Xcode Command Line Tools).
  • Accessibility permission for the terminal/host process used to run the controller. This is required only for live input; status and dry runs do not send events.
  • Codex with the Computer Use capability available for screenshots and ordinary UI interactions. The controller complements Computer Use; it does not replace its confirmation rules.

Install in Codex

Clone the repository and copy the installable skill directory into your personal skills folder:

git clone https://github.com/CHARLESMORGANSOFTWARE/iphone-control.git
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills"
ditto iphone-control/iphone-control "${CODEX_HOME:-$HOME/.codex}/skills/iphone-control"

Restart Codex or start a new task so it discovers the new skill. The skill triggers for requests involving a paired or mirrored iPhone, iPhone Mirroring, Home Screen navigation, long-press, swipe, drag, or folder moves.

To update later:

cd iphone-control
git pull --ff-only
ditto iphone-control "${CODEX_HOME:-$HOME/.codex}/skills/iphone-control"

Give Accessibility permission

Before a live tap or drag, grant Accessibility permission to the application that launches scripts/iphone-control:

  1. Open System Settings > Privacy & Security > Accessibility.
  2. Add and enable the terminal application or host process that runs Codex.
  3. Re-run scripts/iphone-control status to confirm the Mirroring window is found.

The controller fails with an explicit message instead of trying to send input when permission is absent.

How it works

The installable package is iphone-control/:

iphone-control/
├── SKILL.md                     # Codex workflow, safety boundaries, and triggers
├── agents/openai.yaml           # Codex UI metadata
├── scripts/iphone-control       # Cache-aware shell entrypoint
├── scripts/iphone_control.swift # Window detection and CoreGraphics events
├── references/gesture-recipes.md
├── references/recovery.md
└── tests/test_dry_run.sh

The shell entrypoint compiles and caches the Swift executable when the source is newer than the cached binary. The controller enumerates on-screen CoreGraphics windows, keeps phone-like windows belonging to com.apple.ScreenContinuity, and rejects a missing or ambiguous candidate. It converts every requested local coordinate into a global coordinate only after that lookup.

For a drag, it posts a mouse-down, waits for a named timing profile, moves through zero or more waypoints, and posts mouse-up at the final target. folder-drop routes to the folder, waits for it to open, moves into the chosen slot, and releases. Live event emission checks Accessibility trust first.

Controller commands

Run commands from the installed skill's scripts/ directory, or call them through their absolute path.

# Read-only live window discovery
scripts/iphone-control status

# Safe simulated geometry: no device events
scripts/iphone-control --dry-run --fixture-window 100,80,320,640 tap 10 20

# Ordinary gestures; coordinates are local to the Mirroring window
scripts/iphone-control tap 140 260
scripts/iphone-control right-click 140 260
scripts/iphone-control swipe left
scripts/iphone-control shortcut home

# A true press-hold drag that avoids an intermediate target
scripts/iphone-control --timing deliberate drag \
  --from 120,300 --via 190,300 --to 250,360

# Hover over a folder, then release into a confirmed empty slot
scripts/iphone-control folder-drop \
  --from 120,300 --folder 250,360 --slot 285,405

Available timing profiles are quick, standard (the default), and deliberate. Use deliberate if the Mirroring app interprets a drag as a page swipe.

Safe operating model

  1. Capture one baseline screenshot and run status.
  2. State the expected start state, small action batch, verification checkpoint, and invalidators.
  3. Use Computer Use for screenshots, accessibility actions, text, ordinary taps, and keyboard work. Escalate to the direct controller only for real mouse-down gestures, routes, and folder hover behavior.
  4. Verify after a page transition, folder opening, layout mutation, external interruption, or batch completion.
  5. Discard every coordinate and re-plan if the phone disconnects, its window changes, the layout reflows, a folder opens unexpectedly, or the user says stop/pause or moves the pointer.

The skill does not authorize sensitive actions. Existing Codex confirmation requirements still apply to deletion, permissions, communication, financial, medical, account, and other protected actions. A request to move an icon authorizes only that requested move.

Test without an iPhone

bash iphone-control/tests/test_dry_run.sh

The dry-run suite verifies local-to-global coordinate conversion, waypoint drag stages, malformed-input rejection, status output, and shortcut stages. It never interacts with a real iPhone.

For a Swift compile check:

swiftc -typecheck iphone-control/scripts/iphone_control.swift

Troubleshooting

Symptom Action
iPhone Mirroring window not found Open iPhone Mirroring, reconnect/lock the phone as needed, then run status again.
Accessibility error Enable Accessibility for the terminal or Codex host in System Settings, then retry from fresh state.
Drag becomes a page swipe Release safely, take a new baseline, confirm editing mode, and retry with --timing deliberate.
Wrong folder opens or icons reflow Stop, close editing if needed, take a new screenshot, and discard all old coordinates.
User interrupts Stop immediately. Do not resume the earlier action plan.

Development

The implementation plan is in docs/superpowers/plans/2026-07-13-iphone-control.md. The repository contains no live-device tests; exercise live gestures only with explicit authorization and on reversible navigation where possible.

License

Released under the MIT License.

About

macOS skill and CoreGraphics controller for reliable iPhone Mirroring control.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages