Skip to content

E0 · App-driving + screenshot recipe (#180) - #190

Merged
hellno merged 1 commit into
mainfrom
hellno/app-driving-recipe
Jul 3, 2026
Merged

E0 · App-driving + screenshot recipe (#180)#190
hellno merged 1 commit into
mainfrom
hellno/app-driving-recipe

Conversation

@hellno

@hellno hellno commented Jul 3, 2026

Copy link
Copy Markdown
Owner

Commits a documented, reusable way to drive the running GPUI wallet and capture per-window screenshots, so every later v4 GUI issue (E2–E7) can produce before/after evidence. Docs + scripts only — no crate touched.

Part of the v4 build epic #179 (E0 is the prerequisite for every GUI issue's evidence). Closes #180.

What's delivered

  • scripts/deckard-drive.sh (committed, executable, not gitignored) — one entry point:
    • win — Swift CGWindowListCopyWindowInfo finder → the largest layer-0 deckard window's id + bounds.
    • shot <name>screencapture -x -o -l<id>, occlusion-proof (region capture reads back whatever app is composited on top; -l<id> grabs the window's own bitmap).
    • click/clickpt/type/key/unlock/raise/depscliclick + osascript input at window-relative coords. raise uses NSRunningApplication.activate and self-reports isActive=<bool>; unlock warns loudly if the app isn't active.
  • docs/dev/driving-the-app.md — the full macOS recipe (just demo funded Sepolia fork, passphrase deckard-demo; just run/just qa), the three primitives spelled out so you can rebuild them without the script, and the honest caveats below. Cross-links docs/dev/headless-gui-screenshots.md for the Linux/CI path.

Honest caveats (the point of E0 — these save hours)

  • Synthetic input only lands when the app is the OS active app. raise reports isActive; false means nothing you type/click will land.
  • Capture works even when input doesn't — a clean screencapture -l shot never proves your click registered. Verify from the next shot.
  • The ⌘↵/click confirm, the ⌘K query, and Activity j/k/x/Esc are hand-rolled on_key_down and don't fire synthetically; registered actions (⌘,, ⌘⇧D) do.
  • So: capture SCREENS via clicks; drive real BROADCASTS through the headless agent (just demo-deposit + just demo-agent); verify LOGIC via tests, not screenshots.

Proof (followed the recipe)

just demo (funded fork via sepolia.drpc.org, chain 11155111) → the Swift finder returned 36105 316 181 880 620 → window-scoped capture of the demo Unlock screen:

Deckard demo Unlock screen — captured by scripts/deckard-drive.sh via the Swift window finder + screencapture -l

⚠️ On the unlock action: in this agent-driven Conductor session the running app could not be made the OS active app (NSRunningApplication.activate() returns true but isActive stays false on macOS 14+; the GPUI app exposes no AXWindow), so the synthetic passphrase+click did not land — exactly the limitation now documented as the #1 caveat. The launch + finder + occlusion-proof capture pipeline is fully validated (screenshot above). To capture the unlocked home screen, run just demo from a foreground Terminal (where the app can take focus) then scripts/deckard-drive.sh unlock — the script's unlock warns if focus is missing.

Definition of Done (#179 Implementer Contract)

  • cargo fmt --all --check clean ✅
  • just check green (clippy -D warnings, both feature configs) ✅
  • cargo test --workspace green (0 failed) ✅
  • No new/changed deps (Cargo.toml/Cargo.lock untouched) ✅
  • Do-not-touch honored: no crates/ file changed ✅
  • ⌘K coverage: n/a — E0 adds no user-facing app action (tooling only).

Commit a reusable, documented way to drive the running GPUI wallet and
capture per-window screenshots, so every v4 GUI issue can produce
before/after evidence.

- scripts/deckard-drive.sh: window finder (Swift CGWindowListCopyWindowInfo
  -> largest layer-0 "deckard" window), screencapture -l<id> (occlusion-proof
  window capture), cliclick/osascript input, and an unlock helper. Self-
  diagnoses focus via NSRunningApplication.activate -> isActive.
- docs/dev/driving-the-app.md: the full macOS recipe (just demo funded Sepolia
  fork, passphrase deckard-demo) + honest caveats -- synthetic input only lands
  when the app is the active/key app; the Cmd-Return confirm, the Cmd-K query,
  and Activity j/k/x/Esc are hand-rolled and don't fire synthetically; drive
  real broadcasts via the headless agent and verify logic with tests. Cross-
  links the Linux/CI recipe.
- docs/screenshots/driving-the-app-unlock.png: recipe-produced capture of the
  demo Unlock screen (launch + Swift finder + window-scoped capture).

Docs/scripts only -- no crate touched.

Refs #179, #180.
@hellno
hellno merged commit 977c90a into main Jul 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

E0 · App-driving + screenshot recipe

1 participant