when an agent is driving a sim headless it's basically flying blind on device state. the three things we kept needing today:
- is the device locked
- is the screen actually awake (locked + asleep screenshots are just black)
- what app is frontmost
none of that is queryable right now so we ended up hacking around it:
- lock state via
xcrun simctl spawn booted notifyutil -g com.apple.springboard.lockstate
- screen awake by screenshotting and checking the dark pixel fraction with PIL (yeah lol)
- frontmost by screenshotting and eyeballing it
which works but it's slow and gross, and every agent that uses serve-sim is going to reinvent the same hacks.
proposal: a serve-sim state [-d udid] --json that returns something like
{ "locked": true, "screenAwake": false, "frontmostApp": "ceo.ryan.hark" }from poking at the source it looks like some of this is already in reach — axFrontmostAsync exists in native.ts, and lock state is a notifyutil read away. totally open to a different shape if you have opinions, and happy to take a swing at the pr myself.
context: we were doing automated lock screen captures of live activities (openurl → lock → screenshot loops) and state checks were the difference between the loop being reliable and it being a retry mess.
when an agent is driving a sim headless it's basically flying blind on device state. the three things we kept needing today:
none of that is queryable right now so we ended up hacking around it:
xcrun simctl spawn booted notifyutil -g com.apple.springboard.lockstatewhich works but it's slow and gross, and every agent that uses serve-sim is going to reinvent the same hacks.
proposal: a
serve-sim state [-d udid] --jsonthat returns something like{ "locked": true, "screenAwake": false, "frontmostApp": "ceo.ryan.hark" }from poking at the source it looks like some of this is already in reach —
axFrontmostAsyncexists in native.ts, and lock state is a notifyutil read away. totally open to a different shape if you have opinions, and happy to take a swing at the pr myself.context: we were doing automated lock screen captures of live activities (openurl → lock → screenshot loops) and state checks were the difference between the loop being reliable and it being a retry mess.