Repository navigation
feat(using-apps-like-a-user): dogfood web, Electron, and native macOS apps through a no-false-pass guard - #53
Merged
Merged
Conversation
A plain pytest ran 15 live tests that launched apps and changed the frontmost app; concurrent runs collided on the same fixture. They now skip unless LIKEUSER_FIXTURE_TESTS=1, like the fixture suite. Cua's hotkey tool ignored chords inside a save sheet, so hotkey now goes through press_key with modifiers. orca's type with a ref clicked the field without focusing it; it now double-clicks and collapses the selection.
…isible box The grader examined can_pass goals only when they were reported fail, so a report marking every goal blocked passed; blocked and partial now need the same explaining finding a fail does. The web visibility check rejected a partly visible element whose unclipped center was off screen, while the spec counts only a box that misses the viewport as offscreen; it now probes the center of the viewport-clipped box.
…d text redacted Cua renews element tokens on every snapshot, so the value and focus signals looked up the acted-on element by a token that no longer existed and reported a change that did not happen. The element is now found again by index and name, and a missing element is not a change. A password fill counts as a login attempt only after a submit-like action. Typed text stays redactable in later verbs through a 0600 file outside the run folder that end deletes, and secret fills save no screenshot.
…safely A begin that failed after capturing the baseline left the clipboard copy and any web session behind and marked the run ended, so end could not clean up; abort now closes the session and deletes the copy. end sends Escape to a leftover sheet only while the lock record still names the run. App quits pass the name to a fixed JXA script as an argument instead of interpolating it into AppleScript. The Cua pin is exact, and the Cua cache directory must be a 0700 directory owned by the user.
A dead-button goal marked blocked with no summary, evidence, or finding passed the grader, so the bar did not prove the agent reported the bug. A cannot_pass goal now needs a summary plus evidence, or a linked finding that is not one of the automatic console, network, or injection kinds.
…, fail closed on lost credentials A ref-less type compared no value before and after, so --expect value could never be satisfied; it now targets the focused element and finds it again after the act. A pending login is tracked by the password field's element identity, so a field with no accessible name still counts. A run begun with credentials that later cannot reload them exits 5 instead of continuing without redaction; end still closes the run but takes no final screenshot.
…run files private An Electron launch through a wrapper fell back to the generic com.github.Electron bundle id and could bind native dialogs to an unrelated Electron app; the native drivers now bind to the pid that owns the verified CDP listener. A Cua memo that cannot be saved fails the observation, and an act without one needs the foreground. Run folders are 0700 and their files 0600 whatever the umask, the state .gitignore always ends with *, and goal evidence includes the screenshots of NO_EFFECT acts and gate stops.
…ng evidence A blocked can_pass goal passed with any linked finding, so a run could block every working goal; it now needs a scenario accepted explanation, including a drift stop on desktop surfaces. A planted-bug goal must cite evidence files that exist and name what it found in scenario-specific words, so a summary unrelated to the bug no longer counts as surfacing it.
…ds, gate keys on the target name A pre-delivery refusal refreshed the drift baseline, so the foreground retry could not see a user takeover. A rerendered web login form gave the password field a new ref and the failed attempt went uncounted; web fields are now matched by role, name, and position. A third-party press or hotkey with a harmless --ref is gated on --target-name as well.
…ca captures The state root was chmodded through a project-controlled symlink; every state directory is now opened without following links and checked for ownership before use. A begin whose cleanup failed threw the errors away and ended the run; it now reports them and leaves the run for a cleanup-only end that retries what failed. orca's temporary screenshots are deleted once copied into the run folder.
…ssure test 1 to the four-rung ladder
…tive errors, withhold screenshots on the password only, strip sensitive launch env, render app text literally, let end retry its reports
…a-user # Conflicts: # BUGS.md # DEFERRED.md # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds
using-apps-like-a-user, a skill for dogfooding a web, Electron, or native macOS app through its UI the way a person would. It acts only on what's on screen and reports only what it can show.The agent never drives the app directly. Every action goes through
scripts/likeuser.py, a guard CLI that sits in front of playwright-cli (web and Electron renderers), Cua (native, background-first), andorca computer(native fallback, foreground). The guard decides the things a model shouldn't be trusted to decide:OBSERVED,NO_EFFECT, orUNSETTLED, so a dead button can't pass.DRIFT.GATE_REQUIREDunless the act says the user asked for it or approved it.endrestores focus, releases the desktop lock, and quits what the run launched.It also keeps credentials out of everything it writes. Credential values are redacted from all output. A screenshot is withheld after a
--secretact or while the password is readable in the tree. An Electron app is launched with sensitive-looking variables stripped from its environment, and a repeatablebegin --launch-env NAMEpasses a named one back through.The spec lives in
docs/quirk/specs/2026-10-11-using-apps-like-a-user/(logic.mdandtech.md).What's in the PR
skills/using-apps-like-a-user/: SKILL.md, references (safety.md,surfaces.md), the guard scripts, and a pressure test.evals/: web, Electron, and native scenarios for agent runs, a grader (grade.py) that checks report claims against fixture state, and bad reports that prove each grader rule rejects a false pass.tests/fixtures/using-apps-like-a-user/: a web fixture server, an Electron fixture app, a native Swift fixture app, recorded driver goldens, and fake drivers.tests/test_using_apps_like_a_user_*.py: unit and CLI tests. The end-to-end desktop tests are opt-in withLIKEUSER_FIXTURE_TESTS=1, because they need macOS, the drivers, and Accessibility and Screen Recording grants. CI skips them.DEFER-19toDEFER-24for v2 ideas, andBUG-35toBUG-37for three findings from the last review that this PR doesn't fix.How it was verified
pytest -qpasses. CI runs the same command.Known gaps
--launch-envdoesn't check against the defaultLIKEUSER_*credentials when no explicit credential source is given.