Skip to content

feature: let the agent click, type and fill forms on web pages - #62

Closed
asasemahmed wants to merge 1 commit into
CopilotKit:mainfrom
asasemahmed:feature/browser-actions
Closed

asasemahmed wants to merge 1 commit into
CopilotKit:mainfrom
asasemahmed:feature/browser-actions

Conversation

@asasemahmed

Copy link
Copy Markdown
Contributor

Implements #61.

What changed

The agent can now click, type, select, check, press keys and scroll on the page open in the chat browser.

  • New tools: browser_elements, browser_act, save_browser_downloads.
  • Password, payment and one-time-code fields are never typed into, and their values are never shown.
  • Buy, pay, send, submit, delete, book and sign-up buttons need the person's OK in chat first.
  • Each step shows as a browser card with Take control.
  • The chat step limit goes from 6 to 20, so multi-step tasks can finish.

Verification

  • pnpm test:browser: the new real-Chromium test fills a form, blocks the submit until confirmed, and refuses a password field.
  • pnpm typecheck and the worker typecheck pass.
  • pnpm test passes, except the Windows-only Docker subprocess test.
  • Tried in the web app: the agent filled a form and submitted it only after I confirmed.

@davidmckayv

Copy link
Copy Markdown
Contributor

Closing. This breaks two written OpenMuse rules.

CONTRIBUTING: page text "cannot grant tool permissions or approve a write", and writes stay "behind persisted, versioned action reviews". The system prompt on main says "Approvals happen in the native app, never through chat tool arguments." This PR makes confirmedByUser, a chat tool argument the model sets itself, the only approval for clicks and submits that buy, send, delete or sign up. The server never verifies it, and which controls count as risky is decided by a regex over labels the page controls. It also raises the chat step limit from 6 to 20 for every conversation, which the description does not mention.

Agent-operated websites are a ROADMAP item that needs capability boundaries and end-to-end evidence first, and CONTRIBUTING requires agreed scope in an issue before substantial changes. #61 was opened the same day with no discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants