Browser hand — agents that can browse the web live

[Dhwanil25]

Overview

Give agents the ability to browse the web in real time during a universe run — fetch pages, extract content, follow links — so research tasks work on live data instead of training knowledge.

How it should work

1. A browser tool is available to agents: { action: 'fetch', url: '...' }
2. The engine fetches the URL server-side (via the Vite plugin middleware), strips HTML to clean text, and returns it to the agent
3. The agent's canvas node shows a 🌐 indicator when actively browsing
4. Visited URLs are shown in the agent card's output

Implementation

Engine side (vite-plugin-agentis.ts):

• Add POST /agentis/browse endpoint — fetches URL, returns { title, text, url }
• Use Node's https module + basic HTML stripping (no headless browser needed for most sites)

Engine client (multiAgentEngine.ts):

• Add browse to the worker system prompt as an available tool
• Parse agent output for <browse url="..." /> tags and resolve them mid-stream

Related

• src/components/pages/HandsPage.tsx — the Hands page is designed for this capability
• Mentioned in README roadmap

[Dhwanil25]

Enhancement: Full Browser Automation via open source tooling

Beyond simple URL fetching, agents could get full browser control using existing open source projects — clicking, scrolling, filling forms, logging in, reading dynamic JS-rendered pages.

Option A — Web Search API (lightweight, recommended first step)

Integrate a search API (Tavily, Brave Search, or SerpAPI) as a tool agents can call:

• Agent emits <search query="..." /> mid-output
• Engine calls the search API, injects top results back into the agent's context
• No browser needed — fast and reliable for research tasks

Option B — Full browser control via browser-use

browser-use is an open source Python library that lets LLMs control a real Chromium browser via Playwright:

• Agents can navigate URLs, click elements, fill forms, extract content from JS-heavy pages
• Useful for tasks like "check the live price on X", "fill out this form", "log in and pull my dashboard data"
• Would run as a sidecar process alongside the Vite dev server

Option C — Playwright MCP (Model Context Protocol)

Use the Playwright MCP server to expose browser actions as MCP tools — compatible with any agent that supports tool use.

Suggested implementation path

1. Start with Option A (search API) — immediate value, zero infrastructure
2. Add Option B behind a feature flag for power users who want full browser control
3. Surface live browsing activity in the Universe 3D view (agent node pulses 🌐 while browsing)

Related

• The existing HandsPage.tsx is the natural home for configuring browser automation settings (Playwright endpoint, search API key, allowed domains)

[m13v]

the HTML stripping approach will hit a wall fast. most modern sites render content via JS, so a raw fetch + strip gives you empty divs and loading spinners. even server-rendered pages have critical content behind lazy load or intersection observers. the server-side fetch approach also can't handle auth flows, pagination, or any interactive elements. at minimum you'd want a lightweight headless browser with JS execution. the canvas node indicator is a nice UX touch though. what percentage of your target URLs are actually static HTML vs JS-rendered SPAs?

[Dhwanil25]

@m13v yeah fair point, most of what agents would actually browse here are JS-heavy sites, dashboards, GitHub, Notion, etc. Raw fetch probably only works for like 20-30% of real cases.

The real plan is to add proper headless browser support via browser-use or Playwright MCP. I saw you were building Chrome automation pipes in ScreenPipe, same problem space. Which approach would you go with here, and would you be open to contributing?