Flutter Agent

The control plane your AI agent uses to drive a real Flutter app.

Flutter Agent runs a REST API on localhost:8000. AI coding agents (Claude Code, Cursor, Antigravity, your own) and ordinary HTTP clients use it to observe and control a Flutter app on an Android emulator or device — screenshots, UI tree inspection, taps / swipes / typing, hot reload, structured Dart logs, real heap + FPS + CPU profiles.

v1.0.0 — production-ready. ADB control, multi-attribute UI matchers, logcat + Dart-native log streaming, Dart VM Service (memory + FPS + CPU profiling), Flutter daemon protocol (real source-level hot reload), and a generated tool-definition file that lets any modern tool-calling agent framework drive every endpoint. See CHANGELOG.md.

---

Why this exists

AI coding agents can write Flutter code, but they're blind to what's actually rendering. Without a feedback loop, they iterate by guessing — "did my change work? did I break something? what state is the app in?"

Flutter Agent gives them eyes and hands:

[Claude Code / Cursor / Antigravity / your agent]
                │ HTTP / SSE
                ▼
         [Flutter Agent server]
                │
         ┌──────┴──────┐
         ▼             ▼
       ADB         Dart VM Service
         │             │
         ▼             ▼
   [Android emulator or device]
              │
        flutter run --machine
              │
        [your Flutter app]

The agent loop becomes:

1. Make a code change.
2. POST /hot_reload.
3. GET /screenshot to see what rendered.
4. GET /flutter_logs or GET /errors to see what happened.
5. Iterate.

Quick Start

Prerequisites: Python 3.10+, the Android SDK platform-tools (for adb), and a running emulator or USB-attached device.

pip install -r requirements.txt
uvicorn app:app --reload

Open http://localhost:8000/docs for the interactive OpenAPI explorer.

# Sanity check
curl http://localhost:8000/health

# Take a screenshot
curl http://localhost:8000/screenshot > screen.png

# Tap at (150, 250)
curl -X POST http://localhost:8000/tap \
  -H "Content-Type: application/json" \
  -d '{"x":150,"y":250}'

# Launch a Flutter app and auto-connect to its VM Service
curl -X POST http://localhost:8000/flutter_run/start \
  -H 'Content-Type: application/json' \
  -d '{"project_path":"/path/to/your/flutter_project"}'

# Tail Dart `print()` / `developer.log()` in real time
curl -N 'http://localhost:8000/flutter_logs/stream'

See QUICK_START.md for the 5-minute walkthrough, ARCHITECTURE.md for the system design, and API_REFERENCE.md for every endpoint.

Endpoints

Endpoint	Method	Purpose
/health	GET	Server + device status
/screenshot	GET	Raw PNG of the device screen
/app_state	GET	Screenshot (base64) + UI element list
/find_element	GET	Locate an element by text / content_desc / resource_id / class_name
/elements	GET	List elements with filters (clickable, enabled, text_contains, …)
/tap	POST	Tap at coordinates
/tap_element	POST	Find an element by query and tap it
/swipe	POST	Swipe gesture
/type_text	POST	Type into focused field
/key_event	POST	Send Android keycode (BACK, HOME, ENTER, …)
/wait_for_element	POST	Poll the UI tree until an element appears, or timeout
/flutter_run/start	POST	Launch flutter run --machine and auto-connect the VM Service
/flutter_run/stop	POST	Stop the running daemon
/flutter_run/status	GET	Running / appId / wsUri / uptime
/flutter_run/send_command	POST	Escape hatch — send any app.* / daemon.* RPC
/vm_service/connect	POST	Attach to a Dart VM Service WebSocket (manual flow)
/vm_service/disconnect	POST	Tear down the VM Service connection
/vm_service/status	GET	URL + connected isolates
/isolates	GET	List running Dart isolates
/memory	GET	Real heap usage via getMemoryUsage
/performance	GET	FPS / jank from the Dart VM Timeline
/cpu_profile	GET	Top CPU hotspots from getCpuSamples over a window
/hot_reload	POST	Source-level reload (via daemon) or reassemble fallback
/hot_restart	POST	Engine restart
/app_logs	GET	Recent logcat lines, with tag / level / regex filters
/clear_logs	POST	Clear the device's logcat buffer
/errors	GET	Parsed crashes (Flutter unhandled, AndroidRuntime fatals, ANRs)
/frame_warnings	GET	Choreographer skipped-frame and Davey signals
/logs/stream	GET (SSE)	Real-time logcat stream with optional filters
/flutter_logs	GET	Buffered Dart print / developer.log entries since a cursor
/flutter_logs/stream	GET (SSE)	Real-time stream of Dart log entries
/flutter_logs/clear	POST	Empty the in-memory log buffer
/app_info	GET	Package version + foreground state
/full_diagnostics	GET	Combined snapshot

UI matchers

find_element, tap_element, and wait_for_element all accept any combination of text, content_desc, resource_id, class_name, plus a match mode of exact (default), contains, or regex. content_desc is essential for Flutter — Flutter widgets expose their accessibility label there, not in text.

Driving a Flutter app

The recommended flow — Flutter Agent launches and owns flutter run --machine for you, one call end-to-end:

curl -X POST http://localhost:8000/flutter_run/start \
  -H 'Content-Type: application/json' \
  -d '{"project_path":"/path/to/your/flutter_project","device_id":"emulator-5554"}'

On success the response contains app_id, the live VM Service ws_uri, and vm_service_connected: true — meaning /memory, /performance, /hot_reload, and /hot_restart are all immediately wired up, and the reload endpoints route through the daemon's app.restart for real source-level reload rather than the widget-tree reassemble fallback.

Set FLUTTER_PROJECT_PATH to skip project_path in the body:

export FLUTTER_PROJECT_PATH=/path/to/your/flutter_project
curl -X POST http://localhost:8000/flutter_run/start -d '{}'

Manual VM Service flow

If you'd rather keep flutter run open in your own terminal:

flutter run -d emulator-5554
# A Dart VM Service on Pixel 9a is available at: http://127.0.0.1:64321/aBcDef=/

curl -X POST http://localhost:8000/vm_service/connect \
  -H 'Content-Type: application/json' \
  -d '{"url": "http://127.0.0.1:64321/aBcDef=/"}'

…or set VM_SERVICE_URL before starting the server.

In this mode /hot_reload falls back to ext.flutter.reassemble (widget-tree rebuild only — useful for setState / theme changes that have already been applied) and returns a hint field telling you to either /flutter_run/start or press r in your flutter run terminal. The response's mechanism field (flutter_daemon / vm_service_reassemble_fallback / vm_service_forced_reload) makes the choice transparent.

Flutter-native logs vs. logcat

Two different log surfaces, each better for a different job:

Endpoint	Source	Use when
/app_logs, /logs/stream	Android logcat — everything on the device	You want native crash reports, Choreographer warnings, ANRs, or non-Flutter system noise
/flutter_logs, /flutter_logs/stream	Dart VM Service Stdout / Stderr / Logging streams + Flutter daemon app.log	You want just your app's output — print(), debugPrint(), structured developer.log() records with level / name / error / stack-trace preserved

The /flutter_logs endpoint maintains a 5,000-entry ring buffer indexed by a monotonic seq, so an agent can poll with ?since=<last_seq> and get exactly the new entries each call — no log loss across polls.

Features

• ADB bridge — screenshots, UI hierarchy parsing, tap / swipe / type, Android keycodes, multi-attribute element matchers with exact / contains / regex modes, /elements filtered listing, /wait_for_element polling.
• Logs & observability — filtered logcat reads, SSE streaming, parsed crash extraction (Flutter unhandled, AndroidRuntime fatals, ANRs), Choreographer frame warnings, package metadata + foreground detection.
• Dart VM Service — heap usage via getMemoryUsage, FPS + jank from getVMTimeline, top CPU hotspots from getCpuSamples, isolate enumeration, VM-Service-backed hot reload + hot restart.
• Flutter daemon protocol — own flutter run --machine end-to-end, auto-connect the VM Service, route hot reload through app.restart for real source-level reload, escape-hatch RPC for any app.* / daemon.* method.
• Flutter-native log streaming — subscribes to VM Service Stdout / Stderr / Logging and the Flutter daemon's app.log events; both poll (since cursor) and SSE-push API.
• Agent SDK adapters — examples/agent_tools/ ships a generator (generate_tools.py) that converts the server's OpenAPI spec into 32 Anthropic-Messages-API tool definitions, plus a runnable agent example (harness_agent.py) that drives Flutter Agent via Claude tool use. The same JSON serves as the tool contract for Claude Code (via MCP), Cursor, Antigravity, and any other tool-calling framework.

Docker

docker-compose up -d
curl http://localhost:8000/health

On macOS/Windows the container reaches a host-side ADB server over TCP via host.docker.internal:5037 — run adb start-server on the host first. See the comments in docker-compose.yml for details.

Development

pip install -r requirements.txt
pytest -v --cov=core --cov=app

Tests mock subprocess.run, so the suite runs without a real device. CI (GitHub Actions) enforces ≥80% coverage on core/ and app.py.

See CONTRIBUTING.md.

License

MIT — see LICENSE.