SwarmHands

Talk or type from anywhere — a swarm of AI agents lights up on the OpenSwarm canvas and does the thing, across your laptop and phone.

A personal device mesh for the Open Swarm Build Jam. You message a Telegram bot (text or voice note) from any device, from anywhere. A coordinator agent decomposes the request and fans out specialist agents — each a live card on the OpenSwarm dashboard — that actually do the work: open a GitHub PR, call a contact, send a WhatsApp, read an OTP off your phone and type it into your laptop, answer a question. Everything runs on a Claude Max subscription through OpenSwarm, and the chat narrates each step in real time.

You (phone / laptop, anywhere)
   │  text or voice note (Telegram)
   ▼
Coordinator agent  ── plans, routes to specialists
   ├─► github    (gh / git)            → open a PR, list issues
   ├─► phone     (adb)                 → call, WhatsApp, open apps
   ├─► otp       (adb SMS → laptop)    → read code, autofill + submit login
   ├─► general   (web / Q&A)           → answer anything
   └─► shop      (Blinkit)             → search + add to cart
        every box is an OpenSwarm agent on Claude Max, visible on the canvas

What you can say (or send as a voice note)

Message	What the swarm does
"raise a PR for issue 8"	GitHub agent (via gh) reads the issue, branches, opens a real PR
"call Vijeta"	Phone dials the contact
"message Vijeta on WhatsApp saying I'm on my way"	Opens the chat, types, taps send
"open Instagram on my phone"	Launches the app instantly
"read the OTP from my phone and type it into my laptop, then log me in"	Reads the newest SMS code → types it into the focused field → submits
"what are good mechanical keyboards?"	General agent answers (web-capable)
"order milk from Blinkit"	Searches Blinkit, adds the item, opens the cart

Every command also works as a Telegram voice note — transcribed locally and run through the same swarm.

Setup

# 1. OpenSwarm backend (the swarm engine)
cd openswarm && bash run.sh          # backend :8324 + dashboard; connect Claude in Settings

# 2. SwarmHands
cd swarmhands
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 3. Telegram bot: create one via @BotFather, then in secrets.local.json (gitignored):
#    { "telegram_bot_token": "...", "work_dir": "/abs/repo",
#      "contacts": {"vijeta": "9…", "_pin": "1234"} }
python -m channels.telegram_bot      # DM the bot from your phone

Phone (for the phone / otp / shop skills): enable Developer Options → USB debugging (USB) or Wireless debugging (adb connect <ip>:<port>). The swarm auto-detects the serial via adb devices and unlocks with _pin if the screen is locked.

The skills

Skill	Does	How
github	issues → PRs, list issues	gh + git (already authenticated)
phone	call, WhatsApp (with send), open app, keys/text	direct adb intents (instant); wake + PIN-unlock
otp	cross-device 2FA login	reads newest SMS via adb → types on laptop → optional submit
general	research, web lookups, Q&A — anything else	Claude + OpenSwarm WebSearch/WebFetch
shop	add a grocery to the Blinkit cart	deterministic search + one vision call for the ADD button
gmail / slack	email / Slack	connected OpenSwarm MCP servers

New skills are plug-ins: add an entry to brain/mesh.py SKILLS and the coordinator can route to it.

How it works

1. Channel in — the Telegram bot receives a message (voice notes are transcribed locally), and narrates progress back into the chat as the swarm runs.
2. Coordinator — a toolless OpenSwarm agent decomposes the request into sub-tasks, each routed to one specialist skill.
3. Specialists — each sub-task launches its own OpenSwarm agent (Claude Max), tagged with the dashboard id so it shows as a live card. They run concurrently; GitHub writers get an isolated git worktree so parallel PRs never collide.
4. Native actions, no browser — GitHub uses gh, the phone uses adb. OpenSwarm's browser-delegation tools are denied so agents can't fall back to a login-walled web page.
5. Result out — specialist results are composed into one reply sent back on the channel.

Everything is an OpenSwarm agent session on the Claude subscription (no API key), so the whole run is visible on the OpenSwarm canvas.

Reliability

Lead with these — instant, native, on OpenSwarm: github, phone (call / WhatsApp / open app), otp, general / web, voice, live narration.

Best-effort: shop (Blinkit) works when it runs cleanly (adds the item, opens the cart) but is hit-or-miss on a busy 3rd-party app — it fails gracefully (adds nothing) rather than adding the wrong thing. gmail / slack depend on OpenSwarm's MCP activation.

Layout

• channels/telegram_bot.py — the universal remote (text + voice), live progress narration
• brain/mesh.py — MeshCoordinator: coordinator plans → specialists fan out; per-skill runtime prompts (phone serial + contacts, otp, shop); the SKILLS registry
• brain/openswarm_client.py — OpenSwarm REST/WS client (launch agents with tools, auto-approve, large-frame WebSocket collection)
• brain/shop.py — Blinkit search-and-add
• devices/adb_phone.py — Android over adb (screenshot + tap / type / key / call / open app)
• voice/transcribe.py — local speech-to-text for voice notes
• scripts/type_text.py — types into the laptop's focused field (for the OTP autofill)