robvis

A JARVIS-style, voice- and gesture-driven command deck for launching and tracking tokens.

Talk to it. Point at it. Launch with a sentence.

---

robvis is a cinematic, Iron-Man-inspired operator console that fuses four things into one continuous holographic HUD:

• a 3D particle core you steer with your hand (MediaPipe) and your voice,
• a face-tracking targeting reticle that locks onto the operator on camera,
• a live market intelligence panel for the hottest tokens (via GMGN), with holographic candlestick charts, holder distribution and security checks, and
• a voice-driven token launch flow — say a sentence, confirm a buy, and it fires an on-chain launch.

It speaks and listens in English and 中文, replies in the language you used, and identifies only as robvis.

Highlights

You say	What happens
"Open CASHCAT" / "打开 CASHCAT"	Opens the token — live price, volume, market cap and a holographic candlestick chart
"What's hot on Robinhood?"	Volume-ranked hot list (1m / 5m / 1h / 24h)
"Who bought this?"	Smart-money / top-trader read on the selected token
"Is it safe? / honeypot?"	Security & due-diligence panel
"Zoom in / rotate / reset"	Drives the 3D core by voice or hand gesture
"What did Vlad post?"	X radar report on the tracked account
"Launch from his latest post"	Pulls the newest image tweet → AI drafts name / ticker / description → stages a launch
"Buy 1 eth" / "fire, buy 50"	Countdown → on-chain launch → contract address on screen

Hand gestures (webcam): pinch to scale the core, move an open palm to rotate, open-palm hold to recenter; a pinch also acts as a cursor click on HUD panels.

Architecture

Browser — holographic command deck
  • Three.js particle core        (frontend/core3d.js)
  • MediaPipe hands + gestures     (frontend/gestures.js)
  • MediaPipe face reticle         (frontend/facetrack.js)
  • operator HUD + panels          (frontend/ophud.js, app.js)
  • voice capture + SSE stream
        │
        ▼
Local FastAPI backend
  ├─ brain.py            intent router (voice → action), bilingual, screen-aware
  ├─ stt.py             speech-to-text (OpenAI cloud or local faster-whisper)
  ├─ skills/tts_eleven  ElevenLabs "JARVIS" voice (OpenAI TTS fallback)
  ├─ skills/xradar      X account radar (image tweets → launch drafts)
  ├─ skills/gmgn        Robinhood-Chain market data (hot list, candles, holders, security)
  ├─ skills/geckoterminal · virtuals_market   market-data fallbacks
  ├─ launchpad.py       AI draft from a tweet + platform routing + fire
  ├─ skills/virtuals    on-chain launch engine (Virtuals Protocol)
  └─ studio_bridge.py   LLM brain (OpenAI, optional fallback)

Quick start

Requirements: Python 3.11+, a modern Chrome (for camera + mic + WebGL).

git clone https://github.com/baogermakesbao/robvis.git
cd robvis
cp backend/.env.example backend/.env      # then add your keys (see below)
./run.sh                                   # → http://127.0.0.1:8899

Open the URL in Chrome and allow camera and microphone. Click ACTIVATE LENS to bring the deck online, then just speak.

By default robvis runs in launch-simulation mode (ROBVIS_SIM_LAUNCH=1) — the full UI and voice flow work with mock transactions, so you can explore and demo without any wallet or funds.

Configuration

Copy backend/.env.example → backend/.env and fill in what you need:

Variable	Purpose
OPENAI_API_KEY	The brain (LLM routing) and optional cloud STT (STT_BACKEND=openai)
ELEVENLABS_API_KEY, ELEVEN_VOICE_ID	The cinematic "JARVIS" voice (falls back to the browser voice if unset)
GMGN_API_KEY	Live market data (a public demo key is used if unset; get your own at gmgn.ai for reliability)
PRIVATE_KEY	A dedicated hot wallet for real launches — never sent to the browser, never logged
BASE_RPC_URL, BONDING_ADDRESS, VIRTUAL_TOKEN	On-chain launch target (verify the bonding address before a mainnet launch)
ROBVIS_SIM_LAUNCH	1 = always simulate, 0 = real launches, unset = auto

Validate safely first. Before spending real funds, point the RPC at a testnet and confirm a full launch end-to-end, then switch to mainnet.

Deployment

A production layout (systemd + nginx + TLS) is documented in deploy/DEPLOY.md, with sample deploy/robvis.service and deploy/nginx.robvis.conf.

Tech stack

Three.js · MediaPipe (Hands + Face Landmarker) · FastAPI · OpenAI · faster-whisper · ElevenLabs · web3.py · Server-Sent Events.

Security

• Secrets live only in backend/.env, which is gitignored — never rendered to the browser, never logged.
• The launch wallet key stays server-side and is used only to sign launch transactions.
• Run against a testnet and in simulation mode until you have verified the flow.

License

MIT © baogermakesbao