JARVIS: The Autonomous Voice Agent

JARVIS is a high-performance, autonomous desktop assistant that transforms natural speech into complex actions. Unlike traditional voice assistants that only answer questions, JARVIS executes work by navigating, interacting, and verifying tasks across your desktop and browser.

---

Key Features

Autonomous Browser Interaction

JARVIS doesn't just click buttons; it understands web interfaces.

• Smart Scrolling & Navigation: Automatically finds elements below the fold or across multiple pages.
• Semantic DOM Targeting: Prioritizes interactive elements based on visibility and importance.
• Element Stability: Intelligent waits for animations and network hydration to prevent "ghost clicks."

Vision-Verified Execution

Screenshots confirm whether a task actually completed. If it fails, JARVIS diagnoses why and retries with a corrected approach.

Self-Healing & Re-Planning

When a step fails mid-workflow, JARVIS stops, re-scans the page, and generates a recovery plan — no blind retries.

Local Parakeet STT (Default)

Fast, local speech-to-text using Microsoft's Parakeet (DirectML + ORT). No cloud dependency. Falls back to Whisper only on --whisper flag.

260+ Reflexes

Instant voice commands for:

• Window management: minimize, maximize, snap, switch, close
• Browser control: tabs, navigation, zoom, dev tools, bookmarks
• Media: play/pause, volume, next/previous track
• Editing: copy, paste, undo, redo, bold, italic, find
• App launchers: open discord/spotify/vscode/terminal/notepad + 50+ web speed-dial sites
• System: brightness, battery, time, date, IP, screen resolution, clipboard history
• Files: open downloads/documents/pictures, find files, rename, delete
• Dictation: start/stop dictation in any app
• Teach mode: record and save multi-step workflows by demonstration

Teach Mode & Skill Learning

Record a workflow once (e.g., "open Gmail, draft reply to Sarah, send"). JARVIS replays it on command. Repeated patterns auto-propose as reusable skills.

Multi-Agent Personas

Save named agent configurations (custom prompt, model, provider). Switch between "Research Specialist", "Shopping Buddy", etc. by voice.

Cinematic HUD Overlay

Tkinter-based overlay with state animations: LOADING, LISTENING, THINKING, EXECUTING, TEACHING, SUCCESS, ERROR, SPEAKING.

Memory & Context

Long-term memory with embedding-based semantic recall. Remembers preferences, past results, and learned selectors across sessions.

Plugin System

Drop .py files into plugins/ to register new reflexes and operations without modifying core code.

---

How It Works

Voice Input → Parakeet/Whisper STT → Intent Sequencer → Planner
    ↓                                        ↓
Action Router ──→ PC Executor (local apps, files, system)
    │
    ├──→ Browser Executor (CDP-based Chrome control)
    │
    ├──→ Vision Verification (screenshot + analysis)
    │
    └──→ Self-Healing Recovery (re-plan on failure)

---

Setup

1. Installation

.\setup.ps1

2. Configuration

Edit your .env file:

# Intelligence
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini

# Voice Settings (Parakeet is default)
STT_PROVIDER=parakeet      # parakeet | whisper
TTS_PROVIDER=local         # local | edge | kokoro

# Persistence
BROWSER_PROFILE_PATH=C:\Users\YourUser\AppData\Local\JARVIS\profile

3. Run

python core\orchestrator.py

---

Usage

Input	Action
Hold Shift	Push-to-talk
Ctrl+Shift+Space	Text input mode
Esc	Abort current action
Say "start dictation"	Dictate into any text field
Say "teach mode"	Record a multi-step workflow

Example Commands

• "Open LinkedIn, search for AI Engineers in London, and summarize the first profile."
• "Go to Amazon and find a mechanical keyboard under $100 with 4+ stars."
• "Draft a professional reply to the last email from Sarah in Gmail."
• "What time is it?"
• "Open notepad"
• "Volume up"
• "Switch to Research Assistant and summarize this article."

---

Safety & Privacy

• Emergency Stop: Tap Esc at any time to kill all autonomous actions.
• Confirmation Mode: Sensitive actions (sending messages, deleting files) require HUD confirmation.
• Local Fallback: Supports Ollama for 100% private, offline LLM processing.
• All STT is local: Parakeet runs entirely on your GPU. No audio leaves your machine.

---

Built with for the future of Autonomous Desktop Interaction.