AgentTwin Glass 🔮

An always-on AI overlay application with screen capture capabilities, built with Tauri, React, and Rust.

Overview

AgentTwin Glass is a lightweight, transparent desktop overlay that provides instant AI assistance with advanced screen capture and vision analysis capabilities. It sits elegantly on top of your workspace, ready to help whenever you need it.

Key Features

• 🪟 Transparent Glass UI - Beautiful, unobtrusive overlay that stays on top
• 📸 Smart Screen Capture - Automatic screenshot detection and capture
• 🤖 AI Vision Analysis - Send screenshots for AI-powered analysis
• ⚡ Global Shortcuts - Toggle visibility with Cmd+Shift+G
• 🔌 WebSocket Integration - Real-time connection to AgentTwin backend
• 💬 Chat Interface - Natural conversation with context awareness
• 🎯 Auto-detection - Automatically captures screen when asking screen-related questions

Prerequisites

• Node.js (v16 or higher)
• Rust (latest stable version)
• Tauri CLI (installed via npm)
• AgentTwin Backend (running separately on ws://127.0.0.1:8765/ws)

Installation

1. Clone the repository

git clone <repository-url>
cd agenttwin-glass

2. Install dependencies

npm install

3. Run in development mode

npm run tauri dev

4. Build for production

npm run tauri build

Usage

Starting the Application

1. Start the AgentTwin backend server (required for AI functionality):

cd ~/Desktop/agenttwin
source venv/bin/activate
python -m src.main

2. Launch AgentTwin Glass:

npm run tauri dev

Keyboard Shortcuts

• Cmd+Shift+G - Toggle window visibility
• Enter - Send message
• Cmd+Enter - Capture screen

Chat Commands

The application automatically detects screen-related queries and captures screenshots. Keywords that trigger auto-capture include:

• "screen", "what's on", "see my", "look at"
• "browser", "window", "display"
• "what do you see", "analyze my"
• "current page", "this page"
• And more...

Manual Screen Capture

Click the camera icon or press Cmd+Enter to manually capture a screenshot before sending your message.

Architecture

Frontend

• React 19 with TypeScript
• Vite for fast development and building
• Tauri API for native functionality

Backend (Rust)

• Tauri 2.9 framework
• Screenshots crate for screen capture
• Tokio for async operations
• Global shortcuts plugin

Communication

• WebSocket connection to external AI backend
• Real-time bidirectional messaging
• Automatic reconnection on disconnect

Configuration

Window Settings

The window configuration can be modified in src-tauri/tauri.conf.json:

{
  "width": 400,
  "height": 600,
  "alwaysOnTop": true,
  "decorations": false,
  "transparent": true,
  "skipTaskbar": true
}

WebSocket URL

The backend WebSocket URL is defined in src/App.tsx:

const WS_URL = "ws://127.0.0.1:8765/ws";

Development

Project Structure

agenttwin-glass/
├── src/                    # React frontend source
│   ├── App.tsx            # Main application component
│   ├── main.tsx           # React entry point
│   └── styles.css         # Styling
├── src-tauri/             # Rust backend source
│   ├── src/
│   │   ├── lib.rs        # Library code
│   │   └── main.rs       # Tauri main
│   ├── Cargo.toml        # Rust dependencies
│   └── tauri.conf.json   # Tauri configuration
├── package.json           # Node dependencies
└── vite.config.ts        # Vite configuration

Building

# Development build
npm run dev

# Production build
npm run build
npm run tauri build

Debugging

The application uses Tauri's logging plugin. Logs can be viewed in the console during development mode.

WebSocket API

Client → Server Messages

// Chat message
{
  "type": "chat",
  "content": "Your message here"
}

// Vision request with screenshot
{
  "type": "vision",
  "prompt": "What do you see?",
  "screenshot": "base64_encoded_image"
}

Server → Client Messages

// Text response
{
  "type": "response",
  "content": "AI response"
}

// Vision analysis response
{
  "type": "vision_response",
  "analysis": "Image analysis"
}

// Error message
{
  "type": "error",
  "message": "Error details"
}

Platform Support

• ✅ macOS (primary platform with private API support)
• ⚠️ Windows (experimental)
• ⚠️ Linux (experimental)

Known Issues

• Transparent windows may have rendering quirks on some platforms
• Screen capture requires appropriate system permissions

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details

Author

Ashish Dwivedi

Acknowledgments

• Built with Tauri
• UI powered by React
• Screen capture via screenshots

---

AgentTwin Glass - Your AI companion, always within reach.