🔗 Live Web App:
https://flutter-ml-web.web.app
Companion Robot is an AI-powered interactive robotic system designed to operate entirely offline using edge-based intelligence. Compact enough to fit on a desk, the system integrates speech recognition, computer vision, and distance sensing through a Raspberry Pi without relying on cloud infrastructure.
Voice commands are captured via an onboard microphone and processed locally using Large Language Models (LLMs). Visual perception is provided by a camera module, while spatial awareness and collision avoidance are achieved using ultrasonic distance measurement. All computation occurs on-device, ensuring low latency, enhanced privacy, and uninterrupted operation.
The robot provides real-time feedback through an integrated display and audio output. This project demonstrates how affordable hardware combined with efficient AI pipelines can enable autonomous, privacy-preserving robotic assistants.
- Enable fully offline AI interaction
- Achieve real-time perception and response
- Ensure safe autonomous navigation
- Provide web-based control via Flutter
- Maintain modular and scalable architecture
Companion Robot follows a layered architecture, ensuring clean separation of hardware interaction, processing logic, and intelligence.
- Microphone: Captures speech commands.
- Camera Module: Supplies real-time visual data.
- Ultrasonic Sensor: Computes distance using echo timing.
Implemented in Python with Flask, this layer:
- Converts speech to text
- Normalizes sensor signals
- Routes commands between UI and AI
- Maintains real-time API communication
- Runs locally (offline) using LM Studio
- Performs:
- Natural language understanding
- Scene and object analysis
- Contextual decision-making
- Emotion classification
- Speaker: AI-generated speech output
- Display: System state and responses
- Motors: Physical movement execution
- Flutter Web UI: Control, logs, and visualization
| Component | Description |
|---|---|
| Raspberry Pi 4 Model B | Central processing unit |
| Camera Module | Visual perception |
| Ultrasonic Sensor (HC-SR04) | Obstacle detection |
| Microphone | Voice input |
| Speaker | Audio output |
| 3.5″ Touch Display | Visual feedback |
| Motor Driver + Motors | Locomotion |
| Regulated Power Supply | Stable power delivery |
- Frontend: Flutter (Web/Desktop)
- Backend: Python, Flask, Socket.IO
- AI Engine: LM Studio (Local LLMs)
- Vision: OpenCV
- Speech: Web Speech API / PyAudio
- Hardware Control: GPIO
- OS: Raspberry Pi OS
The Flask server acts as the central coordination unit.
Fail-safe logic ensures immediate stop or slowdown when obstacles are detected.
- Emotion mapping from AI responses
- Fullscreen emotion playback
- Duplicate-emotion prevention
- Automatic return to idle state
This enhances human-robot interaction quality.
- Manual control dashboard
- AI chat interface
- Auto Mode hotkey
- Live logs & diagnostics
- Network configuration UI
- Desktop full-screen support
💡 Design Principle:
The interface is inspired by mission-command dashboards, emphasizing safety, clarity, and immediate operator awareness without reliance on keyboard input.
Purpose:
This flow ensures real-time, offline intelligence with minimal latency and maximum safety.
Auto Mode enables autonomous behavior using local AI reasoning and sensor fusion.
Capabilities:
- Continuous environment scanning
- Voice-initiated task execution
- Obstacle-aware motion control
- Emotion-synchronized responses
Safety Behavior:
- Ultrasonic distance monitoring
- Automatic slow-down in congested areas
- Emergency stop override at all times
Auto Mode always yields priority to manual or emergency commands.
| Metric | Description |
|---|---|
| CPU Load | Current Raspberry Pi processing usage |
| Temperature | Live system thermal state |
| Distance (cm) | Ultrasonic obstacle measurement |
| Network Status | API connectivity health |
| Motor State | Active / Idle / Emergency Stop |
Telemetry updates continuously while commands are executed, ensuring operator awareness and system transparency.
The Companion Robot uses an emotion-mapped response system to enhance interaction.
Emotion Triggers:
- AI response classification
- System states (idle, thinking, alert)
- Voice interaction outcomes
UI Behavior:
- Fullscreen emotion playback
- Duplicate emotion prevention
- Automatic fallback to idle state
Examples:
- 😊 Happy → Joke or positive response
- 🤔 Thinking → Scanning or processing
- 🚨 Alert → Obstacle detected
This system improves clarity, relatability, and human-robot communication.
- Ultrasonic obstacle avoidance
- Speed reduction near objects
- Emergency stop mechanism
- Manual override priority
- Non-blocking asynchronous execution
 Manual Control Panel Directional and emergency controls |
 AI Chat Interface Offline command & response system |
 Auto Mode Active AI-driven autonomous behavior |
 Camera Preview Real-time vision input for AI |
 Emotion Display AI emotion-based visual feedback |
 Desktop Full-Screen View Optimized wide-screen layout |
- Capture user input
- Preprocess locally
- Interpret using LLM
- Apply safety constraints
- Execute action
- Provide feedback
- Log system state
- Educational robotics
- Offline AI assistants
- Human-robot interaction research
- Privacy-preserving AI systems
- Smart automation demos
- Follow-me mode
- Patrol / guard mode
- Face recognition (optional)
- Expanded emotion library
- Multi-language interaction
This project is developed for educational and experimental purposes.
Proper safety measures must be followed during physical deployment.
⭐ Companion Robot demonstrates the feasibility of intelligent, private, and autonomous edge-AI robotics using affordable hardware.