House-Controling-System

A ROS1 Noetic local multi-modal home assistant prototype for the Jupiter robot platform.

This project demonstrates how a robot can use camera-based hand gestures and offline voice commands to control simulated smart-home devices through ROS topics. It is designed as a lightweight robotics AI demo that can run on:

Jupiter ROS machine
Ubuntu 20.04 + ROS Noetic machine
WSL ROS-Noetic environment with camera / microphone / speaker access

GitHub repository:

https://github.com/KevinRayScottUM/House-Controling-System

1. What This Project Does

The system controls three simulated rooms through a Pygame GUI.

Room	Display Name	Devices from left to right
Room 1	Bedroom	1. Light, 2. Fan, 3. Air Conditioner, 4. Door
Room 2	Living Room	1. Light, 2. Fan, 3. Curtain, 4. Door
Room 3	Toilet	1. Light, 2. Exhaust Fan, 3. Water Heater, 4. Door

The project supports:

Function	Description
Room-level gesture control	Use hand gestures to turn on/off every device in a selected room.
Device-level gesture control	Select a specific device inside a room and then control only that device.
Room-level voice control	Say commands such as `turn on room one` or `turn off toilet`.
Device-level voice control	Say commands such as `turn on the curtain in room two`.
Multi-command voice control	Say commands such as `turn on room one and room two`.
Global voice control	Say commands such as `turn off all rooms`.
Audio feedback	A command bell and device/room sound effects are played through the speaker.
Pygame GUI	A 2D visualizer shows the state of each room and device.
ROS topic routing	Both gesture and voice commands finally publish to `/home_assistant/command`.

The most important design rule is:

All control paths, whether gesture or voice, finally publish JSON commands to /home_assistant/command.

That means the project is easy to debug. If the GUI or speaker does not react, check /home_assistant/command first.

2. Final System Architecture

Camera / Android IP Camera / USB Camera
        |
        v
camera_stream_node.py
        |
        v
/usb_cam/image_raw
        |
        v
mediapipe_gesture_node.py
        |
        v
/home_assistant/gesture_raw
        |
        v
gesture_interpreter_node.py
        |
        v
/home_assistant/command
        |
        +------------------> state_manager_node.py ----> /home_assistant/state ----> home_visualizer_node.py
        |
        +------------------> audio_feedback_node.py ----> speaker sound feedback


Microphone
        |
        v
vosk_asr_arecord_node.py
        |
        v
/recognizer/output
        |
        v
voice_command_node.py
        |
        v
/home_assistant/command

Main data flow:

Input	AI / processing node	Output topic	Final effect
Camera frames	`mediapipe_gesture_node.py`	`/home_assistant/gesture_raw`	Raw gesture labels such as `left_1`, `right_4`, `right_open`, `standby`.
Raw gesture labels	`gesture_interpreter_node.py`	`/home_assistant/command`	JSON room/device control command.
Microphone audio	`vosk_asr_arecord_node.py`	`/recognizer/output`	Recognized text such as `turn on room one`.
Recognized text	`voice_command_node.py`	`/home_assistant/command`	JSON room/device control command.
Final command	`state_manager_node.py`	`/home_assistant/state`	Updated room/device state.
Final command	`audio_feedback_node.py`	speaker	Command bell and room/device sound effect.
State	`home_visualizer_node.py`	Pygame window	Visual room/device status.

3. Expected Environment

Recommended environment:

Ubuntu 20.04
ROS1 Noetic
Python 3.8
catkin workspace
Local display for Pygame GUI
Camera: USB camera, built-in camera, ROS image topic, or Android IP camera
Microphone: ALSA / PulseAudio / default device
Speaker: system speaker, USB speaker, Jupiter speaker output, or WSL audio output

The expected project path is:

~/catkin_ws/src/local_home_assistant

Part A. Fresh Installation on Jupiter / ROS-Noetic Machine

This section is written for a reader who wants to copy and paste commands from zero until the project runs.

4. Check Ubuntu, ROS, and Catkin

Open a terminal.

lsb_release -a
source /opt/ros/noetic/setup.bash
rosversion -d
which catkin_make

Expected ROS version:

noetic

Expected catkin_make path usually looks like:

/opt/ros/noetic/bin/catkin_make

5. Install System Dependencies

sudo apt update

sudo apt install -y \
  git \
  wget \
  unzip \
  curl \
  python3-pip \
  python3-venv \
  python3-opencv \
  python3-tk \
  ros-noetic-cv-bridge \
  ros-noetic-image-transport \
  pulseaudio-utils \
  alsa-utils \
  libasound2-plugins \
  portaudio19-dev \
  espeak \
  v4l-utils

Why these packages are needed:

Package	Purpose
`git`	Clone the GitHub repository.
`wget`, `unzip`, `curl`	Download and check the Vosk model and IP camera URL.
`python3-pip`, `python3-venv`	Create and manage the Python environment.
`python3-opencv`	Read camera frames and IP camera streams.
`ros-noetic-cv-bridge`	Convert ROS Image messages to OpenCV images.
`ros-noetic-image-transport`	ROS image topic support.
`pulseaudio-utils`, `alsa-utils`, `libasound2-plugins`	Test and route microphone/speaker devices.
`portaudio19-dev`	Audio backend support.
`espeak`	Optional local speech/audio support.
`v4l-utils`	Check `/dev/videoX` camera devices.

6. Create or Prepare the Catkin Workspace

mkdir -p ~/catkin_ws/src
cd ~/catkin_ws

source /opt/ros/noetic/setup.bash
catkin_make
source ~/catkin_ws/devel/setup.bash

Add ROS setup to .bashrc so every new terminal loads ROS automatically:

grep -qxF 'source /opt/ros/noetic/setup.bash' ~/.bashrc || echo 'source /opt/ros/noetic/setup.bash' >> ~/.bashrc
grep -qxF 'source ~/catkin_ws/devel/setup.bash' ~/.bashrc || echo 'source ~/catkin_ws/devel/setup.bash' >> ~/.bashrc
source ~/.bashrc

7. Clone This Project

cd ~/catkin_ws/src

git clone https://github.com/KevinRayScottUM/House-Controling-System.git local_home_assistant

cd ~/catkin_ws/src/local_home_assistant
ls

Expected files:

CMakeLists.txt
LICENSE
README.md
assets
config
dataset
docs
launch
models
package.xml
scripts

8. Create the Python Virtual Environment

Use --system-site-packages. This is important because ROS Python modules such as rospy and cv_bridge are installed at the system level.

mkdir -p ~/venvs
python3 -m venv --system-site-packages ~/venvs/ROS_HOME_ASSISTANT
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

Upgrade Python tools. The pinned setuptools / importlib_metadata setting avoids a common Noetic + virtual environment compatibility problem.

python -m pip install --upgrade pip wheel
python -m pip install --upgrade "setuptools<70" "importlib_metadata>=4.13"

Install project Python packages:

python -m pip install pygame pyyaml vosk requests sounddevice
python -m pip install mediapipe==0.10.11

If MediaPipe installation fails, try:

python -m pip install mediapipe==0.10.9

Verify imports:

python - <<'PY'
import rospy
import cv2
import pygame
import yaml
import vosk
from cv_bridge import CvBridge

print("rospy OK")
print("cv2 OK:", cv2.__version__)
print("pygame OK:", pygame.version.ver)
print("yaml OK")
print("vosk OK")
print("cv_bridge OK")

try:
    import mediapipe as mp
    print("mediapipe OK:", mp.__version__)
except Exception as e:
    print("mediapipe import failed:", repr(e))
PY

Expected: the output should show OK for all required modules.

9. Download the Vosk Offline ASR Model

The Vosk model is not included in this repository because it is a large local model file.

mkdir -p ~/catkin_ws/src/local_home_assistant/models
cd ~/catkin_ws/src/local_home_assistant/models

wget https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip

Confirm the model exists:

ls ~/catkin_ws/src/local_home_assistant/models/vosk-model-small-en-us-0.15

Expected:

am
conf
graph
ivector
README

If the model already exists, do not download it again.

10. Build Only This ROS Package

Recommended build command:

cd ~/catkin_ws

source /opt/ros/noetic/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

catkin_make --pkg local_home_assistant

source ~/catkin_ws/devel/setup.bash

Check whether ROS can find the package:

rospack find local_home_assistant

Expected example:

/home/ros/catkin_ws/src/local_home_assistant

Important Jupiter / Workspace Note

If your ~/catkin_ws/src contains other ROS packages such as TurtleBot3 examples, a full catkin_make may fail because of unrelated packages.

A common unrelated error is:

AttributeError: module 'importlib_metadata' has no attribute 'EntryPoints'

For this project, use:

catkin_make --pkg local_home_assistant

If you need to repair the Python compatibility issue inside the virtual environment:

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate
python -m pip install --upgrade "setuptools<70" "importlib_metadata>=4.13"

Part B. Test Devices Before Running the Full System

Do this before the final three-terminal launch. Most runtime problems are caused by the wrong camera, microphone, or speaker device.

11. Test Speaker Output

11.1 Basic Linux Speaker Test

speaker-test -t sine -f 880 -l 1

You should hear a short tone.

11.2 Test Pygame Audio

This is closer to how the project plays command bell and device sound effects.

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

python - <<'PY'
import math, wave, struct, time
from pathlib import Path

wav = Path("/tmp/jupiter_speaker_test.wav")
sr = 44100
duration = 0.45
freq = 880
amp = 0.45

with wave.open(str(wav), "w") as f:
    f.setnchannels(1)
    f.setsampwidth(2)
    f.setframerate(sr)
    for i in range(int(sr * duration)):
        v = int(32767 * amp * math.sin(2 * math.pi * freq * i / sr))
        f.writeframes(struct.pack("<h", v))

print("Created:", wav)

import pygame
pygame.mixer.init(frequency=sr, size=-16, channels=1, buffer=512)
snd = pygame.mixer.Sound(str(wav))
snd.set_volume(1.0)
ch = snd.play()
print("Playing test beep...")
while ch and ch.get_busy():
    time.sleep(0.05)
pygame.mixer.quit()
print("Speaker test done.")
PY

If there is no sound, check available output devices:

pactl info
pactl list short sinks
pactl get-default-sink

Set a default output sink if needed:

pactl set-default-sink YOUR_SINK_NAME

In WSL, also check the Windows sound output device and Volume Mixer. The Python or WSL audio stream may be muted.

12. Test Microphone Input

For this project, the most stable microphone parameter is often:

_audio_device:=default

Start by testing default.

12.1 List Microphones

arecord -l
pactl list short sources
pactl info | grep "Default Source"

12.2 Test Default Microphone

arecord -D default -f S16_LE -r 16000 -c 1 -d 5 /tmp/test_mic_default.wav
aplay /tmp/test_mic_default.wav

If you hear your recording, use:

_audio_device:=default
_sample_rate:=16000

12.3 Test PulseAudio Microphone

arecord -D pulse -f S16_LE -r 16000 -c 1 -d 5 /tmp/test_mic_pulse.wav
aplay /tmp/test_mic_pulse.wav

If it works, use:

_audio_device:=pulse
_sample_rate:=16000

12.4 Test Hardware Microphone

If arecord -l shows:

card 1: USB Audio, device 0

test:

arecord -D hw:1,0 -f S16_LE -r 16000 -c 1 -d 5 /tmp/test_mic_hw.wav
aplay /tmp/test_mic_hw.wav

If it works, use:

_audio_device:=hw:1,0
_sample_rate:=16000

12.5 Quick Microphone Decision Table

Test result	Use this Terminal 2 parameter
`arecord -D default ...` works	`_audio_device:=default`
`arecord -D pulse ...` works	`_audio_device:=pulse`
`arecord -D hw:1,0 ...` works	`_audio_device:=hw:1,0`
none of them works	fix OS microphone routing before running Vosk

13. Test Camera Input

13.1 USB or Built-in Camera

List video devices:

ls /dev/video*
v4l2-ctl --list-devices

Test camera index 0:

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

python - <<'PY'
import cv2
camera_index = 0
cap = cv2.VideoCapture(camera_index)
print("camera opened:", cap.isOpened())
ok, frame = cap.read()
print("read frame:", ok)
print("frame shape:", None if frame is None else frame.shape)
cap.release()
PY

If camera 0 fails, test camera 1:

python - <<'PY'
import cv2
camera_index = 1
cap = cv2.VideoCapture(camera_index)
print("camera opened:", cap.isOpened())
ok, frame = cap.read()
print("read frame:", ok)
print("frame shape:", None if frame is None else frame.shape)
cap.release()
PY

Use the working index later:

camera_source:=0

or:

camera_source:=1

13.2 Android IP Camera

If the Jupiter camera is not ready, use an Android phone or tablet as an IP camera.

Example:

http://192.168.0.205:8080/video

Test it:

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

python - <<'PY'
import cv2
url = "http://192.168.0.205:8080/video"
cap = cv2.VideoCapture(url)
print("camera opened:", cap.isOpened())
ok, frame = cap.read()
print("read frame:", ok)
print("frame shape:", None if frame is None else frame.shape)
cap.release()
PY

If the test fails:

curl -I http://192.168.0.205:8080
curl -I http://192.168.0.205:8080/video

The Android device and the Jupiter robot must be on the same network.

13.3 Quick Camera Decision Table

Test result	Use this Terminal 1 parameter
OpenCV camera index `0` works	`camera_source:=0`
OpenCV camera index `1` works	`camera_source:=1`
Android IP camera works	`camera_source:=http://YOUR_PHONE_IP:8080/video`
camera opens but left/right appears reversed	add `swap_hands:=true` or `mirror_image:=false`
camera FPS is low but stable	keep it; 7-15 FPS is acceptable for this demo

Part C. Final Demo Run with Three Terminals

The final project is intentionally run with three terminals. This is clearer and safer because camera, microphone, and text-to-command parsing can be debugged separately.

Terminal	Purpose	Main nodes
Terminal 1	Camera + gesture + GUI + speaker feedback	`camera_stream_node.py`, `mediapipe_gesture_node.py`, `gesture_interpreter_node.py`, `state_manager_node.py`, `home_visualizer_node.py`, `audio_feedback_node.py`
Terminal 2	Microphone + Vosk speech recognition	`vosk_asr_arecord_node.py`
Terminal 3	Recognized text to final command	`voice_command_node.py`

14. Clean Old ROS Processes

Before a new demo run:

pkill -f roslaunch || true
pkill -f roscore || true
pkill -f rosmaster || true
pkill -f home_state_manager || true
pkill -f home_visualizer_node || true
pkill -f camera_stream_node || true
pkill -f mediapipe_gesture_node || true
pkill -f gesture_interpreter_node || true
pkill -f voice_command_node || true
pkill -f vosk_asr || true

15. Terminal 1: Gesture + Camera + GUI + Speaker Feedback

This terminal starts:

ROS master
Pygame room simulator
state manager
audio feedback node
camera stream node
MediaPipe gesture recognition
gesture interpreter

Case A: Android IP Camera

Replace the IP address with your own phone/tablet camera IP if needed.

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

roslaunch local_home_assistant home_assistant_camera_mediapipe.launch \
  camera_source:=http://192.168.0.205:8080/video \
  enable_bell:=true \
  enable_device_sfx:=true \
  audio_volume:=1.0 \
  play_startup_sound:=true \
  command_cooldown_sec:=1.5 \
  selection_timeout_sec:=8.0 \
  device_action_timeout_sec:=8.0 \
  standby_publish_interval:=0.25 \
  process_every_n:=1 \
  show_gesture_debug:=true

Case B: Built-in or USB Camera

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

roslaunch local_home_assistant home_assistant_camera_mediapipe.launch \
  camera_source:=0 \
  enable_bell:=true \
  enable_device_sfx:=true \
  audio_volume:=1.0 \
  play_startup_sound:=true \
  command_cooldown_sec:=1.5 \
  selection_timeout_sec:=8.0 \
  device_action_timeout_sec:=8.0 \
  standby_publish_interval:=0.25 \
  process_every_n:=1 \
  show_gesture_debug:=true

If index 0 fails, try:

camera_source:=1

Expected result:

1. Pygame home simulator window opens.
2. MediaPipe gesture debug window opens.
3. Terminal shows state manager, audio feedback, camera, MediaPipe, and gesture interpreter logs.

16. Terminal 2: Vosk Offline Speech Recognition

This terminal listens to microphone audio and publishes recognized text to:

/recognizer/output

Recommended Jupiter setting:

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

rosrun local_home_assistant vosk_asr_arecord_node.py \
  _audio_device:=default \
  _sample_rate:=16000 \
  _use_grammar:=true \
  _phrase_publish_delay_sec:=1.25 \
  _max_phrase_buffer_sec:=7.0

If default does not work, try PulseAudio:

rosrun local_home_assistant vosk_asr_arecord_node.py \
  _audio_device:=pulse \
  _sample_rate:=16000 \
  _use_grammar:=true \
  _phrase_publish_delay_sec:=1.25 \
  _max_phrase_buffer_sec:=7.0

If your tested hardware device is hw:1,0:

rosrun local_home_assistant vosk_asr_arecord_node.py \
  _audio_device:=hw:1,0 \
  _sample_rate:=16000 \
  _use_grammar:=true \
  _phrase_publish_delay_sec:=1.25 \
  _max_phrase_buffer_sec:=7.0

Expected output:

[vosk_asr_arecord_node] FINAL_MERGED: turn on the light in room one

17. Terminal 3: Voice Text-to-Command Parser

This terminal converts recognized text from:

/recognizer/output

into final home control commands on:

/home_assistant/command

Run:

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

rosrun local_home_assistant voice_command_node.py \
  _command_cooldown_sec:=0.6 \
  _multi_command_gap_sec:=0.35

Expected output:

[voice_command_node] Ready: robust multi-command parser

18. Minimum Functional Test After Starting the Three Terminals

After the three terminals are running, test in this order:

Test	Action	Expected result
Room gesture test	Left hand 1 + right open palm	Room 1 all devices turn on.
Device gesture test	Left hand 1 + right hand 1, both hands down, right open palm	Room 1 light turns on.
Door gesture test	Left hand 1 + right hand 4, both hands down, right open palm	Room 1 door opens.
Voice room test	Say `turn on room one`	Room 1 all devices turn on.
Voice device test	Say `turn on the curtain in room two`	Room 2 curtain opens.
Global voice test	Say `turn off all rooms`	Room 1, Room 2, and Room 3 all turn off.
Sound test	Trigger any command	Bell + room/device sound effect plays.

Part D. Supported Gesture Commands

This part is the most important part for operating the project. The gesture system has two levels:

Room-level gesture:
  choose a room and immediately turn the whole room ON/OFF.

Device-level gesture:
  choose a room, choose a device, put both hands down until Standby appears, then use only the right hand to turn that selected device ON/OFF.

The Standby step is intentional. It avoids the conflict between:

Left hand 1 + right open palm = Room 1 all ON

and:

Left hand 1 + right hand 1 + right open palm = intended Room 1 Device 1 ON

Without Standby, the system may think the second sequence is just a room-level ON command.

19. Gesture Symbols

Gesture symbol	What the camera should see	System label	Meaning
Left hand 1	Left hand shows one finger	`left_1`	Select Room 1 / Bedroom
Left hand 2	Left hand shows two fingers	`left_2`	Select Room 2 / Living Room
Left hand 3	Left hand shows three fingers	`left_3`	Select Room 3 / Toilet
Right hand fist	Right hand shows zero fingers / closed fist	`right_fist`	OFF / close selected room or device
Right hand 1	Right hand shows one finger	`right_1`	Select device 1 in the selected room
Right hand 2	Right hand shows two fingers	`right_2`	Select device 2 in the selected room
Right hand 3	Right hand shows three fingers	`right_3`	Select device 3 in the selected room
Right hand 4	Right hand shows four fingers	`right_4`	Select device 4 in the selected room
Right open palm	Right hand shows five fingers clearly open	`right_open`	ON / open selected room or device
Both hands down	No hands visible in camera	`standby`	Confirm device selection and wait for action

Important:

right_4 means selecting the fourth device.
right_open means the right hand must be fully open with five fingers.

If the thumb is not visible, right_open may be detected as right_4. For the ON action, show a clear open palm with the thumb also open.

20. Device Mapping by Room

The devices are ordered from left to right in the Pygame interface.

Room selected by left hand	Right hand 1	Right hand 2	Right hand 3	Right hand 4
Left hand 1 = Room 1 / Bedroom	Light	Fan	Air Conditioner	Door
Left hand 2 = Room 2 / Living Room	Light	Fan	Curtain	Door
Left hand 3 = Room 3 / Toilet	Light	Exhaust Fan	Water Heater	Door

21. Room-Level Gesture Commands

Room-level control does not require Standby.

Operation	Gesture sequence	ROS command target	Expected GUI effect	Expected audio
Turn on Room 1	Left hand 1 + right open palm	`room_1`, `master_switch`, `on`	All Room 1 devices turn on/open	Room ON sound
Turn off Room 1	Left hand 1 + right fist	`room_1`, `master_switch`, `off`	All Room 1 devices turn off/close	Room OFF sound
Turn on Room 2	Left hand 2 + right open palm	`room_2`, `master_switch`, `on`	All Room 2 devices turn on/open	Room ON sound
Turn off Room 2	Left hand 2 + right fist	`room_2`, `master_switch`, `off`	All Room 2 devices turn off/close	Room OFF sound
Turn on Room 3	Left hand 3 + right open palm	`room_3`, `master_switch`, `on`	All Room 3 devices turn on/open	Room ON sound
Turn off Room 3	Left hand 3 + right fist	`room_3`, `master_switch`, `off`	All Room 3 devices turn off/close	Room OFF sound

Recommended way to perform room-level gestures:

1. Keep both hands visible and separated.
2. Show the left hand room number.
3. Show right open palm for ON or right fist for OFF.
4. Hold the gesture for about 0.5 to 1 second.

22. Device-Level Gesture Logic

Device-level gestures use four steps.

Step	What to do	What should happen
Step 1	Show left hand room number.	The system selects the room.
Step 2	Show right hand device number 1/2/3/4.	The Pygame GUI highlights the selected device with a blue box.
Step 3	Put both hands down until the MediaPipe window shows `Standby`.	The selected device becomes armed.
Step 4	Raise only the right hand: open palm for ON, fist for OFF.	Only the selected device changes state.

Key rule:

For device-level control, do not directly change from right hand device number to right open palm while the left hand is still visible.
Always put both hands down first and wait for Standby.

23. Device-Level Gesture Commands for Room 1 / Bedroom

Desired action	Gesture sequence	ROS command target	Expected GUI effect
Turn on Room 1 light	Left hand 1 + right hand 1 -> both hands down / Standby -> right open palm	`room_1`, `light`, `on`	Room 1 light turns on
Turn off Room 1 light	Left hand 1 + right hand 1 -> both hands down / Standby -> right fist	`room_1`, `light`, `off`	Room 1 light turns off
Turn on Room 1 fan	Left hand 1 + right hand 2 -> both hands down / Standby -> right open palm	`room_1`, `fan`, `on`	Room 1 fan turns on
Turn off Room 1 fan	Left hand 1 + right hand 2 -> both hands down / Standby -> right fist	`room_1`, `fan`, `off`	Room 1 fan turns off
Turn on Room 1 air conditioner	Left hand 1 + right hand 3 -> both hands down / Standby -> right open palm	`room_1`, `ac`, `on`	Room 1 AC turns on
Turn off Room 1 air conditioner	Left hand 1 + right hand 3 -> both hands down / Standby -> right fist	`room_1`, `ac`, `off`	Room 1 AC turns off
Open Room 1 door	Left hand 1 + right hand 4 -> both hands down / Standby -> right open palm	`room_1`, `door`, `on`	Room 1 door opens
Close Room 1 door	Left hand 1 + right hand 4 -> both hands down / Standby -> right fist	`room_1`, `door`, `off`	Room 1 door closes

24. Device-Level Gesture Commands for Room 2 / Living Room

Desired action	Gesture sequence	ROS command target	Expected GUI effect
Turn on Room 2 light	Left hand 2 + right hand 1 -> both hands down / Standby -> right open palm	`room_2`, `light`, `on`	Room 2 light turns on
Turn off Room 2 light	Left hand 2 + right hand 1 -> both hands down / Standby -> right fist	`room_2`, `light`, `off`	Room 2 light turns off
Turn on Room 2 fan	Left hand 2 + right hand 2 -> both hands down / Standby -> right open palm	`room_2`, `fan`, `on`	Room 2 fan turns on
Turn off Room 2 fan	Left hand 2 + right hand 2 -> both hands down / Standby -> right fist	`room_2`, `fan`, `off`	Room 2 fan turns off
Open Room 2 curtain	Left hand 2 + right hand 3 -> both hands down / Standby -> right open palm	`room_2`, `curtain`, `on`	Room 2 curtain opens
Close Room 2 curtain	Left hand 2 + right hand 3 -> both hands down / Standby -> right fist	`room_2`, `curtain`, `off`	Room 2 curtain closes
Open Room 2 door	Left hand 2 + right hand 4 -> both hands down / Standby -> right open palm	`room_2`, `door`, `on`	Room 2 door opens
Close Room 2 door	Left hand 2 + right hand 4 -> both hands down / Standby -> right fist	`room_2`, `door`, `off`	Room 2 door closes

25. Device-Level Gesture Commands for Room 3 / Toilet

Desired action	Gesture sequence	ROS command target	Expected GUI effect
Turn on Room 3 light	Left hand 3 + right hand 1 -> both hands down / Standby -> right open palm	`room_3`, `light`, `on`	Room 3 light turns on
Turn off Room 3 light	Left hand 3 + right hand 1 -> both hands down / Standby -> right fist	`room_3`, `light`, `off`	Room 3 light turns off
Turn on Room 3 exhaust fan	Left hand 3 + right hand 2 -> both hands down / Standby -> right open palm	`room_3`, `exhaust`, `on`	Room 3 exhaust fan turns on
Turn off Room 3 exhaust fan	Left hand 3 + right hand 2 -> both hands down / Standby -> right fist	`room_3`, `exhaust`, `off`	Room 3 exhaust fan turns off
Turn on Room 3 water heater	Left hand 3 + right hand 3 -> both hands down / Standby -> right open palm	`room_3`, `heater`, `on`	Room 3 water heater turns on
Turn off Room 3 water heater	Left hand 3 + right hand 3 -> both hands down / Standby -> right fist	`room_3`, `heater`, `off`	Room 3 water heater turns off
Open Room 3 door	Left hand 3 + right hand 4 -> both hands down / Standby -> right open palm	`room_3`, `door`, `on`	Room 3 door opens
Close Room 3 door	Left hand 3 + right hand 4 -> both hands down / Standby -> right fist	`room_3`, `door`, `off`	Room 3 door closes

26. Keyboard Gesture Testing Table

Use this when testing without a real camera.

Launch keyboard demo:

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

roslaunch local_home_assistant home_assistant_keyboard_demo.launch \
  enable_bell:=true \
  enable_device_sfx:=true \
  command_cooldown_sec:=1.5 \
  device_action_timeout_sec:=8.0

Then type the keys in the gesture> prompt.

Keyboard sequence	Meaning	Expected command
`1` then `o`	Room 1 all ON	`room_1 master_switch on`
`1` then `f`	Room 1 all OFF	`room_1 master_switch off`
`2` then `o`	Room 2 all ON	`room_2 master_switch on`
`2` then `f`	Room 2 all OFF	`room_2 master_switch off`
`3` then `o`	Room 3 all ON	`room_3 master_switch on`
`3` then `f`	Room 3 all OFF	`room_3 master_switch off`
`1` then `r1` then `s` then `o`	Room 1 light ON	`room_1 light on`
`1` then `r1` then `s` then `f`	Room 1 light OFF	`room_1 light off`
`1` then `r2` then `s` then `o`	Room 1 fan ON	`room_1 fan on`
`1` then `r3` then `s` then `o`	Room 1 AC ON	`room_1 ac on`
`1` then `r4` then `s` then `o`	Room 1 door OPEN	`room_1 door on`
`2` then `r3` then `s` then `o`	Room 2 curtain OPEN	`room_2 curtain on`
`2` then `r4` then `s` then `f`	Room 2 door CLOSE	`room_2 door off`
`3` then `r2` then `s` then `o`	Room 3 exhaust ON	`room_3 exhaust on`
`3` then `r3` then `s` then `o`	Room 3 heater ON	`room_3 heater on`
`3` then `r4` then `s` then `f`	Room 3 door CLOSE	`room_3 door off`

Keyboard symbols:

Key	Simulated gesture
`1`, `2`, `3`	left hand 1/2/3 room selection
`r1`, `r2`, `r3`, `r4`	right hand 1/2/3/4 device selection
`s`	standby / both hands down
`o`	right open palm / ON
`f`	right fist / OFF
`q`	quit keyboard gesture input

27. Common Gesture Mistakes

Mistake	What happens	Correct action
Left hand and right hand overlap	MediaPipe may confuse left/right hands	Keep hands separated in the camera frame
Right hand 4 is used as ON	The system selects device 4 instead of turning ON	Use a clear five-finger open palm for ON
Device number directly changes to open palm while left hand remains visible	It may conflict with room-level master switch control	Put both hands down and wait for Standby first
Fist still shows one finger	It may be detected as `right_1`	Close the fist clearly
Open palm hides the thumb	It may be detected as `right_4`	Spread all five fingers including thumb
Room selection times out	No command is triggered	Repeat the room selection and continue faster
No blue box appears after device selection	Device was not selected	Hold right-hand device number longer or check `/home_assistant/gesture_raw`

Part E. Supported Voice Commands

28. Room-Level Voice Commands

turn on room one
turn off room one
turn on room two
turn off room two
turn on room three
turn off room three
turn on bedroom
turn off bedroom
turn on living room
turn off living room
turn on toilet
turn off toilet

29. Device-Level Voice Commands

Room 1 examples:

turn on the light in room one
turn off the light in room one
turn on the fan in room one
turn off the fan in room one
turn on the ac in room one
turn off the ac in room one
open the door in room one
close the door in room one

Room 2 examples:

turn on the light in room two
turn off the light in room two
turn on the fan in room two
turn off the fan in room two
turn on the curtain in room two
turn off the curtain in room two
open the curtain in room two
close the curtain in room two
open the door in room two
close the door in room two

Room 3 examples:

turn on the light in room three
turn off the light in room three
turn on the exhaust in room three
turn off the exhaust in room three
turn on the heater in room three
turn off the heater in room three
open the door in room three
close the door in room three

30. Multi-Room Voice Commands

turn on room one and room two
turn off room one and room two
turn on room one and room three
turn off room two and room three
turn on bedroom and living room
turn off living room and toilet

31. Multi-Device Voice Commands

turn on the light and fan in room one
turn off the light and fan in room one
turn on the light and ac in room one
turn off the heater and exhaust in room three
turn on the light in room one and the curtain in room two
turn off the fan in room one and the curtain in room two
turn on the light in bedroom and the curtain in living room
turn on the door in room one and the door in room two

32. Global All-Room Voice Commands

These commands control all rooms at once.

Turn everything off:

turn off all rooms
turn off all the rooms
switch off all rooms
power off all rooms
turn off every room
switch off every room
turn off the whole house
turn off the entire house
turn off everything
shut down all rooms
disable all rooms

Turn everything on:

turn on all rooms
turn on all the rooms
switch on all rooms
power on all rooms
turn on every room
turn on the whole house
turn on everything
enable all rooms

Example:

turn off all rooms

is converted internally into:

room_1 master_switch off
room_2 master_switch off
room_3 master_switch off

Part F. Testing Without Real Devices

33. Test Voice Parser Without Microphone

Use this when the microphone is not ready.

Terminal 1: start the main system.

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

roslaunch local_home_assistant home_assistant_camera_mediapipe.launch \
  camera_source:=0 \
  enable_bell:=true \
  enable_device_sfx:=true

Terminal 2: start voice parser.

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

rosrun local_home_assistant voice_command_node.py

Terminal 3: manually publish recognized text.

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash

rostopic pub /recognizer/output std_msgs/String "turn on the light in room one" -1

Test global all-room command:

rostopic pub /recognizer/output std_msgs/String "turn off all rooms" -1

Test door command:

rostopic pub /recognizer/output std_msgs/String "open the door in room one" -1

34. Test Keyboard Gesture Without Camera

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

roslaunch local_home_assistant home_assistant_keyboard_demo.launch \
  enable_bell:=true \
  enable_device_sfx:=true \
  command_cooldown_sec:=1.5 \
  device_action_timeout_sec:=8.0

Then use examples from Section 26.

Part G. Debug Commands

35. Check Active ROS Nodes

rosnode list

Expected nodes in the final run may include:

/audio_feedback_node
/camera_stream_node
/gesture_interpreter
/home_state_manager
/home_visualizer_node
/mediapipe_gesture_node
/voice_command_node
/vosk_asr_arecord_node

36. Check Important ROS Topics

rostopic list

Important topics:

/recognizer/output
/home_assistant/gesture_raw
/home_assistant/gesture_debug
/home_assistant/gesture_selection
/home_assistant/command
/home_assistant/state
/usb_cam/image_raw

37. Echo Recognized Voice Text

rostopic echo /recognizer/output

38. Echo Raw Gesture Output

rostopic echo /home_assistant/gesture_raw

Expected raw gesture labels include:

left_1
left_2
left_3
right_fist
right_1
right_2
right_3
right_4
right_open
standby

39. Echo Final Commands

rostopic echo /home_assistant/command

This is the most important debug topic. Both voice and gesture should eventually publish to this topic.

Expected examples:

{"source":"gesture_room_master","room":"room_1","target":"master_switch","action":"on"}
{"source":"gesture_device_standby","room":"room_1","target":"door","action":"on"}
{"source":"voice","room":"room_2","target":"curtain","action":"off"}

40. Echo Gesture Debug

rostopic echo /home_assistant/gesture_debug

Useful messages include:

selected_room=room_1
device_selected_waiting_standby
Standby detected
published ...

41. Echo Home State

rostopic echo /home_assistant/state

42. Show ROS Graph

For report or demo evidence:

rqt_graph

Take a screenshot and explain the node-topic relationship in the report.

Part H. Common Problems and Fixes

43. Duplicate Package Error

Error example:

RuntimeError: Multiple packages found with the same name "local_home_assistant"

Cause: a backup folder was created inside ~/catkin_ws/src, for example:

local_home_assistant_backup_20260526_202909

Fix:

cd ~/catkin_ws/src
mkdir -p ~/catkin_ws_backups
mv local_home_assistant_backup_* ~/catkin_ws_backups/ 2>/dev/null || true

Then rebuild:

cd ~/catkin_ws
source /opt/ros/noetic/setup.bash
catkin_make --pkg local_home_assistant
source ~/catkin_ws/devel/setup.bash

44. Full `catkin_make` Fails in Unrelated Packages

If full catkin_make fails in packages such as TurtleBot3, build only this package:

cd ~/catkin_ws
catkin_make --pkg local_home_assistant

If the error is:

AttributeError: module 'importlib_metadata' has no attribute 'EntryPoints'

repair the Python environment:

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate
python -m pip install --upgrade "setuptools<70" "importlib_metadata>=4.13"

45. Pygame GUI Does Not Open

Check display:

echo $DISPLAY

If running directly on Jupiter with monitor, use the robot local graphical desktop. If running over SSH, enable X11 forwarding or use the robot local display.

46. Camera Does Not Open

Test camera with OpenCV first:

source ~/venvs/ROS_HOME_ASSISTANT/bin/activate

python - <<'PY'
import cv2
cap = cv2.VideoCapture(0)
print("opened:", cap.isOpened())
ok, frame = cap.read()
print("read:", ok, "shape:", None if frame is None else frame.shape)
cap.release()
PY

Try:

camera_source:=1

or use Android IP camera:

camera_source:=http://YOUR_PHONE_IP:8080/video

47. Left and Right Hands Are Reversed

Use:

roslaunch local_home_assistant home_assistant_camera_mediapipe.launch \
  camera_source:=0 \
  swap_hands:=true

If using Android IP camera:

roslaunch local_home_assistant home_assistant_camera_mediapipe.launch \
  camera_source:=http://192.168.0.205:8080/video \
  swap_hands:=true

48. Right Hand 1 Is Detected as Fist

The correct rule should be:

right hand 0 fingers  -> right_fist
right hand 1 finger   -> right_1
right hand 2 fingers  -> right_2
right hand 3 fingers  -> right_3
right hand 4 fingers  -> right_4
right hand 5 fingers  -> right_open

Check raw gesture output:

rostopic echo /home_assistant/gesture_raw

When showing right hand 1, expected:

right_1

not:

right_fist

49. Right Open Palm Is Detected as Right Hand 4

Cause: the thumb may not be clearly visible, so MediaPipe counts four fingers instead of five.

Fix:

Spread all five fingers clearly.
Make the thumb visible.
Do not hide the thumb behind the palm.

For ON action, the system expects:

right_open

not:

right_4

50. Device Gesture Accidentally Opens Whole Room

Use the Standby rule.

Correct device-level gesture:

left hand room number + right hand device number
put both hands down until Standby
raise only right hand open/fist

Do not directly change from:

left hand 1 + right hand 1

to:

left hand 1 + right open

because that may conflict with the room-level master switch gesture.

51. Vosk Cuts Off Long Sentences

Use the improved continuous ASR node with phrase buffering:

rosrun local_home_assistant vosk_asr_arecord_node.py \
  _audio_device:=default \
  _sample_rate:=16000 \
  _use_grammar:=true \
  _phrase_publish_delay_sec:=1.25 \
  _max_phrase_buffer_sec:=7.0

If it still publishes too early, increase the delay:

_phrase_publish_delay_sec:=1.8
_max_phrase_buffer_sec:=8.0

52. Words Such as Curtain/Fan/Light Become `[unk]`

Make sure Vosk is running with grammar enabled:

_use_grammar:=true

Speak clearly and use supported phrases such as:

turn on the curtain in room two
turn on the fan in room one
turn on the light in room one
open the door in room one

53. Voice Recognition Works but GUI Does Not Change

Check ASR output:

rostopic echo /recognizer/output

Check final command output:

rostopic echo /home_assistant/command

If /recognizer/output has text but /home_assistant/command has no command, make sure Terminal 3 is running:

rosrun local_home_assistant voice_command_node.py

54. No Sound Feedback

Check that audio_feedback_node is running:

rosnode list | grep audio

Manually test one command:

rostopic pub /home_assistant/command std_msgs/String '{"source":"manual_audio_test","room":"room_1","target":"light","action":"on"}' -1

If the terminal logs the command but no sound is heard, test speaker output again:

speaker-test -t sine -f 880 -l 1

and test Pygame audio again with the command in Section 11.2.

Part I. Notes for Report / Demo Recording

55. Recommended 5-Minute Demo Flow

Show the three terminals.
Show Pygame GUI and MediaPipe window.
Show room-level gesture control.
Show device-level gesture control with Standby.
Show door control using right hand 4.
Show voice command for one room.
Show voice command for one device.
Show global command: turn off all rooms.
Show /home_assistant/command briefly or show rqt_graph.

56. Recommended Report Explanation

The project implements a ROS-based local home assistant for the Jupiter robot. It combines:

camera-based gesture recognition
offline speech recognition
rule-based command parsing
ROS topic routing
room/device state management
audio feedback
Pygame visualization

The project focuses on functionality and usefulness rather than unnecessary complexity.

57. Main ROS Topics

Topic	Message Type	Purpose
`/usb_cam/image_raw`	`sensor_msgs/Image`	Camera frames
`/home_assistant/gesture_raw`	`std_msgs/String` JSON	Raw gesture label from MediaPipe
`/home_assistant/gesture_debug`	`std_msgs/String`	Debug explanation for gesture state machine
`/home_assistant/gesture_selection`	`std_msgs/String` JSON	Selected room/device for GUI highlight
`/recognizer/output`	`std_msgs/String`	Recognized speech text from Vosk
`/home_assistant/command`	`std_msgs/String` JSON	Final room/device command from gesture or voice
`/home_assistant/state`	`std_msgs/String` JSON	Current room/device state

58. Main ROS Nodes

Node	Purpose
`camera_stream_node.py`	Reads camera/IP camera and publishes ROS image frames
`mediapipe_gesture_node.py`	Converts hand landmarks into raw gesture labels
`gesture_interpreter_node.py`	Converts gesture labels into room/device commands
`vosk_asr_arecord_node.py`	Converts microphone audio into recognized text
`voice_command_node.py`	Converts recognized text into room/device commands
`state_manager_node.py`	Updates and publishes room/device state
`home_visualizer_node.py`	Shows Pygame 2D room simulator
`audio_feedback_node.py`	Plays command bell and device/room sound effects
`keyboard_gesture_mock_node.py`	Allows gesture testing without a camera

59. License

This project is released under the Apache-2.0 License.

Name		Name	Last commit message	Last commit date
assets/sounds/generated		assets/sounds/generated
config		config
dataset		dataset
launch		launch
models		models
scripts		scripts
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md
package.xml		package.xml

Latest commit

History

Folders and files

Repository files navigation