Hawkeye System

Autonomous Visual Tracking for Gimbal Systems

Hawkeye System is a ROS 2-based software pipeline for autonomous human detection, tracking, and gimbal control on UAV platforms. The system uses deep learning-based perception (YOLOv8 + ByteTrack) coupled with image-based visual servoing (IBVS) to keep a target centered in the camera frame, while simultaneously stabilizing the gimbal against platform disturbances.

The full pipeline was developed and validated inside NVIDIA Isaac Sim (v5.0.0), which provides photorealistic rendering, physics-based simulation, and native ROS 2 integration.

---

System Architecture

The system consists of two main ROS 2 nodes communicating via standard message types:

                        NVIDIA Isaac Sim (v5.0.0)
  ┌─────────────────────────────────────────────────────────┐
  │  UAV Platform ──► Motorized Gimbal ──► RGB Camera       │
  │  (Kinematic)      (Yaw/Pitch/Roll)                      │
  └──────┬──────────────────┬───────────────────┬───────────┘
         │                  │                   │
    /cam_vel           /current_orientation    /rgb
  geometry_msgs/Twist  geometry_msgs/Vector3  sensor_msgs/Image
         ▲                  │                   │
         │                  │                   ▼
  ┌──────┴──────┐    ┌──────┴───────────────────┴───────────┐
  │  Controller │◄───│  Perception Node                     │
  │  Node       │    │  (YOLOv8m + ByteTrack + Target Mgr)  │
  │  (IBVS)    │    │                                       │
  └─────────────┘    └───────────────────────────────────────┘
                           /target_coord
                        geometry_msgs/Point

Perception Node (hawkeye_perception)

• Detection: YOLOv8m, person-only (class 0), confidence threshold 0.35, input resized to 640px width
• Tracking: ByteTrack with custom configuration optimized for gimbal camera motion (sparse optical flow GMC, extended track buffer, low thresholds)
• Target Identity Manager: State machine with TRACKING → RECOVERY → GRACE PERIOD → LOST states, spatial re-association, and manual ID switching via keyboard

Controller Node (hawkeye_control)

• Dual-mode architecture: Tracking mode (visual servoing) and Stabilization mode (hold orientation)
• Pixel-to-angle conversion using pinhole camera model (focal length + aperture parameters)
• PID controllers for yaw, pitch (tracking), and roll (stabilization)
• NPSO-optimized gains: PID parameters tuned via Novel Particle Swarm Optimization with a multi-objective fitness function (ITAE, settling time, band violation, oscillation, overshoot, target loss penalties)
• 1 kHz control loop with sample-and-hold of perception outputs

---

Repository Structure

hawkeye-system/
├── ros2_ws/
│   └── src/
│       ├── hawkeye_perception/          # Perception ROS 2 package
│       │   ├── hawkeye_perception/
│       │   │   ├── __init__.py
│       │   │   ├── detection_node.py    # YOLOv8 + ByteTrack + target management
│       │   │   └── detection_node_test.py
│       │   ├── config/
│       │   │   └── bytetrack.yaml       # ByteTrack tracker configuration
│       │   ├── resource/
│       │   │   └── hawkeye_perception
│       │   ├── package.xml
│       │   ├── setup.py
│       │   └── setup.cfg
│       │
│       └── hawkeye_control/             # Control ROS 2 package
│           ├── hawkeye_control/
│           │   ├── __init__.py
│           │   ├── controller_node.py   # Dual-mode gimbal controller (IBVS + stabilization)
│           │   ├── controller_node_test.py
│           │   └── stabilization_node.py  # Standalone roll stabilizer
│           ├── resource/
│           │   └── hawkeye_control
│           ├── package.xml
│           ├── setup.py
│           └── setup.cfg
│
├── optimization/
│   ├── npso_optimizer.py                # NPSO-PID optimizer (multi-objective)
│   └── run_npso.py                      # CLI runner for optimization
│
├── evaluation/
│   └── performance_analyzer.py          # Real-time performance monitoring & plotting
│
├── results/
│   └── npso-pid/
│       ├── Smoothness_optimization.json # NPSO results (smoothness-focused)
│       └── Tracking_optimization.json   # NPSO results (tracking-focused)
│
├── docs/
│   └── SIMULATION_SETUP.md             # Guide for Isaac Sim scene setup
│
├── .gitignore
├── LICENSE
└── README.md

---

Prerequisites

• ROS 2 (Humble or later)
• Python 3.10+
• NVIDIA Isaac Sim v5.0.0 (for simulation — not included in this repo)
• YOLOv8 weights (yolov8m.pt) — download from Ultralytics

Python Dependencies

pip install ultralytics opencv-python numpy matplotlib

---

Building & Running

1. Build the ROS 2 workspace

cd ros2_ws
colcon build --symlink-install
source install/setup.bash

2. Launch the perception node

ros2 run hawkeye_perception detection_node

3. Launch the controller node

ros2 run hawkeye_control controller_node

4. (Optional) Run the performance analyzer

python3 evaluation/performance_analyzer.py

Note: The perception node expects camera images on the /rgb topic (sensor_msgs/Image) and the controller expects gimbal orientation on /current_orientation (geometry_msgs/Vector3). In simulation, these are published by Isaac Sim's ROS 2 bridge.

---

ROS 2 Topics

Topic	Type	Direction	Description
/rgb	sensor_msgs/Image	Isaac Sim → Perception	Camera frames (~60 Hz)
/target_coord	geometry_msgs/Point	Perception → Controller	Pixel error (x, y) + validity flag (z)
/current_orientation	geometry_msgs/Vector3	Isaac Sim → Controller	Gimbal orientation (pitch, roll, yaw) in degrees
/cam_vel	geometry_msgs/Twist	Controller → Isaac Sim	Angular velocity commands (pitch, yaw, roll) in rad/s
/pid_log	Float64MultiArray	Controller → Analyzer	Controller telemetry for evaluation

---

PID Optimization (NPSO)

The PID gains were tuned using Novel Particle Swarm Optimization. The fitness function combines tracking and stabilization objectives:

Fitness = 0.7 × J_tracking + 0.3 × J_stabilization

To run your own optimization:

cd optimization
python3 run_npso.py

Pre-computed results are available in results/npso-pid/.

---

Simulation Setup (Isaac Sim)

The simulation environment, action graphs, and USD scene files are not included in this repository because they contain proprietary NVIDIA assets and binary scene data. See docs/SIMULATION_SETUP.md for instructions on recreating the simulation environment.

Key simulation components (set up manually in Isaac Sim):

• Full Warehouse environment asset
• UAV + 3-DoF Gimbal (URDF-based, with yaw/pitch/roll joints)
• Animated human characters for tracking targets
• Action Graphs for ROS 2 bridge, camera publishing, joint control, and clock
• Physics scene with appropriate time step settings

---

Performance Summary

Validated in closed-loop simulation (NVIDIA Isaac Sim v5.0.0) with a UAV executing sinusoidal translation (±10 m) and rotation (±30°) disturbances:

Metric	Value
Steady-state tracking error (mean)	20.8 px
Steady-state tracking error (std)	9.2 px
Settling time (after target acquisition)	~2.0 s
Roll stabilization error (mean)	0.31°
Roll disturbance rejection ratio	~60:1 (−35 dB)
Perception throughput	~57.4 FPS
Detection confidence (mean)	0.82
Identity switches (steady-state)	0 / min

---

Contributors

This project was developed as part of a group assignment at THWS (Technical University of Applied Sciences Würzburg-Schweinfurt), under the supervision of Prof. Dr.-Ing. Volker Willert.

Name	Contributions
Ferran Artero	System architecture & overview, perception pipeline (visual tracking)
Mario Lazzarini	State of the art review, control system design & NPSO optimization
Ekaitz Uria	Methodology, simulation environment setup (Isaac Sim)
Roque Ballesteros	Introduction, experimental results & evaluation

Supervisor: Prof. Dr.-Ing. Volker Willert — THWS, Faculty of Electrical Engineering

---

License

MIT License — see LICENSE for details.