A real-time vision and control system for UAV detection, tracking, active camera alignment, and 3D state estimation using a monocular camera mounted on a two-axis gimbal.
The project combines deep-learning-based object detection, optical-flow tracking, Kalman filtering, geometric triangulation, and embedded gimbal control in one closed-loop engineering system.
Portfolio note: This was a collaborative university project developed by Faris Dautović and Ajla Midžić. Both authors contributed equally to the system design, implementation, experiments, evaluation, and technical report.
- Real-time UAV detection using YOLOv8
- Inter-frame tracking with Lucas–Kanade optical flow
- Position and velocity estimation using a Kalman filter
- Active target centering using a PD-controlled two-axis gimbal
- Arduino-based motor actuation through PWM
- Monocular 3D position estimation using camera motion and triangulation
- Validation against motion-capture ground truth
- Integration of computer vision, estimation, control, and embedded hardware
flowchart LR
A[USB Camera] --> B[Video Frame]
B --> C[YOLOv8 UAV Detection]
C --> D[Bounding Box and Target Center]
B --> E[Lucas-Kanade Optical Flow]
D --> E
E --> F[Position and Velocity Measurements]
F --> G[Kalman Filter]
G --> H[Estimated UAV State]
H --> I[Pixel Error Relative to Image Center]
I --> J[PD Gimbal Controller]
J --> K[Arduino PWM Commands]
K --> L[Two-Axis Gimbal]
L --> A
H --> M[Camera Pose and Image Measurements]
M --> N[Temporal Median Filtering]
N --> O[Triangulation]
O --> P[Estimated 3D UAV Position]
The detector provides periodic target corrections, while optical flow maintains tracking between detections. The Kalman filter smooths the measured image-plane position and velocity. The estimated target position is converted into a centering error, which is used by the PD controller to rotate the gimbal and keep the UAV near the image center.
| Area | Technologies |
|---|---|
| Programming | Python, C++, MATLAB, Arduino C/C++ |
| Computer Vision | OpenCV, YOLOv8, Lucas–Kanade optical flow |
| State Estimation | Kalman filtering, temporal median filtering |
| 3D Geometry | Camera calibration, coordinate transformations, triangulation |
| Control | PD control, image-based error regulation |
| Embedded Systems | Arduino Uno, PWM, servo/gimbal control |
| Evaluation | Motion-capture ground truth, RMSE analysis |
drone-tracking-system/
├── Arduino/ # Embedded gimbal-control implementation
├── Data/ # Experimental and evaluation data
├── Matlab/ # Analysis, plotting, and validation scripts
├── Python/ # Detection, tracking, and state-estimation pipeline
├── docs/
│ └── UAV_Detection_and_State_Estimation_Report.pdf
└── README.md
YOLOv8 detects the UAV and returns a bounding box and confidence score. To balance detection accuracy with CPU execution time, the detector is not required to run on every frame.
Lucas–Kanade sparse optical flow tracks image features between detector updates. This helps maintain continuous target tracking when detection is temporarily unavailable or computationally too expensive to execute on every frame.
The filter estimates the image-plane state
x = [horizontal position, vertical position,
horizontal velocity, vertical velocity]
by combining optical-flow measurements with a constant-velocity process model. This reduces noise and provides smoother target motion estimates.
The estimated UAV center is compared with the image center. Two independent PD control loops generate roll and pitch commands for the gimbal.
A dead zone is used around the image center to avoid unnecessary high-frequency micro-adjustments caused by small pixel errors and mechanical limitations.
The system estimates the UAV position from multiple camera views created by gimbal motion. Median filtering reduces the effect of outliers in image measurements and orientation data before triangulation is performed using the calibrated camera model.
The system was evaluated indoors using live video from the gimballed camera. Experiments included low-clutter and moderately cluttered scenes, with motion-capture measurements used as ground truth for selected test sequences.
| Metric | Result |
|---|---|
| Mean centering error | 63.41 px |
| Position RMSE — X | 0.4411 m |
| Position RMSE — Y | 1.0127 m |
| Position RMSE — Z | 0.9657 m |
| Velocity RMSE — X | 0.2886 m/s |
| Velocity RMSE — Y | 0.5024 m/s |
| Velocity RMSE — Z | 0.6786 m/s |
The system maintained stable visual tracking and captured the general UAV motion. The largest estimation errors occurred in depth, which is expected because monocular triangulation is sensitive to camera-pose uncertainty and limited viewpoint changes.
- Maintaining tracking during temporary detector failures
- Balancing detection frequency and real-time performance
- Reducing noisy optical-flow and orientation measurements
- Integrating vision software with physical gimbal actuation
- Limiting control oscillation and mechanical micro-adjustments
- Estimating depth from a moving monocular camera
- Validating estimated motion against independent ground truth
- AI model integration in an engineering workflow
- Real-time computer vision
- State estimation and sensor-data filtering
- Feedback-control implementation
- Embedded communication and motor actuation
- Camera geometry and coordinate transformations
- System integration, testing, and debugging
- Quantitative evaluation and technical documentation
- Monocular depth estimation is sensitive to camera calibration and gimbal-orientation errors.
- Detection may become less reliable during occlusion or against cluttered backgrounds.
- CPU-based execution limits how frequently YOLOv8 can be evaluated.
- Rapid motion and motion blur can reduce optical-flow accuracy.
- Mechanical backlash and limited gimbal response affect centering precision.
- Stereo or depth-camera integration for more accurate 3D estimation
- Embedded GPU deployment for higher detection frequency
- Improved gimbal-angle sensing and calibration
- More robust confidence-aware sensor fusion
- Advanced temporal filtering and outlier rejection
- Model-predictive control for smoother gimbal motion
- Testing outdoors and under more challenging lighting conditions
The complete report includes the system model, algorithms, control design, experimental setup, evaluation metrics, results, and discussion:
View the full technical report
Faris Dautović
Ajla Midžić
Department of Automatic Control and Electronics
Faculty of Electrical Engineering, University of Sarajevo
Both authors contributed equally to the conception, design, implementation, experiments, data analysis, interpretation of results, and preparation of the technical report.
This repository is maintained as Ajla Midžić's portfolio copy of the collaborative project.
Original collaborative repository:
fdautovic1/drone-tracking-system