Kestrel MK-1

Autonomous vision-guided drone with a ground station, simulated in Webots.

A Webots simulation of a Mavic 2 Pro quadrotor flown by an autonomous perception–and–control stack, paired with a lightweight ground-control station (GCS) for live monitoring and command. The drone perceives its surroundings with monocular depth estimation and open-vocabulary object detection, avoids obstacles reactively, and can search for, lock onto, and hold a named target ("car", "person", …) entirely on its own.

It runs as two processes that talk over a simple file bus, so the heavy perception models never block the 8 ms flight-control loop.

---

Table of contents

1. Features
2. How it works (architecture)
3. Requirements
4. Installation
5. Configuration
6. Running it
7. Using the ground station
8. Autonomous target search & lock
9. Tuning
10. Troubleshooting
11. Project layout

---

Features

• Monocular depth perception — Depth-Anything-V2 (Small) produces a per-frame depth map used for reactive obstacle avoidance.
• Open-vocabulary detection — YOLO-World detects arbitrary classes by text prompt (no retraining); the target class is set live from the GCS.
• Two complementary trackers — an NCC pixel tracker for a stable per-frame lock, plus an IoU multi-target tracker that bridges detector gaps.
• Reactive obstacle avoidance — depth-based "follow-the-gap" steering with roll/yaw/brake/climb response, active even while hovering.
• Autonomous target behaviour — volume search → confirm → halt-and-lock, with a high-altitude orbit-sweep fallback when the target can't be found low.
• Command hypervisor — a small verb queue (takeoff, land, forward, turn, climb, hold, abort, look_for, approach, orbit) with cooperative preemption.
• Optional VLM pilot — a vision-language model can drive the drone toward a natural-language goal, using a rolling clip of recent frames.
• Live ground station — a Tkinter viewer that shows the camera feed with an on-screen overlay (detections, avoidance arrow, altitude/HUD) and a command console.

---

How it works (architecture)

 ┌─────────────────────────────┐         file bus (C:\tmp\ … on Windows)        ┌───────────────────────────┐
 │  Webots  +  controller       │   mavic_view.png   (annotated camera frame)   │  Ground station (viewer)  │
 │  (mavic_avoider.py)          │ ─────────────────────────────────────────────▶│  - live image + HUD       │
 │                              │   mavic_status.json (telemetry, detections)    │  - buttons & console      │
 │  • 8 ms flight-control loop  │                                                │  - optional VLM pilot     │
 │  • PID stabilisation         │◀───────────────────────────────────────────── │                           │
 │  • depth + YOLO worker       │   mavic_commands.jsonl (one command per line)  └───────────────────────────┘
 │    threads (off the loop)    │
 └─────────────────────────────┘

• The controller (controllers/mavic_avoider/mavic_avoider.py) runs inside Webots as an <extern> client. Each 8 ms tick it reads the IMU/GPS/camera, runs the Mavic 2 Pro stabilisation PIDs, merges in obstacle avoidance and the active command's intent, and drives the four rotors. Depth and detection run on background worker threads (drop-old queues) so their latency only lowers the perception rate — it never stalls flight control.
• The ground station (viewer.py) is a separate process. It only reads the shared image/telemetry files and appends command lines — it never imports the controller, so you can restart either side independently.
• The shared files live under /tmp/…, which resolves to C:\tmp\ on Windows. Both processes run on the same machine, so this is just a fast local mailbox.

---

Requirements

• Webots R2025a (the world targets the Mavic 2 PRO proto).
• Python 3.10+ with a virtual environment (venv/).
• A GPU is optional — depth runs on CUDA (fp16) if available, falls back to CPU. Detection (YOLO-World) runs on CPU by design (see note below).
• Model weights yolov8s-world.pt (and yolo11n.pt) are included in the repo. Depth-Anything-V2-Small is fetched from the Hugging Face Hub on first run.
• (Optional) An OpenAI-compatible LLM/VLM server (e.g. LM Studio) for the /ask visual query and the autonomous VLM pilot.

Why YOLO-World runs on CPU: its runtime set_classes() (called whenever the requested target changes) builds the class text-embeddings on CPU and does not move them to the GPU, so on CUDA predict() fails with a device-mismatch error and silently returns zero detections. CPU sidesteps it; the detector thread absorbs the latency, so the flight loop is unaffected.

---

Installation

# from the repo root
python -m venv venv
.\venv\Scripts\python.exe -m pip install --upgrade pip
.\venv\Scripts\python.exe -m pip install -r requirements.txt
# Windows-specific pins (CUDA wheels etc.) if present:
.\venv\Scripts\python.exe -m pip install -r requirements-win.txt

Then copy the example environment file and adjust it:

Copy-Item .env.example .env

---

Configuration

All runtime knobs live in .env (git-ignored). The most important ones:

Variable	Meaning	Default
DEPTH_DEVICE	auto / cuda:0 / cpu for the depth model	auto
YOLO_DEVICE	Keep cpu (see note above)	cpu
LLM_BACKEND	lm-studio (local) or nano-gpt (cloud)	lm-studio
LM_STUDIO_URL	OpenAI-compatible endpoint for the VLM	local 1234
PILOT_*	VLM-pilot cadence / frame-buffer tuning	see file

See .env.example for the full annotated list.

---

Running it

The simulation and the controller start in a specific order, because the controller connects to Webots as an external client.

1. Open the world in Webots. Load worlds/obstacle_avoidance.wbt. The robot's controller field is <extern>, so Webots prints:

INFO: 'Mavic 2 PRO' extern controller: Waiting for local or remote connection on port 1234.

2. Start the controller. In a PowerShell window:

.\run_controller.ps1

This sets the Webots env vars and launches the controller from the venv. It loads the models, connects to Webots, and begins the flight loop. Output is mirrored to controller.log for inspection.

3. Start the ground station.

.\venv\Scripts\python.exe viewer.py

Shortcut: .\start_all.ps1 opens the controller and the viewer in two separate windows for you (Webots must already be running first).

The run_controller.ps1 script exists because the bundled webots-controller.exe does not quote forwarded paths, so a space in the project path (e.g. Kutup TAN) breaks the launch. The script sets WEBOTS_CONTROLLER_URL itself and runs the venv Python directly.

---

Using the ground station

The viewer window shows the live annotated camera feed and a command console at the bottom. You can drive the drone with buttons or by typing commands.

Buttons

Button	Action
Takeoff	Climb to the default altitude and hover
Hold	Hover in place
Land	Descend and land
ABORT	Hard hover; clears the queue (preempts anything running)
Approach	Fly toward the target typed in the target box
Orbit L/R	Circle the target left / right
Find & Orbit	Search for the target, then orbit it

Console commands

Type into the console and press Enter. Verbs accept positional shorthand:

takeoff 2          # climb to 2 m
forward 5 0.7      # go forward 5 s at 0.7 speed
turn 90 0.5        # turn 90° at 0.5 rate
climb 1.5          # change altitude by +1.5 m
altitude 6         # go to absolute 6 m
hold 3             # hover 3 s
land
abort
force <verb> ...   # run a command at force priority (bypasses avoidance)

Target / perception commands:

find car           # search for, then lock onto a "car"  (alias: look car)
classes set car person     # always run the detector for these classes
classes clear              # stop persistent detection
classes list               # print the current class set
/what is in front of me?   # ask the VLM about the current view (needs a VLM server)
pilot fly to the red car   # start the autonomous VLM pilot toward a goal
pilot stop                 # stop the pilot

You can also paste a raw command as JSON, e.g. {"verb": "look_for", "args": {"description": "car", "sweep_alt": 6}}.

---

Autonomous target search & lock

find <thing> (the look_for verb) runs a small state machine:

1. SEARCH — a 3-phase volume sweep: hover and yaw a full revolution (SCAN), re-aim to a fresh heading (TURN), then translate to a new vantage (CRUISE), stepping through altitude bands so it scans a volume, not just one ring.
2. CONFIRM — the instant a matching detection appears, the sweep stops and the drone centres it and verifies it persists, so a one-frame glimpse during a fast yaw isn't lost.
3. LOCK — the drone halts and holds the target centred (gentle, rate-limited yaw; it does not fly in). The lock is preemptible: issue any other command (Orbit, Approach, Hold) and it hands off immediately — no ABORT needed.
4. HIGH-SWEEP fallback — if the low search never finds the target within its timeout, the drone climbs to a high vantage (default 8 m, override with sweep_alt) and flies a banked orbit, scanning a wide area from above, before giving up. Reactive avoidance stays on during the climb/orbit (with a capped bank angle so it can't over-roll).

Robustness built in: detections below a confidence floor and detections that "teleport" far from the current lock are rejected (kills spurious edge false-positives); the NCC tracker is reset if it drifts off the lock; and a short grace window lets a momentarily-lost target be re-acquired before re-searching.

---

Tuning

Most behaviour is controlled by named constants at the top of controllers/mavic_avoider/commands.py and mavic_avoider.py:

• Search sweep: SEARCH_SCAN_RATE, SEARCH_CRUISE_TIME, SEARCH_ALT_BANDS, SEARCH_MIN_ALT.
• Lock-on: LOCK_YAW_GAIN, LOCK_YAW_MAX, LOCK_CENTER_DEADBAND, LOCK_MIN_CONF, LOCK_MAX_JUMP.
• High sweep: HIGH_SWEEP_ALT, HIGH_SWEEP_YAW, HIGH_SWEEP_FWD, HIGH_SWEEP_TIMEOUT, HIGH_SWEEP_AVOID_AUTH.
• Avoidance: NEAR_THRESHOLD, AVOID_ROLL_CLAMP, AVOID_SMOOTH.
• Detector tracker: DetectionTracker(miss_budget=…) (in control ticks).

---

Troubleshooting

• No detection boxes appear. Make sure YOLO_DEVICE=cpu in .env, the target is reasonably framed (a whole-object silhouette, not an extreme close-up), and you've issued find <thing> or classes set <thing>. Check controller.log.
• Controller won't connect. Webots must show the "Waiting for connection on port 1234" message before you run run_controller.ps1.
• Nothing updates / stale image. The controller probably stopped — check its PowerShell window / controller.log.
• Drone drifts after a search. There is no GPS position-hold (real-FPV style); when it strafes away from something it stays at the new spot.

---

Project layout

.
├── controllers/mavic_avoider/
│   ├── mavic_avoider.py   # main controller: flight loop, PIDs, worker threads
│   ├── commands.py        # hypervisor + verb executors (search/lock/orbit/…)
│   ├── detector.py        # IoU multi-target tracker
│   ├── mission.txt        # optional startup commands (one JSON per line)
│   └── *.pt               # model weights
├── models.py              # loads Depth-Anything-V2 + YOLO-World
├── target_tracker.py      # NCC pixel tracker
├── viewer.py              # ground-station GUI + command console
├── vlm_pilot.py           # autonomous vision-language pilot
├── llm_backend.py         # LM Studio / nano-gpt client
├── worlds/obstacle_avoidance.wbt   # Webots world
├── run_controller.ps1     # launch the controller (Windows)
├── start_all.ps1          # launch controller + viewer
└── requirements*.txt