Skip to content

GUIDE.md

Latest commit

 

History

History
184 lines (136 loc) · 7.78 KB

GUIDE.md

File metadata and controls

184 lines (136 loc) · 7.78 KB
Raw

Guide: connect a fruit fly brain to a drone

This is the long version of the README, step by step, from zero to a drone in the air.

Try it in the browser first

Open spikecalls.github.io/FlyDrones. Press S to swing a fly swatter at the drone and see whether the giant fiber reacts in time, click any neuron group in the Brain panel to stimulate it, keys 1-6 are gestures, USE MY HAND turns on webcam hand tracking, CHAIR RUN flies at an obstacle, N switches day and night, C cycles camera views. The browser runs MiniFly with a JavaScript port of the same engine (docs/live/engine.js), checked against the Python package in CI.

0. The idea in one minute

A fly does not have a "flight controller". It has two compound eyes (roughly 750-800 ommatidia each), motion-sensing neurons, and a few hundred descending neurons that carry decisions from the brain down to the wing motor in the thorax.

FlyDrones rebuilds that chain on a computer:

camera frame ─► optic flow per eye cell ─► spikes on T4/T5, LPLC2, LC4, R1-R6
                                             │
                          connectome (signed synapse counts) + LIF neurons
                                             │
      spikes on DNg02 (wing stroke), DNp03 (saccade), DNp01 (giant fiber escape)
                                             │
             firing rate − resting rate ─► throttle, yaw, forward ─► safety ─► drone

The drone's own flight controller keeps it level. The fly brain decides where to go.

1. Install

Python 3.10 or newer.

git clone https://github.com/SpikeCalls/FlyDrones.git
cd FlyDrones
python -m venv .venv
# macOS / Linux
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1

pip install -e ".[vision]"         # core + OpenCV window
pip install -e ".[all,dev]"        # everything: gestures, data, all drones, tests

Check it: flydrones --version and pytest -q.

2. Fly in the simulator

flydrones demo --live

A window opens with four panels: the drone camera with the fly-eye grid, a live spike raster, the room seen from above, and the commands after the safety governor. The demo uses a scripted hand: open palm, fist, move right, rush at the camera, drop.

Save a GIF instead of a window: flydrones demo --record demo.gif. Save a flight log: --log flight.csv.

Other simulator runs:

flydrones swarm --live                                   # 3 drones, 3 copies of one brain
flydrones fly --drone sim --input camera --seconds 30 --live   # no hand, pure optic flow

3. Look inside the brain

flydrones inspect

prints every input and output group, how many neurons it matched, and then stimulates each input group for 500 ms and shows how the descending neurons respond. This is the fastest way to see whether your wiring makes sense. Examples:

  • T4c_L (upward motion, left eye) should raise DNg02 (more lift).
  • LPLC2_L (looming on the left) should fire DNp01_L (giant fiber) and DNp03_L (turn away).

Write your own experiments with examples/01_poke_neurons.py.

4. Load the real connectome

pip install -e ".[data]"
flydrones download malecns          # 3 files, ~1.2 GB, resumable
flydrones build-brain --out data/malecns_brain.npz

build-brain:

  1. reads cell annotations and drops glia / unannotated bodies,
  2. signs every neuron by predicted neurotransmitter (acetylcholine +, GABA / glutamate / histamine −),
  3. streams the 1.1 GB weights table and keeps pairs with ≥ 3 synapses,
  4. matches the groups from defaults.yaml by cell type and side,
  5. saves a compact .npz.

You need ~6 GB of free RAM while building. After that the .npz loads in seconds. Too slow for real time on your machine? Build a sensorimotor core:

flydrones build-brain --core-hops 3 --out data/malecns_core3.npz
flydrones bench --brain data/malecns_core3.npz

Details: CONNECTOME_DATA.md.

5. Calibrate the read-out

The real connectome does not come with labels like "this neuron means climb". Calibration shows the brain seven visual situations (rest, sinking, rising, rotating left/right, looming left/right), records the descending neurons and fits a small linear read-out:

flydrones calibrate --brain data/malecns_brain.npz --out readout_malecns.json

It prints R² for throttle and yaw. Then fly with it:

flydrones demo --config configs/malecns.yaml

Only the read-out is fitted. The connectome is never changed.

If DNg02 is silent or saturated, adjust brain.bias in configs/malecns.yaml (tonic flight drive, in mV) and re-run inspect.

6. Pick a drone

Tello Crazyflie 2.1 + Flow deck ArduPilot / PX4 Betaflight + ESP32
price (approx.) low medium medium-high low-medium
camera for the fly yes (Wi-Fi video) no (use webcam hand) optional no (use webcam hand)
altitude hold yes yes yes no (angle mode only)
difficulty easiest easy medium hardest
indoor safe yes, with guards yes no, outdoors cage / net

Start with Tello if you want the "camera → fly eyes" story, or Crazyflie for the smallest, safest indoor drone. Full setup for each: HARDWARE.md.

7. First real flight (Tello)

  1. Charge the battery, fit prop guards, clear a 3 × 3 m space.
  2. pip install -e ".[tello,gestures]"
  3. Connect your computer to the TELLO-XXXXXX Wi-Fi.
  4. Dry run, nothing is sent: flydrones fly --drone tello --config configs/tello.yaml --input both --live (the dry run does not connect to the drone; it shows the brain and the commands it would send)
  5. Real flight: flydrones fly --drone tello --config configs/tello.yaml --input both --live --send --seconds 60
  6. The drone takes off after the brain warm-up (it measures resting firing rates while still on the ground).
  7. Hand gestures in front of the laptop webcam: open palm → climb, fist → hold, drop hand → descend.
  8. Ctrl+C or q in the window lands. Low battery, 3 minutes, or a brain stall also land or hover.

8. Record a video for social media

  • Screen-record the --live dashboard and film the drone with a phone at the same time. Put them side by side.
  • Or record the dashboard as a GIF: flydrones demo --record out.gif --every 2.
  • Show the numbers honestly: the title bar of the dashboard prints neuron count, connection count, brain name and how fast the brain runs compared to real time.

9. Tuning cheat-sheet

symptom knob (YAML)
drone drifts up / down at rest decoder.settle_s longer, or brain.bias for DNg02
climbs too fast with open palm decoder.axes.throttle.gain lower, safety.max_throttle lower
yaw wobbles decoder.smoothing lower, decoder.axes.yaw.gain lower, imu.yaw_saturation_dps higher
escapes too often decoder.escape.threshold_hz higher, vision.loom_floor higher
never escapes vision.loom_gain higher, inputs.LPLC2_*.max_hz higher
optic flow too noisy (real camera) vision.blur 2, vision.flow_gain higher
brain slower than real time build-brain --core-hops 3, brain.lif.dt 1.0, fewer record_neurons