Haltere

A connectome-constrained fruit-fly brain controlling an FPV drone in Liftoff. The goal is a system that can complete unfamiliar races and freestyle tasks. Visual pilot assistance and learned navigation are welcome when they improve that goal; the fly brain remains a meaningful motor controller.

Quickstart · Current status · Models · Game setup · Training · For agents and contributors

fast-brain-08 on Straw Bale, hilltop through the descending checkpoints, at 4× speed: connectome activity (left) beside gameplay (right). The brain drives throttle, roll and pitch; the disclosed fast race-cue pilot supplies the velocity goal and yaw. It finished the full three-lap race in 5:17.898 and 5:17.805 on this seen course, without the side-to-side weave on the downhill; it has not finished Minus Two or Pine Valley. Release, videos and limits.

Current status

Reviewed against code and flight evidence on 2026-09-25. Haltere can finish specific seen races with assistance, but reliable, fast general race and freestyle flight remains unsolved. The newest published weights are fast-brain-08, an experimental download: with brain motors under the fast race-cue pilot at 6 m/s it finished Straw Bale twice (5:17.898, 5:17.805; fast-brain-07 5:45.792; fast PD 5:02.933 with the earlier 2026-09-23 pilot, not re-flown with the current one) with no yaw weave on the downhill, but has not finished Minus Two or Pine Valley, where obstacles on or beside the line to the checkpoint also stop the PD.

The session checkpoint records the unfinished dense geometry comparison and where to resume. Automatic telemetry monitoring now reports progress and stalls without routine screenshots; automatic game finish recognition remains unfinished.

The current acceptance target is three clean full races on each of five tracks, within 20% of the user's matching full-race time, on one frozen stack using the same original [Copy] New Drone:

Three-lap race	User's full-race time	Maximum target time
Straw Bale / Field Day	1:19.006	1:34.807
Pine Valley / Forest For The Trees	2:07.049	2:32.459
Minus Two / Turn Signals	1:29.277	1:47.132
Autumn Fields / Walk In The Park	1:04.494	1:17.392
Hangar C03 / Shipments	1:24.076	1:40.891

These are saved race, not single-lap, times. Exact IDs, values and provenance are in main_track_targets.json. They are scoring metadata and are never supplied to the flight controller.

An earlier frozen full-race comparison used motor10 and a corrected PD baseline under the same visual pilot at 2.5 m/s: brain 0/2 finishes; PD 1/2. PD finished Straw Bale in 13:04.047; both motors hit a pillar on Minus Two. There were no camera or control-deadline stops. All four untrimmed standard brain/gameplay videos fully decode; PD footage explicitly labels the brain as running in shadow. This diagnostic supports geometry as the next priority and also exposes a motor-tracking gap.

Component	Current evidence and limits
fast-brain-11 + obstacle stack (development)	First braking brain motor to fly a full Straw Bale lap: lap 1 in 1:42.988 (fast-brain-08 1:46) with one audited downhill contact and brain-08-level smoothness; not selected by its frozen gates and no race finish (on Straw: a start-arch leg after a false marker reading, an arch leg, and an arch's top bar; the Minus Two hairpin). Round-6 update: with the newer pilot stack it became the first brain through the Minus Two hairpin and flew the Straw downhill without ground contact (still no finish). Release.
fast-brain-10b / fast-brain-09b + obstacle stack (development)	Braking readouts distilled with synthetic governor speed caps; not selected by their frozen gates and no race finish. Minus Two only: both cleared pillar A and braked at the arches; fast-brain-10b is as smooth as brain-08 (0.0031 per tick) but missed the caps at the hairpin during a false terrain climb and hit its wall; fast-brain-09b was stopped short of that wall by the turn-first rule, then sank to the floor accelerating out. Release.
fast-brain-08 + fast race-cue pilot	Re-distilled under the downhill-fixed pilot (vertical goal x 0.4 s, lower ridge, smoothing): Straw Bale 5:17.898 and 5:17.805; downhill fast-yaw time 3-5% (brain-07 31-32%), smoother stick commands, about 10% higher roll/pitch body rates, slower and slightly rougher on sharp switches. Pilot also recognises slope contact and searches gently. Minus Two pillar and Pine Valley impacts. Release.
fast-brain-07 + fast race-cue pilot	Re-distilled under arc turns: Straw Bale 5:45.792 and 5:46.846; 50% more speed through 45-75° checkpoint switches and 25% less stick chatter than brain-06. Minus Two pillar and Pine Valley impacts. Release.
fast-brain-06 + fast race-cue pilot	Brain motors at 6 m/s: Straw Bale 5:50.204 and 5:50.489 (2/2 after a telemetry-guard fix); Minus Two pillar and Pine Valley terrain crashes. Only readout rows 0-2 changed; scene currents blanked. Release.
Fast PD + fast race-cue pilot	Baseline with the 2026-09-23 pilot, brain in shadow: Straw Bale 5:02.933 (not flown with the downhill-fixed pilot; with arc turns it hit an arch in lap 3); Minus Two and Pine Valley obstacle crashes. --looming-brake climbed the Pine mound before a boulder stopped it. Flight card.
Scene09 + visible race cues	Earlier full three-lap finishes: Straw Bale 14:05.703, Minus Two 9:27.415. Both are seen courses.
Motor10 candidate05	Only three motor-readout rows/biases changed; recurrent weights are unchanged. Its release batch finished 0/3; a separate open development-loop repeat finished in 3:01.576. Later diagnostic results are retained separately.
Full Rabbit visual assistance	Gate selection, target smoothing, speed and heading run with the visual brain. Its frozen five-attempt baseline completed no laps. Guide.
Visible race cues	Causal checkpoint-marker guidance is disclosed. It supplies a direction, not free space or a freestyle objective. Inputs.
Navigation predictor	Runtime use is allowed. No current flight runner loads the motion-forecasting predictor; it lacks task-directed flight evidence. Direction.
Generalization	Five first-exposure courses failed across successive development revisions. A completed open generated loop does not establish obstacle avoidance or unseen racing.

The development program uses generated and Workshop courses, frozen comparisons, camera-derived geometry, then broader brain training. Manual map design is unnecessary. The obstacle-section generator and offline scorer include physical gate frames, descents and occlusions. The first box-calibration autonomous flight hit the last wall. A separate oracle-guided PD collection finished that course in 3:35.704, qualifying its playability; this is not an autonomous result. A causal image/motion geometry prototype detects the wall in the failed-flight replay. On the complete collection trajectory it produced no warnings, with 6.7% mean relative depth error on matched primitive surfaces, but coverage remains sparse. The latest frozen geometry batch finished 2/3 on this known development course, in 2:31.958 and 6:51.857. The third attempt hit the final wall after descending through a gap in the observed geometry. All three complete standard brain/gameplay videos decode. PD controlled the motors and the brain ran in shadow; no neural weights changed. Earlier batches and failures remain in the linked flight record. Reliable obstacle avoidance, brain-controlled transfer and the five-track speed target remain unmet. The experimental runner now also accepts geometry guidance with a motor-tracking brain, preserving its throttle/roll/pitch outputs; that integration still needs complete flight evidence before promotion. An optional pretrained dense obstacle input is available for a matched experiment. Its metric scale is unqualified for Liftoff and it is off by default; offline depth checks are not race results. Live comparison and limits. Attempts, diagnostic and limits. Course geometry is excluded from autonomous runtime; oracle collection is explicitly labelled and scored separately.

The dynamics measurement mode adds bounded input pulses and measured recovery for calibrating the original drone before faster flight training. It is a separate PD calibration tool, not a brain flight result. The original drone has completed all 12 throttle pulses and, after correcting an initial roll overshoot, all 72 revised angular pulses with stable recovery. The frozen angular fit also passed an independent 72-pulse batch at different amplitudes. The experimental simulator and flight-cost trainer retain explicit limits on unvalidated high-speed translation; no new racing result is implied. The first flight-cost checkpoint regressed and was rejected; the trainer now selects snapshots on complete maneuvers and checks a separate test seed. The second checkpoint improved simulated cost but was slower and less precise in Liftoff. A generic launch/braking fix let the unchanged motor10 brain finish the generated S-bend motor task in 29.315 s, versus 28.505 s for PD and 39.557 s for the new weights. This is privileged trajectory guidance, not autonomous navigation or a race. All six attempts, including the earlier failed batch, are in the motor comparison videos and evidence. The third training attempt completed that same motor task in 29.929 s, but remained slower and less precise than its unchanged parent. None of these flight-cost checkpoints replaces the published model.

Validation: the last full suite passed 787 tests at 09898a1 (existing PyTorch warnings). The CPU quickstart and checkpoint loading were checked in this checkout, and recorded flights verified actual processed controls. A fresh installation has not been revalidated. See the flight index for historical attempts and project direction for acceptance rules.

Quickstart

The commands below are PowerShell, run from the repository root. Python 3.11+ is declared in pyproject.toml; this checkout was checked with Python 3.13.2 and PyTorch 2.11.0+cu128 on Windows with an RTX 4090. Install uv and Git first.

git clone https://github.com/skulitom/haltere.git
Set-Location haltere
uv venv --python 3.13 .venv
uv pip install --python .venv/Scripts/python.exe "torch==2.11.0" --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv/Scripts/python.exe -e ".[dev,vision]"
.venv/Scripts/python.exe -m haltere.cli --help
.venv/Scripts/python.exe -m haltere.cli eval artifacts/ftSmooth_best.pt --device cpu --batch 1 --steps 30

The last command is a short loading and simulator smoke check and prints JSON metrics. It needs no game, gamepad or raw connectome download. It is too short to assess flight quality. The graph and published inference checkpoints are already tracked in this repository.

PyTorch is installed separately so you can select the build for your machine; use the official installer selector for a different CUDA or CPU setup. Training is designed for an NVIDIA GPU. The live game/capture/controller workflow is Windows-specific; other platforms have not been verified in this update.

All examples use the environment's Python directly, so activation is optional. With that environment activated, haltere is shorthand for python -m haltere.cli. Optional extras are liftoff (virtual pad), fast-capture (Windows DXGI), neuprint, replays and publish. Install the gamepad driver before adding liftoff; see game setup.

Models and data

Checkpoint	Role
fast-brain-11 bundle	Experimental braking motor-readout weights (fast-brain-11-b-cw13) for the obstacle stack on branch m5; one full Straw Bale lap (1:42.988). Development candidate: no race finish, not selected by its frozen gates. Download separately.
fast-brain-10 bundle	Experimental braking motor-readout weights (fast-brain-10b, fast-brain-09b) for the obstacle stack on branch m4. Development candidates: no race finish, not selected by their frozen gates. Download separately.
fast-brain-08 bundle	Experimental fast motor-readout weights distilled under the downhill-fixed pilot (vertical goal x 0.4 s, scene currents blanked). Two Straw Bale finishes (5:17.805); Minus Two and Pine Valley failed. Download separately.
fast-brain-07 bundle	Experimental fast motor-readout weights distilled under the arc-turn pilot (scene currents blanked). Two Straw Bale finishes (5:45.792); Minus Two and Pine Valley failed. Download separately.
fast-brain-06 bundle	Experimental fast motor-readout weights (scene currents blanked) for the fast race-cue pilot at 6 m/s. Two Straw Bale finishes; Minus Two and Pine Valley failed. Download separately.
Motor10 candidate05 bundle	New experimental motor-readout weights with recorded-scene robustness training. One development-loop finish; obstacle races failed. Download separately; not promoted over scene09.
Scene09 bundle	Published learned-scene reference with its exact detector and original-drone mapping. Download separately; evaluated with explicit visual assistance.
artifacts/ftSmooth_best.pt	Published connectome controller for hover and movement patterns.
artifacts/ftPath2_best.pt	Published connectome controller for taught paths and the older visual-pilot stack.
artifacts/ftRobust_best.pt	Earlier controller trained with broad physics randomization.
artifacts/imJ_best.pt	Imitation-trained connectome controller; simulator reference.
artifacts/mlp_baseline.pt	Non-connectome motor teacher/control baseline.
artifacts/gatenet_best.pt	Published gate detector for the older visual stack.
artifacts/gatenet_colourblind.pt	Detector augmentation experiment; not a demonstrated transfer improvement.
artifacts/experimental/navigation_*.pt	Offline path predictors; versions and limits.

Published assets are also available on GitHub Releases and Hugging Face. Inference checkpoints omit optimizer state and depend on the matching data/built/flight.npz, flight.nodes.parquet and flight.meta.json. Use full training checkpoints from a run for train --resume / imitate --resume.

runs/, raw connectome tables and recordings under data/vision/ and data/liftoff/ are local and ignored by Git. Research guides refer to those local experiments; those paths will not exist in a fresh clone. Dataset plans pin particular source takes, not downloadable public training data.

How it works

The checked-in graph selects 30,000 neurons and 2,767,698 directed edges from the male CNS v1.0 connectome. It is a rate-network abstraction of a selected subgraph, not the entire biological fly. Graph selection and transmitter-sign rules are in configs/flight.yaml; actual graph counts are in flight.meta.json.

Telemetry is converted into gyro, load, flow, attitude and other sensory channels. Encoders drive selected fly populations; the recurrent network produces motor activity. Published working brains read from wing motor and premotor populations. Visual variants add image or frozen scene features; the checkpoint specifies the exact sensory encoding and detector calibration.

The deployed arrangement can include a visual pilot or learned planner before the brain's goal inputs, and explicit heading assistance after its output. The assisted visual mode records which commands come from each component. Teacher models and known routes can also supply training labels. Distinguish training-only labels from the declared runtime inputs.

Location	Responsibility
haltere/connectome/	Source tables, population selection and graph construction.
haltere/brain/	Sparse recurrent network, sensory encoders and motor readout.
haltere/sim/	Differentiable quadrotor, rates, PID, mixer and training tasks.
haltere/train/	Motor imitation, flight-cost training, human/gate/scene learning and evaluation.
haltere/vision/	Gate detection, datasets, scene features and path-prediction experiments.
haltere/liftoff/	Telemetry, control mapping, capture, pilots, recording and scoring.
configs/, tests/, docs/	Configurations, automated checks and evidence/workflow guides.

Training

Start motor imitation with the included MLP teacher and the premotor readout:

.venv/Scripts/python.exe -m haltere.cli imitate --config configs/train_premotor.yaml --imitate-config configs/imitate_premotor.yaml --teacher artifacts/mlp_baseline.pt --run runs/imitate-new
.venv/Scripts/python.exe -m haltere.cli eval runs/imitate-new/best.pt --batch 8 --steps 400

The explicit teacher path avoids the configuration's historical runs/mlp300/best.pt dependency. Use a new run directory. This is a full training job, not part of the quickstart. Flight-cost fine-tuning uses haltere train; configs/train_premotor_smooth.yaml and configs/train_path2.yaml describe smoothing and moving-target curricula. Basic configs/train.yaml is an experimental starting configuration, not a recipe that reproduces the shipped brain.

Task	Guide / entry point
Record and prepare human demonstrations	Human recordings.
Train the actual fly brain from recordings	Human, gate and scene learning.
Train/evaluate the separate predictor	Navigation training, v2 evidence.
Collect oracle-guided demonstrations	Route collection, bot-route extraction.
Test unfamiliar courses and tasks	Project acceptance criteria, race protocol.

Training a predictor alone does not update the fly brain. Report the actual changed parameters for every brain-training claim. Evaluate the intended full stack, and use no-predictor/no-assistance comparisons to attribute changes. Preserve raw recordings and whole-take/course holdouts.

To rebuild a graph, download the public tables and write to a new output stem:

.venv/Scripts/python.exe -m haltere.cli fetch
.venv/Scripts/python.exe -m haltere.cli populations
.venv/Scripts/python.exe -m haltere.cli build --out runs/flight-rebuilt

The raw download is approximately 570 MB and needs no login. A rebuilt or modified graph must not silently replace the graph used by an existing model. Optional neuPrint access requires the neuprint extra and NEUPRINT_APPLICATION_CREDENTIALS; haltere neuprint check checks access.

Flying and recording

Follow Liftoff setup for telemetry, driver installation, calibration and the persistent pad bridge. Run the game and its capture/controller processes in Anode, with the viewer hidden unless requested. Retain the original [Copy] New Drone, verify throttle-low and the game's processed controls, pause before disconnecting the pad, and keep Liftoff open between tests. With Anode 0.9.0 or later and HidHide installed, Anode keeps the virtual pad inside its seat, where the user's own games cannot open it, so flights continue while the user plays: the pad asks anode gamepad state and stands its game guard down once Anode reports it seatOnly and verifiedFromDesktop, and the flight preflight lists running games instead of refusing. Otherwise the virtual pad is machine-wide, so every game sees it: it refuses to plug in while another game runs, unplugs itself within about a second when one starts, and the flight preflight refuses too. A game is an install in a Steam, Epic, C:\XboxGames, GOG, EA, Ubisoft, Riot or Battle.net library, a Store game package, a known emulator, or any process outside the pad's session with a controller library (XInput, GameInput, DirectInput) loaded other than Explorer, browsers, Steam and overlays. A Liftoff outside the pad's own session counts.

• Published motor checkpoints use haltere liftoff fly.
• New visual checkpoints use python -m haltere.liftoff.visual_brain and their recorded detector/mapping contract. The legacy fly command rejects these.
• Add --pilot-assistance rabbit for the new assisted visual mode.
• --record PATH.mp4 produces the standard brain-activity panel beside gameplay. Use fresh log/video paths and label shadow footage as shadow.

The flight cards record predictions, complete commands, model provenance and observed outcomes. Verify race completion from the game, not just the pilot's estimated gate count. Scripted patterns and known-route demonstrations are useful tracking tests, but do not establish unseen freestyle or race navigation.

For agents and contributors

Read AGENTS.md and project_direction.md first. Inspect the working tree before editing and preserve unrelated changes. Use the current command help and checkpoint metadata as the implementation reference; historical guides document particular experiments, not universal defaults.

.venv/Scripts/python.exe -m pytest -q
.venv/Scripts/python.exe -m haltere.cli liftoff fly --help
.venv/Scripts/python.exe -m haltere.liftoff.visual_brain --help

For changes to live control, verify the actual processed input and record the whole stack's assistance mode, checkpoint hashes, code revision, camera/mapping and outcome. Publish completed, verified code to GitHub. Publish new weights only with accurate provenance and evaluation limits; offline and synthetic checks alone do not qualify an improved flight controller.

The previous long experiment narrative remains in the historical README at 6e86b8b. Its dates, recommendations and training-only policy are historical.

Attribution

Project code is MIT licensed. The source connectome is the Janelia male CNS dataset, created by Janelia FlyEM, Cambridge Connectomics and Google collaborators and released under CC BY. The included flight graph is derived data; retain source attribution. The full raw tables are downloaded separately.