Skip to content

Add sensory action bar and transparent neural inspector #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bitdeep merged 1 commit into from
Sep 12, 2026
Merged

Add sensory action bar and transparent neural inspector #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,11 @@
Explore the **NETSPHERE** in a local browser, inspect one anatomical flybody animal,
and stimulate a **127,400-neuron FlyWire reference model** while watching measured
neural activity. An experimental **MN9-to-proboscis link** now drives one body
actuator from simulated motor-neuron spikes, with explicit baseline and blocked-link
controls. [Open the development environment](#interactive-environment-in-the-browser)
actuator from simulated motor-neuron spikes. A separate action bar offers sugar,
water, bitter and antennal stimuli, with baseline and blocked-link comparisons.
A transparent inspector keeps measured signals visible and puts explanations in
hover, focus and touch tooltips.
[Open the development environment](#interactive-environment-in-the-browser)
or read the [motor link and its limits](docs/motor-link.md).

The animal is the anatomically detailed [flybody](https://github.com/TuragaLab/flybody) model of *Drosophila melanogaster* (Google DeepMind and HHMI Janelia, *Nature* 2025). In the separate recorded-flight pipeline, its pretrained controller runs on CUDA, MuJoCo Warp integrates the body and wing aerodynamics at 20 kHz, and a geometric navigator steers the fly through the collidable interior. Every frame of those recordings comes from the integrated physical state.
Expand All @@ -25,7 +28,7 @@ The full 60-second take, its metrics and the validation files are attached to th
- It **is** whole-body physics: joints, wings with ellipsoid fluid forces, and the official DMPO flight policy (wingbeat pattern generator plus a residual MLP) driving the actuators.
- It **is** a real 3D world: walls, pillars, ducts, cables and walkways with collision in the same MuJoCo model that integrates the fly. The camera moves through that space.
- The **recorded flight** uses the official MLP policy. The browser development lab
runs the published FlyWire 630 connectome and an experimental sugar-response
runs the published FlyWire 630 connectome and an experimental taste-response
motor link. Its firing-rate-to-servo adapter is engineered; world sensing,
neural walking and neural flight remain unimplemented.
- The navigator is **not** learned vision. It reads the known world geometry and the measured position at 100 Hz and picks turns and climbs with clearance for wings and body. It only changes the reference command; it never writes the animal's pose or velocity.
Expand Down Expand Up @@ -83,13 +86,18 @@ verify_take.py independent checks: duration, decoded frames, clearance, ha
The local browser observatory contains **one physical flybody animal**, selectable
with a close-up orbit camera, plus three city viewpoints and a gravity/contact
experiment. It settles passively between on-demand trials, with no autonomous
walking or flight policy. Open **Neural activity → Stimulate sugar neurons** to
run the entire FlyWire 630 graph (127,400 neurons, 14,687,178 stored connections)
and drive the rostrum from its two MN9 motor neurons. **View proboscis** focuses
the observer on the head. Neural and physical time advance together in this
500 ms experiment. Compare no input or a blocked motor link; the UI retains
measured spikes, drive and joint motion after completion.
The expandable antennal reference assay retains its separate clock and readouts.
walking or flight policy. The always-visible action bar offers **Feed** (sugar),
**Water**, **Bitter** and **Antenna**. Taste trials run the entire FlyWire 630 graph
(127,400 neurons, 14,687,178 stored connections); its MN9 spikes can drive the
rostrum. Feed models feeding initiation, not eating or digestion. Water activates
water-sensing neurons; bitter alone produces neural activity without movement in
this protocol. **Wings** and **Walk** are labeled **Not connected**.
Use **Focus** to inspect the head and **Neural activity** to open the transparent
inspector. Hover, focus or tap its components and numbers for explanations;
point along a graph to inspect measured samples. Actions work with the panel closed.
**Baseline**, **Block link** (or **Block sensory** for Antenna) and **Stop** apply
to the selected protocol. Taste trials share 500 ms of neural and physical time;
the antennal assay retains its separate clock and has no muscle coupling.
Three.js renders measured body poses on demand; native MuJoCo sleep reduces idle
work. The backend is capped at 2 CPUs and 1 GiB. The browser requests
high-performance GPU graphics and targets 30 FPS while moving, with bounded
Expand Down Expand Up @@ -182,7 +190,7 @@ scripts/
dev_environment.py container-owned backend watcher; one dev stack on port 8089
neural_reference.py incremental LIF dynamics over the full FlyWire 630 graph
neural_lab.py bounded on-demand trials and measured activity telemetry
motor_bridge.py shared-clock sugar → FlyWire → MN9 → native rostrum servo
motor_bridge.py shared-clock taste → FlyWire → MN9 → native rostrum servo
validate_motor_bridge.py causal motor controls, actuator isolation and clock checks
fetch_neural_reference.py, prepare_neural_reference.py, validate_neural_reference.py
passive_fly.py cached anatomy attachment and passive initialization
Expand Down
47 changes: 38 additions & 9 deletions docs/browser-environment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,39 @@ The three city view buttons return to known observation points. Touch screens ha
movement buttons and support dragging to look.

The animal settles passively and has **one experimental neural motor link**.
Open **Neural activity → Stimulate sugar neurons** to route calculated MN9 spikes
through an engineered adapter to the native rostrum servo. **View proboscis**
focuses the observer on the head. Compare **Baseline · no input** and **Block motor
link**; **Stop** removes actuator authority. New trials wait for physical rest,
The separate action bar offers **Feed** (sugar), **Water**, **Bitter** and
**Antenna**. Taste trials route calculated MN9 spikes through an engineered adapter
to the native rostrum servo. Feed models feeding initiation, not eating or
digestion; Water stimulates water-sensing neurons. Bitter alone produces neural
activity without movement in this protocol. **Wings** and **Walk** are labeled
**Not connected** and explain the missing motor circuitry.
**Focus** points the observer at the head. Compare **Baseline** and **Block link**
for the selected taste stimulus; **Stop** removes actuator authority.
New motor trials wait for physical rest,
and pause/resume affects both clocks. All other actuators remain disabled.
Anatomical springs, joints, gravity and contact stay active. It does not walk or
fly autonomously. [Motor protocol and causal validation](motor-link.md).

The expandable **Antennal reference assay** retains a separate connectome assay:
**Antenna** retains a separate connectome assay:
antennal stimulus → FlyWire LIF dynamics → neural readouts → disconnected muscles.
Its structure view shows a selection of actual connections with schematic positions;
the entire published graph is retained in the simulation. Counts and voltages come
from calculated activity, and the timeline groups measured spikes in 5 ms bins.
The trial clock is separate from the physical clock. Physical contact, mouse
Its **Block sensory** comparison blocks outgoing sensory connections; **Stop**
cancels the assay without stopping the environment. The trial clock is separate
from the physical clock. Physical contact, mouse
movement and camera direction do not stimulate the brain yet.
[Protocol, preparation and validation](neural-reference.md).
Select it with **Fly 001**, the specimen card, or its marker when visible.

**Neural activity** opens a transparent inspector with **Taste → body** and
**Antenna** tabs. Actions remain outside it, available with the panel closed.
Hover or focus components, numbers and traces for explanations and measured
values; click/tap also opens inspector tooltips. Graph position selects the
nearest 5 ms sample. Escape dismisses the tooltip. Short **Experimental link**
and **Neural only** labels keep the coupling boundary visible.
Transparency uses a flat alpha background without an additional blur pass.

Select the animal with **Fly 001**, the specimen card, or its marker when visible.
Selection and camera movement never command its muscles.
The separate recorded-flight pipeline remains available.
The amber sphere is an explicitly initialized gravity experiment, not an animal.
Expand Down Expand Up @@ -163,9 +179,11 @@ activating the `dev` profile cannot start a flight recording in the background.
The server only serves an allowlist of frontend assets and read endpoints:
`/api/world`, `/api/state`, `/api/events`, `/health`, plus `/api/dev-version` in
development mode. `POST /api/command` accepts bounded drop, pause, speed,
`neural_trial`, `motor_trial` and `motor_stop` commands with same-origin JSON.
`neural_trial`, `neural_stop`, `motor_trial` and `motor_stop` commands with same-origin JSON.
Trial mode must be `stimulus`, `baseline` or `blocked`; only one neural or motor
trial runs at a time. Motor commands cannot select arbitrary neurons, actuator
trial runs at a time. `motor_trial` optionally accepts `stimulus`: exactly
`sugar`, `water` or `bitter`, defaulting to `sugar` for older clients.
Motor commands cannot select arbitrary neurons, actuator
names, gains, torques or durations. It cannot read
arbitrary project files or edit the scene. At most eight event streams are open
at once. The container has a read-only filesystem and limits of 2 CPUs, 1 GiB
Expand All @@ -174,6 +192,12 @@ The separate antennal worker has a duty budget of 0.15 core, runs only on reques
and publishes changes at up to 10 Hz. It adds no continuous scene render loop.
This local server is not intended to be exposed directly to the internet.

`web/actions.js` owns trial commands. `web/neural.js` and `web/motor.js` only
inspect state; they draw changed data while their tab is visible.
`web/inspect.js` manages one shared, event-driven tooltip, including the bounded
sample lookup. Neither the inspector nor the action bar adds an idle animation
loop, and unchanged control state causes no repeated DOM writes.

## Validation

```bash
Expand Down Expand Up @@ -202,6 +226,11 @@ baseline and blocked sensory output, verified their displayed counts, and
inspected the panel on desktop and at 390×844 with no horizontal overflow.
There were no JavaScript exceptions or failed requests; a software graphics
context reset during viewport resizing recovered. The session was then closed.
The subsequent action-bar round exercised all four stimuli, blocked water input's
motor link with the inspector closed, and stopped/restarted an antennal trial.
Desktop and mobile tooltips worked with pointer, click and keyboard focus.
The mobile panel and action bar remained separate with no horizontal overflow.
Detailed counts and causal comparisons are in the [motor guide](motor-link.md#validation).
Page/process live-reload and 30 FPS on the user's hardware remain separate from
those checks.

Expand Down
Loading
Loading