diff --git a/README.md b/README.md
index d4f8912..32ccfb0 100644
--- a/README.md
+++ b/README.md
@@ -5,8 +5,11 @@
Explore the **NETSPHERE** in a local browser, inspect one anatomical flybody animal,
and stimulate a **127,400-neuron FlyWire reference model** while watching measured
neural activity. An experimental **MN9-to-proboscis link** now drives one body
-actuator from simulated motor-neuron spikes, with explicit baseline and blocked-link
-controls. [Open the development environment](#interactive-environment-in-the-browser)
+actuator from simulated motor-neuron spikes. A separate action bar offers sugar,
+water, bitter and antennal stimuli, with baseline and blocked-link comparisons.
+A transparent inspector keeps measured signals visible and puts explanations in
+hover, focus and touch tooltips.
+[Open the development environment](#interactive-environment-in-the-browser)
or read the [motor link and its limits](docs/motor-link.md).
The animal is the anatomically detailed [flybody](https://github.com/TuragaLab/flybody) model of *Drosophila melanogaster* (Google DeepMind and HHMI Janelia, *Nature* 2025). In the separate recorded-flight pipeline, its pretrained controller runs on CUDA, MuJoCo Warp integrates the body and wing aerodynamics at 20 kHz, and a geometric navigator steers the fly through the collidable interior. Every frame of those recordings comes from the integrated physical state.
@@ -25,7 +28,7 @@ The full 60-second take, its metrics and the validation files are attached to th
- It **is** whole-body physics: joints, wings with ellipsoid fluid forces, and the official DMPO flight policy (wingbeat pattern generator plus a residual MLP) driving the actuators.
- It **is** a real 3D world: walls, pillars, ducts, cables and walkways with collision in the same MuJoCo model that integrates the fly. The camera moves through that space.
- The **recorded flight** uses the official MLP policy. The browser development lab
- runs the published FlyWire 630 connectome and an experimental sugar-response
+ runs the published FlyWire 630 connectome and an experimental taste-response
motor link. Its firing-rate-to-servo adapter is engineered; world sensing,
neural walking and neural flight remain unimplemented.
- The navigator is **not** learned vision. It reads the known world geometry and the measured position at 100 Hz and picks turns and climbs with clearance for wings and body. It only changes the reference command; it never writes the animal's pose or velocity.
@@ -83,13 +86,18 @@ verify_take.py independent checks: duration, decoded frames, clearance, ha
The local browser observatory contains **one physical flybody animal**, selectable
with a close-up orbit camera, plus three city viewpoints and a gravity/contact
experiment. It settles passively between on-demand trials, with no autonomous
-walking or flight policy. Open **Neural activity → Stimulate sugar neurons** to
-run the entire FlyWire 630 graph (127,400 neurons, 14,687,178 stored connections)
-and drive the rostrum from its two MN9 motor neurons. **View proboscis** focuses
-the observer on the head. Neural and physical time advance together in this
-500 ms experiment. Compare no input or a blocked motor link; the UI retains
-measured spikes, drive and joint motion after completion.
-The expandable antennal reference assay retains its separate clock and readouts.
+walking or flight policy. The always-visible action bar offers **Feed** (sugar),
+**Water**, **Bitter** and **Antenna**. Taste trials run the entire FlyWire 630 graph
+(127,400 neurons, 14,687,178 stored connections); its MN9 spikes can drive the
+rostrum. Feed models feeding initiation, not eating or digestion. Water activates
+water-sensing neurons; bitter alone produces neural activity without movement in
+this protocol. **Wings** and **Walk** are labeled **Not connected**.
+Use **Focus** to inspect the head and **Neural activity** to open the transparent
+inspector. Hover, focus or tap its components and numbers for explanations;
+point along a graph to inspect measured samples. Actions work with the panel closed.
+**Baseline**, **Block link** (or **Block sensory** for Antenna) and **Stop** apply
+to the selected protocol. Taste trials share 500 ms of neural and physical time;
+the antennal assay retains its separate clock and has no muscle coupling.
Three.js renders measured body poses on demand; native MuJoCo sleep reduces idle
work. The backend is capped at 2 CPUs and 1 GiB. The browser requests
high-performance GPU graphics and targets 30 FPS while moving, with bounded
@@ -182,7 +190,7 @@ scripts/
dev_environment.py container-owned backend watcher; one dev stack on port 8089
neural_reference.py incremental LIF dynamics over the full FlyWire 630 graph
neural_lab.py bounded on-demand trials and measured activity telemetry
- motor_bridge.py shared-clock sugar → FlyWire → MN9 → native rostrum servo
+ motor_bridge.py shared-clock taste → FlyWire → MN9 → native rostrum servo
validate_motor_bridge.py causal motor controls, actuator isolation and clock checks
fetch_neural_reference.py, prepare_neural_reference.py, validate_neural_reference.py
passive_fly.py cached anatomy attachment and passive initialization
diff --git a/docs/browser-environment.md b/docs/browser-environment.md
index 485f305..ace4c49 100644
--- a/docs/browser-environment.md
+++ b/docs/browser-environment.md
@@ -9,23 +9,39 @@ The three city view buttons return to known observation points. Touch screens ha
movement buttons and support dragging to look.
The animal settles passively and has **one experimental neural motor link**.
-Open **Neural activity → Stimulate sugar neurons** to route calculated MN9 spikes
-through an engineered adapter to the native rostrum servo. **View proboscis**
-focuses the observer on the head. Compare **Baseline · no input** and **Block motor
-link**; **Stop** removes actuator authority. New trials wait for physical rest,
+The separate action bar offers **Feed** (sugar), **Water**, **Bitter** and
+**Antenna**. Taste trials route calculated MN9 spikes through an engineered adapter
+to the native rostrum servo. Feed models feeding initiation, not eating or
+digestion; Water stimulates water-sensing neurons. Bitter alone produces neural
+activity without movement in this protocol. **Wings** and **Walk** are labeled
+**Not connected** and explain the missing motor circuitry.
+**Focus** points the observer at the head. Compare **Baseline** and **Block link**
+for the selected taste stimulus; **Stop** removes actuator authority.
+New motor trials wait for physical rest,
and pause/resume affects both clocks. All other actuators remain disabled.
Anatomical springs, joints, gravity and contact stay active. It does not walk or
fly autonomously. [Motor protocol and causal validation](motor-link.md).
-The expandable **Antennal reference assay** retains a separate connectome assay:
+**Antenna** retains a separate connectome assay:
antennal stimulus → FlyWire LIF dynamics → neural readouts → disconnected muscles.
Its structure view shows a selection of actual connections with schematic positions;
the entire published graph is retained in the simulation. Counts and voltages come
from calculated activity, and the timeline groups measured spikes in 5 ms bins.
-The trial clock is separate from the physical clock. Physical contact, mouse
+Its **Block sensory** comparison blocks outgoing sensory connections; **Stop**
+cancels the assay without stopping the environment. The trial clock is separate
+from the physical clock. Physical contact, mouse
movement and camera direction do not stimulate the brain yet.
[Protocol, preparation and validation](neural-reference.md).
-Select it with **Fly 001**, the specimen card, or its marker when visible.
+
+**Neural activity** opens a transparent inspector with **Taste → body** and
+**Antenna** tabs. Actions remain outside it, available with the panel closed.
+Hover or focus components, numbers and traces for explanations and measured
+values; click/tap also opens inspector tooltips. Graph position selects the
+nearest 5 ms sample. Escape dismisses the tooltip. Short **Experimental link**
+and **Neural only** labels keep the coupling boundary visible.
+Transparency uses a flat alpha background without an additional blur pass.
+
+Select the animal with **Fly 001**, the specimen card, or its marker when visible.
Selection and camera movement never command its muscles.
The separate recorded-flight pipeline remains available.
The amber sphere is an explicitly initialized gravity experiment, not an animal.
@@ -163,9 +179,11 @@ activating the `dev` profile cannot start a flight recording in the background.
The server only serves an allowlist of frontend assets and read endpoints:
`/api/world`, `/api/state`, `/api/events`, `/health`, plus `/api/dev-version` in
development mode. `POST /api/command` accepts bounded drop, pause, speed,
-`neural_trial`, `motor_trial` and `motor_stop` commands with same-origin JSON.
+`neural_trial`, `neural_stop`, `motor_trial` and `motor_stop` commands with same-origin JSON.
Trial mode must be `stimulus`, `baseline` or `blocked`; only one neural or motor
-trial runs at a time. Motor commands cannot select arbitrary neurons, actuator
+trial runs at a time. `motor_trial` optionally accepts `stimulus`: exactly
+`sugar`, `water` or `bitter`, defaulting to `sugar` for older clients.
+Motor commands cannot select arbitrary neurons, actuator
names, gains, torques or durations. It cannot read
arbitrary project files or edit the scene. At most eight event streams are open
at once. The container has a read-only filesystem and limits of 2 CPUs, 1 GiB
@@ -174,6 +192,12 @@ The separate antennal worker has a duty budget of 0.15 core, runs only on reques
and publishes changes at up to 10 Hz. It adds no continuous scene render loop.
This local server is not intended to be exposed directly to the internet.
+`web/actions.js` owns trial commands. `web/neural.js` and `web/motor.js` only
+inspect state; they draw changed data while their tab is visible.
+`web/inspect.js` manages one shared, event-driven tooltip, including the bounded
+sample lookup. Neither the inspector nor the action bar adds an idle animation
+loop, and unchanged control state causes no repeated DOM writes.
+
## Validation
```bash
@@ -202,6 +226,11 @@ baseline and blocked sensory output, verified their displayed counts, and
inspected the panel on desktop and at 390×844 with no horizontal overflow.
There were no JavaScript exceptions or failed requests; a software graphics
context reset during viewport resizing recovered. The session was then closed.
+The subsequent action-bar round exercised all four stimuli, blocked water input's
+motor link with the inspector closed, and stopped/restarted an antennal trial.
+Desktop and mobile tooltips worked with pointer, click and keyboard focus.
+The mobile panel and action bar remained separate with no horizontal overflow.
+Detailed counts and causal comparisons are in the [motor guide](motor-link.md#validation).
Page/process live-reload and 30 FPS on the user's hardware remain separate from
those checks.
diff --git a/docs/motor-link.md b/docs/motor-link.md
index 0ba41d7..36b9ab9 100644
--- a/docs/motor-link.md
+++ b/docs/motor-link.md
@@ -1,19 +1,23 @@
# Experimental MN9-to-proboscis link
The browser can now turn **simulated FlyWire motor-neuron spikes into a physical
-flybody joint response**. This first link uses sugar-responsive sensory neurons
+flybody joint response**. This first link uses published taste-sensory neurons
and the MN9 motor pair. It drives only the rostrum, part of the proboscis.
-Open **http://localhost:8089 → Neural activity → Stimulate sugar neurons**.
-**View proboscis** focuses the observer on the head; drag to choose an angle.
-The panel retains measured MN9 spikes, adapter drive and joint angle after the
-movement. Compare **Baseline · no input** and **Block motor link**.
+Open **http://localhost:8089** and use **Feed**, **Water** or **Bitter** in the
+action bar. **Feed** stimulates sugar neurons: it models feeding initiation, not
+eating or digestion. **Focus** points the observer at the head; drag to choose
+an angle. **Neural activity** opens a transparent, read-only inspector retaining
+measured MN9 spikes, adapter drive and joint angle. Hover, focus or tap components
+and numbers for details; point along the graph to inspect 5 ms samples.
+The action bar stays available with the panel closed. Compare **Baseline** and
+**Block link** for the selected stimulus.
New trials wait for the body to settle naturally. **Stop** cancels the trial;
the footer pauses or slows both physical and neural time together.
```mermaid
flowchart LR
- S["21 sugar GRNs Direct input"] --> C["Full FlyWire 630 graph 127,400 LIF neurons"]
+ S["Sugar / water / bitter GRNs Direct input"] --> C["Full FlyWire 630 graph 127,400 LIF neurons"]
C --> M["MN9 left and right Calculated spikes"]
M --> A["Engineered adapter Filtered rate → servo target"]
A --> B["Native flybody rostrum MuJoCo joint dynamics"]
@@ -22,19 +26,26 @@ flowchart LR
This is an on-demand motor experiment. Input is delivered directly to identified
neurons; no sugar object, taste receptor mechanics, world sensing or physical
feedback into the brain is implemented. There is no autonomous walking/flight
-controller or complete ventral nerve cord. The antennal assay remains separately
-available and has no motor authority.
+controller or complete ventral nerve cord. **Wings** and **Walk** therefore remain
+visibly **Not connected**, with explanations and no command handler. **Antenna**
+runs the separate assay and has no motor authority.
## Biological source and engineering boundary
The pinned [Shiu/Spiller notebook](https://github.com/philshiu/Drosophila_brain_model/blob/91bdd1e7dcf193f3e7ca5a8933497fcef63b7960/figures.ipynb)
-identifies 21 right labellar sugar GRNs (`neu_sugar`) and two MN9 neurons
-(`ids_mn9`). The runtime verifies that notebook's SHA-256 against the graph
+identifies 21 right labellar sugar GRNs (`neu_sugar`), 18 water GRNs (`neu_water`),
+21 bitter GRNs (`neu_bitter`) and two MN9 neurons (`ids_mn9`).
+The runtime verifies that notebook's SHA-256 against the graph
manifest and extracts literal IDs without executing cells. The motor IDs are
`720575940660219265` (left) and `720575940645521262` (right).
The [neural reference guide](neural-reference.md) describes the entire retained
graph, LIF equations, numerical agreement and source licenses.
+The sensory groups come from the model accompanying
+[Shiu et al., *A Drosophila computational brain model reveals sensorimotor processing*, Nature 2024](https://www.nature.com/articles/s41586-024-07763-9).
+Our **Bitter** preset stimulates bitter neurons alone. It does not reproduce the
+paper's combined sugar-plus-bitter inhibition experiment.
+
MN9 innervates the rostrum protractor muscle, providing a documented motor
association for this first link. Proboscis movement involves additional muscles
and joints; connecting one servo does not reconstruct that complete system.
@@ -60,15 +71,17 @@ The waveform and gain are not calibrated against an animal's measured kinematics
Each trial resets neural state and uses seed `20260912`, while preserving the
body's integrated state and global physical clock. It lasts **500 ms**: 300 ms
-of 200 Hz input per sugar neuron, then 200 ms without input. The published IDs
+of 200 Hz input per selected sensory neuron, then 200 ms without input. The published IDs
are reused; this specific finite-duration protocol is a local assay, not a claim
to reproduce every published feeding experiment.
| Trial | Neural input | Motor link |
|---|---|---|
| Sugar response | Fixed sugar events | MN9 drives the rostrum adapter |
+| Water response | Fixed water events | Same MN9-to-rostrum adapter |
+| Bitter response | Fixed bitter events | Same adapter; no MN9 spikes in this protocol |
| Baseline | None | No spikes and no actuator drive |
-| Block motor link | Identical sugar events | Normal brain activity; actuation disabled |
+| Block link | Identical selected sensory events | Normal brain activity; actuation disabled |
One neural step precedes one native physical step, both **0.1 ms**, on the same
worker. A mismatch stops the trial. Pause freezes both clocks; paused time does
@@ -103,36 +116,52 @@ The comparison clones the same settled **test** state for each intervention usin
MuJoCo's complete data copy. The live service never resets the fly for a trial.
Checks cover unchanged state at start and before the first motor spike, shared
clocks, native wake, excluded actuators, concurrent trial rejection, cancellation
-while paused, delta telemetry and failure on clock mismatch.
+while paused, delta telemetry and failure on clock mismatch. They also reject
+unsupported stimulus presets through the command interface and check that
+antennal cancellation allows a fresh trial.
Measured on 2026-09-12:
| Trial | MN9 left / right spikes | Downstream spikes | Peak rostrum movement | Other actuator force |
|---|---:|---:|---:|---:|
| No input | 0 / 0 | 0 | 0° | 0 |
-| Motor link blocked | 30 / 17 | 3,839 | 0° | 0 |
+| Sugar, link blocked | 30 / 17 | 3,839 | 0° | 0 |
| Sugar response | 30 / 17 | 3,839 | 40.03° | 0 |
-
-The first motor force occurs at the first MN9 spike, **25.5 ms of simulated
-time**, not wall time. Both stimulated trials have 5,076 total spikes, including
-1,237 input spikes. Neural and physical clocks each advance 500 ms; all checks
+| Water, link blocked | 14 / 7 | 1,293 | 0° | 0 |
+| Water response | 14 / 7 | 1,293 | 29.96° | 0 |
+| Bitter, link blocked | 0 / 0 | 677 | 0° | 0 |
+| Bitter response | 0 / 0 | 677 | 0° | 0 |
+
+The first motor force occurs at the first MN9 spike: **25.5 ms for sugar** and
+**76.2 ms for water**, in simulated time. Sugar has 5,076 total spikes (1,237
+input); water has 2,348 (1,055 input); bitter has 1,914 (1,237 input). Blocking
+the motor link preserves each preset's neural counts while removing actuation.
+Neural and physical clocks each advance 500 ms; all checks
complete without physics warnings. These establish a causal software-to-actuator
path, not biological equivalence, real-time performance or autonomous behavior.
The ignored `out/neural-reference/motor-validation.json` records code/cache
hashes, protocol, counts, force, clock alignment, CPU/wall time and peak RSS.
-The isolated validation process peaked near 215 MiB RSS; the stimulated 500 ms
-run used about 2.3 CPU seconds. After the browser trials, a short resting-service
-sample showed 149.8 MiB and 8.43% of one CPU core. These are observations of
-specific workloads, not performance guarantees.
+The expanded validation process peaked near 231 MiB RSS; the sugar trial used
+about 2.2 CPU seconds and water about 1.9 CPU seconds. After closing the isolated
+browser, a resting-service sample showed 146.5 MiB and 7.82% of one CPU core.
+These are observations of specific workloads, not performance guarantees.
-Browser checks exercised all three modes, head focus, pause/resume with both
+Earlier browser checks exercised all three sugar modes, head focus, pause/resume with both
clocks frozen, cancellation with drive off, and the retained antennal assay
(2,906 total / 706 downstream spikes). The baseline and motor-blocked trials
-added no new 3D frames to the settled scene. Desktop and 390×844 mobile panels
-were inspected with no horizontal overflow. All 295 captured requests returned
-HTTP 200, with no JavaScript exceptions. Resizing the software-rendered viewport
-caused a graphics-context reset that recovered. This does not establish hardware
-FPS. The isolated browser was closed after validation.
+added no new 3D frames to the settled scene.
+
+The action-bar check exercised all four sensory presets, water with the motor
+link blocked while the panel was closed, and antennal Stop followed by a fresh
+trial. Water produced about 30° of movement; bitter produced 677 downstream
+spikes, no MN9 spikes, no movement and no additional scene frames. A later sugar
+trial moved about 39.5°: live joint excursions depend on the naturally settled
+starting pose. The fixed-state validation above is the reproducible comparison.
+Desktop and 390×844 mobile layouts had no horizontal overflow or panel/bar
+overlap. Pointer, click and keyboard-focus tooltips showed measured values, and
+Escape dismissed them. No JavaScript exceptions or failed requests were observed.
+Resizing the software-rendered viewport caused a graphics-context reset that
+recovered. This does not establish hardware FPS. The isolated browser was closed.
The [browser guide](browser-environment.md) covers rendering limits and operation.
diff --git a/docs/neural-reference.md b/docs/neural-reference.md
index 10b4993..5de9562 100644
--- a/docs/neural-reference.md
+++ b/docs/neural-reference.md
@@ -8,7 +8,7 @@ counts preserved. It uses the data and model parameters from
This is not the newer MaleCNS dataset. Specimens and releases remain distinct.
The antennal assay described here has its own clock, starts from neural rest
-and does not drive the body. A separate [sugar-response motor experiment](motor-link.md)
+and does not drive the body. A separate [taste-response motor experiment](motor-link.md)
now couples the same full graph's MN9 neurons to the flybody rostrum through an
explicit engineered adapter, with a shared neural/physical clock.
World-to-sensory mapping, broader muscle control, walking and flight remain
@@ -16,7 +16,8 @@ future work; this is not a continuously embodied animal.
## Protocol and visible signals
-Open **Neural activity → Antennal reference assay** in the Fly 001 card.
+Choose **Antenna** in the action bar. Open **Neural activity → Antenna** to
+inspect the result; the action bar works with the inspector closed.
Each antennal trial lasts 150 ms of neural
time: 100 ms of input followed by 50 ms of recovery. The 146 published JON
antennal neurons come from the CE, F and D groups in the reference notebook.
@@ -25,14 +26,15 @@ mechanics. A fixed seed (`20260912`) makes comparisons repeatable.
| Control | Input | Intervention |
|---|---|---|
-| Stimulate antennal neurons | 150 Hz per input neuron | Normal published connections |
+| Antenna | 150 Hz per input neuron | Normal published connections |
| Baseline | No external input | Normal published connections |
-| Block sensory output | Same input events as stimulus | Zero outgoing weights from input neurons |
+| Block sensory | Same input events as stimulus | Zero outgoing weights from input neurons |
Silencing follows the reference code's outgoing-weight intervention. It does not
-delete neurons. The JON neurons still spike in the blocked trial.
+delete neurons. The JON neurons still spike in the blocked trial. **Stop** cancels
+the current trial at the next bounded block; another trial starts fresh.
-The panel shows accumulated trial spikes, membrane voltages, a 5 ms binned
+The transparent panel shows accumulated trial spikes, membrane voltages, a 5 ms binned
timeline and published aBN1, aDN1 and aDN2 readouts. The last two are descending
neurons; aBN1 is an interneuron. The schematic displays 21 selected actual neurons
and their connections, using no invented anatomical coordinates. The selection
@@ -40,11 +42,13 @@ does not prune the simulated graph. Node sizes/colors encode measured accumulate
spikes and remain visible after completion; no decorative animation runs at rest.
Excitatory/inhibitory edges show signed structural weights, not measured current
on each connection.
+Hover, focus or tap the components and values for their interpretation, IDs and
+measured voltages. Point along the timeline to inspect a sample.
The neural panel updates from the existing event stream without invalidating the
3D scene. It redraws its diagram/timeline only after a changed sample, while open
and visible. Muscle disconnection stays explicit in the antennal workflow;
-the sugar motor link has its own labeled workflow and measured body trace.
+the taste motor link has its own tab, labeled workflow and measured body trace.
## Model and numerical comparison
diff --git a/package.json b/package.json
index a6057f8..1aaa741 100644
--- a/package.json
+++ b/package.json
@@ -5,7 +5,7 @@
"type": "module",
"packageManager": "pnpm@11.24.0",
"scripts": {
- "check": "node --check web/environment.js && node --check web/fly.js && node --check web/neural.js && node --check web/motor.js && node --check web/dev-reload.js"
+ "check": "node --check web/environment.js && node --check web/fly.js && node --check web/neural.js && node --check web/motor.js && node --check web/actions.js && node --check web/inspect.js && node --check web/dev-reload.js"
},
"dependencies": {
"three": "0.180.0"
diff --git a/scripts/motor_bridge.py b/scripts/motor_bridge.py
index 0933ba1..ab9c4b2 100644
--- a/scripts/motor_bridge.py
+++ b/scripts/motor_bridge.py
@@ -21,6 +21,7 @@
CPU_LIMIT = 10.
SPIKE_LIMIT = 100000
MODES = ("stimulus", "baseline", "blocked")
+STIMULI = {"sugar": 21, "water": 18, "bitter": 21}
SOURCE = Path(__file__).resolve().parents[1]/"data/neural-reference/figures.ipynb"
@@ -38,13 +39,14 @@ def feeding_identifiers(graph, source=SOURCE):
if not isinstance(node, ast.Assign):
continue
for target in node.targets:
- if isinstance(target, ast.Name) and target.id in ("neu_sugar", "ids_mn9"):
+ if isinstance(target, ast.Name) and target.id in ("neu_sugar", "neu_water", "neu_bitter", "ids_mn9"):
values = ast.literal_eval(node.value)
if (not isinstance(values, list) or not all(type(x) is int for x in values)
or target.id in found and found[target.id] != values):
raise ValueError("Ambiguous feeding neuron identifiers")
found[target.id] = values
- if len(found.get("neu_sugar", [])) != 21 or len(found.get("ids_mn9", [])) != 2:
+ if (any(len(found.get(f"neu_{name}", [])) != count for name, count in STIMULI.items())
+ or len(found.get("ids_mn9", [])) != 2):
raise ValueError("Published feeding neuron groups are missing")
# IDs exceed JavaScript's safe integer range. Expose their strings, not numbers.
lookup = {int(value): i for i, value in enumerate(graph.ids)}
@@ -72,7 +74,9 @@ def __init__(self, graph, model, data, fly_tree):
ids = feeding_identifiers(graph)
if abs(model.opt.timestep-PARAMETERS.dt_ms/1000) > 1e-12:
raise ValueError("Motor link requires matching 0.1 ms clocks")
- self.inputs = np.array([item["index"] for item in ids["neu_sugar"]])
+ self.input_groups = {name: np.array([item["index"] for item in ids[f"neu_{name}"]])
+ for name in STIMULI}
+ self.inputs = self.input_groups["sugar"]
self.readouts = [{**item, "label": label} for item, label in
zip(ids["ids_mn9"], ("MN9 left", "MN9 right"))]
self.motor_indices = np.array([item["index"] for item in self.readouts])
@@ -90,6 +94,8 @@ def __init__(self, graph, model, data, fly_tree):
self.decay = float(np.exp(-PARAMETERS.dt_ms/FILTER_MS))
self.descriptor = {
"available": True, "input_count": len(self.inputs), "inputs": ids["neu_sugar"],
+ "stimuli": [{"id": name, "input_count": count, "inputs": ids[f"neu_{name}"]}
+ for name, count in STIMULI.items()],
"readouts": self.readouts, "rate_hz": RATE_HZ, "seed": SEED,
"total_ms": TICKS*PARAMETERS.dt_ms,
"stimulus_ms": STIMULUS_TICKS*PARAMETERS.dt_ms,
@@ -116,15 +122,20 @@ def disable(self):
self.data.qfrc_applied[self.dof] = 0.
self.model.tree_sleep_policy[self.fly_tree] = mujoco.mjtSleepPolicy.mjSLEEP_ALLOWED
- def start(self, mode):
+ def start(self, mode, stimulus="sugar"):
if not isinstance(mode, str) or mode not in MODES:
raise ValueError("Motor mode must be stimulus, baseline or blocked")
+ if not isinstance(stimulus, str) or stimulus not in STIMULI:
+ raise ValueError("Taste stimulus must be sugar, water or bitter")
if not self.descriptor["available"]:
raise ValueError(self.state["error"])
if self.active:
raise ValueError("A motor trial is already running")
self.disable()
self.mode = mode
+ self.inputs = self.input_groups[stimulus]
+ self.sensory[:] = False
+ self.sensory[self.inputs] = True
self.neurons = LIF(self.graph, self.inputs)
self.events = poisson_events(self.inputs, TICKS,
rate_hz=0 if mode == "baseline" else RATE_HZ,
@@ -138,7 +149,8 @@ def start(self, mode):
self.peak_excursion = self.peak_force = 0.
self.history = []
self.state = {"revision": self.state["revision"]+1, "status": "running",
- "trial": self.state["trial"]+1, "mode": mode, "drive_enabled": False,
+ "trial": self.state["trial"]+1, "mode": mode, "stimulus": stimulus,
+ "input_count": len(self.inputs), "drive_enabled": False,
"history": [], "readouts": [], "simulated_ms": 0, "physical_ms": 0,
"total_spikes": 0, "downstream_spikes": 0,
"peak_excursion_deg": 0, "peak_force": 0}
diff --git a/scripts/neural_lab.py b/scripts/neural_lab.py
index e945637..c4b0910 100644
--- a/scripts/neural_lab.py
+++ b/scripts/neural_lab.py
@@ -23,6 +23,7 @@ class NeuralLab:
def __init__(self, stop, directory=CACHE):
self.stop = stop
self.lock = threading.Lock()
+ self.cancel = threading.Event()
self.thread = None
self.graph = None
self.state = {"revision": 0, "status": "unavailable", "motor_connected": False}
@@ -72,6 +73,7 @@ def start(self, mode):
if self.stop.is_set():
raise ValueError("The environment is stopping")
trial = self.state["trial"]+1
+ self.cancel.clear()
self.state = {
"revision": self.state["revision"]+1, "status": "running",
"motor_connected": False, "trial": trial, "mode": mode,
@@ -83,6 +85,9 @@ def start(self, mode):
name="neural-assay", daemon=True)
self.thread.start()
+ def cancel_trial(self):
+ self.cancel.set()
+
def _publish(self, values):
with self.lock:
self.state = {**self.state, **values, "revision": self.state["revision"]+1}
@@ -105,7 +110,7 @@ def _run(self, mode):
bins = np.zeros(3, dtype=np.int64)
next_publish = 0.
for first in range(0, TICKS, 25):
- if self.stop.is_set():
+ if self.stop.is_set() or self.cancel.is_set():
self._publish({"status": "cancelled"})
return
for external in events[first:first+25]:
@@ -126,7 +131,7 @@ def _run(self, mode):
raise RuntimeError("Trial stopped at its resource limit; results are incomplete.")
# Duty-cycle one worker; unused neurons and all graph files stay
# shared/read-only. No continuous assay runs while the UI is idle.
- if self.stop.wait(max(0, cpu_used/CPU_BUDGET-wall_used)):
+ if self.cancel.wait(max(0, cpu_used/CPU_BUDGET-wall_used)) or self.stop.is_set():
self._publish({"status": "cancelled"})
return
now = time.monotonic()
diff --git a/scripts/serve_environment.py b/scripts/serve_environment.py
index 3baba57..556a814 100644
--- a/scripts/serve_environment.py
+++ b/scripts/serve_environment.py
@@ -185,7 +185,9 @@ def command(self, command):
with self.lock:
if self.stop.is_set():
raise ValueError("The environment is stopping")
- if action in ("neural_trial", "motor_trial") and set(command) == {"action", "mode"}:
+ if (action in ("neural_trial", "motor_trial")
+ and (set(command) == {"action", "mode"}
+ or action == "motor_trial" and set(command) == {"action", "mode", "stimulus"})):
if self.neural is None:
raise ValueError("The neural lab is not enabled in this environment")
if self.motor.active or self.neural.snapshot()["status"] == "running":
@@ -195,13 +197,16 @@ def command(self, command):
raise ValueError("Resume the physical simulation before starting a motor trial")
if self.data.tree_asleep[self.fly_tree] < 0:
raise ValueError("Let the fly settle before starting a motor trial")
- self.motor.start(command["mode"])
+ self.motor.start(command["mode"], command.get("stimulus", "sugar"))
self.next_pose_time = 0.
else:
self.neural.start(command["mode"])
elif action == "motor_stop" and set(command) == {"action"}:
if self.motor:
self.motor.finish()
+ elif action == "neural_stop" and set(command) == {"action"}:
+ if self.neural:
+ self.neural.cancel_trial()
elif action == "pause" and set(command) == {"action", "paused"}:
if not isinstance(command["paused"], bool):
raise ValueError("paused must be boolean")
@@ -460,6 +465,9 @@ def do_GET(self):
"/fly.js": ("web/fly.js", "text/javascript; charset=utf-8"),
"/neural.js": ("web/neural.js", "text/javascript; charset=utf-8"),
"/motor.js": ("web/motor.js", "text/javascript; charset=utf-8"),
+ "/actions.js": ("web/actions.js", "text/javascript; charset=utf-8"),
+ "/inspect.js": ("web/inspect.js", "text/javascript; charset=utf-8"),
+ "/actions.css": ("web/actions.css", "text/css; charset=utf-8"),
"/neural.css": ("web/neural.css", "text/css; charset=utf-8"),
"/dev-reload.js": ("web/dev-reload.js", "text/javascript; charset=utf-8"),
"/style.css": ("web/style.css", "text/css; charset=utf-8"),
diff --git a/scripts/validate_motor_bridge.py b/scripts/validate_motor_bridge.py
index 16dda5c..bfe6583 100644
--- a/scripts/validate_motor_bridge.py
+++ b/scripts/validate_motor_bridge.py
@@ -27,12 +27,15 @@ def validate():
mujoco.mj_copyData(resting, model, data)
other = np.arange(model.nu) != bridge.actuator
results = {}
- for mode in ("baseline", "blocked", "stimulus"):
+ cases = [("sugar", mode) for mode in ("baseline", "blocked", "stimulus")]
+ cases += [(taste, mode) for taste in ("water", "bitter") for mode in ("blocked", "stimulus")]
+ expected = {"sugar": ([30, 17], 5076), "water": ([14, 7], 2348), "bitter": ([0, 0], 1914)}
+ for taste, mode in cases:
bridge.disable()
mujoco.mj_copyData(data, model, resting)
before = data.qpos.copy(), data.qvel.copy(), data.time
started, cpu_started = time.monotonic(), time.thread_time()
- env.command({"action": "motor_trial", "mode": mode})
+ env.command({"action": "motor_trial", "mode": mode, "stimulus": taste})
np.testing.assert_array_equal(before[0], data.qpos)
np.testing.assert_array_equal(before[1], data.qvel)
assert before[2] == data.time
@@ -68,24 +71,35 @@ def validate():
if mode == "baseline":
assert counts == [0, 0] and result["total_spikes"] == 0
else:
- assert counts == [30, 17] and result["total_spikes"] == 5076
- if mode != "stimulus":
+ assert (counts, result["total_spikes"]) == expected[taste]
+ assert result["stimulus"] == taste
+ if mode != "stimulus" or taste == "bitter":
assert result["peak_force"] == 0 and result["peak_excursion_deg"] == 0
np.testing.assert_array_equal(data.qpos, resting.qpos)
else:
- assert first_force_tick == first_motor_tick == 255
- assert result["peak_excursion_deg"] > 10
+ assert first_force_tick == first_motor_tick and first_motor_tick is not None
+ if taste == "sugar":
+ assert first_motor_tick == 255
+ assert result["peak_excursion_deg"] > 5
assert 0 < result["peak_force"] <= .1
assert data.tree_asleep[env.fly_tree] < 0, "Sleeping animal did not wake"
- results[mode] = {
+ key = mode if taste == "sugar" else f"{taste}_{mode}"
+ results[key] = {
key: result[key] for key in ("total_spikes", "input_spikes", "downstream_spikes",
"peak_force", "peak_excursion_deg", "physical_ms", "simulated_ms")
}
- results[mode].update(mn9_spikes=counts, other_actuator_peak_force=peak_other,
+ results[key].update(mn9_spikes=counts, other_actuator_peak_force=peak_other,
first_force_ms=None if first_force_tick is None else first_force_tick*PARAMETERS.dt_ms,
wall_seconds=time.monotonic()-started,
cpu_seconds=time.thread_time()-cpu_started)
- print(mode, json.dumps(results[mode]), flush=True)
+ print(key, json.dumps(results[key]), flush=True)
+ # The public endpoint accepts only named presets, not arbitrary channels.
+ for stimulus in ("wing", "", [], {"neurons": [1]}):
+ try:
+ env.command({"action": "motor_trial", "mode": "stimulus", "stimulus": stimulus})
+ raise AssertionError("Unbounded sensory selector accepted")
+ except ValueError:
+ pass
# Cancellation must remove actuator authority, including after it has fired.
bridge.start("stimulus")
for _ in range(400):
@@ -128,12 +142,21 @@ def validate():
pass
assert bridge.state["status"] == "error" and not bridge.active
assert model.opt.disableflags & int(mujoco.mjtDisableBit.mjDSBL_ACTUATION)
+ env.command({"action": "neural_trial", "mode": "stimulus"})
+ env.command({"action": "neural_stop"})
+ env.neural.thread.join(timeout=5)
+ assert env.neural.snapshot()["status"] == "cancelled"
+ # A fresh trial must not inherit a cancelled predecessor's stop flag.
+ env.command({"action": "neural_trial", "mode": "baseline"})
+ env.neural.thread.join(timeout=10)
+ assert env.neural.snapshot()["status"] == "complete"
+ assert env.neural.snapshot()["total_spikes"] == 0
env.stop.set()
root = Path(__file__).resolve().parents[1]
return {
"schema": 1, "passed": True,
"code_sha256": {name: sha256(root/"scripts"/name) for name in
- ("motor_bridge.py", "serve_environment.py", "neural_reference.py",
+ ("motor_bridge.py", "serve_environment.py", "neural_reference.py", "neural_lab.py",
"validate_motor_bridge.py")},
"cache_manifest_sha256": sha256(root/"out/neural-reference/manifest.json"),
"protocol": bridge.descriptor, "results": results,
diff --git a/web/actions.css b/web/actions.css
new file mode 100644
index 0000000..9c4e840
--- /dev/null
+++ b/web/actions.css
@@ -0,0 +1,25 @@
+.action-bar{position:absolute;z-index:5;bottom:99px;left:50%;transform:translateX(-50%);width:min(660px,calc(100vw - 32px));padding:12px 15px;border:1px solid #bbd5cc40;border-radius:10px;background:#0c181ee8;box-shadow:0 6px 24px #0003}
+.action-heading{display:flex;justify-content:space-between;gap:16px;margin-bottom:9px;font:9px monospace;color:#a6c0b8}
+.action-heading>span:first-child{letter-spacing:1.6px}
+.action-presets{display:grid;grid-template-columns:repeat(6,1fr);gap:6px}
+.action-presets button{position:relative;min-height:53px;display:flex;flex-direction:column;align-items:center;justify-content:center;gap:3px;border:1px solid var(--line);border-radius:5px;background:#afcec208;font-size:10px}
+.action-symbol{font:21px sans-serif;line-height:22px;color:#c3e0d0}
+.action-presets button[aria-pressed=true]{background:#bad6c321;border-color:#bad6c380;color:#e4f2e8}
+.action-presets small{font-size:7px;color:#b0bcbf}
+.action-tools{display:flex;justify-content:space-between;gap:10px;margin-top:10px}
+.action-tools>div{display:flex;gap:5px}
+.action-tools button{font-size:10px;padding:7px 10px;border:1px solid var(--line);border-radius:4px;background:#c3d9dc08;white-space:nowrap}
+.action-tools #neural-open{color:#c9e1d3;border-color:#b7d6c34d}
+.action-bar button[aria-disabled=true]{color:#8d9ca1;cursor:help}
+.action-bar button[aria-disabled=true] .action-symbol{color:#80948e}
+.action-bar button[aria-disabled=true]:hover{border-color:#bfceca66;background:#c3d9dc0b}
+#fly-panel{bottom:272px;width:225px;background:#101c224d;backdrop-filter:none}
+#fly-panel .eyebrow{font-size:8px;letter-spacing:1.2px;line-height:1.6}
+#fly-panel .specimen-facts{margin-top:12px;gap:8px}
+#inspector-tooltip{position:fixed;z-index:30;width:max-content;max-width:min(290px,calc(100vw - 20px));padding:11px 13px;border:1px solid #c0d9cb66;border-radius:6px;background:#101e25fa;color:#e0ece5;font:11px/1.65 Inter,"Segoe UI",Arial,sans-serif;box-shadow:0 4px 20px #0005;white-space:pre-line}
+[data-tooltip]:not(button):not(a){cursor:help}
+[data-tooltip]:focus-visible{outline:1px solid #bedac8;outline-offset:3px}
+#toast{z-index:9;bottom:269px}
+@media(max-width:1000px){#fly-panel{bottom:275px;left:26px;width:200px;padding:12px}}
+@media(max-width:700px){.action-bar{bottom:91px;padding:10px;width:calc(100vw - 24px)}.action-presets{gap:4px}.action-presets button{font-size:9px;min-height:53px}.action-presets small{font-size:6px}.action-symbol{font-size:19px}.action-tools{gap:5px;margin-top:8px}.action-tools>div{gap:4px}.action-tools button{font-size:8px;padding:8px 6px}.action-heading{font-size:8px;margin-bottom:7px}#fly-panel{bottom:259px;left:20px;width:185px}#toast{bottom:255px}}
+@media(max-height:650px) and (min-width:701px){#fly-panel{display:none}.action-bar{bottom:93px}.action-presets button{min-height:44px}.action-symbol{font-size:18px;line-height:18px}}
diff --git a/web/actions.js b/web/actions.js
new file mode 100644
index 0000000..d48cbbd
--- /dev/null
+++ b/web/actions.js
@@ -0,0 +1,87 @@
+const $ = (id) => document.getElementById(id);
+const names = { sugar: 'Feed', water: 'Water', bitter: 'Bitter', antenna: 'Antenna' };
+
+export function createActions(neural, motor, command, selectReadout) {
+ const presets = [...document.querySelectorAll('[data-preset]')];
+ let selected = 'sugar', state = null, connected = false, pending = false;
+ let seenMotor = 0, seenNeural = 0;
+ let rendered = '';
+ $('fly-actions').hidden = false;
+ function disabled(button, value) {
+ // aria-disabled keeps reasons available to keyboard and touch users.
+ button.setAttribute('aria-disabled', String(value));
+ }
+ function allowed(preset) {
+ return connected && !pending && !state?.motor_running && !state?.neural_running
+ && (preset === 'antenna' ? neural?.available
+ : motor?.available && !state?.paused && state?.fly.sleeping);
+ }
+ function render() {
+ const key = [selected, connected, pending, state?.motor_running, state?.neural_running,
+ state?.paused, state?.fly.sleeping, state?.motor?.stimulus].join('|');
+ if (key === rendered) return;
+ rendered = key;
+ for (const button of presets) {
+ disabled(button, !allowed(button.dataset.preset));
+ button.setAttribute('aria-pressed', String(button.dataset.preset === selected));
+ }
+ disabled($('action-baseline'), !allowed(selected));
+ disabled($('action-block'), !allowed(selected));
+ disabled($('trial-stop'), !connected || pending || !(state?.motor_running || state?.neural_running));
+ const active = state?.motor_running ? state.motor : state?.neural_running ? state.neural : null;
+ $('action-status').textContent = !connected ? 'Reconnecting…'
+ : pending ? pending === 'stop' ? 'Stopping…' : 'Starting…'
+ : active ? `${state.motor_running ? names[active.stimulus || 'sugar'] : 'Antenna'} · ${state.motor_running && state.paused ? 'Paused' : 'Running'}`
+ : !neural?.available ? 'Neural cache unavailable'
+ : state?.paused ? 'Body paused'
+ : !state?.fly.sleeping ? 'Body settling'
+ : 'Ready';
+ $('action-block').textContent = selected === 'antenna' ? 'Block sensory' : 'Block link';
+ $('action-block').dataset.tooltip = selected === 'antenna'
+ ? 'Same antennal input, with outgoing sensory connections blocked. Input neurons still spike.'
+ : `Same ${selected} input and brain activity, with motor actuation disabled.`;
+ $('action-baseline').dataset.tooltip = `${names[selected]} protocol with no external input. The physical fly is never reset for a trial.`;
+ }
+ async function run(mode) {
+ if (!allowed(selected)) return;
+ selectReadout(selected === 'antenna' ? 'antenna' : 'motor');
+ pending = true;
+ render();
+ try {
+ await command(selected === 'antenna' ? { action: 'neural_trial', mode }
+ : { action: 'motor_trial', mode, stimulus: selected });
+ } finally { pending = false; render(); }
+ }
+ for (const button of presets) {
+ button.addEventListener('click', () => {
+ if (!allowed(button.dataset.preset)) return;
+ selected = button.dataset.preset;
+ run('stimulus');
+ });
+ }
+ $('action-baseline').addEventListener('click', () => run('baseline'));
+ $('action-block').addEventListener('click', () => run('blocked'));
+ $('trial-stop').addEventListener('click', async () => {
+ if ($('trial-stop').getAttribute('aria-disabled') === 'true') return;
+ pending = 'stop'; render();
+ try { await command({ action: state.motor_running ? 'motor_stop' : 'neural_stop' }); }
+ finally { pending = false; render(); }
+ });
+ render();
+ return {
+ update(next) {
+ state = next;
+ if (next.motor_running && next.motor.trial !== seenMotor) {
+ seenMotor = next.motor.trial;
+ selected = next.motor.stimulus || 'sugar';
+ selectReadout('motor');
+ } else if (next.neural_running && next.neural.trial !== seenNeural) {
+ seenNeural = next.neural.trial;
+ selected = 'antenna';
+ selectReadout('antenna');
+ }
+ render();
+ },
+ setConnected(value) { connected = value; render(); },
+ };
+}
diff --git a/web/environment.js b/web/environment.js
index ae6205e..b7a81da 100644
--- a/web/environment.js
+++ b/web/environment.js
@@ -1,5 +1,7 @@
import * as THREE from 'three';
import { createFly } from './fly.js';
+import { createActions } from './actions.js';
+import { setupTooltips } from './inspect.js';
const $ = (selector) => document.querySelector(selector);
const canvas = $('#world');
@@ -23,7 +25,7 @@ let needsRender = true, lastRender = 0, renderLimit = 30, pixelBudget = 1500000;
let graphicsLost = false;
let graphicsDevice = '', softwareGraphics = false, renderCostMs = 0;
let cameraMovedSinceHud = false;
-let neuralPanel;
+let neuralPanel, actions;
let fly, flyView = false, headView = false, orbitYaw = -.95, orbitPitch = .45, orbitDistance = .65;
let scheduledFrame = 0, hudTimer = 0;
const freeHelp = $('.navigation-help').innerHTML;
@@ -62,7 +64,7 @@ function setConnected(value) {
$('#connection-dot').className = `dot ${value ? 'live' : 'offline'}`;
$('#connection').textContent = value ? 'Environment connected' : 'Reconnecting…';
for (const id of ['#drop', '#pause', '#speed']) $(id).disabled = !value;
- neuralPanel?.setConnected(value);
+ actions?.setConnected(value);
}
async function command(payload) {
@@ -89,7 +91,8 @@ function updateState(next) {
if (!next.neural || (state?.neural?.revision ?? -1) > next.neural.revision) next.neural = state?.neural;
if (!next.motor || (state?.motor?.revision ?? -1) > next.motor.revision) next.motor = state?.motor;
state = next;
- neuralPanel?.update(next.neural, next.motor, next.paused, next.fly.sleeping);
+ neuralPanel?.update(next.neural, next.motor);
+ actions?.update(next);
lastEvent = performance.now();
setConnected(!next.error);
if (next.error) {
@@ -636,7 +639,8 @@ async function start() {
world = await response.json();
if (world.neural) {
const { createNeuralPanel } = await import('./neural.js');
- neuralPanel = createNeuralPanel(world.neural, world.motor, command);
+ neuralPanel = createNeuralPanel(world.neural, world.motor, setupTooltips());
+ actions = createActions(world.neural, world.motor, command, (kind) => neuralPanel.select(kind));
}
await buildWorld();
const [animal, initial] = await Promise.all([
diff --git a/web/index.html b/web/index.html
index 3fe29f7..bd00e63 100644
--- a/web/index.html
+++ b/web/index.html
@@ -4,10 +4,11 @@
-
+
NETSPHERE — Fly environment
+
@@ -52,100 +53,103 @@