Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ The animal is the anatomically detailed [flybody](https://github.com/TuragaLab/f

The full 60-second take, its metrics and the validation files are attached to the [v0.1.0 release](https://github.com/bitdeep/fly-netsphere/releases/tag/v0.1.0).

**New preview — a fly living in the NETSPHERE, from its viewpoint.**
[Watch the stabilized 60-second recording](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4) of the updated megastructure. The camera follows the physical head with a level horizon; the same flight completes 1.2 million physics steps with zero world contacts. [Camera comparison, validation and replay instructions](docs/stabilized-pov.md).

## What this is, and what it is not

- It **is** whole-body physics: joints, wings with ellipsoid fluid forces, and the official DMPO flight policy (wingbeat pattern generator plus a residual MLP) driving the actuators.
Expand Down Expand Up @@ -41,7 +44,7 @@ Perturbation tests with the same seed and initial pose: moving the first pillar
## How it works

```
city_world.py MJCF world: procedural brutalist district, 412 geoms including the fly, lengths in cm
city_world.py MJCF world: procedural brutalist district, 542 geoms including the fly, lengths in cm
│
city_navigation.py every 10 ms: candidate manoeuvres vs. distance to the world → reference command
▼
Expand All @@ -58,9 +61,9 @@ verify_take.py independent checks: duration, decoded frames, clearance, ha

- **Policy.** `scripts/flight_mlp.py` is a NumPy port of the official Acme checkpoint: Linear → LayerNorm → tanh, then ELU layers and a mean head. `scripts/flight_cuda.py` runs the same controller with NVIDIA Warp, and the two are compared numerically on every run.
- **Observations and actions.** A 104-D observation (the `walker/*` keys in lexicographic order) maps to a 12-D canonical action in [-1, 1], then to actuator ranges. Controller step 2e-4 s, physics step 5e-5 s.
- **World.** `scripts/city_world.py` builds a central core, pillars, ducts, cables, walkways, access plates, grilles and wear. Textures are procedural and drawn in surface coordinates. The NETSPHERE plates are physical geometry.
- **World.** `scripts/city_world.py` builds a deep shaft, stacked galleries, a central core, pillars, ducts and sagging cables. Large concrete volumes have formwork seams, cracks and mineral streaks; service elements use metal panels. Cold light and blue-grey distance fog give depth to the interior. Textures are procedural and drawn in surface coordinates. The NETSPHERE plates and every cable segment are physical geometry. The published `netsphere_60s` take uses the earlier, 412-geom district.
- **Navigation.** `scripts/city_navigation.py` is a receding-horizon navigator over known geometry. It explores four regions of the district and keeps wing and body clearance.
- **Camera.** A near-lateral third-person view (azimuth 85°, elevation 2°) shortens its distance with a ray test when scenery is in the way. A first-person view is available. Visibility is checked on every frame with an unfiltered object-ID render.
- **Camera.** First-person places the camera at the integrated `walker/head` position, with a 65° vertical field of view. The default comfort mode keeps pitch and roll at zero and follows the tangent of the measured head path after symmetric Gaussian smoothing (σ = 0.25 s). It uses neighboring recorded states to remove the rapid gaze oscillation caused by sampling instantaneous flapping velocity at video rate. Only viewing direction is filtered; the eye stays at the physical head in every exposure sample. The observer hides the fly's own anatomy to avoid filming inside its head; physical anatomy and collisions remain active. This is a human viewing camera, not a compound-eye simulation or the navigator's sensor. An optional near-lateral third-person view (azimuth 85°, elevation 2°) shortens its distance with a ray test when scenery is in the way. Visibility is checked on every frame with an unfiltered object-ID render; first-person positions are independently checked against the saved head kinematics.
- **Timing.** The video clock is the physics clock. A run that falls or touches the world is rejected and its diagnostics are kept. No episode is stretched or restarted.

## Quick start
Expand All @@ -85,9 +88,28 @@ docker compose run --rm fly python scripts/verify_take.py out/my_take --seconds

`simulate_city.sh SECONDS DIR [VIEW]` runs simulation, render and verification; `VIEW` defaults to `first-person`, and the validated take uses `third-person`. `docker compose run --rm fly` with no arguments runs the script with its defaults.

For the updated district from the fly's viewpoint:

```bash
make take VIEW=first-person TAKE=out/my_pov
```

To stabilize an existing recording without recomputing or changing its physics:

```bash
docker compose run --rm fly python scripts/render_city.py out/my_pov \
--view first-person --stabilization comfort --output out/my_pov/stabilized.mp4
docker compose run --rm fly python scripts/verify_take.py out/my_pov \
--seconds 60 --video stabilized.mp4
```

`--stabilization legacy` reproduces the previous velocity-following gaze.
Comfort validation checks the actual rendered camera basis, level horizon and
angular acceleration, alongside the unchanged state/model hashes.

A minute of flight is 300,000 controller steps and 1.2 million physics steps. Expect roughly 15 minutes of compute per simulated minute on an RTX 4090, plus rendering. The first run compiles the CUDA kernels, which are cached in the `warp-cache` volume.

Each take directory contains `fly_city.mp4`, `states.npz` (poses, velocities, physics clock, navigator commands and the sub-poses of every frame), `model.mjb` (the compiled model with its geometry and textures), `metrics.json` and `validation.json`. States can be re-rendered with another camera without recomputing physics.
Each take directory contains `fly_city.mp4`, `states.npz` (poses, velocities, physics clock, navigator commands and the sub-poses of every frame), `model.mjb` (the compiled model with its geometry and textures), `metrics.json`, `fly_city.render.json` (camera and per-frame visibility) and `validation.json`. States can be re-rendered with another camera without recomputing physics. Use `verify_take.py DIR --seconds 60 --video wide.mp4` to validate an alternative recording.

Options of `scripts/run_city.py`: `--seconds`, `--seed`, `--backend cuda|cpu`, `--obstacle-shift`, `--disable-avoidance` (negative control), `--body-pitch` (cruise reference, default 30°) and `--shutter-samples`.

Expand Down
90 changes: 90 additions & 0 deletions docs/stabilized-pov.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# A fly living in the NETSPHERE — stabilized POV

[**Watch the full 60-second video · 720p · 30 fps**](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4)

[![Frames from the continuous recording inside the NETSPHERE](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-contact_sheet.png)](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4)

An anatomically detailed fruit fly flies through a collidable, BLAME!-inspired interior. The official flybody flight policy drives its actuators on CUDA, MuJoCo Warp integrates its body and wing aerodynamics, and a geometric navigator chooses reference commands around obstacles. This recording follows the fly from the position of its physical head.

The previous first-person camera followed instantaneous flight velocity. Sampling that rapidly oscillating signal at video rate produced distracting changes in viewing direction. The new `comfort` mode reduces yaw acceleration by **97.18%** on this take and keeps the horizon level. Both recordings use the same integrated flight states and compiled world.

## What changed

The renderer averages the measured head positions within each exposure, smooths that recorded path with a symmetric Gaussian (σ = 0.25 s), and uses its tangent for yaw. Pitch and roll stay at zero. Only the viewing direction is filtered: the camera remains at the integrated head position in every exposure sample.

This is an offline camera for human viewers. The symmetric filter uses neighboring past and future recorded states; it is not a real-time controller, a model of compound-eye vision, or a claim about what a fly perceives. The navigator uses known geometry, not learned vision or a connectome. The fly's anatomy is hidden from this camera to avoid rendering the inside of its head; its physical body, wings and collision geometry remain active.

The world in this preview is the updated 542-geom interior, including the fly: a deep shaft, stacked galleries, concrete pillars, metal ducts and sagging cables. It is already part of the simulation; this camera change does not modify its geometry or the flight controller. The earlier third-person district remains available in the [v0.1.0 release](https://github.com/bitdeep/fly-netsphere/releases/tag/v0.1.0).

## Measured result

Take: `blame_pov_60s`, seed 0, continuous GPU simulation on an RTX 4090.

| Measure | Result |
|---|---:|
| Simulated time / decoded video duration | 60.000 s / 60.000 s |
| Video | 1280 × 720, 30 fps, 1,800 frames |
| Exposure | 8 physical sub-poses per frame, 1/120 s |
| Physics steps / controller steps | 1,200,000 / 300,000 |
| World contacts / episode resets / solver overflow flags | 0 / 0 / 0 |
| Distance flown | 12.014 m |
| Altitude range | 3.21–7.18 cm |
| Maximum reference tracking error | 0.671 mm |
| Minimum conservative whole-body clearance | 8.44 mm |
| Median anatomical pitch | 35.32° |
| Navigator goals reached / avoidance updates | 16 / 5,532 |
| Maximum checked camera-to-head position error | 0.0000381 mm |
| Yaw acceleration RMS, previous → comfort | 1,624.89 → 45.89 °/s² |
| Pitch acceleration RMS, previous → comfort | 1,469.42 → 0.00 °/s² |

The angular comparison reconstructs the previous camera's exact yaw/pitch recurrence from the saved velocities and measures the new gaze from the recorded OpenGL camera forward vectors. Acceleration is the second difference of unwrapped angles at 30 fps. The percentage describes this recording and this metric; it is not a perceptual comfort study.

`verify_take.py` checks the decoded frame count and duration, physical continuity, saved-state and model hashes, whole-body clearance, head attachment, per-frame world visibility, level camera basis and yaw acceleration RMS below 150 °/s². All 1,800 frames passed. A separate six-second preview also passed before rendering the complete minute.

## Download and verify

The [preview release](https://github.com/bitdeep/fly-netsphere/releases/tag/pov-stabilization-preview-1) includes the original MP4 bytes, the physical states and compiled world, simulation metrics, camera measurements, validation, comparison and a contact sheet. Filenames use the `blame_pov_60s-` prefix; `SHA256SUMS` covers every asset except itself.

| Evidence | Download |
|---|---|
| Simulation and source hashes | [metrics.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-metrics.json) |
| Physical and video acceptance | [fly_city_stabilized.validation.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.validation.json) |
| Per-frame camera measurements | [fly_city_stabilized.render.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.render.json) |
| Previous/comfort angular comparison | [stabilization_comparison.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-stabilization_comparison.json) |
| Asset integrity | [SHA256SUMS](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/SHA256SUMS) |

Video SHA-256:

```text
b7b88f119f66ceaf782f92d5bcbedbe7b3b22d258d8416f0bd260323e60786f7
```

After the [repository setup](../README.md#quick-start), download the published take and restore its filenames:

```bash
mkdir -p out/pov-preview
base=https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1
curl -fL "$base/SHA256SUMS" -o out/pov-preview/SHA256SUMS
for asset in fly_city_stabilized.mp4 fly_city_stabilized.render.json \
fly_city_stabilized.validation.json metrics.json states.npz model.mjb \
stabilization_comparison.json contact_sheet.png; do
curl -fL "$base/blame_pov_60s-$asset" -o "out/pov-preview/blame_pov_60s-$asset"
done
(cd out/pov-preview && sha256sum -c SHA256SUMS)
for asset in out/pov-preview/blame_pov_60s-*; do
mv "$asset" "out/pov-preview/${asset##*/blame_pov_60s-}"
done
docker compose run --rm fly python scripts/verify_take.py out/pov-preview \
--seconds 60 --video fly_city_stabilized.mp4
```

Re-render those states with the comfort camera, without recomputing physics:

```bash
docker compose run --rm fly python scripts/render_city.py out/pov-preview \
--view first-person --stabilization comfort --output out/pov-preview/rerender.mp4
docker compose run --rm fly python scripts/verify_take.py out/pov-preview \
--seconds 60 --video rerender.mp4
```

Use `--stabilization legacy` to compare the previous gaze. Rendering on different drivers or encoders may produce different MP4 bytes; the published SHA identifies the original recording. The state and model hashes identify the physical take independently of the camera.
Loading
Loading