diff --git a/README.md b/README.md index 37acad7..ea464e1 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,9 @@ The animal is the anatomically detailed [flybody](https://github.com/TuragaLab/f The full 60-second take, its metrics and the validation files are attached to the [v0.1.0 release](https://github.com/bitdeep/fly-netsphere/releases/tag/v0.1.0). +**New preview — a fly living in the NETSPHERE, from its viewpoint.** +[Watch the stabilized 60-second recording](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4) of the updated megastructure. The camera follows the physical head with a level horizon; the same flight completes 1.2 million physics steps with zero world contacts. [Camera comparison, validation and replay instructions](docs/stabilized-pov.md). + ## What this is, and what it is not - It **is** whole-body physics: joints, wings with ellipsoid fluid forces, and the official DMPO flight policy (wingbeat pattern generator plus a residual MLP) driving the actuators. @@ -41,7 +44,7 @@ Perturbation tests with the same seed and initial pose: moving the first pillar ## How it works ``` -city_world.py MJCF world: procedural brutalist district, 412 geoms including the fly, lengths in cm +city_world.py MJCF world: procedural brutalist district, 542 geoms including the fly, lengths in cm │ city_navigation.py every 10 ms: candidate manoeuvres vs. distance to the world → reference command ▼ @@ -58,9 +61,9 @@ verify_take.py independent checks: duration, decoded frames, clearance, ha - **Policy.** `scripts/flight_mlp.py` is a NumPy port of the official Acme checkpoint: Linear → LayerNorm → tanh, then ELU layers and a mean head. `scripts/flight_cuda.py` runs the same controller with NVIDIA Warp, and the two are compared numerically on every run. - **Observations and actions.** A 104-D observation (the `walker/*` keys in lexicographic order) maps to a 12-D canonical action in [-1, 1], then to actuator ranges. Controller step 2e-4 s, physics step 5e-5 s. -- **World.** `scripts/city_world.py` builds a central core, pillars, ducts, cables, walkways, access plates, grilles and wear. Textures are procedural and drawn in surface coordinates. The NETSPHERE plates are physical geometry. +- **World.** `scripts/city_world.py` builds a deep shaft, stacked galleries, a central core, pillars, ducts and sagging cables. Large concrete volumes have formwork seams, cracks and mineral streaks; service elements use metal panels. Cold light and blue-grey distance fog give depth to the interior. Textures are procedural and drawn in surface coordinates. The NETSPHERE plates and every cable segment are physical geometry. The published `netsphere_60s` take uses the earlier, 412-geom district. - **Navigation.** `scripts/city_navigation.py` is a receding-horizon navigator over known geometry. It explores four regions of the district and keeps wing and body clearance. -- **Camera.** A near-lateral third-person view (azimuth 85°, elevation 2°) shortens its distance with a ray test when scenery is in the way. A first-person view is available. Visibility is checked on every frame with an unfiltered object-ID render. +- **Camera.** First-person places the camera at the integrated `walker/head` position, with a 65° vertical field of view. The default comfort mode keeps pitch and roll at zero and follows the tangent of the measured head path after symmetric Gaussian smoothing (σ = 0.25 s). It uses neighboring recorded states to remove the rapid gaze oscillation caused by sampling instantaneous flapping velocity at video rate. Only viewing direction is filtered; the eye stays at the physical head in every exposure sample. The observer hides the fly's own anatomy to avoid filming inside its head; physical anatomy and collisions remain active. This is a human viewing camera, not a compound-eye simulation or the navigator's sensor. An optional near-lateral third-person view (azimuth 85°, elevation 2°) shortens its distance with a ray test when scenery is in the way. Visibility is checked on every frame with an unfiltered object-ID render; first-person positions are independently checked against the saved head kinematics. - **Timing.** The video clock is the physics clock. A run that falls or touches the world is rejected and its diagnostics are kept. No episode is stretched or restarted. ## Quick start @@ -85,9 +88,28 @@ docker compose run --rm fly python scripts/verify_take.py out/my_take --seconds `simulate_city.sh SECONDS DIR [VIEW]` runs simulation, render and verification; `VIEW` defaults to `first-person`, and the validated take uses `third-person`. `docker compose run --rm fly` with no arguments runs the script with its defaults. +For the updated district from the fly's viewpoint: + +```bash +make take VIEW=first-person TAKE=out/my_pov +``` + +To stabilize an existing recording without recomputing or changing its physics: + +```bash +docker compose run --rm fly python scripts/render_city.py out/my_pov \ + --view first-person --stabilization comfort --output out/my_pov/stabilized.mp4 +docker compose run --rm fly python scripts/verify_take.py out/my_pov \ + --seconds 60 --video stabilized.mp4 +``` + +`--stabilization legacy` reproduces the previous velocity-following gaze. +Comfort validation checks the actual rendered camera basis, level horizon and +angular acceleration, alongside the unchanged state/model hashes. + A minute of flight is 300,000 controller steps and 1.2 million physics steps. Expect roughly 15 minutes of compute per simulated minute on an RTX 4090, plus rendering. The first run compiles the CUDA kernels, which are cached in the `warp-cache` volume. -Each take directory contains `fly_city.mp4`, `states.npz` (poses, velocities, physics clock, navigator commands and the sub-poses of every frame), `model.mjb` (the compiled model with its geometry and textures), `metrics.json` and `validation.json`. States can be re-rendered with another camera without recomputing physics. +Each take directory contains `fly_city.mp4`, `states.npz` (poses, velocities, physics clock, navigator commands and the sub-poses of every frame), `model.mjb` (the compiled model with its geometry and textures), `metrics.json`, `fly_city.render.json` (camera and per-frame visibility) and `validation.json`. States can be re-rendered with another camera without recomputing physics. Use `verify_take.py DIR --seconds 60 --video wide.mp4` to validate an alternative recording. Options of `scripts/run_city.py`: `--seconds`, `--seed`, `--backend cuda|cpu`, `--obstacle-shift`, `--disable-avoidance` (negative control), `--body-pitch` (cruise reference, default 30°) and `--shutter-samples`. diff --git a/docs/stabilized-pov.md b/docs/stabilized-pov.md new file mode 100644 index 0000000..74489fc --- /dev/null +++ b/docs/stabilized-pov.md @@ -0,0 +1,90 @@ +# A fly living in the NETSPHERE — stabilized POV + +[**Watch the full 60-second video · 720p · 30 fps**](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4) + +[![Frames from the continuous recording inside the NETSPHERE](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-contact_sheet.png)](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.mp4) + +An anatomically detailed fruit fly flies through a collidable, BLAME!-inspired interior. The official flybody flight policy drives its actuators on CUDA, MuJoCo Warp integrates its body and wing aerodynamics, and a geometric navigator chooses reference commands around obstacles. This recording follows the fly from the position of its physical head. + +The previous first-person camera followed instantaneous flight velocity. Sampling that rapidly oscillating signal at video rate produced distracting changes in viewing direction. The new `comfort` mode reduces yaw acceleration by **97.18%** on this take and keeps the horizon level. Both recordings use the same integrated flight states and compiled world. + +## What changed + +The renderer averages the measured head positions within each exposure, smooths that recorded path with a symmetric Gaussian (σ = 0.25 s), and uses its tangent for yaw. Pitch and roll stay at zero. Only the viewing direction is filtered: the camera remains at the integrated head position in every exposure sample. + +This is an offline camera for human viewers. The symmetric filter uses neighboring past and future recorded states; it is not a real-time controller, a model of compound-eye vision, or a claim about what a fly perceives. The navigator uses known geometry, not learned vision or a connectome. The fly's anatomy is hidden from this camera to avoid rendering the inside of its head; its physical body, wings and collision geometry remain active. + +The world in this preview is the updated 542-geom interior, including the fly: a deep shaft, stacked galleries, concrete pillars, metal ducts and sagging cables. It is already part of the simulation; this camera change does not modify its geometry or the flight controller. The earlier third-person district remains available in the [v0.1.0 release](https://github.com/bitdeep/fly-netsphere/releases/tag/v0.1.0). + +## Measured result + +Take: `blame_pov_60s`, seed 0, continuous GPU simulation on an RTX 4090. + +| Measure | Result | +|---|---:| +| Simulated time / decoded video duration | 60.000 s / 60.000 s | +| Video | 1280 × 720, 30 fps, 1,800 frames | +| Exposure | 8 physical sub-poses per frame, 1/120 s | +| Physics steps / controller steps | 1,200,000 / 300,000 | +| World contacts / episode resets / solver overflow flags | 0 / 0 / 0 | +| Distance flown | 12.014 m | +| Altitude range | 3.21–7.18 cm | +| Maximum reference tracking error | 0.671 mm | +| Minimum conservative whole-body clearance | 8.44 mm | +| Median anatomical pitch | 35.32° | +| Navigator goals reached / avoidance updates | 16 / 5,532 | +| Maximum checked camera-to-head position error | 0.0000381 mm | +| Yaw acceleration RMS, previous → comfort | 1,624.89 → 45.89 °/s² | +| Pitch acceleration RMS, previous → comfort | 1,469.42 → 0.00 °/s² | + +The angular comparison reconstructs the previous camera's exact yaw/pitch recurrence from the saved velocities and measures the new gaze from the recorded OpenGL camera forward vectors. Acceleration is the second difference of unwrapped angles at 30 fps. The percentage describes this recording and this metric; it is not a perceptual comfort study. + +`verify_take.py` checks the decoded frame count and duration, physical continuity, saved-state and model hashes, whole-body clearance, head attachment, per-frame world visibility, level camera basis and yaw acceleration RMS below 150 °/s². All 1,800 frames passed. A separate six-second preview also passed before rendering the complete minute. + +## Download and verify + +The [preview release](https://github.com/bitdeep/fly-netsphere/releases/tag/pov-stabilization-preview-1) includes the original MP4 bytes, the physical states and compiled world, simulation metrics, camera measurements, validation, comparison and a contact sheet. Filenames use the `blame_pov_60s-` prefix; `SHA256SUMS` covers every asset except itself. + +| Evidence | Download | +|---|---| +| Simulation and source hashes | [metrics.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-metrics.json) | +| Physical and video acceptance | [fly_city_stabilized.validation.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.validation.json) | +| Per-frame camera measurements | [fly_city_stabilized.render.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-fly_city_stabilized.render.json) | +| Previous/comfort angular comparison | [stabilization_comparison.json](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/blame_pov_60s-stabilization_comparison.json) | +| Asset integrity | [SHA256SUMS](https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1/SHA256SUMS) | + +Video SHA-256: + +```text +b7b88f119f66ceaf782f92d5bcbedbe7b3b22d258d8416f0bd260323e60786f7 +``` + +After the [repository setup](../README.md#quick-start), download the published take and restore its filenames: + +```bash +mkdir -p out/pov-preview +base=https://github.com/bitdeep/fly-netsphere/releases/download/pov-stabilization-preview-1 +curl -fL "$base/SHA256SUMS" -o out/pov-preview/SHA256SUMS +for asset in fly_city_stabilized.mp4 fly_city_stabilized.render.json \ + fly_city_stabilized.validation.json metrics.json states.npz model.mjb \ + stabilization_comparison.json contact_sheet.png; do + curl -fL "$base/blame_pov_60s-$asset" -o "out/pov-preview/blame_pov_60s-$asset" +done +(cd out/pov-preview && sha256sum -c SHA256SUMS) +for asset in out/pov-preview/blame_pov_60s-*; do + mv "$asset" "out/pov-preview/${asset##*/blame_pov_60s-}" +done +docker compose run --rm fly python scripts/verify_take.py out/pov-preview \ + --seconds 60 --video fly_city_stabilized.mp4 +``` + +Re-render those states with the comfort camera, without recomputing physics: + +```bash +docker compose run --rm fly python scripts/render_city.py out/pov-preview \ + --view first-person --stabilization comfort --output out/pov-preview/rerender.mp4 +docker compose run --rm fly python scripts/verify_take.py out/pov-preview \ + --seconds 60 --video rerender.mp4 +``` + +Use `--stabilization legacy` to compare the previous gaze. Rendering on different drivers or encoders may produce different MP4 bytes; the published SHA identifies the original recording. The state and model hashes identify the physical take independently of the camera. diff --git a/scripts/render_city.py b/scripts/render_city.py index 6cd183a..313a323 100644 --- a/scripts/render_city.py +++ b/scripts/render_city.py @@ -16,6 +16,39 @@ from flight_runtime import make_environment +def comfort_headings(physics, states, head_id, ns, sigma_seconds=.25): + """Offline gaze from measured displacement; never feeds the physical model. + + Instantaneous flapping velocity aliases badly at video rate. Average each + exposure's actual head positions, smooth that path symmetrically, and use + its tangent for yaw. Only orientation is filtered; the eye stays at the + integrated head position in every exposure sample. + """ + poses = states["shutter_qpos"] if "shutter_qpos" in states else states["qpos"] + samples = [] + for pose in poses: + physics.data.qpos[:] = pose + mujoco.mj_kinematics(physics.model.ptr, physics.data.ptr) + samples.append(physics.data.xpos[head_id].copy()) + positions = np.asarray(samples).reshape(-1, ns, 3).mean(axis=1) + if len(positions) < 3: + raise ValueError("Comfort camera needs at least three recorded frames") + dt = 1/30 + radius = int(np.ceil(4*sigma_seconds/dt)) + offsets = np.arange(-radius, radius+1) + kernel = np.exp(-.5*(offsets*dt/sigma_seconds)**2) + kernel /= kernel.sum() + # Linear extension preserves constant flight speed at the two endpoints. + padded = np.concatenate([ + positions[0]+np.arange(-radius, 0)[:, None]*(positions[1]-positions[0]), + positions, + positions[-1]+np.arange(1, radius+1)[:, None]*(positions[-1]-positions[-2])]) + smooth = np.stack([np.convolve(padded[:, j], kernel, "valid") + for j in range(3)], axis=-1) + velocity = np.gradient(smooth, dt, axis=0) + return np.unwrap(np.arctan2(velocity[:, 1], velocity[:, 0])) + + def main(): parser = argparse.ArgumentParser() parser.add_argument("take", type=Path) @@ -28,6 +61,8 @@ def main(): parser.add_argument("--view", choices=["third-person", "first-person"], default="third-person") parser.add_argument("--fov", type=float, default=65, help="Vertical field of view for the head-mounted first-person camera") + parser.add_argument("--stabilization", choices=["comfort", "legacy"], default="comfort", + help="First-person gaze: level horizon and averaged displacement, or original velocity following") parser.add_argument("--output", type=Path) args = parser.parse_args() metrics = json.loads((args.take/"metrics.json").read_text()) @@ -47,6 +82,9 @@ def main(): head_id = physics.model.name2id("walker/head", "body") if first_person: physics.model.vis.global_.fovy = args.fov + ns = metrics.get("shutter_samples", 1) + comfort = first_person and args.stabilization == "comfort" + headings = comfort_headings(physics, states, head_id, ns) if comfort else None cam = MovableCamera(physics, height=args.height, width=args.width) cam.scene.flags[mujoco.mjtRndFlag.mjRND_FOG] = True cam.option.flags[mujoco.mjtVisFlag.mjVIS_CONTACTPOINT] = False @@ -77,12 +115,17 @@ def main(): hit_geom = np.array([-1], dtype=np.int32) visible_pixels = [] world_pixels, camera_positions, head_positions, luminance = [], [], [], [] + camera_forwards, camera_ups = [], [] try: for i, (t, qpos, qvel) in enumerate(zip(states["time"], states["qpos"], states["qvel"])): - target_yaw = np.arctan2(qvel[1], qvel[0]) - yaw += (.5 if first_person else .13)*np.arctan2(np.sin(target_yaw-yaw), np.cos(target_yaw-yaw)) - target_pitch = np.arctan2(qvel[2], np.linalg.norm(qvel[:2])) - gaze_pitch += .25*(target_pitch-gaze_pitch) + if comfort: + yaw = headings[i] + gaze_pitch = 0. + else: + target_yaw = np.arctan2(qvel[1], qvel[0]) + yaw += (.5 if first_person else .13)*np.arctan2(np.sin(target_yaw-yaw), np.cos(target_yaw-yaw)) + target_pitch = np.arctan2(qvel[2], np.linalg.norm(qvel[:2])) + gaze_pitch += .25*(target_pitch-gaze_pitch) azimuth = np.rad2deg(yaw)+args.azimuth_offset az, el = np.deg2rad([azimuth, args.elevation]) toward_camera = -np.array([np.cos(az)*np.cos(el), np.sin(az)*np.cos(el), np.sin(el)]) @@ -94,7 +137,6 @@ def main(): if not first_person and safe_distance < .5: raise RuntimeError(f"Camera corridor too narrow at frame {i}: {safe_distance}") camera_distance = min(camera_distance+.12*(safe_distance-camera_distance), safe_distance) - ns = metrics.get("shutter_samples", 1) subframes = states["shutter_qpos"][i*ns:(i+1)*ns] if "shutter_qpos" in states else [qpos] if not len(subframes): break @@ -105,8 +147,8 @@ def main(): physics.data.time = t mujoco.mj_forward(physics.model.ptr, physics.data.ptr) if first_person: - # Real head position; direction follows measured flight velocity. - # This is a stabilized human POV, not a compound-eye model. + # Real head position with a human viewing direction. + # Gaze stabilization never changes the integrated anatomy. forward = np.array([np.cos(yaw)*np.cos(gaze_pitch), np.sin(yaw)*np.cos(gaze_pitch), np.sin(gaze_pitch)]) eye = physics.data.xpos[head_id].copy() @@ -120,6 +162,8 @@ def main(): encoder.stdin.write(image.tobytes()) if first_person: camera_positions.append(np.mean([c.pos for c in cam.scene.camera], axis=0).tolist()) + camera_forwards.append(np.mean([c.forward for c in cam.scene.camera], axis=0).tolist()) + camera_ups.append(np.mean([c.up for c in cam.scene.camera], axis=0).tolist()) head_positions.append(eye.tolist()) luminance.append(float(image.mean())) visibility_physics.data.qpos[:] = subframes[-1] @@ -167,9 +211,16 @@ def main(): "camera_collision_avoidance"): qa.pop(key) qa.update(camera_position_source="integrated walker/head body position", - gaze_source="smoothed measured velocity, horizon stabilized", + stabilization=args.stabilization, + gaze_source=("tangent of Gaussian-smoothed measured head displacement; fixed level horizon" + if comfort else "smoothed instantaneous velocity, horizon stabilized"), + gaze_sigma_seconds=.25 if comfort else None, + gaze_offline_symmetric=comfort, anatomy_hidden_in_observer=True, vertical_fov_deg=args.fov, camera_positions_cm=camera_positions, head_positions_cm=head_positions, + camera_forwards=camera_forwards, camera_ups=camera_ups, + source_states_sha256=metrics["states_sha256"], + source_model_sha256=metrics["model_sha256"], minimum_mean_luminance=min(luminance), maximum_head_camera_error_cm=float(np.linalg.norm( np.asarray(camera_positions)-head_positions, axis=1).max())) diff --git a/scripts/verify_take.py b/scripts/verify_take.py index 2b9bd8b..7de449f 100644 --- a/scripts/verify_take.py +++ b/scripts/verify_take.py @@ -92,6 +92,22 @@ def main(): check(len(render["visible_world_pixels"]) == len(state["qpos"]), "Incomplete POV visibility check") check(render["minimum_visible_world_pixels"] >= 1280*720*.25, "POV world not readable") check(render["minimum_mean_luminance"] >= 8, "POV too dark") + if render.get("stabilization") == "comfort": + forward = np.asarray(render["camera_forwards"]) + up = np.asarray(render["camera_ups"]) + check(forward.shape == up.shape == (len(state["qpos"]), 3), "Missing camera basis") + check(np.isfinite(forward).all() and np.isfinite(up).all(), "Nonfinite camera basis") + check(np.max(np.abs(np.linalg.norm(forward, axis=1)-1)) < 1e-5, "Invalid gaze direction") + check(np.max(np.abs(forward[:, 2])) < 1e-5, "Comfort gaze bobs vertically") + check(np.max(np.abs(up-[0, 0, 1])) < 1e-5, "Comfort horizon rolls") + check(render["source_states_sha256"] == meta["states_sha256"], "Changed physical take") + check(render["source_model_sha256"] == meta["model_sha256"], "Changed physical world") + yaw = np.unwrap(np.arctan2(forward[:, 1], forward[:, 0])) + yaw_acceleration = np.rad2deg(np.diff(yaw, n=2))*30**2 + # The original take had >1,600 deg/s² RMS from sampled wingbeat + # velocity. Check actual rendered orientations, not filter settings. + yaw_acceleration_rms = float(np.sqrt(np.mean(yaw_acceleration**2))) + check(yaw_acceleration_rms < 150, "Comfort camera still has abrupt angular motion") else: check(len(render["visible_fly_pixels"]) == len(state["qpos"]), "Incomplete visibility check") check(render["minimum_visible_fly_pixels"] >= 600, "Fly not visible throughout video") @@ -107,6 +123,11 @@ def main(): if first_person: report.update(maximum_head_camera_error_cm=float(max(head_errors)), minimum_visible_world_pixels=render["minimum_visible_world_pixels"]) + if render.get("stabilization") == "comfort": + report.update(stabilization="comfort", + yaw_acceleration_rms_deg_s2=yaw_acceleration_rms, + maximum_vertical_gaze_component=float(np.max(np.abs(forward[:, 2]))), + unchanged_physical_states=True) report_path = take/"validation.json" if args.video == "fly_city.mp4" else video_path.with_suffix(".validation.json") report_path.write_text(json.dumps(report, indent=2)+"\n") print(json.dumps(report, indent=2))