Add OnePlanner driver + LiDAR pipeline (depends on #9)#10
Draft
hakuturu583 wants to merge 13 commits into
Draft
Conversation
Wire the OnePlanner E2E planner (tier4/OnePlanner NVlabs#112) as a wizard driver option: `driver=oneplanner` swaps in the `oneplanner:local` container in place of the default vavam/alpamayo driver services. - New `wizard/configs/driver/oneplanner.yaml` overrides `services.driver` with the OnePlanner image, mounts the shared USDZ scene from the splatsim renderer, and bind-mounts the E2E planner checkpoint read-only at `/opt/oneplanner/ckpts/e2e_planner.pth`. - Add `defines.oneplanner_ckpt` to `base_config.yaml`, defaulting to the local Downloads copy. Override with e.g. `defines.oneplanner_ckpt=/path/to.ckpt`. Not yet verified end-to-end; also blocked on the host CUDA driver being too old for the OnePlanner CUDA 13 base image (currently falls back to CPU). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Snapshot before branching to tier4-dev to fix physics/CARLA tick sync. Do not merge as-is — will be split into proper commits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The OnePlanner driver_server expects --bevfusion-config (YAML topology) and
--bevfusion-ckpt (mmcv-format state_dict) alongside --planner-ckpt for E2E
checkpoints trained with ``use_grid_sample_bev=True`` (e.g. phase2 gridbev).
Without them the planner runs without ``_bev_feature_map`` and produces
degraded trajectories on those weights.
Add two defines to the wizard base config so the host paths flow through
Hydra, and bind-mount both files into the driver container at the paths the
image expects (``/opt/oneplanner/configs/perception/model.yaml`` and
``/opt/oneplanner/ckpts/bevfusion.pth``). The BEVFusion ckpt is a separate
file because ``load_mmcv_checkpoint`` does not strip the ``model.bevfusion.``
Lightning-style prefix carried by combined E2E checkpoints; users extract it
once with ``torch.save({'state_dict': {k.removeprefix('model.bevfusion.'): v
for k, v in ckpt['state_dict'].items() if k.startswith('model.bevfusion.')}},
'bevfusion.pth')``.
Verified end-to-end: driver_server loads planner + BEVFusion cleanly against
``lidar_30e_j6gen2_120m.yaml`` + a BEVFusion-only state_dict extracted from
``oneplanner_phase2_gridbev_last.ckpt``. No shape mismatches or missing keys
reported by ``weight_converter``.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two small fixes surfaced while wiring the OnePlanner driver against a scene whose id (`odaibatest_alpasim`) is neither a clipgt UUID nor an XODR-derived identifier: - TrafficService: skip the traffic-session bootstrap (which regexes the scene id for a map UUID) when the trafficsim endpoint is skipped, while still populating `_traffic_objs` so `simulate_traffic` can replay recorded trajectories. - SensorsimService: clamp the camera-render trajectory query timestamps to the trajectory's [start, stop) so a sub-millisecond clock skew at the very first tick does not raise an out-of-range interpolation error. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduce a `Lidar`/`Lidars` data model mirroring `Camera`/`Cameras`, accumulate `driver_lidar_point_cloud` entries from ASL / broadcaster, and expose the sweeps on `ScenarioEvalInput` and `SimulationResult`. The `DEFAULT` video layout gains an opt-in projection of the latest LiDAR sweep onto the rendered camera axis, colored by forward depth. Points are uniformly subsampled to a configurable cap (`lidar_overlay_max_points`, default 8000) to keep matplotlib scatter fast — raw PANDAR128 sweeps produce ~250k points which would otherwise dominate render time. The overlay is gated by `video.overlay_lidar_on_camera` (default off) and picks its sensor via `video.lidar_id_to_overlay` (defaults to the first available sensor when unset). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Clamp LiDAR trajectory_to_pose_pair timestamps like the camera path so the final sweep does not raise when end_us hits the trajectory end. - Consolidate duplicated _traffic_objs assignment in TrafficService by moving it before the skip guard. - Drop the num_points reconciliation branch in Lidar.add_point_cloud; trust the reshape output and the buffer as the source of truth. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Alpasim sends camera-in-world poses in ROS ENU convention (Z-up), but splatsim's tile-local scene lives in a Y-up RUB frame. The adapter was subtracting `tile_local_centroid` without any rotation, so the camera optical axis ended up pointing along splatsim's +Y (up) — the rendered CAM_FRONT_WIDE view came out as a bird's-eye image instead of a forward-looking one. Route both pose_to_viewmat and pose_to_sensor_to_world through a shared `_pose_to_tile_local` helper that applies `R_ENU_TO_TILE = [[1,0,0], [0,0,1],[0,-1,0]]` before recentering. Update existing tests to reflect the new frame semantics and add a regression test verifying +Z alpasim maps to +Y splatsim. Also mount render_adapter.py alongside server.py in the splatsim renderer wizard config so this fix reaches the running container without a rebuild. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rollouts previously always began at ``rig.trajectory.time_range_us.start``,
so a 20 s simulation on a 56 s USDZ recording could only replay the first
20 s of ego motion. Introduce ``SimulationConfig.trajectory_start_us_offset``
(microseconds) so users can pivot the anchor forward — useful for skipping
uninteresting recording preambles and for feeding OnePlanner an EgoHistory
context window from an arbitrary point in the trace.
Threaded through ``_build_rollout_timing`` and exposed on
``Rig.first_camera_frame_{ranges,end}_us`` via a new ``min_frame_end_us``
kwarg so the camera anchor advances to the first frame at or after the
shifted trajectory start.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
44c44c0 introduced a Z-up→Y-up rotation on the sensor pose based on the assumption that splatsim's tile-local frame is Y-up. That assumption is wrong: the USDZ scene.json for the Odaiba capture reports ``up_axis: "z"`` and its ``tile_local_centroid`` has a small Z component (3.5m) with large horizontal offsets — the tile is Z-up like alpasim's ENU world, so no rotation is needed. Applying the rotation was rolling the camera 90° about X and producing a top-down bird's-eye view. Drop the rotation from ``_pose_to_tile_local`` and update the tests. Both ``pose_to_viewmat`` and ``pose_to_sensor_to_world`` now pass the pose orientation straight through, subtracting only ``world_origin`` (tile-local centroid) from the translation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`T_sensor_rig` in NuRec/3dgs_io USDZ exports is the rig→sensor matrix (OpenCV +Z-forward), not sensor-in-rig — documented in splatsim's own _usdz.py and verified empirically here (a probe rendering three candidate viewmats against splatsim's `iter_world_to_camera` matches this convention to fp precision, and the raw matrix places the camera 1.3m underground). The runtime composes `ego.compose(rig_to_camera)`, which requires camera-in-rig semantics, so emit the inverse of the raw field. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Masaya Kataoka <ms.kataoka@gmail.com>
The wizard's driver=oneplanner service used to bind-mount four ad-hoc
files from /tmp over the OnePlanner image, with no way to recover them
if the local /tmp was cleared. Move the still-needed ones into the repo
and reference them via ${repo-relative:'...'} so the wizard config is
self-contained.
Kept (still not upstreamed in oneplanner:local):
* encoder.py — cast has_speed_limit to bool before torch.where.
* model_loader.py — load StateNormalizer/ObservationNormalizer from
the JSON that ships in the image, instead of falling back to identity.
* driver_preprocessing.py — arc-length-upsample 20 route waypoints to
the 500 that OnePlanner expects (25 segments × 20 points).
Dropped: driver_server.py and driver_servicer.py — a rebuilt
oneplanner:local now ships them (with a newer per-session .rrd dump
loop that supersedes our per-tick fork). README documents the
extract+diff flow so we can retire more of these when future images
land the fixes.
Also defines oneplanner_normalization_json under defines: so the mount
path in oneplanner.yaml resolves without a hydra key error.
Signed-off-by: Masaya Kataoka <ms.kataoka@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lidar.py: set sensor_pose_delta to an approximate PANDAR128 rig->lidar transform so the LiDAR renders from roof height instead of ground level, which otherwise gives OnePlanner an unusable point cloud. driver_preprocessing.py: add env-gated ALIGN_DEBUG logging and NPZ sample dumping (default off) to compare alpasim's local frame against OnePlanner's alignment frame while debugging the road_border/ego frame mismatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds tier4/OnePlanner as an alternative driver container and wires up the LiDAR sensing pipeline that OnePlanner needs (point-cloud consumption from splatsim → PredictionInput.lidar_clouds → planner).
Depends on #9 (splatsim renderer integration). Base is set to
feature/splatsim-integration— please merge #9 first, then rebase this ontotier4-devfor a clean history.Commits (2 on top of #9)
src/wizard/configs/driver/oneplanner.yaml(Hydra profile that spawns the externaloneplanner:localimage)defines.oneplanner_ckptinbase_config.yamlfor host-side.ckptbind-mount at/opt/oneplanner/ckpts/e2e_planner.pthsrc/runtime/alpasim_runtime/events/lidar.py(newLidarFrameEvent)event_loop.py,unbound_rollout.py,sensorsim_service.pylidars/1lidar_pandar.yamlExternal dependencies (not in this repo)
EgoDrivergRPC-facing mode.ckptset viadefines.oneplanner_ckpt=/path/to/e2e_planner.ckptdefines.splatsim_scene_usdzdocker build -t oneplanner:local .inside a tier4/OnePlanner checkout on the wizard hostData flow
Test plan (post-cleanup)
driver=oneplanner renderer=splatsimboots end-to-end with a valid ckpt + USDZuv run pytest src/runtime/testsFollow-ups
oneplanner_ckptlayout + expected keys somewhere in docs/🤖 Generated with Claude Code