Dense LiDAR SDP hurts 3D lifting — --no_sdp outperforms on non-Aria indoor scans

[hycZoujy06]

Background:
I'm adapting Boxer to indoor LiDAR scans (pinhole RGB tiles + dense visible point clouds, not Aria). Geometry is verified: point cloud projects correctly onto the image, 2D detections look good. I use rotated0=True for Aria compatibility.

Core issue:
on some frames, adding SDP from dense LiDAR causes 3D boxes to shift or get filtered out. Most frames work fine with SDP, but a subset of frames degrade noticeably, while --no_sdp stays stable across all frames.

I suspect sdp_to_patches() assumes sparse Aria MPS feature points. Dense scan points (walls, floors, ceilings) dominate per-patch median depth, giving BoxerNet a misleading depth prior. I tried edge-guided, nearest-depth, and patch-balanced sampling — none improved over simple random downsampling, and all hurt previously good frames.

Questions:

1. Does sdp_to_patches() assume Aria-style sparse SDP distribution?
2. Any guidance on converting dense scans to Boxer-compatible SDP?

Example of badcase:

Below is the corresponding point cloud, downsampled to ≤ 3,000 points (the raw scan contains tens of thousands). Increasing the limit to 10,000 points actually makes the 3D predictions worse:

[ddetone]

its should work with lidar. we trained with both sparse (SDP) and dense (RGB-D) depth maps

can you visualize the patch encodings from sdp_to_patches() overlaid on the image? they should align with the image like this (see right side):

you can look in run_boxer.py for how to visualize them:

python run_boxer.py --input nym10_gen1 --max_n=5

[ddetone]

also just double checking -- do you know the extrinsics calibration between the LIDAR and the RGB image? you will need that in order to put the LIDAR points in the same "world coordinate frame" as the RGB camera

[hycZoujy06]

Thanks for the suggestion. I prepared a minimal sample for this affected frame and will upload it as a zip.

The zip contains:

• image.jpg: the pinhole RGB image
• points.ply: the original dense LiDAR point cloud in the global/world frame
• sdp_3000.ply / sdp_3000.npz: the randomly sampled 3000-point SDP used for my Boxer run
• rt.json: camera intrinsics and T_global_cam, with convention p_global = T_global_cam * p_cam
• boxer_result.jpg: my 2D / raw 3D / final 3D visualization generated with sdp_3000

For boxer_result.jpg, I used:

• rotated0=True
• random 3000 LiDAR points as SDP, seed=0
• thresh2d=0.25
• thresh3d=0.5

The RGB/LiDAR projection looks geometrically aligned to me, but using LiDAR-derived SDP on this frame still shifts or filters some 3D boxes compared with --no_sdp. I hope this minimal sample helps you check whether the dense point cloud should be converted differently before being used as SDP.

frame26_minimal.zip

[ddetone]

I'll try to test it soon. In the meantime, can you answer my questions above?

[ddetone]

Update -- I am able to reproduce the issue. My hypothesis is that this is actually a pretty fundamental issue in the training dataloader learning the wrong biases. I'm re-training the model now and will report back as soon as possible

[hycZoujy06]

Thanks a lot for testing and reproducing the issue.
I’ll wait for your retraining results and would be happy to test a new checkpoint if helpful.

[ddetone]

@hycZoujy06 I trained a new model and I think it fixed the issue. Its an early snapshot (still needs 2-3 days to finish converging), but looks promising:

[hycZoujy06]

Great news, thanks for the update!
The result looks much better on my sample. I’ll wait for the converged checkpoint, and I’d be happy to test it on my full data once it’s available.

[ddetone]

Hi, can you try this checkpoint referenced in the code? If you re-run the download checkpoint script it should work: fd6bf73. You rebase on top of the latest main and it should work.

Note this has only trained about 7 days and takes 21 days to fully train, so I may update it later to share the best possible model.

Let me know how it works.

[hycZoujy06]

Thanks, I’ll rebase onto the latest main, download the new checkpoint, and test it on my minimal sample first.
If it looks good, I’ll run more tests on my data and report back.

[hycZoujy06]

Thanks for the update — we’ve rerun the minimal case and it now works end-to-end on our side.
The key issue we were seeing before is resolved, and the result is stable now.

We also tried it on more complex scenes and the behavior looks good there as well. We’re looking forward to the results after training.