up to 3D?

[Tianyi20]

Hi, it seems the demos are 2D tabletop cases. How to upgrade to 3D? Could the author share some good insights?

[Vector-Wangel]

Hi, thanks for the interest! Scaling to 3D doesn't really change the ManiDreams loop. The TSIP, cage, and solver are already dimension-agnostic. There are some primitive idea to scale up to 3d.

1. 2D decomposition. Keep the planar pipeline that already runs on the real robot (SAM2 on the xy plane, planar cages) and deal with z by splitting the task into planar steps, or by running two views (top-down and side) with a composite cage. Cheapest, reuses everything we already have. Downside is it assumes a dominant manipulation plane, so out-of-plane motion like toppling won't show up.
2. Object-centric 3D movement. Let the DRIS hold SE(3) poses instead of pixels. Each instance becomes a 6-DoF pose, and the cage becomes a 3D region, position ball plus orientation cone. Most direct extension of the sim pipeline, and the Geometric3DCage is already a special case of this. The cost shows up in perception: you need a 6-DoF pose tracker instead of planar segmentation, which gets hard in clutter.
3. Point cloud prediction. Let the DRIS hold a point cloud. It's just the 3D version of the pixel-space DRIS we use on the real robot. The TSIP predicts the next cloud and the cage goes volumetric. Most general, and the right call for deformables or anything where flattening the geometry throws away what matters. Catch is prediction cost, a point-cloud TSIP is heavy, so it fits best in plan-then-execute mode. Similar works can be seen in PointWorld