Skip to content

Derive the MolmoSpaces recorded control frame from their robot description, not a matching link name #593

Description

@vertix

A recorded episode carries the robot model its poses are measured in — URDF, joint names, and the control frame. The 3D viewer, offline IK, and any policy asking for a different frame all anchor to that frame, so it has to be the one the sim actually measures at.

For MolmoSpaces the two sides are joined by name, never by value:

  • The eval config supplies the model from positronic's side, because the molmo server cannot import positronic to emit its own: bundled_franka_model(GRASP_SITE_LINK) in positronic/cfg/eval/sim/molmo.py. That builds positronic's own bundled FR3 URDF with a Robotiq 2F-85 grafted on, and declares DEFAULT_FRAME at the gripper_grasp_site link.
  • Server-side, _assert_measures_at_grasp_site checks the arm move group's leaf frame is a MuJoCo site named mapping.MOLMO_GRASP_SITE.

Nothing compares where that point actually sits. A scene whose same-named site had a different flange offset would misframe every recorded pose — silently, not as a crash.

This is latent rather than live: MolmoSpaces ships one DROID rig (Franka + Robotiq 2F-85) across every benchmark, so the name determines the transform in practice.

Preferred approach: vendor their model

Rather than sending a transform across the wire, keep a copy of MolmoSpaces' own robot description in the positronic wrapper and build the recorded model from that, so both sides derive from one source instead of positronic supplying a look-alike.

This is simpler than the wire-protocol alternative and removes the mismatch by construction rather than detecting it.

Open questions for whoever picks this up — they need the asset packs, which is why they are not answered here:

  • Does MolmoSpaces distribute a URDF for the DROID rig, or only MJCF? keys.URDF is what the viewer and offline IK consume, so an MJCF-only distribution needs a conversion step or a checked-in derived URDF.
  • Is the grasp site expressible as a link/frame in that description? It is a MuJoCo site in their MJCF; the recorded model needs it as the control frame.
  • Licensing and size of the vendored assets, alongside the existing assets/fr3_collision meshes.
  • Whether the same treatment should apply to RoboLab, which takes the other route already — its model rides the env's robot_meta over the wire, since its launcher can serialize it.

Fallback if their description cannot be vendored

Send the live flange-to-grasp transform in the reset frame, have the env compute it, and derive the static metadata from it instead of from a link name. That touches protocol, server, env and the config.

Verification

Either way this needs a parity re-run on a box with the asset packs, confirming the recorded control frame is unchanged for an existing benchmark episode — the failure mode is silent, so a green test suite is not evidence.

Raised by Codex on #504 and deferred there: #504 (comment)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions