Add the MolmoSpaces env-server integration

.basedpyright/baseline.json

@@ -5628,16 +5628,6 @@
                 }
             }
         ],
-        "./positronic/simulator/env_server/proxy.py": [
-            {
-                "code": "reportOptionalMemberAccess",
-                "range": {
-                    "startColumn": 28,
-                    "endColumn": 32,
-                    "lineCount": 1
-                }
-            }
-        ],
         "./positronic/simulator/env_server/tests/mujoco_env.py": [
             {
                 "code": "reportArgumentType",
@@ -5687,14 +5677,6 @@
                     "lineCount": 1
                 }
             },
-            {
-                "code": "reportOptionalMemberAccess",
-                "range": {
-                    "startColumn": 60,
-                    "endColumn": 64,
-                    "lineCount": 1
-                }
-            },
             {
                 "code": "reportAttributeAccessIssue",
                 "range": {
@@ -5705,14 +5687,6 @@
             }
         ],
         "./positronic/simulator/env_server/tests/test_remote_env.py": [
-            {
-                "code": "reportOptionalSubscript",
-                "range": {
-                    "startColumn": 22,
-                    "endColumn": 32,
-                    "lineCount": 1
-                }
-            },
             {
                 "code": "reportOptionalMemberAccess",
                 "range": {

ARCHITECTURE.md

@@ -165,3 +165,25 @@ leaks a timing vocabulary into the dataset core. Hence telemetry is a set of sid
 process, next to the dataset but never inside it: nested spans for the phase split and a free-running
 machine-load sampler, wall-clock native, owned by `positronic/telemetry.py`. The pass report is an
 offline reduce over those raw files, so nothing is stored twice and the dataset stays clock-agnostic.
+
+**An adoption loses nothing.** A task is defined in real or in one simulator, and carries that
+simulator with it — object poses, success criteria and horizons included. What a customer buys is
+one API across all of them: they implement a single Positronic policy interface and their model
+runs on every supported env, giving up nothing the env offers natively. Two requirements hold that
+up, one on each side of the interface:
+
+- Given a policy that already drives an env directly, it must be possible to construct a Positronic
+  `Policy` equivalent to it. This binds policy construction as much as it binds the adoption.
+- An adoption's capabilities match what its env provides natively, so a Positronic run reproduces
+  the env's own run: a deterministic env to byte-identical outcomes (modulo wire format), a
+  non-deterministic env to an identical sim/inference call sequence (same count, same order).
+
+Every sim-env adoption ships a native-vs-Positronic parity test that drives one pinned episode
+through both stacks and asserts this, re-run on every bump of the sim's pinned version.
+
+The episode horizon is one case of that reproduction rather than a rule of its own: a task that
+defines a horizon has it enforced by the env, which reports expiry through the same terminal `done`
+a success uses; a task that defines none leaves nothing to reproduce. The harness `Task.timeout` is
+only a runaway-cost safety net, so the config that knows the benchmark derives the timeout from the
+horizon it declares rather than taking one on faith — a budget below the horizon would silently
+truncate valid episodes and score them as failures.

CLAUDE.md

@@ -115,4 +115,6 @@
 # Infrastructure
 - Machines, Docker contexts and images: `docker/CONTEXTS.md`
 - Model-specific workflows: `positronic/vendors/{lerobot,gr00t,openpi}/README.md`
+- Inference serving, and the adapter/codec/wire-client separation of responsibilities (read BEFORE
+  writing a sim/rig adapter): `positronic/offboard/README.md`
 - Reconstructing previous runs: read `run_metadata_*.yaml` and episode `static.json` from output directory

docs/evaluation.md

@@ -9,6 +9,7 @@ You ship a new checkpoint and want a clean answer to one question: is it actuall
 ## What you get
 
 - **One checkpoint, every target.** Sim: LIBERO, RoboLab (NVIDIA Isaac Lab), MolmoSpaces. Real hardware: the DROID setup (Franka FR3 + Robotiq 2F-85), bimanual next. Serve a DROID policy once and it runs across all of them, and on the rig, with nothing to port. Sim for cheap, broad iteration; real hardware as ground truth.
+- **Native benchmarks, comparable scores.** Each benchmark runs with its own task definitions and horizons — the run reproduces the native benchmark rather than a re-interpretation of it — so your score is comparable with the benchmark's own published numbers.
 - **Blinded A/B.** Your checkpoint against your own previous checkpoints, or against our maintained baselines (π0.5, GR00T, SmolVLA, ACT) — randomized and blinded, so lighting and setup drift don't bias the result.
 - **Every run returned.** Multi-view video, full telemetry, and the complete run dataset — not just a success rate. Yours to analyze.
 - **Latency-honest execution.** On real hardware, inference and network delay are real — a slow model is scored as slow. In sim the world pauses during inference by default (as in other harnesses), but you can charge the model's measured inference time with `--charge_inference_time=True`, so sim scores reflect the delay the robot would actually feel — something sim-only harnesses can't model.

positronic/cfg/eval/sim/molmo.py

@@ -0,0 +1,99 @@
+import logging
+from pathlib import Path
+
+import configuronic as cfn
+
+from positronic.cfg.eval import number_trials, spec
+from positronic.drivers.roboarm.models import GRASP_SITE_LINK, bundled_franka_model
+from positronic.eval import Eval, Observation, Task
+from positronic.eval import keys as eval_keys
+from positronic.simulator.env_server.proxy import RemoteEnvControlSystem, remote_franka_embodiment
+from positronic.simulator.molmo_spaces import keys as molmo_keys
+from positronic.simulator.molmo_spaces import mapping
+from positronic.simulator.molmo_spaces.adapter import DEFAULT_CAMERA_DICT, MolmoAdapter
+from positronic.simulator.molmo_spaces.launcher import serve_molmo_spaces
+
+# How far the harness deadline sits above the benchmark horizon. Being sim-time, the spare budget costs
+# nothing unless the sim stops terminating, which is the only thing the deadline is there to catch.
+_TIMEOUT_MARGIN_SEC = 10.0
+
+
+@cfn.config(camera_dict=DEFAULT_CAMERA_DICT, episodes=None, trial_count=1, timeout=None, seed=None)
+def _molmo_eval(
+    benchmark_dir: str,
+    episodes: int | list[int] | None,
+    trial_count: int,
+    timeout: float | None,
+    camera_dict: dict[str, str],
+    seed: int | None,
+) -> Eval:
+    """A MolmoSpaces eval: the embodiment proxies a remote MolmoSpaces env, the task carries the scenario.
+
+    MolmoSpaces (https://github.com/allenai/molmospaces) is AllenAI's MuJoCo manipulation benchmark on the DROID
+    rig (Franka arm + Robotiq 2F-85) across ProcTHOR scenes; a benchmark is a directory holding a ``benchmark.json``
+    (a JSON list of episode specs — house, task, exact object poses, cameras, language goal), so
+    ``--eval.benchmark_dir`` names that directory and ``--eval.episodes`` optionally pins a subset of episode
+    indices (default: the whole benchmark). The asset packs live under ``MLSPACES_ASSETS_DIR``.
+
+    positronic launches a single task-agnostic env server in MolmoSpaces' own interpreter; the proxy drives it
+    over the socket, the env answers which episodes the sweep runs, and the episode index rides each trial's reset
+    token. The instruction is never pinned: the task reads its language live from the env, which reports the
+    episode's resolved goal in every reset's meta. Episodes are exact-pose deterministic, so ``trial_count``
+    defaults to 1.
+
+    ``timeout`` is not the benchmark horizon — the sim owns that (the benchmark's ``task_horizon_sec``, enforced
+    env-side and delivered as a terminal ``done``). It is only a runaway-cost safety net for a sim that never
+    terminates, so its default is the benchmark's own horizon plus a margin. An explicit value can only lower the
+    deadline, never raise it, and one at or below the horizon truncates valid episodes — so any value that
+    differs from the default is warned about.
+    """
+    # A non-positive count yields no trials at all, and an empty plan reads to the self-driving harness as a
+    # finished run — the command would exit 0 having evaluated nothing.
+    if trial_count < 1:
+        raise ValueError(f'--eval.trial_count must be at least 1, got {trial_count}')
+    proxy = RemoteEnvControlSystem(MolmoAdapter(camera_dict), serve_molmo_spaces(Path(benchmark_dir)))
+    # MolmoSpaces drives a Franka DROID rig; recordings carry the same model (URDF + meshes + joint names +
+    # control frame) for the 3D viewer and offline IK, supplied here since the molmo server can't import
+    # positronic to emit it via ``robot_meta``. ``DEFAULT_FRAME`` is declared on the gripper's grasp site,
+    # which is where ``env.py`` reports ``robot_state.ee_pose`` and resolves Cartesian targets, so a policy
+    # frame reached from it via ``ChangeEEFrame`` and offline IK over a recording both anchor correctly.
+    embodiment = remote_franka_embodiment(
+        proxy, camera_dict, descriptor='remote.molmo_spaces.droid', static_meta=bundled_franka_model(GRASP_SITE_LINK)
+    )
+    # The env's full MuJoCo state is recorded as privileged ground truth, never fed to the policy.
+    privileged = {mapping.OBS_SIM_STATE: Observation(proxy.privileged[mapping.OBS_SIM_STATE], None)}
+
+    def tasks() -> list[Task]:
+        params = proxy.tasks(spec(episodes=episodes))
+        # The benchmark declares one horizon over all its episodes (the env refuses an inconsistent one), so one
+        # backstop deadline covers the run.
+        backstop = params[0][molmo_keys.TASK_HORIZON] + _TIMEOUT_MARGIN_SEC
+        if timeout is not None and timeout != backstop:
+            logging.warning(
+                '--eval.timeout %ss overrides the benchmark backstop of %ss (the %ss horizon plus a margin); '
+                'running with %ss. The deadline only catches a sim that stopped terminating, and a deadline at '
+                'or below the horizon cuts valid episodes short and scores them as failures.',
+                timeout,
+                backstop,
+                params[0][molmo_keys.TASK_HORIZON],
+                min(timeout, backstop),
+            )
+        deadline = backstop if timeout is None else min(timeout, backstop)
+        task = Task(instruction_source=lambda: proxy.meta[mapping.META_TASK], timeout_sec=deadline)
+        # Benchmark episodes are exact-pose deterministic and carry their own seed. An unset ``seed`` leaves
+        # ``eval.seed`` off the trial, so the env falls back to the episode's spec seed (reproducing the
+        # benchmark); an explicit ``seed`` overrides it, sweeping ``seed .. seed + trial_count - 1``.
+        return number_trials([
+            (task, {**p, **({eval_keys.SEED: seed + t} if seed is not None else {})})
+            for p in params
+            for t in range(trial_count)
+        ])
+
+    return Eval(embodiment, tasks, privileged=privileged, done=proxy.done)
+
+
+# The whole benchmark in one run (every episode in ``--eval.benchmark_dir``'s benchmark.json).
+benchmark = _molmo_eval
+
+# A single-episode smoke target: the first episode of the benchmark.
+first_episode = _molmo_eval.override(episodes=0)

positronic/drivers/roboarm/models.py

@@ -3,7 +3,7 @@
 real arm and the live franka driver."""
 
 import xml.etree.ElementTree as ET
-from functools import lru_cache
+from functools import cache, lru_cache
 from pathlib import Path
 from typing import NamedTuple
 
@@ -99,10 +99,17 @@ def _2f85_finger(side: str, sign: int, base_rpy: str) -> list[_UrdfRow]:
     ]
 
 
+GRASP_SITE_LINK = 'gripper_grasp_site'
+# The 2F-85's grasp point — where the closed pads meet — 155mm along the flange approach axis, sharing the
+# flange's orientation. MuJoCo Menagerie's own 2F-85 places its ``grasp_site`` here, and a MuJoCo rig driving
+# this gripper measures and accepts poses at that site rather than at the arm's flange.
+_2F85_GRASP_XYZ = '0 0 0.155'
+
 # The coupler stays fixed: given an axis, the outer link hangs 19 mm out at full grip.
 _ROBOTIQ_2F85 = [
     _UrdfRow('gripper_base_mount', FLANGE_LINK, None, '0 0 0.007', _2F85_MOUNT_RPY, None, 'base_mount.stl', None),
     _UrdfRow('gripper_base', 'gripper_base_mount', None, '0 0 0.0038', '0 0 -1.5707963268', None, 'base.stl', None),
+    _UrdfRow(GRASP_SITE_LINK, FLANGE_LINK, None, _2F85_GRASP_XYZ, '0 0 0', None, None, None),
     *_2f85_finger('right', 1, '0 0 0'),
     *_2f85_finger('left', -1, '0 0 3.1415926536'),
     # RoboLab's ``eef_frame`` (``Robotiq_2F_85/base_link`` ∘ ``EEF_OFFSET_ROT``), measured off its DROID USD
@@ -174,19 +181,21 @@ def attach_robotiq_2f85(arm_root: ET.Element, meshes: dict[str, bytes]) -> dict:
     return gripper[roboarm_keys.GRIPPER]
 
 
-@lru_cache(maxsize=1)
-def bundled_franka_model() -> dict:
+@cache
+def bundled_franka_model(default_frame_at: str = EE_LINK) -> dict:
     """The bundled real franka arm + Robotiq 2F-85 for the 3D viewer: the FR3 URDF and its collision
     meshes with the 2F-85 grafted onto the flange, plus the canonical joint names and control frame.
 
-    Backfills real-robot datasets recorded before they stored their own model.
+    Backfills real-robot datasets recorded before they stored their own model. ``default_frame_at`` names the
+    link ``DEFAULT_FRAME`` is declared on: a rig that measures and drives at the gripper's grasp point passes
+    ``GRASP_SITE_LINK``, so it publishes poses in the frame it drives rather than at the franka EE.
     """
     here = Path(__file__).resolve()
     arm_root = ET.fromstring((here.parent / 'fr3.urdf').read_text())
     mesh_dir = here.parents[2] / 'assets' / 'fr3_collision'
     meshes = {f.name: f.read_bytes() for f in sorted(mesh_dir.glob('*.stl'))}
     gripper = attach_robotiq_2f85(arm_root, meshes)
-    add_default_frame(arm_root, EE_LINK)
+    add_default_frame(arm_root, default_frame_at)
     return {
         roboarm_keys.URDF: ET.tostring(arm_root, encoding='unicode'),
         'meshes': meshes,

positronic/drivers/roboarm/tests/test_ik.py

@@ -23,6 +23,8 @@
     DROID_EE_FRAME,
     DROID_EEF_LINK,
     EE_LINK,
+    FLANGE_LINK,
+    GRASP_SITE_LINK,
     bundled_franka_model,
     bundled_panda_model,
 )
@@ -245,3 +247,20 @@ def test_pickle_roundtrip(solver_cls):
     q_result = restored.solve(q_start, target_pose)
     result_pose = _fk(PANDA_URDF, q_result)
     np.testing.assert_allclose(result_pose[:3], target_pose[:3], atol=1e-3)
+
+
+def test_grasp_site_sits_at_the_2f85_grasp_point():
+    """The 2F-85's grasp point is 155mm along the flange approach axis, in the flange's own orientation —
+    where MolmoSpaces' franka_droid model places its ``gripper/grasp_site``."""
+    transform = frame_transform(bundled_franka_model()[roboarm_keys.URDF], FLANGE_LINK, GRASP_SITE_LINK)
+    np.testing.assert_allclose(transform.translation, [0.0, 0.0, 0.155], atol=1e-9)
+    np.testing.assert_allclose(transform.rotation.as_rotation_matrix, np.eye(3), atol=1e-9)
+
+
+def test_grasp_site_model_declares_the_frame_it_reports_in():
+    """A rig measuring at the grasp point declares ``DEFAULT_FRAME`` there, so the frame it publishes poses
+    in is the frame it drives."""
+    model = bundled_franka_model(GRASP_SITE_LINK)
+    assert model[roboarm_keys.CONTROL_FRAME] == DEFAULT_FRAME
+    transform = frame_transform(model[roboarm_keys.URDF], DEFAULT_FRAME, GRASP_SITE_LINK)
+    np.testing.assert_allclose(transform.as_matrix, np.eye(4), atol=1e-9)

positronic/eval/__init__.py

@@ -67,7 +67,9 @@ class Task:
     """One trial: the goal the policy conditions on, the time budget it runs under, and what sets it up."""
 
     instruction_source: str | Callable[[], str]
-    # Time budget for a rollout; ``None`` ends on ``Eval.done`` alone.
+    # Time budget for a rollout; ``None`` ends on ``Eval.done`` alone. A benchmark sim enforces the task's own
+    # horizon and reports expiry as a terminal, so there the budget is a runaway-cost net set well beyond any
+    # healthy horizon; a real or attended eval has no such terminal and the budget is the trial's actual bound.
     timeout_sec: float | None
     # What to ask for, keyed as ``Embodiment.prepare_handlers`` is; a handler this does not name goes unasked
     prepare_args: dict[str, Any] = field(default_factory=dict)

positronic/offboard/README.md

@@ -2,6 +2,35 @@
 
 This package implements the protocol and utilities for offboard policy inference, allowing robots or simulators to stream observations to a remote server and receive actions.
 
+## Separation of responsibilities: adapter vs codec vs wire client
+
+Three layers touch an observation on its way to a model, and each owns exactly one concern.
+When writing a new sim/rig adapter, check this table before adding any transform to it:
+
+| Layer | Owns | Examples |
+|---|---|---|
+| **Adapter** (per sim/rig, e.g. `simulator/molmo_spaces/adapter.py`) | Rig semantics ONLY: mapping the rig's observation/action vocabulary onto positronic's raw keys | Camera-key mapping, gripper qpos → `[0, 1]` closure, decoded commands → the rig's action format |
+| **Codec** (per model family, `policy/codec.py` subclasses) | Model preprocessing: everything the checkpoint's input distribution requires | Resize-with-pad to model resolution, prompt normalization (e.g. DROID lowercasing), state assembly |
+| **Wire client** (`InferenceClient` / `RemotePolicy`) | Transport optimization, negotiated — never semantics | Downscaling frames to the server-advertised `image_sizes` (aspect-preserving, never upscaling), optional JPEG compression |
+
+Consequences:
+
+- **An adapter never resizes, pads, normalizes prompts, or otherwise preprocesses for the model.**
+  It passes frames and text through at native fidelity. If the same transform appears in an adapter
+  and a codec, the adapter's copy is the bug: a drifted duplicate silently changes eval inputs.
+- **Bandwidth is not the adapter's problem.** The client already downsizes to what the server says
+  it needs: every `Codec` advertises its expected input sizes via the reserved `image_sizes` meta
+  key (see `Codec.meta`), the server returns it in the session handshake, and the client fits
+  frames to it before sending. This is default-on — an adapter that resizes "to keep the wire
+  payload small" is duplicating it.
+- **Codecs run on either side of the wire** — the client being the process driving the robot or sim,
+  the server being the process holding the model. positronic-native evals compose the codec around
+  `RemotePolicy` on the client (`cfg/policy.py` — the wire then carries model-sized encoded inputs,
+  and the client-side resize is disabled since `codec.meta` already reports `image_sizes`).
+  Thin-client deployments (a sim adapter in a foreign venv talking to a serverless endpoint) host
+  the codec on the server — the wire carries raw positronic keys, downsized by the negotiation
+  above. Both placements are supported; pick by where the dependencies can live.
+
 ## Protocol v1
 
 The unified WebSocket protocol is built to enable ANY hardware to connect to ANY model. All Positronic inference servers (LeRobot, GR00T, OpenPI) implement this protocol, allowing a single `.remote` policy client to work across all vendors.