Skip to content

feat(robots): add opt-in direct action primitive - #203

Open
Elessar123 wants to merge 2 commits into
RLinf:mainfrom
Elessar123:feat/direct-action
Open

Elessar123 wants to merge 2 commits into
RLinf:mainfrom
Elessar123:feat/direct-action

Conversation

@Elessar123

@Elessar123 Elessar123 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Agents can use VLA and scripted primitives such as move_to, but cannot submit a native environment action as a planner tool. This adds execute_action(values=[...]) alongside those primitives for LIBERO, RoboCasa, and RoboTwin, exposed only when the run starts with --enable-direct-action.

Environment-owned action contract

  • The connected environment supplies native action layouts and per-coordinate bounds through the read-only env.get_action_spec RPC. The shared client generates the tool schema and validates inputs from that specification; the primitive has no fixed action dimensions or blanket [-1, 1] limit.
  • LIBERO and RoboCasa read the active simulator's native action_spec. RoboTwin describes native joint/pose waypoint layouts from its configured robot. Typed environments advertise their supported action_type values and default mode.
  • Existing environment clients, cancellation, observation updates, recordings, and recipe export are reused. RoboCasa invalidates VLA history after a direct action so subsequent VLA calls reseed from the current scene.
  • Default runs do not register the new tool or query its specification. No new dependencies. External environment servers must be updated with the client before enabling this optional tool.
  • Paired README documentation and offline tests cover enablement, action forwarding, different environment dimensions and bounds, invalid inputs, cancellation, state records, and VLA history continuity.

This makes the direct-action primitive independent of a particular robot's vector size. It does not broaden the set of robot/controller configurations supported by the existing execution backends.

Validation

  • pre-commit run --all-files and pre-commit run --files rpent/robots/components/action_spec.py — passed.
  • .venv/bin/python -m pytest tests/unit_tests/robots/test_direct_action.py -q --tb=short36 passed.
  • .venv/bin/python -m pytest tests/unit_tests -q --tb=short613 passed, 3 skipped, 4 failed. All four failures are in test_pi05_vla_server_contracts.py because the test environment lacks torch; the same four failures reproduced on unmodified main.
  • Manual simulation smoke tests on the revised implementation used the changed checkout, production robot runtime hooks with only the environment component, real RPC clients, and Toolkit.execute_tool("execute_action", ...) on GPU 3. Each test fetched the live environment action specification:
    • LIBERO-PRO, libero_object_swap, task 2, seed 0: five actions moved the end effector 16.42 mm; six state records, updated images, and an episode video were produced.
    • RoboCasa, OpenDrawer, target split, seed 0: eight actions moved the base 1.59 mm; nine state records, updated images, and an episode video were produced.
    • Both owned environment processes shut down successfully.

The simulation checks establish action execution and observation/artifact updates, not autonomous planner performance or benchmark task success. RoboTwin simulation execution and live VLA/direct-action alternation remain unverified. RoboTwin native-layout tests require its optional runtime and were skipped locally; RoboCasa's VLA-history reseeding is covered offline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant