feat(robots): add opt-in direct action primitive - #203
Open
Elessar123 wants to merge 2 commits into
Open
Elessar123 wants to merge 2 commits into
Elessar123 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents can use VLA and scripted primitives such as
move_to, but cannot submit a native environment action as a planner tool. This addsexecute_action(values=[...])alongside those primitives for LIBERO, RoboCasa, and RoboTwin, exposed only when the run starts with--enable-direct-action.Environment-owned action contract
env.get_action_specRPC. The shared client generates the tool schema and validates inputs from that specification; the primitive has no fixed action dimensions or blanket[-1, 1]limit.action_spec. RoboTwin describes native joint/pose waypoint layouts from its configured robot. Typed environments advertise their supportedaction_typevalues and default mode.This makes the direct-action primitive independent of a particular robot's vector size. It does not broaden the set of robot/controller configurations supported by the existing execution backends.
Validation
pre-commit run --all-filesandpre-commit run --files rpent/robots/components/action_spec.py— passed..venv/bin/python -m pytest tests/unit_tests/robots/test_direct_action.py -q --tb=short— 36 passed..venv/bin/python -m pytest tests/unit_tests -q --tb=short— 613 passed, 3 skipped, 4 failed. All four failures are intest_pi05_vla_server_contracts.pybecause the test environment lackstorch; the same four failures reproduced on unmodified main.Toolkit.execute_tool("execute_action", ...)on GPU 3. Each test fetched the live environment action specification:libero_object_swap, task 2, seed 0: five actions moved the end effector 16.42 mm; six state records, updated images, and an episode video were produced.OpenDrawer, target split, seed 0: eight actions moved the base 1.59 mm; nine state records, updated images, and an episode video were produced.The simulation checks establish action execution and observation/artifact updates, not autonomous planner performance or benchmark task success. RoboTwin simulation execution and live VLA/direct-action alternation remain unverified. RoboTwin native-layout tests require its optional runtime and were skipped locally; RoboCasa's VLA-history reseeding is covered offline.