Skip to content

feat(explore): add minimal RoboCasa and RoboTwin support - #164

Open
ZhuWeiwen315 wants to merge 6 commits into
RLinf:mainfrom
ZhuWeiwen315:feat/exploration-robotwin-robocasa
Open

feat(explore): add minimal RoboCasa and RoboTwin support#164
ZhuWeiwen315 wants to merge 6 commits into
RLinf:mainfrom
ZhuWeiwen315:feat/exploration-robotwin-robocasa

Conversation

@ZhuWeiwen315

@ZhuWeiwen315 ZhuWeiwen315 commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown

PR title

feat(explore): add minimal RoboCasa and RoboTwin support

Description

This PR adds minimal exploration-mode support for RoboCasa and RoboTwin, following RPent's existing robot-specific extension layout.

Main changes:

  • Add exploration prompts and prompt selection for RoboCasa and RoboTwin.

  • Expose each environment's ordinary episode reset through its robot-local toolkit.

  • Track per-session attempt budgets and refuse an unsolved finish while attempts remain.

  • Determine success from each environment's native signal:

    • RoboCasa: state.success
    • RoboTwin: eval_success

  • Export only the successful trajectory after the latest reset as the winning recipe.

  • Route exploration notes through a local memory inbox, with optional post-run merging.

  • Pass exploration session settings through the existing CLI and dashboard orchestration.

  • Update the existing RoboTwin tool-schema expectation for the newly exposed reset tool.

The implementation intentionally keeps environment-specific reset and success logic inside each robot adapter. It does not introduce a new shared lifecycle abstraction or modify LIBERO, shared EnvClient reset behavior, environment servers, VLA protocols, dependencies, or workflows.

Dedicated exploration contract and memory tests are intentionally left outside this minimal production-focused PR. No model checkpoints, simulator assets, credentials, run logs, or machine-local configuration are included.

Testing

Focused exploration, memory-profile, registry, CLI, and dashboard contract tests: 95 passed, 1 warning.

Full compatible unit-test suite: 589 passed, 2 deselected, 4 warnings.

The two deselected tests exercise the optional LeRobot exporter. The installed local LeRobot version rejects the existing task keyword passed to LeRobotDataset.add_frame; this is unrelated to this PR, and neither the flywheel implementation nor its tests are modified here.

The warnings come from optional pytest plugins and dependency deprecations in the local environment.

The updated contracts cover:

  • RoboTwin's exploration-only reset tool exposure.
  • HF versus local memory-profile selection for RoboCasa and RoboTwin.
  • Preservation of RoboCasa Target50's frozen results/ artifact layout.
  • Qualified local artifact identities for task configuration and split.
  • RoboCasa local task, suite, and global memory lookup.
  • Prompt rendering through the public robot registry.

Additional checks:

  • Ruff lint: passed
  • Ruff format check: passed
  • git diff --check: passed
  • Added-Python-line Chinese-character scan: passed

Manual verification

RoboCasa

  • Hardware: NVIDIA GeForce RTX 4090

  • Task: OpenDrawer

  • Split/seed: target, seed 1

  • Exploration budget: 1 session × 2 attempts

  • Planner: Codex, gpt-5.5, xhigh

  • VLA: local RLDX checkpoint

  • Result: solved on attempt 1 with one uninterrupted rldx_skill call

  • Native evidence:

    • state.success=true
    • robocasa_terminated=true
    • joint_p=1.1437

  • Artifacts:

    • valid result.json
    • semantic audit
    • one-action winning recipe
    • suite and global inbox proposals

  • Elapsed time: 220.3 seconds

RoboTwin

  • Hardware: NVIDIA GeForce RTX 4090

  • Task: beat_block_hammer

  • Configuration/seed: demo_randomized, seed 100000

  • Exploration budget: 1 session × 2 attempts

  • Planner: Codex, gpt-5.5, xhigh

  • VLA: local LingBot-VLA checkpoint

  • Reset evidence:

    • attempt 1 was closed as unsuccessful
    • the ordinary exploration reset completed
    • solving continued on attempt 2

  • Result: native eval_success=true on attempt 2 at step 12

  • Winning recipe: exactly three post-reset lingbot_act calls; attempt 1 and the reset were excluded

  • Artifacts:

    • semantic audit
    • three-action winning recipe
    • local inbox proposal

  • Elapsed time: 512.7 seconds

Both runs exited with status 0, stopped their owned environment and VLA processes, and released all GPU memory. Automatic memory merging was disabled during manual verification so that only temporary inbox output was produced.

RoboTwin emitted the existing non-blocking episode.mp4 serialization warning after success; the audit, recipe, transcript, observations, and native success evidence were preserved.

Full multi-task and multi-seed benchmark evaluation is outside the scope of this PR.

Checklist

  • My code follows the code style of this project.
  • I have updated related documentation when needed.
  • I have added tests to cover my changes when needed.
  • All affected existing tests passed.

@read-the-docs-community

read-the-docs-community Bot commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 RPent-cn | 🛠️ Build #34625531 | 📁 Comparing 6ad3b57 against latest (24541d6)

  🔍 Preview build  

24 files changed · + 5 added · ± 18 modified · - 1 deleted

+ Added

± Modified

- Deleted

@read-the-docs-community

read-the-docs-community Bot commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 RPent | 🛠️ Build #34625530 | 📁 Comparing 6ad3b57 against latest (24541d6)

  🔍 Preview build  

24 files changed · + 5 added · ± 18 modified · - 1 deleted

+ Added

± Modified

- Deleted

@ZhuWeiwen315
ZhuWeiwen315 force-pushed the feat/exploration-robotwin-robocasa branch from 3f259a8 to 755e863 Compare September 14, 2026 11:06
@ZhuWeiwen315 ZhuWeiwen315 changed the title feat(explore): add exploration support for RoboTwin and RoboCasa feat(explore): add minimal RoboCasa and RoboTwin support Sep 14, 2026
@ZhuWeiwen315
ZhuWeiwen315 force-pushed the feat/exploration-robotwin-robocasa branch from 755e863 to 0aa3a26 Compare September 15, 2026 03:17
@ZhuWeiwen315
ZhuWeiwen315 marked this pull request as ready for review September 15, 2026 03:47
@ZhuWeiwen315
ZhuWeiwen315 force-pushed the feat/exploration-robotwin-robocasa branch from 0aa3a26 to 24541a3 Compare September 15, 2026 04:36
qurakchin
qurakchin reviewed Sep 16, 2026
View reviewed changes
Comment thread docs/source-zh/rst_source/usage/robocasa.rst Outdated
Comment thread robots/robocasa/prompt_bundle.py Outdated
Comment thread rpent/cli/main.py Outdated
Comment thread robots/robocasa/prompt_bundle.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robocasa/prompt_bundle.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/robot_spec.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/prompts/explore.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/toolkit.py
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/robot_spec.py
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robocasa/prompts/explore.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/prompts/explore.py Outdated
@qurakchin

qurakchin commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

please rebase to the latest main branch

@ZhuWeiwen315
ZhuWeiwen315 force-pushed the feat/exploration-robotwin-robocasa branch from cb3f9ff to 7223b99 Compare September 17, 2026 04:06
@ZhuWeiwen315

ZhuWeiwen315 commented Sep 17, 2026

Copy link
Copy Markdown
Author

Rebased the PR onto the latest main branch at 902ac6b and addressed the follow-up exploration review comments in 7223b99. Focused tests pass (51 passed, 1 optional-plugin warning), and pre-commit lint and format checks pass.

Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robotwin/robot_spec.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread robots/robocasa/prompts/evaluate.py Outdated
Sonorous281
Sonorous281 reviewed Sep 17, 2026
View reviewed changes
Comment thread tests/unit_tests/robots/robocasa/test_memory_contracts.py Outdated
@ZhuWeiwen315
ZhuWeiwen315 force-pushed the feat/exploration-robotwin-robocasa branch from 30e66c9 to d6c5b22 Compare September 17, 2026 08:57
@ZhuWeiwen315

ZhuWeiwen315 commented Sep 17, 2026

Copy link
Copy Markdown
Author

Addressed the latest review feedback in d6c5b22 and rebased onto the latest main at d6876ca. All review threads are resolved. Local verification passes: 95 focused tests and 589 compatible unit tests, with 2 unrelated optional-LeRobot tests deselected; Ruff lint and format also pass. @qurakchin, the Unit tests and Pre-commit workflows for the current head SHA are awaiting maintainer approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qurakchin qurakchin qurakchin left review comments
@Sonorous281 Sonorous281 Sonorous281 approved these changes

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ZhuWeiwen315 @qurakchin @Sonorous281