Skip to content

feat(franka): add supervised visual task-card generation and replay - #187

Draft
Elessar123 wants to merge 1 commit into
RLinf:mainfrom
Elessar123:feat/franka-visual-task-cards
Draft

Elessar123 wants to merge 1 commit into
RLinf:mainfrom
Elessar123:feat/franka-visual-task-cards

Conversation

@Elessar123

Copy link
Copy Markdown
Contributor

Description

Recorded Franka runs already contain ordered primitive commands and RGB-D snapshots, but could not be replayed as visual task cards. This adds supervised generation and replay for single and dual Franka through the existing TaskCardPlanner / RobotSpec.replay_card hook.

  • Generate a versioned card from states.json, an operator-confirmed successful source, and per-step semantic/rotation annotations. Molmo grounds source images; calibrated RGB-D supplies 3-D anchor offsets.
  • Re-localize each movement on current images and recompute move_delta. Preserve sequential left/right primitive calls and current VLA primitives; rotate by an explicitly annotated relative delta or fixed quaternion target.
  • Follow upstream's shared right_base dual-arm motion convention, transform left-arm targets only for native workspace checks, and reject legacy unlabelled dual-arm state records.
  • Require terminal confirmation before environment initialization/reset, before execution, and for final task success. Save attempted recipe JSONL, outcome JSON and source-to-replay step mapping alongside normal state/image artifacts; stop subsequent actions on projection or primitive failure.
  • Reuse Toolkit, EnvState, Molmo client and robot projection helpers. LIBERO execution heuristics and card format remain unchanged. Includes English/Chinese usage docs.

Rebased onto 9040f84 and adapted to the new dual-Franka state/perception/operator interfaces and molmo.ground RPC. Existing dual-Franka exploration remains separate.

Draft limitations: annotations are explicit, not automatically inferred from hidden reasoning; object-relative orientation estimation, Dashboard confirmation UI and automatic memory publication are not implemented. Cards require matching configuration/calibration hashes. Workspace and motion bounds are not collision detection or a hardware emergency stop.

Testing

  • python -m pytest -q tests/unit_tests/robots/franka tests/unit_tests/robots/dual_franka tests/unit_tests/robots/libero/test_task_card.py tests/unit_tests/rpent/cli/test_main_contracts.py: 109 passed. Environment reports an existing unknown timeout pytest config warning.
  • Ten loopback scenarios use real HTTP/socket RPC, the real dual-Franka Toolkit and production RGB-D projection with a software rig and deterministic image grounder: shifted targets, human rejection, arm failure, invalid depth and abort.
  • Ruff lint/format on changed Python files and git diff --check.

Manual verification

No physical robot, real Molmo model, VLA checkpoint or simulator physics was run. Hardware/model-service validation remains required; this PR is intentionally draft. Loopback tests validate software orchestration and geometry, not real manipulation success.

Checklist

  • My code follows the code style of this project.
  • I have updated related documentation when needed.
  • I have added tests to cover my changes when needed.
  • All new and existing tests passed. (Focused suites above passed; full repository/hardware suites were not run.)

Signed-off-by: Yixian Zhang <105852015+Elessar123@users.noreply.github.com>
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 RPent-cn | 🛠️ Build #34565955 | 📁 Comparing 2d47f7a against latest (9040f84)

  🔍 Preview build  

4 files changed
+ rst_source/usage/franka_task_card.html
± rst_source/usage/dual_franka.html
± rst_source/usage/franka.html
± rst_source/usage/so101.html

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 RPent | 🛠️ Build #34565954 | 📁 Comparing 2d47f7a against latest (9040f84)

  🔍 Preview build  

4 files changed
+ rst_source/usage/franka_task_card.html
± rst_source/usage/dual_franka.html
± rst_source/usage/franka.html
± rst_source/usage/so101.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant