Merge final side-grip MediaPipe handover flow - #60
Closed
oyeong011 wants to merge 109 commits into
Closed
Conversation
Add an offline HSV classifier and regression gate so dispenser color logic can be tested from synthetic patches or saved image crops before camera bringup. Constraint: Current re-experiment has no live camera, and robot motion must stay out of the color-classification gate. Rejected: Testing color discrimination only through live camera/ROS nodes | unavailable and couples perception validation to hardware. Confidence: high Scope-risk: narrow Directive: Keep this path perception-only; robot execution belongs in a separate gated integration branch. Tested: python3 -m py_compile tools/perception/color_discrimination.py tools/perception/offline_color_discrimination_test.py tools/checks/check_offline_color_discrimination.py; python3 tools/checks/check_offline_color_discrimination.py Not-tested: Real dispenser images and live camera lighting conditions.
Add a dry-run-only post-shake hand tracking and handover planning sequence so the workflow can end with a human handover concept without enabling live motion near a person. Constraint: Handover near a human is an HRI safety-critical step and must not be activated by perception alone. Rejected: Adding direct robot handover motion after shaking | unsafe without hand tracking validation, operator approval, force limits, and HRI review. Confidence: high Scope-risk: moderate Directive: Keep HANDOVER_CUP_TO_HUMAN_DISABLED disabled until a separate live-executor safety review is completed. Tested: python3 -m py_compile src/azas_task_manager/azas_task_manager/cocktail_workflow_plan.py tools/checks/check_cocktail_workflow_plan.py; python3 tools/checks/check_cocktail_workflow_plan.py Not-tested: Live hand detector, camera depth, and real robot handover execution.
…ty and functionality
…-color-discrimination-20260601
Integrate the upstream cup detector model and image-only orientation utilities while keeping Azas motion pose generation behind the existing detection-to-TF pipeline. Constraint: Cup poses and robot motion must continue through /azas/cup_detection and /jarvis/tumbler_dispenser/tumbler_pose; upstream mock/base-coordinate motion code was not imported. Rejected: Vendoring yolo_pick_demo as an executable motion package | it directly computes base coordinates and includes mock pose values that conflict with Azas safety rules. Confidence: high Scope-risk: moderate Directive: Do not reintroduce upstream motion or mock coordinate paths without replacing them with Azas topic/TF contracts and real calibration evidence. Tested: PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -q src/azas_perception/test/test_depth_and_detection_logic.py; python3 -m py_compile perception and launch files; colcon build --packages-select azas_perception azas_bringup --symlink-install --allow-overriding azas_bringup Not-tested: Live RealSense/YOLO inference and real robot motion.
…and collision handling
The side-grip cleanup path was terminating the measured dispenser collision scene, and panel logs hid server-inserted prerequisites. Keep collision-scene nodes out of side-grip cleanup and report the actual server execution order. Constraint: Real-motion panel steps must fail closed, but cleanup must not remove required collision-scene prerequisites for the same run. Rejected: Disabling side-grip collision handling | the fix is to preserve prerequisite scenes, not to run without them. Confidence: high Scope-risk: narrow Directive: Keep cleanup patterns scoped to the step they clean; do not include shared prerequisite nodes in one-shot step cleanup lists. Tested: python3 -m py_compile tools/run/robot_pipeline_control_server.py; imported panel module and verified side_grip command keeps start_joint_state_relay:=true when installed; grep confirmed removed panel card keys remain absent. Not-tested: Live side_grip retry on hardware.
Measured front-hold poses are taught as link_6 targets. The panel was compensating the active GripperDA TCP into a current-TCP MoveLine target, which pushed dispenser_1 to x=0.838 and tripped the direct-move bounds. Switch panel dispenser moves to select the zero-offset link_6 TCP before moving and disable current-TCP compensation. Constraint: Do not alter measured dispenser coordinates; preserve fail-closed bounds and use the script's existing TCP selection path. Rejected: Raising direct MoveLine x_max to accept x=0.838 | that would widen the safety envelope instead of using the measured link_6 frame correctly. Confidence: high Scope-risk: narrow Directive: Keep front_hold_poses interpreted as link_6 poses unless they are re-taught with explicit measured_target_frame changes. Tested: python3 -m py_compile tools/run/robot_pipeline_control_server.py; imported panel module and verified collision-scene dedupe plus move_to_dispenser_1 command contains --set-current-tcp-before-move and --no-compensate-current-tcp. Not-tested: Live dispenser move retry on hardware.
Use dispenser ID press poses from calibration.yaml for the panel path so HTML execution no longer selects legacy color/taught posx targets. Treat the measured press pose as the final pressed target instead of deriving a lower Z from legacy press depth.\n\nConstraint: Robot coordinates must come only from measured calibration data, not LLM-generated or legacy color aliases.\nRejected: Keep target_dispenser red/green/yellow/blue mapping | It reproduces the observed legacy coordinate motion.\nConfidence: high\nScope-risk: narrow\nDirective: Do not reintroduce target_dispenser/use_taught_posx for panel press_dispenser_N paths.\nTested: python3 -m py_compile tools/run/robot_pipeline_control_server.py src/azas_dispenser/azas_dispenser/dispenser_press_node.py; measured press command generation for dispensers 1-4; colcon build --symlink-install --packages-select azas_dispenser\nNot-tested: Live robot press after restarting the running panel server
Let the one-word launcher restart only the panel HTTP server when its Python source changed, and provide an explicit --restart escape hatch, so operators do not need to remember the long cd/source/pkill sequence.\n\nConstraint: Keep robot/hardware stack cleanup separate; this launcher restarts only robot_pipeline_control_server.py.\nRejected: Tell users to keep typing pkill/source/run commands | It is error-prone and was the direct usability complaint.\nConfidence: high\nScope-risk: narrow\nDirective: Preserve azas-panel as the canonical operator entrypoint.\nTested: bash -n tools/run/open_robot_pipeline_control_panel.sh; azas-panel --restart; curl -fsS http://127.0.0.1:8765/\nNot-tested: Non-ssu workstation with missing /home/ssu/ros2_ws setup
Move measured dispenser front-hold through current-TCP compensation with a final link6 verification instead of requiring the Doosan controller to accept azas_link6_tcp. This keeps the measured link6 target while avoiding false blocks when the controller rejects TCP registration/selection.\n\nConstraint: Cup/dispenser poses remain measured front_hold_poses; no generated coordinates.\nRejected: Require operators to register azas_link6_tcp manually | It blocks the panel path and is avoidable with current-TCP compensation plus link6 verification.\nConfidence: high\nScope-risk: narrow\nDirective: Keep --verify-link6-target enabled when using current-TCP compensation.\nTested: python3 -m py_compile tools/run/move_to_measured_dispenser_front_hold.py tools/run/robot_pipeline_control_server.py; generated move_to_dispenser_1 command; azas-panel --restart; live /api/steps resolved_command check\nNot-tested: Live robot execution after compensation change
Use the same widened direct-move X guard for pick_from_dispenser_N that front-hold placement uses, and pass those bounds through the internal measured front-hold approach. This prevents the current-TCP compensation path from being accepted during placement but blocked during regrasp/lift.\n\nConstraint: Regrasp still reuses measured front_hold_poses only; no operator/LLM cup coordinates.\nRejected: Keep x_max=0.72 for regrasp | Compensated current TCP can be around x=0.84 while link6 verifies against the measured pose.\nConfidence: high\nScope-risk: narrow\nDirective: Keep pick_from_dispenser_N bounds consistent with move_to_dispenser_N compensation assumptions.\nTested: python3 -m py_compile tools/run/pick_from_measured_dispenser_front_hold.py tools/run/move_to_measured_dispenser_front_hold.py tools/run/robot_pipeline_control_server.py; generated command assertions; azas-panel --restart; live /api/steps assertions for move/press/pick\nNot-tested: Live robot execution of the full move-press-pick sequence
Constraint: commit requested before further geometry changes. Confidence: medium Scope-risk: broad Directive: Keep cup coordinates sourced from perception topics or measured calibration only. Tested: not run; checkpoint commit of existing workspace state. Not-tested: runtime ROS motion and hardware validation.
Merge the four measured nozzle/head boxes into one upright cuboid so the empty gaps between outlets are also treated as occupied during planning review. Add a link_6-attached RG2-style collision envelope so MoveIt previews no longer assume an empty flange. Constraint: Do not invent cup coordinates or measured calibration values; derive nozzle span only from existing measured nozzle boxes and keep real-motion enable flags false. Rejected: Keeping four tilted nozzle boxes | it left gaps and orientation artifacts the operator explicitly wanted removed. Confidence: high Scope-risk: moderate Directive: Keep the gripper envelope synchronized with rg2_link6_tcp.urdf.xacro if dimensions change. Tested: python3 -m py_compile for changed Python/launch files; YAML merged-block assertion; xacro rg2_link6_tcp.urdf.xacro; colcon build --packages-select azas_motion azas_bringup --symlink-install; ros2 run azas_motion link6_gripper_collision_node --ros-args -p publish_once:=true; ros2 launch azas_bringup rg2_link6_tcp.launch.py --show-args; python3 tools/checks/check_measured_dispenser_geometry.py Not-tested: live robot motion and physical collision clearance.
Replace the merged vertical nozzle block with four separate horizontal spout boxes so RViz shows each outlet and leaves inter-outlet gaps visible again. Constraint: Keep measured calibration and outlet/press coordinates unchanged; restore geometry from the pre-merge measured draft instead of inventing new robot coordinates. Rejected: Single conservative merged vertical block | it hid the physical spout shape and overfilled the gaps between outlets. Confidence: high Scope-risk: narrow Directive: Use separate visual/collision objects when operator review depends on seeing individual dispenser nozzles. Tested: YAML assertion for no merged block and four horizontal nozzle boxes in bringup/perception configs; python3 tools/checks/check_measured_dispenser_geometry.py; colcon build --packages-select azas_bringup azas_motion --symlink-install; timeout 4s ros2 run azas_motion measured_dispenser_collision_scene_node loaded and published dispenser_1..4_head_nozzle_box. Not-tested: full RViz visual inspection and live robot motion.
Represent the four measured nozzle lanes as one horizontal cuboid so RViz shows a single long outlet block spanning the four nozzles and the gaps between them. Constraint: Keep calibration outlet and press poses unchanged; derive the merged block only from the existing measured nozzle boxes. Rejected: Four separate nozzle boxes | user wanted the nozzle area combined into one block, not individually visible. Confidence: high Scope-risk: narrow Directive: Do not turn this into a vertical column; the merged nozzle must remain a horizontal spout block. Tested: YAML assertion for one merged horizontal spout and no per-nozzle boxes; python3 -m py_compile measured_dispenser_collision_scene_node.py; python3 tools/checks/check_measured_dispenser_geometry.py; colcon build --packages-select azas_bringup azas_motion --symlink-install; timeout 4s ros2 run azas_motion measured_dispenser_collision_scene_node published dispenser_head_nozzle_merged_horizontal_spout_box. Not-tested: full RViz visual inspection and live robot motion.
The dispenser press course path was failing before the press-contact move because a side_grip workspace wall collided with link_2, unrelated to the dispenser nozzle geometry. Have the course script request removal of those wall objects before planning while keeping the measured dispenser body and merged horizontal spout visible. Constraint: Preserve measured dispenser calibration and do not relax dispenser collision geometry. Rejected: Changing nozzle geometry again | the reported collision was side_grip_workspace_x_min_wall vs link_2, not a nozzle/body contact. Confidence: high Scope-risk: narrow Directive: Keep REMOVE_COURSE_WORKSPACE_WALLS enabled for this RViz course script unless the workspace walls are revalidated for the full manipulator sweep. Tested: python3 -m py_compile measured_dispenser_collision_scene_node.py; bash -n run_course_dispenser_press_cycle_rviz.sh; python3 tools/checks/check_measured_dispenser_geometry.py; colcon build --packages-select azas_motion azas_bringup --symlink-install; timeout 4s ros2 run azas_motion measured_dispenser_collision_scene_node --ros-args -p remove_course_workspace_collision_objects:=true logged removal of side_grip_workspace_* walls and published dispenser_head_nozzle_merged_horizontal_spout_box. Not-tested: full dispenser press cycle execution in RViz after wall removal.
Lengthen the merged horizontal nozzle block to the full Y span of dispenser_combined_body_box and keep its orientation identical to the combined body so the two boxes are axis-aligned in RViz. Constraint: Preserve measured calibration outlet and press poses; only adjust draft collision/visual geometry. Rejected: Keeping the shorter four-nozzle span | user requested the combined portion be longer and angle-aligned with the combined box. Confidence: high Scope-risk: narrow Directive: Keep the merged nozzle block horizontal and aligned with dispenser_combined_body_box unless new measured geometry supersedes it. Tested: YAML assertions that nozzle Y center/size/bounds and orientation match dispenser_combined_body_box; python3 tools/checks/check_measured_dispenser_geometry.py; colcon build --packages-select azas_bringup --symlink-install; timeout 4s ros2 run azas_motion measured_dispenser_collision_scene_node --ros-args -p remove_course_workspace_collision_objects:=true. Not-tested: full RViz visual inspection and live robot motion.
Make the merged nozzle block match the combined dispenser body footprint in X and Y while keeping the spout in front of the body. Launch link_6 gripper visualization in the course RViz path and publish explicit RG2 markers so the gripper remains visible even when the supplemental RobotModel is not obvious. Constraint: Preserve measured outlet/press calibration; adjust only draft collision/visual geometry and RViz visualization helpers. Rejected: Relying only on the supplemental RobotModel | the course RViz path did not show the gripper reliably. Confidence: high Scope-risk: moderate Directive: Keep /azas/link6_gripper/markers enabled in the course RViz config when debugging link_6 TCP/gripper clearance. Tested: python3 -m py_compile link6_gripper_collision_node.py measured_dispenser_collision_scene_node.py; bash -n run_course_dispenser_press_cycle_rviz.sh; YAML assertions for nozzle X/Y footprint matching combined body and touching body front face; python3 tools/checks/check_measured_dispenser_geometry.py; colcon build --packages-select azas_motion azas_bringup --symlink-install; ros2 run azas_motion link6_gripper_collision_node --ros-args -p publish_once:=true; ros2 launch azas_bringup rg2_link6_tcp.launch.py --show-args; xacro rg2_link6_tcp.urdf.xacro. Not-tested: full RViz visual inspection and live robot motion.
Align the dispenser course RViz script with the project's standard Doosan virtual port so the emulator/controller stack can provide /joint_states consistently. Add explicit bringup-failure diagnostics when ros2_control dies before joint_state_broadcaster becomes available. Constraint: Do not change robot geometry or measured calibration; this only fixes launch defaults and diagnostics. Rejected: Treating the failure as a dispenser/gripper collision issue | logs show ros2_control hardware initialization failed before /joint_states existed. Confidence: high Scope-risk: narrow Directive: Keep virtual course scripts on port 12345 unless the Doosan emulator launch contract changes globally. Tested: bash -n tools/run/run_course_dispenser_press_cycle_rviz.sh; grep verified PORT default and diagnostic messages. Not-tested: full Doosan emulator launch after port change.
Unify the operator-facing panel around the actual hardware sequence: connect robot/gripper/camera, acquire the cup through side-grip, scan dispenser colors, run the measured dispenser recipe cycle, and place the cup holder result. Keep RViz preview and one-click scripts available while guarding real-motion setup against stale virtual Doosan sessions.\n\nConstraint: Cup poses remain vision-supplied; no generated or hardcoded cup coordinates were introduced.\nConstraint: New behaviors include developer-provided lid-pick and lying-cup pick integration surfaces plus dispenser/cocktail panel controls.\nRejected: Leaving the full-cocktail button to skip side-grip | the dispenser recipe assumes a cup is already grasped.\nConfidence: medium\nScope-risk: moderate\nDirective: Restart azas-panel after this commit so the updated server prerequisite ordering and HTML buttons are loaded.\nTested: node --check extracted robot_pipeline_control.html script; python3 -m py_compile changed Python launch/nodes/server; verified full-flow queue expansion order.\nNot-tested: Real hardware motion and camera/gripper physical execution.
Bring the updated develop branch into the integration branch while preserving the cocktail panel flow and accepting the new lid-grip and perception additions.\n\nConstraint: Develop added lid sticker/grip planning, hand-eye/static TF assets, and perception setup changes.\nRejected: Dropping either config file family from azas_perception setup.py | both .pt model assets and .json hand-eye data are needed.\nConfidence: medium\nScope-risk: moderate\nDirective: After restarting azas-panel, verify the full queue still expands through side-grip before dispenser recipe execution.\nTested: node --check extracted robot_pipeline_control.html script; python3 -m py_compile merged Python launch/nodes/server/perception files; verified full-flow queue expansion order.\nNot-tested: Real hardware motion, camera stream, gripper actuation, and physical lid/cup pickup.
This reverts commit d44217e.
- Updated RG2 launch files to disable gripper collision publishing for improved performance. - Changed default mount orientation for RG2 in URDF and launch files to identity (0, 0, 0). - Introduced a new parameter `side_target_x_offset_m` in the YoloCupPickNode for side-grip motion planning. - Enhanced side-grip logic to apply target offset during planning and added warnings for offset usage. - Updated launch files to include the new parameter and adjusted related commands for consistency. - Added a new script to wait for lid grip status, ensuring proper sequence execution after lid closure. - Improved error handling and logging in various scripts to enhance debugging and user feedback.
Bring the develop router path into the current integration branch while preserving the local lid-marker ArUco tolerance work and adding the develop-side parameter guard. Constraint: User requested origin/develop, not the current branch upstream. Rejected: Rebase local integration commits | branch already had local integration history and a merge preserves the field branch shape. Confidence: high Scope-risk: moderate Directive: Do not run auto_cup_flow_router with enable_real_motion:=true without supervised hardware readiness. Tested: colcon build --packages-up-to azas_task_manager azas_bringup --symlink-install; ros2 launch azas_bringup auto_cup_flow_router.launch.py --show-args; python3 -m py_compile src/azas_perception/azas_perception/lid_marker.py Not-tested: Real robot auto routing launch with enable_real_motion:=true.
- Added new UI elements and logic for displaying cocktail recipes in voice.js. - Introduced a new SVG icon for the cocktail representation. - Enhanced the Python scripts for managing the cocktail preparation sequence with resume capabilities. - Created shell scripts to automate the voice command flow and manage the Azas voice stack in a tmux session. - Updated the run scripts to support new features and ensure proper execution of the cocktail preparation pipeline.
…n automation script
Constraint: User requested committing current branch work before merging fix/eliminate-eleven-mediapipe-handover. Rejected: Stashing local edits | User explicitly asked not to stash. Confidence: medium Scope-risk: moderate Directive: Preserve current side-grip behavior and dispenser 4 cup-holder offset during the merge. Tested: Not run by user directive. Not-tested: Dry-run, build, syntax checks, hardware validation.
Constraint: Preserve current branch side-grip behavior and dispenser 4 final cup-holder offset while merging fix/eliminate-eleven-mediapipe-handover. Rejected: Adopting fix branch CUP_HOLDER_PLACE_FINAL_X_OFFSET_M=0.003 wrapper default | It would override the current branch cup-holder X offset requirement. Confidence: medium Scope-risk: moderate Directive: Keep side-grip inset/clearance and dispenser 4 extra -1cm cup-holder offset unless re-measured on hardware. Tested: Read-only conflict-marker and symbol checks only. Not-tested: Dry-run, build, syntax checks, hardware validation per user directive.
Constraint: GitHub protected main rejects force push and direct PR merge had conflicts. Rejected: Resolving toward main content | User identified current branch as the completed robot behavior. Confidence: medium Scope-risk: moderate Directive: This merge intentionally keeps the current side-grip MediaPipe handover tree while making origin/main an ancestor for protected-branch PR merge. Tested: Not run by user directive. Not-tested: Dry-run, build, syntax checks, hardware validation.
Contributor
Author
|
Closing: operator clarified this branch itself is the final reference branch, not a request to update main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final verified robot flow from fix/handover-sidegrip-mediapipe.
Not-tested: dry-run/build/syntax/hardware validation not run per operator directive.