Behavior by scilwb · Pull Request #133 · RLinf/RPent

scilwb · 2026-08-31T13:17:48Z

Rework progress

Branch rebuilt on latest main (includes #136, #132, #142, #148).

Validation

Official task success on turning_on_radio (seed 0), twice: real Codex planner, info["done"]["success"] == true, terminal receipt SHA-verified, recipe_<tag>.jsonl (LIBERO format), bounded streaming episode.mp4
Dashboard: three physical cameras mapped correctly, solved text gated on official success, TaskRun recipe flow works for BEHAVIOR
Fresh dual-venv install + checkpoint SHA-256 binding + assets verified; pytest 259 passed, pre-commit, en/zh Sphinx -W

Key commits

Deferred

Motion primitives (navigate_to/move_to/rotate_wrist/open/close/press) return motion_unavailable; cuRobo motion stack ports in a follow-up PR

Copilot

Pull request overview

Adds a new robots/behavior integration to run BEHAVIOR-1K tasks in RPent, including runtime sidecars (ENV/VLA/DINO), Dashboard support, pinned asset/checkpoint verification, and end-to-end English/Chinese documentation for installation and usage.

Changes:

Introduce BEHAVIOR robot plugin (RobotSpec, toolkit/primitives, task specs, success semantics, outer explore harness).
Add runtime sidecars + clients for ENV RPC, Pi0.5 VLA HTTP serving, and DINOv2 memory embeddings with pinned asset identities.
Add reproducible dual-venv install/verification/dashboard scripts and update docs/READMEs + pyproject extras.

Reviewed changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/verify_behavior_assets.sh	Validates required BEHAVIOR assets + DINO/Policy checkpoint contracts.
scripts/run_behavior_dashboard.sh	Launch wrapper for BEHAVIOR Dashboard sessions (env/model GPU binding, paths).
scripts/install_behavior_runtime.sh	Reproducible dual-venv installer + pinning/manifest generation for BEHAVIOR runtime.
rpent/dashboard/static/dashboard.js	Dashboard realtime frame refresh logic update for unavailable-frame retry behavior.
rpent/dashboard/state.py	Applies frame-path projection from tool results to keep Dashboard frames in sync.
rpent/cli/dashboard.py	Allows robot-provided Dashboard server/state classes + runtime backend binding hooks.
robots/behavior/vla_server.py	Pi0.5 FastAPI sidecar for BEHAVIOR action inference (/predict + action gating).
robots/behavior/vla_client.py	HTTP client wrapper for the BEHAVIOR VLA sidecar.
robots/behavior/tools.py	BEHAVIOR primitive handlers + result sanitization and official-success latching.
robots/behavior/toolkit.py	Toolkit bridge exposing BEHAVIOR tools + dashboard event emission + receipt writing.
robots/behavior/terminal_success.py	Implements “official success only” semantics and receipt validation helpers.
robots/behavior/task_specs.py	Source-controlled task/seed mapping + per-task policies (radio/trash tasks).
robots/behavior/selfcheck.py	Lightweight import/config/tool-surface selfcheck for the BEHAVIOR plugin.
robots/behavior/robot_spec.py	BEHAVIOR RobotSpec + dashboard spec + toolkit factory wiring.
robots/behavior/prompts/user.py	BEHAVIOR user prompt section body.
robots/behavior/prompts/system.py	BEHAVIOR system prompt section bodies (evidence/termination discipline).
robots/behavior/prompts/init.py	Package marker for BEHAVIOR prompt sections.
robots/behavior/prompt_bundle.py	Prompt assembly from runtime variables with safe opaque runtime text blocks.
robots/behavior/policy_checkpoint.py	Checkpoint identity contract + binding/verification for Pi0.5 BEHAVIOR checkpoint.
robots/behavior/memory_schema.py	Deterministic validation helpers for episode-memory schema hashing/canonical JSON.
robots/behavior/memory_embeddings_dinov2.py	Pinned DINOv2 embedding identity + safe source extraction + CUDA backend encoder.
robots/behavior/harness.py	Outer multi-attempt Explore harness (process-isolated retries + receipt summarization).
robots/behavior/env_server.py	Main-thread HTTP RPC server for OmniGibson/BEHAVIOR env backend adapter.
robots/behavior/env_client.py	RPC client enforcing terminal-success semantics and decoding bytes payloads.
robots/behavior/dino_server.py	DINOv2 encoder RPC server with pinned asset validation + CUDA requirement.
robots/behavior/dino_client.py	RPC client wrapper for the optional DINO component (normalization + meta checks).
robots/behavior/dashboard/static/behavior_controls.css	BEHAVIOR-specific Dashboard controls styling and layout.
robots/behavior/init.py	Exports BEHAVIOR entrypoints (`get_robot_spec`, `get_toolkit`).
README.zh-CN.md	Adds BEHAVIOR entry to Chinese “What’s new” + feature matrix.
README.md	Adds BEHAVIOR entry to English “What’s new” + feature matrix.
pyproject.toml	Adds `behavior` extra dependency set for BEHAVIOR-facing RPent deps.
docs/source-zh/rst_source/usage/behavior.rst	Full Chinese BEHAVIOR install/asset/runtime/dashboard workflow docs.
docs/source-zh/rst_source/installation.rst	Documents BEHAVIOR extra and explains why it’s excluded from `.[full]`.
docs/source-zh/rst_source/development/architecture.rst	Updates architecture docs to include `behavior` robot package.
docs/source-zh/rst_source/development/add_robot.rst	Updates add-robot docs to reflect current standard robot packages.
docs/source-zh/index.rst	Adds BEHAVIOR to Chinese docs index and front-page overview text.
docs/source-en/rst_source/usage/behavior.rst	Full English BEHAVIOR install/asset/runtime/dashboard workflow docs.
docs/source-en/rst_source/installation.rst	Documents BEHAVIOR extra and explains why it’s excluded from `.[full]`.
docs/source-en/rst_source/development/architecture.rst	Updates architecture docs to include `behavior` robot package.
docs/source-en/rst_source/development/add_robot.rst	Updates add-robot docs to reflect current standard robot packages.
docs/source-en/index.rst	Adds BEHAVIOR to English docs index and front-page overview text.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

qurakchin · 2026-09-01T02:30:19Z

+    return 0 if successful_attempts else 1
+
+
+def main(argv: Sequence[str] | None = None) -> int:


is this file for explore mode? please use the style like libero.

QuanluZhang · 2026-09-01T03:33:28Z

+
+See :doc:`../development/add_primitive` for the tool-extension walkthrough.
+
+Reproducing results


please show the reproduced results

Sonorous281 · 2026-09-07T06:55:52Z

+- Explore 只构造一个 ``inbox_write`` MemoryManager，写入范围限定为
+  ``<memory-dir>/_inbox/<recipe-tag>``；
+- ``MEMORY.md``、``global/``、``suite/``、``task/``、``_inbox/`` 和
+  ``_merged/`` 保持 RPent 标准语义。


Are these memory paths still aligned with the current memory contract? the implementation now uses _internal/inbox and task_only, while this section still documents _inbox, task, and _merged.

Sonorous281 · 2026-09-07T08:07:17Z

+        help="Alias for --public-seed for dashboard and legacy launchers.",
+    )
+    parser.add_argument(
+        "--behavior-mode",


do we need both --explore and --behavior-mode to represent exploration? could behavior use the shared exploration state like libero, so cli and dashboard do not need separate mode handling?

Sonorous281 · 2026-09-07T08:10:00Z

+    output_dir = Path(output_dir).expanduser().resolve()
+    requested_memory_profile = getattr(args, "memory_profile", None)
+    memory_profile = str(requested_memory_profile or "local")
+    if memory_profile != "local":


Is there a behavior-specific reason to require the local memory profile for normal evaluation? libero follows the shared pattern of hf for normal evaluation and local memory for exploration.

Sonorous281 · 2026-09-07T08:11:05Z

+
+    owned_daemons: dict[str, ProcessDaemon] = {}
+    primitives_kwargs: dict[str, Any] = {}
+    pending_env: tuple[ProcessDaemon | None, RpcClient] | None = None


could this use the same starters, connectors, and timeouts pattern as libero and robocasa? the shared spawn helpers are already used, but the env, vla, and dino paths are still expanded separately.

Sonorous281 · 2026-09-07T08:12:07Z

+        "env.press": 1800.0,
+    }
+
+    def __init__(self, client: RpcClient, *, expected_meta: dict[str, Any]) -> None:


this subclass currently bypasses base envclient initialization and reimplements much of the reset, step, chunk_step, and camera flow.

Sonorous281 · 2026-09-07T08:12:48Z

+    "side_or_indeterminate",
+)
+
+PUBLIC_TOOL_CONTRACTS: dict[int, tuple[str, ...]] = {


Do we need to keep public tool contract versions 1 through 4 for a new integration that has not shipped yet? could we keep only the current tool surface unless there is an actual compatibility requirement?

Sonorous281 · 2026-09-07T08:14:13Z

+        self._last_info = info_dict
+        return info_dict
+
+    def _reset_raw(self) -> tuple[Any, dict[str, Any]]:


Since this integration installs a pinned rlinf revision, do we need to support reset_raw, env_reset, and generic reset at the same time?

Sonorous281 · 2026-09-07T08:16:48Z

+    return ()
+
+
+def _result_official_success(result: dict[str, Any], *, terminated: bool) -> bool:


could the behavior layer normalize this into a generic official_success field instead? that would avoid making the shared dashboard understand the behavior-specific receipt schema and info done success path.

Sonorous281 · 2026-09-07T08:18:51Z

+    )
+    from rpent.robots.components.pi05_vla_client import Pi05VLAClient
+
+    expected_binding = validate_policy_checkpoint(args.policy_checkpoint)


Do we need to hash the full checkpoint again in _connect_vla after the manifest was already generated and validated before the server started?

Sonorous281 · 2026-09-07T08:19:53Z

+    compile_catalog.add_argument("--source-archive", required=True, type=Path)
+    compile_catalog.add_argument("--weights", required=True, type=Path)
+    compile_catalog.add_argument("--cache-dir", type=Path, default=None)
+    compile_catalog.add_argument("--cuda-device", choices=("2", "7"), required=True)


do we still need a second cli here now that behavior-build-memory is the public entry point? this one also hardcodes cuda devices 2 and 7, which looks machine-specific.

Sonorous281 · 2026-09-07T08:21:34Z

+    ).encode("utf-8")
+
+
+def official_success_receipt_sha256(receipt: Mapping[str, Any]) -> str:


do we need a separate sha-verified receipt for success inside the same runtime path?

Sonorous281 · 2026-09-07T08:22:51Z

+    ):
+        assert value in system or value in user
+
+    ordered_sections = [


could this test focus on successful rendering and required variables rather than the exact prompt section order and wording? otherwise normal prompt cleanup would require updating the test as well.

Sonorous281 · 2026-09-07T08:25:33Z

+========
+
+`BEHAVIOR-1K <https://behavior.stanford.edu/>`_ 基于 OmniGibson 提供长程家庭任务。
+RPent 当前把 ``turning_on_radio`` 和 ``picking_up_trash`` 作为标准 sibling


could we make the chinese wording here more natural and keep english mainly for established api or project names? phrases such as 接入合同, 标准 sibling robot plugin,源码 editable...

Sonorous281 · 2026-09-07T08:26:29Z

 [tool.setuptools.packages.find]
 where = ["."]
-include = ["rpent*"]
+include = ["rpent*", "robots", "robots.behavior*"]


Is there a reason to package only robots.behavior while the sibling robot integrations remain source-checkout plugins? could the behavior entry points follow the same packaging model as the other robots?

Sonorous281 · 2026-09-07T08:37:50Z

+    "rpent[rlinf]",
+    "rpent-openpi @ git+https://github.com/RLinf/openpi.git@rpent",
+    # OpenPI's tool.uv source is not propagated through package metadata.
+    "lerobot @ git+https://github.com/huggingface/lerobot.git@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5",


Could we drop the explicit LeRobot pin here as well to stay aligned with latest main? #154 removed the same pin from the OpenPI-based LIBERO extras

scilwb and others added 12 commits August 30, 2026 22:00

Add Behavior robot plugin prototype

7b21142

Decouple dashboard runtime binding

0186419

Polish Behavior plugin integration

7f3992d

Clarify Behavior setup self-check docs

e57a8ed

Clarify Behavior integration documentation

4e985b8

docs: focus Behavior integration notes

2a1cc37

docs: publish Behavior checkpoint download

ff15e0a

Prune unused behavior helpers

034a69a

robots: remove Behavior private checkpoint paths

74aa26c

docs: align Behavior guide with Libero

74d46bd

fix: add Behavior source copyright headers

110d35f

fix: make BEHAVIOR reproduction runnable

5ecb178

Copilot AI lite review requested due to automatic review settings August 31, 2026 13:17

Copilot started reviewing on behalf of scilwb August 31, 2026 13:18 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Comment thread robots/behavior/vla_client.py Outdated

Comment thread robots/behavior/vla_server.py Outdated

Comment thread README.zh-CN.md Outdated

Comment thread robots/behavior/task_specs.py

scilwb added 4 commits August 31, 2026 21:41

fix(behavior): tighten VLA request contract

4c4d842

chore: merge upstream main into behavior

3552c16

feat(behavior): enable bounded dashboard controls

c6a0534

feat(behavior): add world-frame dashboard arm controls

d887362