Piper OpenPI JAX: RTC and Temporal Ensemble

中文说明

本仓库为双臂 Piper 机械臂提供两套相互独立的 OpenPI JAX action chunk 执行方案。策略模型从 JAX Orbax checkpoint 恢复，GPU 服务器负责策略推理，Piper 端 client 负责采集观测、请求动作并执行双臂控制。

RTC（Real-Time Chunking）：重新推理时保留上一段完整轨迹，通过 hard/soft 时间约束减少相邻 action chunk 切换时的不连续，同时保持短期计划的一致性。
TE（Temporal Ensemble）：持续请求相互重叠的 action chunk，将不同 chunk 中对应同一真实控制时刻的动作对齐后加权平均，用于减小局部预测噪声。

本仓库中的 client 是双臂 client，不是单臂 client，并保留了 prompt、策略双臂顺序和强制交换双臂动作块的入口。

这是一套 Piper/OpenPI 集成代码，不是适用于所有机器人的通用配置。真实部署前，checkpoint、OpenPI config、norm stats、相机键名、动作维度、双臂顺序以及 Piper 驱动必须与训练设置一致。

仓库结构

openpi-jax-rtc-te/
├── rtc/
│   ├── openpi_piper_jax_rtc_policy_server.py
│   ├── openpi_piper_dual_arm_jax_rtc_client.py
│   ├── start_jax_rtc_server.sh
│   └── start_jax_rtc_dual_arm_client.sh
├── temporal_ensemble/
│   ├── openpi_piper_jax_temporal_ensemble_policy_server.py
│   ├── openpi_piper_dual_arm_jax_temporal_ensemble_client.py
│   ├── start_jax_temporal_ensemble_server.sh
│   └── start_jax_temporal_ensemble_dual_arm_client.sh
├── configs/
│   ├── policy.example.yaml
│   ├── server.example.env
│   ├── client.example.env
│   ├── rtc.example.env
│   └── temporal_ensemble.example.env
├── policy/
│   └── README.md
├── references/
│   ├── original_jax_orbax_policy_server.py
│   ├── piper_inference_dependency.py
│   ├── piper_motor_driver.py
│   └── piper_robot_driver.py
├── DEPENDENCIES.md
└── CHECKSUMS.sha256

rtc/：RTC JAX server、双臂 client 和启动脚本。
temporal_ensemble/：独立的 TE server、双臂 client 和启动脚本。
configs/：不含真实信息的 policy manifest，以及可复制为本机文件的 server/client/RTC/TE 环境变量模板。
policy/README.md：Policy Adapter 的请求、返回、metadata 和替换 policy 接口约定。
references/：本项目依赖的原始 JAX/Piper/LeRobot 集成参考代码。发布前需要检查上游许可证。
DEPENDENCIES.md：server/client 依赖边界以及复现实验时需要记录的版本。
CHECKSUMS.sha256：用于确认发布的源文件、启动脚本、配置模板和接口说明没有发生意外变化。

仓库不会包含实际 policy 权重。使用者通过 configs/policy.example.yaml 确认兼容条件，再复制 .env 示例并填写自己的 checkpoint、norm stats、OpenPI 路径和设备配置：

cd /path/to/openpi-jax-rtc-te
cp configs/server.example.env configs/server.local.env
cp configs/client.example.env configs/client.local.env

.local.env 已被 Git 忽略，不应上传。

RTC 与 TE 的区别

项目	RTC	Temporal Ensemble
默认端口	`8011`	`8012`
默认执行 chunk	25 actions	50 actions
新旧轨迹关系	新轨迹受到上一完整轨迹的 hard/soft 约束	每个 chunk 独立生成，client 端再做加权
时间处理	冻结立即需要执行的动作，soft 约束重叠区	按绝对 control step 对齐多个 chunk
主要参数	delay、execution horizon、soft decay、guidance scale	inference interval、decay、max chunks
更适合	需要动作计划连贯性的长时序、双臂搬运任务	各 chunk 计划兼容、主要问题是局部预测噪声的任务

不要把 TE client 连接到启用了 RTC 的 server。TE client 会主动拒绝这种组合，因为普通时间加权可能破坏 RTC 已经冻结或约束的动作。

动作与双臂假设

当前实现基于以下约定：

Piper 双臂状态和执行动作均为 14D。
前 7D 和后 7D 分别表示一只机械臂的六个关节与一个夹爪。
物理顺序通常为 left(can0), right(can1)。
server 可以处理原生 14D Piper 输出，也可以处理 80D unified policy 输出。
对 80D 输出使用明确的 Piper slot mapping，不使用不安全的 action[:14]。
server 将模型输出恢复为绝对 14D Piper target，client 再根据双臂顺序完成映射并发送命令。

相机配置：当前是四相机 client、三相机模型

当前双臂 client 会初始化并读取四路 RealSense：

Client key	视角	当前 JAX policy adapter 是否使用
`head`	头部或全局视角	是
`left_wrist`	左腕视角	是
`right_wrist`	右腕视角	是
`front_view`	额外正面视角	否

因此，当前代码的准确描述是：四路相机被 client 采集并发送，其中三路进入模型。

使用现有代码时必须提供四个相机序列号：

export HEAD_CAMERA_SERIAL='<HEAD_CAMERA_SERIAL>'
export LEFT_WRIST_CAMERA_SERIAL='<LEFT_WRIST_CAMERA_SERIAL>'
export RIGHT_WRIST_CAMERA_SERIAL='<RIGHT_WRIST_CAMERA_SERIAL>'
export FRONT_VIEW_CAMERA_SERIAL='<FRONT_VIEW_CAMERA_SERIAL>'

也可以直接提供完整 JSON：

export ROBOT_CAMERAS_JSON='<YOUR_CAMERA_JSON>'

如果需要真正的 3-camera client，需要同时修改 client 的 build_remote_observation() 和 launcher 中的相机配置，使 front_view 不再初始化和读取。只删除 FRONT_VIEW_CAMERA_SERIAL 会导致当前启动脚本直接报缺少变量。

环境要求

GPU 推理服务器

Linux 与可用的 CUDA GPU。
Python 3.11。
OpenPI 源码目录，其中至少存在 src/ 和 packages/openpi-client/src/。
与该 OpenPI checkout 匹配的 JAX/Python 环境。
完整的 Orbax/OCDBT checkpoint，至少包含：
- _CHECKPOINT_METADATA
- params/manifest.ocdbt
- params/array_metadatas
与 checkpoint 和数据配置匹配的 norm_stats.json。
能被本地 cotrain config 代码解析的 CONFIG_NAME。

Piper 客户端

Python 3.11 与 OpenPI WebSocket client 依赖。
兼容的 LeRobot-Piper checkout。
对应的 Piper SDK、RealSense 和机器人驱动依赖。
两路已配置 CAN，通常左臂为 can0，右臂为 can1。
当前 launcher 需要四个可见的 RealSense 设备。
Piper 电脑能够访问 GPU server 的策略端口。

仓库不包含 checkpoint、norm stats、数据集、机器人视频、真实路径、SSH 配置、凭据或相机序列号。

主要环境变量

Server 端

变量	含义	占位示例
`OPENPI_ROOT`	OpenPI checkout	`/path/to/openpi`
`CHECKPOINT_DIR`	需要测试的准确 checkpoint step 目录	`/path/to/checkpoints/checkpoint_step`
`NORM_STATS_PATH`	对应的 norm stats	`/path/to/assets/norm_stats.json`
`OPENPI_PYTHON_BIN`	server 端 Python 3.11 绝对路径	`/path/to/python3.11`
`CONFIG_NAME`	OpenPI 配置名称	`your_config_name`
`PROMPT`	当前任务指令	`your task instruction`
`POLICY_HOST`	监听地址；远程 client 通常使用 `0.0.0.0`	`0.0.0.0`
`POLICY_PORT`	WebSocket 端口	RTC `8011`、TE `8012`

启动脚本会检查 Python 路径，因此建议提供 Python 可执行文件的绝对路径，而不是只填写 python3。

Client 端

变量	含义	占位示例
`OPENPI_ROOT`	client 可访问的 OpenPI checkout	`/path/to/openpi`
`LEROBOT_PIPER_ROOT`	LeRobot-Piper checkout	`/path/to/lerobot-piper`
`CLIENT_PYTHON_BIN`	client Python 3.11 绝对路径	`/path/to/python3.11`
`POLICY_HOST`	GPU server 地址	`<SERVER_IP>`
`POLICY_PORT`	与 server 完全一致的端口	RTC `8011`、TE `8012`
`PROMPT`	发送给策略的任务指令	`your task instruction`
`LEFT_CAN_NAME`	左臂 CAN	`can0`
`RIGHT_CAN_NAME`	右臂 CAN	`can1`

真实部署流程

RTC 与 TE 应作为两组独立实验。一次只启动一组匹配的 server 和 client。

第一步：确认 checkpoint、config 和 norm stats

启动前确认：

CHECKPOINT_DIR 是需要测试的准确 checkpoint step。
NORM_STATS_PATH 来自相同数据集和动作表示。
CONFIG_NAME 能恢复相同的模型结构和 transforms。
模型 action horizon 不小于 ACTIONS_PER_INFERENCE。
相机 key 和 prompt 格式与训练一致。
checkpoint 的 14D 双臂顺序已经确认。

只更换 checkpoint 路径并不能自动修复 config、norm stats 或动作映射不匹配。

第二步：先验证 server，不启动网络服务

在 GPU server 上准备公共变量：

cd /path/to/openpi-jax-rtc-te

export OPENPI_ROOT='/path/to/openpi'
export CHECKPOINT_DIR='/path/to/checkpoints/checkpoint_step'
export NORM_STATS_PATH='/path/to/assets/norm_stats.json'
export OPENPI_PYTHON_BIN='/path/to/python3.11'
export CONFIG_NAME='your_config_name'
export PROMPT='your task instruction'
export POLICY_HOST='0.0.0.0'

验证 RTC：

export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export VALIDATE_ONLY='true'

bash rtc/start_jax_rtc_server.sh

RTC validate-only 会加载 checkpoint，执行合成观测推理，检查可执行 action shape、完整 action horizon 以及 hard/soft/free mask。验证完成后程序退出，不会持续监听端口。

验证 TE：

export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export VALIDATE_ONLY='true'

bash temporal_ensemble/start_jax_temporal_ensemble_server.sh

TE 的时间加权发生在 client 端。server validate-only 负责确认相同的 checkpoint 与 JAX 推理路径能够工作。

第三步：启动策略 server

RTC server：

cd /path/to/openpi-jax-rtc-te

export POLICY_HOST='0.0.0.0'
export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export VALIDATE_ONLY='false'

bash rtc/start_jax_rtc_server.sh

TE server：

cd /path/to/openpi-jax-rtc-te

export POLICY_HOST='0.0.0.0'
export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export VALIDATE_ONLY='false'

bash temporal_ensemble/start_jax_temporal_ensemble_server.sh

如果 client 在另一台机器上，需要允许对应端口通过 server 防火墙。不要把策略服务直接暴露到不可信网络。

第四步：配置 Piper client

在 Piper 电脑上：

cd /path/to/openpi-jax-rtc-te

export OPENPI_ROOT='/path/to/openpi'
export LEROBOT_PIPER_ROOT='/path/to/lerobot-piper'
export CLIENT_PYTHON_BIN='/path/to/python3.11'

export POLICY_HOST='<SERVER_IP>'
export PROMPT='your task instruction'
export LEFT_CAN_NAME='can0'
export RIGHT_CAN_NAME='can1'

export HEAD_CAMERA_SERIAL='<HEAD_CAMERA_SERIAL>'
export LEFT_WRIST_CAMERA_SERIAL='<LEFT_WRIST_CAMERA_SERIAL>'
export RIGHT_WRIST_CAMERA_SERIAL='<RIGHT_WRIST_CAMERA_SERIAL>'
export FRONT_VIEW_CAMERA_SERIAL='<FRONT_VIEW_CAMERA_SERIAL>'

export FPS='30'
export STEPS='100'

STEPS=0 表示持续运行直到按下 Ctrl+C。首次验证时建议使用有限步数。

第五步：确认双臂顺序

代码中有两种不同的交换入口：

FORCE_SWAP_ARMS：修改策略 14D 动作中两个 7D block 的解释顺序。
COMMAND_ARM_ORDER：在动作发送到物理机械臂之前，再交换两个 7D block。

正常物理命令顺序应保持：

export COMMAND_ARM_ORDER='left-right'
export COMMAND_SWAP_MODE='absolute'

根据 checkpoint 训练时的 action order 设置：

# Policy 输出为 [left 7D, right 7D]
export FORCE_SWAP_ARMS='false'

# Policy 输出为 [right 7D, left 7D]
export FORCE_SWAP_ARMS='true'

不要因为 policy order 相反就同时设置 COMMAND_ARM_ORDER=right-left。FORCE_SWAP_ARMS 已经负责把 policy order 恢复到物理左右臂顺序；command-level swap 是独立的兼容或诊断入口。

RTC 与 TE launcher 当前的 FORCE_SWAP_ARMS 默认值不同，所以真实部署时应显式设置，不要依赖默认值。

第六步：先运行 dry-run

Dry-run 仍然会连接 Piper、读取双臂状态和相机、连接策略 server 并计算最终命令，但不会调用 robot.send_action()。

RTC dry-run：

cd /path/to/openpi-jax-rtc-te

export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export PREFETCH_THRESHOLD='4'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export FORCE_SWAP_ARMS='false'
export ENABLE_ARM='false'
export DRY_RUN='true'

bash rtc/start_jax_rtc_dual_arm_client.sh \
  --debug-action-details \
  --print-command-counts

TE dry-run：

cd /path/to/openpi-jax-rtc-te

export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export ENSEMBLE_INFERENCE_INTERVAL='4'
export ENSEMBLE_DECAY='0.25'
export ENSEMBLE_MAX_CHUNKS='4'
export FORCE_SWAP_ARMS='false'
export ENABLE_ARM='false'
export DRY_RUN='true'

bash temporal_ensemble/start_jax_temporal_ensemble_dual_arm_client.sh

RTC debug 输出包含 observation、model action、delta、final action 和可选的电机命令计数。TE 输出包含最终 14D action，以及当前控制步使用的 chunk index 和权重。启用真机运动前，应确认左右臂动作块对应正确、没有意外全零、NaN 或 infinite value。

第七步：启动真实机械臂执行

RTC：

cd /path/to/openpi-jax-rtc-te

export STEPS='0'
export ENABLE_ARM='true'
export DRY_RUN='false'
export DISABLE_ARM_ON_EXIT='false'

bash rtc/start_jax_rtc_dual_arm_client.sh

TE：

cd /path/to/openpi-jax-rtc-te

export STEPS='0'
export ENABLE_ARM='true'
export DRY_RUN='false'
export DISABLE_ARM_ON_EXIT='false'

bash temporal_ensemble/start_jax_temporal_ensemble_dual_arm_client.sh

launcher 当前将 action scale 设为 1.0、额外平滑 alpha 设为 1.0、deadband 设为 0，并把 client delta 上限设置为近似开放值，因此不会额外加入保守动作滤波。真实行为主要由策略输出、RTC/TE controller、Piper driver 和底层机械臂限制决定。

按 Ctrl+C 停止。如果设置 DISABLE_ARM_ON_EXIT=true，退出时还会请求关闭机械臂使能。

RTC 参数说明

当前 RTC 设计保留模型完整的 50-action horizon，同时只返回前 25 个 action 作为可执行窗口：

full_actions：保存完整模型轨迹，用于下一次 RTC 请求。
actions：返回给 client 的可执行前缀。
RTC_DELAY_STEPS：新推理完成前，需要保持不变的立即执行动作数量。
RTC_EXECUTION_HORIZON：一次允许进入执行队列的最大有效动作数。
RTC_SOFT_MASK_DECAY：soft overlap 第一个位置的权重及后续指数衰减。
RTC_GUIDANCE_SCALE：RTC guided sampling 的贡献比例。

server 和 client 的 ACTIONS_PER_INFERENCE、delay、execution horizon、soft decay 与 guidance scale 应保持一致。client 在 chunk 切换时会打印 skip、hard、soft、free、queued 和 full_prior，可用于确认完整 prior 被保留且 soft 区域不是空的。

默认配置：

model horizon:             50
ACTIONS_PER_INFERENCE:     25
FPS:                       30
PREFETCH_THRESHOLD:        4
RTC_DELAY_STEPS:           3
RTC_EXECUTION_HORIZON:     25
RTC_SOFT_MASK_DECAY:       0.6
RTC_GUIDANCE_SCALE:        1.0

如果 checkpoint 原生输出 50 actions，不要为了执行 25 actions 就把模型完整 horizon 截断为 25。RTC 需要保留 50-action prior 才能在 hard 区域之后形成有效的 soft overlap。

TE 参数说明

TE client 使用下式对齐动作：

action_index = control_step - request_step

只有指向当前绝对 control step 的动作会参与加权。旧 chunk 在当前时刻对应更大的 action_index，因此会获得更低的指数权重。

默认配置：

ACTIONS_PER_INFERENCE:       50
FPS:                         30
ENSEMBLE_INFERENCE_INTERVAL: 4
ENSEMBLE_DECAY:              0.25
ENSEMBLE_MAX_CHUNKS:         4

减小 inference interval 会增加 chunk 重叠和 server 负载。
增大 decay 会让新 chunk 的权重更高。
增大 max chunks 会保留更多历史计划，可能降低局部噪声，也可能在任务阶段切换时造成犹豫或原地抖动。

常见问题

`Missing required path: python3`

使用 Python 可执行文件的绝对路径：

export OPENPI_PYTHON_BIN='/absolute/path/to/python3.11'
export CLIENT_PYTHON_BIN='/absolute/path/to/python3.11'

Client 无法连接 server

检查：

server 是否设置 VALIDATE_ONLY=false。
远程连接时 server 是否设置 POLICY_HOST=0.0.0.0。
server/client 端口是否一致。
防火墙是否允许对应端口。
是否存在旧 server 进程占用端口。

Checkpoint 验证失败

检查准确的 checkpoint 目录、Orbax metadata、Python 3.11 环境、CONFIG_NAME 和匹配的 NORM_STATS_PATH。来自不同动作表示或数据配置的 checkpoint 不能只通过更换路径解决。

相机初始化失败

确认四个序列号在 Piper 电脑上均可见且没有重复。使用 ROBOT_CAMERAS_JSON 时，每个相机条目必须包含数值形式的 serial_number 和兼容的 FPS/分辨率。

左右臂动作相反

先保持 COMMAND_ARM_ORDER=left-right，只切换 FORCE_SWAP_ARMS 并检查 dry-run 日志。只有确定需要交换物理输出 block 时，才使用 command-level swap。

RTC soft 区域为空

确认 server 返回完整模型 horizon 的 full_actions，可执行 prefix 小于完整 horizon，并且 server/client 参数一致。client chunk-switch 日志中应出现非零 soft 数量。

TE 平滑但在阶段切换时犹豫

TE 可能正在平均语义上不兼容的旧计划和新计划。可以减少保留 chunk 的数量或存活时间、增大 decay 让新计划占主导，或者改用 RTC。不要在 RTC 之后再叠加普通 TE。

校验与发布说明

本仓库已将机器路径、设备序列号、主机名、数据集标识和任务 prompt 替换为占位符。可使用以下命令检查复制的源文件：

cd /path/to/openpi-jax-rtc-te
shasum -a 256 -c CHECKSUMS.sha256

发布到 GitHub 前：

检查 OpenPI、LeRobot、Piper SDK、RealSense 以及 references/ 中代码的许可证。
添加仓库许可证和上游 attribution。
固定已经验证的依赖版本或 commit。
不要提交 checkpoint、norm stats、数据集、机器人视频、SSH key、凭据或真实相机序列号。
Shell/Python 语法检查、合成验证和 dry-run 只能证明对应路径可运行，不能证明真实机械臂行为安全或任务一定成功。

English

This repository provides two independent action-chunk execution methods for a dual-arm Piper robot using an OpenPI policy restored from a JAX Orbax checkpoint:

RTC (Real-Time Chunking): replans while preserving part of the previous action chunk through hard and soft temporal constraints. It is intended to reduce discontinuities between consecutive plans while retaining a coherent short-term trajectory.
Temporal Ensemble (TE): requests overlapping action chunks, aligns their predictions to the same control step, and computes a weighted average. It can reduce small prediction noise, but may average incompatible plans near semantic stage transitions.

The implementation is split into a GPU policy server and a robot-side client. The client in this repository is dual-arm only and retains explicit controls for policy arm order and forced action-block swapping.

This is integration code, not a universal Piper configuration. Your checkpoint, OpenPI config, normalization statistics, camera layout, action dimensions, arm order, and robot drivers must match each other before real-hardware execution.

Repository layout

openpi-jax-rtc-te/
├── rtc/
│   ├── openpi_piper_jax_rtc_policy_server.py
│   ├── openpi_piper_dual_arm_jax_rtc_client.py
│   ├── start_jax_rtc_server.sh
│   └── start_jax_rtc_dual_arm_client.sh
├── temporal_ensemble/
│   ├── openpi_piper_jax_temporal_ensemble_policy_server.py
│   ├── openpi_piper_dual_arm_jax_temporal_ensemble_client.py
│   ├── start_jax_temporal_ensemble_server.sh
│   └── start_jax_temporal_ensemble_dual_arm_client.sh
├── configs/
│   ├── policy.example.yaml
│   ├── server.example.env
│   ├── client.example.env
│   ├── rtc.example.env
│   └── temporal_ensemble.example.env
├── policy/
│   └── README.md
├── references/
│   ├── original_jax_orbax_policy_server.py
│   ├── piper_inference_dependency.py
│   ├── piper_motor_driver.py
│   └── piper_robot_driver.py
├── DEPENDENCIES.md
└── CHECKSUMS.sha256

The configs/ directory contains a policy compatibility manifest and sourceable deployment templates. policy/README.md defines the adapter request/response contract. The references/ directory records upstream integration points used by the clients; the launchers do not automatically install or replace the corresponding OpenPI, LeRobot, RealSense, or Piper SDK dependencies.

Actual policy weights are intentionally excluded. Users verify compatibility with configs/policy.example.yaml, then copy the environment templates and fill in their own checkpoint, norm stats, paths, and devices:

cd /path/to/openpi-jax-rtc-te
cp configs/server.example.env configs/server.local.env
cp configs/client.example.env configs/client.local.env

The .local.env files are ignored by Git and should not be uploaded.

RTC and Temporal Ensemble compared

Item	RTC	Temporal Ensemble
Server port default	`8011`	`8012`
Executable chunk default	25 actions	50 actions
Replanning behavior	Generates a new chunk with constraints from the previous full trajectory	Generates independent overlapping chunks
Transition handling	Hard-freezes the immediate prefix and softly constrains the overlap	Averages all valid predictions for the same control step
Main client parameters	delay, execution horizon, soft-mask decay, guidance scale	inference interval, decay, maximum retained chunks
Recommended use	Long-horizon or bimanual tasks where plan commitment matters	Tasks where local prediction noise is the main problem and overlapping plans remain compatible

Do not run the TE client against an RTC-enabled server. The TE client explicitly rejects this combination because ordinary temporal averaging can overwrite the trajectory commitment introduced by RTC.

Action and robot assumptions

This implementation assumes:

A dual-arm Piper observation/action has 14 values: 7 for the left arm and 7 for the right arm.
Each 7D block contains six arm joints and one gripper command.
Physical robot order is normally left(can0), right(can1).
The server supports either a native 14D Piper policy output or an 80D unified policy output.
For an 80D unified policy, the Piper slots are mapped explicitly; the code does not use a generic action[:14] slice.
The checkpoint is a JAX Orbax/OCDBT checkpoint and the normalization statistics belong to the same training configuration.

The server converts model outputs into absolute 14D Piper targets. The client then applies the selected arm mapping before sending commands to the robot.

Camera configuration

The current dual-arm client initializes four RealSense streams:

Client key	Intended view	Used by the current JAX model adapter
`head`	head or global view	Yes
`left_wrist`	left wrist	Yes
`right_wrist`	right wrist	Yes
`front_view`	additional front view	No

The client currently captures and sends all four images, while the server-side policy adapter consumes only head, left_wrist, and right_wrist. Therefore, the checked-in launcher is operationally a four-camera client feeding a three-camera policy.

To use the repository unchanged, provide all four camera serial numbers:

export HEAD_CAMERA_SERIAL='<HEAD_CAMERA_SERIAL>'
export LEFT_WRIST_CAMERA_SERIAL='<LEFT_WRIST_CAMERA_SERIAL>'
export RIGHT_WRIST_CAMERA_SERIAL='<RIGHT_WRIST_CAMERA_SERIAL>'
export FRONT_VIEW_CAMERA_SERIAL='<FRONT_VIEW_CAMERA_SERIAL>'

Alternatively, provide the complete camera object yourself:

export ROBOT_CAMERAS_JSON='<YOUR_CAMERA_JSON>'

Changing to a true three-camera client requires editing build_remote_observation() and the launcher camera configuration so that the unused front_view camera is neither initialized nor read. Simply omitting FRONT_VIEW_CAMERA_SERIAL will make the current launcher stop with a missing-variable error.

Requirements

Policy server machine

Linux machine with a CUDA-capable GPU.
Python 3.11.
An OpenPI source checkout containing:
- src/
- packages/openpi-client/src/
The Python/JAX environment required by that OpenPI checkout.
A complete Orbax checkpoint containing at least:
- _CHECKPOINT_METADATA
- params/manifest.ocdbt
- params/array_metadatas
A matching norm_stats.json.
A valid OpenPI configuration name resolvable by your local cotrain configuration code.

Piper client machine

Python 3.11 with the OpenPI WebSocket client dependencies.
A compatible LeRobot-Piper checkout.
Piper SDK and RealSense dependencies required by that checkout.
Two configured CAN interfaces, normally can0 for the left arm and can1 for the right arm.
Four visible RealSense devices for the checked-in launcher.
Network access to the selected server port.

This repository does not include checkpoints, datasets, normalization files, robot calibration, SSH configuration, API credentials, or camera serial numbers.

Required environment variables

Server variables

Variable	Meaning	Example placeholder
`OPENPI_ROOT`	OpenPI checkout	`/path/to/openpi`
`CHECKPOINT_DIR`	Exact Orbax checkpoint step directory	`/path/to/checkpoints/checkpoint_step`
`NORM_STATS_PATH`	Matching normalization JSON	`/path/to/assets/norm_stats.json`
`OPENPI_PYTHON_BIN`	Absolute path to the server Python 3.11 executable	`/path/to/python3.11`
`CONFIG_NAME`	OpenPI training/inference config name	`your_config_name`
`PROMPT`	Task instruction	`your task instruction`
`POLICY_HOST`	Bind address; use `0.0.0.0` for a remote client	`0.0.0.0`
`POLICY_PORT`	WebSocket server port	RTC `8011`, TE `8012`

Use an absolute Python path. The launchers validate executable paths before starting, so a bare command such as python3 may not pass the path check.

Client variables

Variable	Meaning	Example placeholder
`OPENPI_ROOT`	OpenPI checkout available on the client	`/path/to/openpi`
`LEROBOT_PIPER_ROOT`	Compatible LeRobot-Piper checkout	`/path/to/lerobot-piper`
`CLIENT_PYTHON_BIN`	Absolute path to client Python 3.11	`/path/to/python3.11`
`POLICY_HOST`	IP or hostname of the GPU server	`<SERVER_IP>`
`POLICY_PORT`	Must match the selected server	RTC `8011`, TE `8012`
`PROMPT`	Must describe the same task intended for the policy	`your task instruction`
`LEFT_CAN_NAME`	Left-arm CAN interface	`can0`
`RIGHT_CAN_NAME`	Right-arm CAN interface	`can1`
Camera variables	Four serial numbers, or `ROBOT_CAMERAS_JSON`	placeholders above

Deployment workflow

Use RTC and TE as separate experiments. Start one matching server/client pair at a time.

1. Check checkpoint and normalization compatibility

Before starting either method, verify that:

CHECKPOINT_DIR points to the exact checkpoint step you intend to test.
NORM_STATS_PATH was produced for the same dataset/action representation.
CONFIG_NAME reconstructs the same model architecture and transforms.
The model action horizon is at least the requested ACTIONS_PER_INFERENCE.
The camera keys and prompt format match training.
The policy arm order matches the 14D action layout used by the checkpoint.

Changing only the checkpoint path is insufficient when its config, normalization statistics, or action mapping differs.

2. Validate the server without opening a network service

On the GPU server:

cd /path/to/openpi-jax-rtc-te

export OPENPI_ROOT='/path/to/openpi'
export CHECKPOINT_DIR='/path/to/checkpoints/checkpoint_step'
export NORM_STATS_PATH='/path/to/assets/norm_stats.json'
export OPENPI_PYTHON_BIN='/path/to/python3.11'
export CONFIG_NAME='your_config_name'
export PROMPT='your task instruction'
export POLICY_HOST='0.0.0.0'

Validate RTC:

export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export VALIDATE_ONLY='true'

bash rtc/start_jax_rtc_server.sh

RTC validation loads the checkpoint, runs synthetic inference, verifies the executable action shape and full action horizon, and checks that the RTC hard/soft/free mask regions are internally consistent. It does not start the WebSocket service.

Validate Temporal Ensemble:

export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export VALIDATE_ONLY='true'

bash temporal_ensemble/start_jax_temporal_ensemble_server.sh

TE validation loads the same server-side policy path and checks a synthetic inference result. Temporal ensembling itself runs on the client.

3. Start the selected policy server

After validation succeeds, keep the same checkpoint/config/norm-stat variables and disable validation-only mode.

Start RTC server:

cd /path/to/openpi-jax-rtc-te

export POLICY_HOST='0.0.0.0'
export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export VALIDATE_ONLY='false'

bash rtc/start_jax_rtc_server.sh

Start TE server instead:

cd /path/to/openpi-jax-rtc-te

export POLICY_HOST='0.0.0.0'
export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export VALIDATE_ONLY='false'

bash temporal_ensemble/start_jax_temporal_ensemble_server.sh

Allow the selected TCP port through the server firewall if the robot client runs on another machine. Do not expose the policy server directly to an untrusted network.

4. Configure the Piper client

On the robot computer:

cd /path/to/openpi-jax-rtc-te

export OPENPI_ROOT='/path/to/openpi'
export LEROBOT_PIPER_ROOT='/path/to/lerobot-piper'
export CLIENT_PYTHON_BIN='/path/to/python3.11'

export POLICY_HOST='<SERVER_IP>'
export PROMPT='your task instruction'
export LEFT_CAN_NAME='can0'
export RIGHT_CAN_NAME='can1'

export HEAD_CAMERA_SERIAL='<HEAD_CAMERA_SERIAL>'
export LEFT_WRIST_CAMERA_SERIAL='<LEFT_WRIST_CAMERA_SERIAL>'
export RIGHT_WRIST_CAMERA_SERIAL='<RIGHT_WRIST_CAMERA_SERIAL>'
export FRONT_VIEW_CAMERA_SERIAL='<FRONT_VIEW_CAMERA_SERIAL>'

export FPS='30'
export STEPS='100'

STEPS=0 means run continuously until Ctrl+C. Use a finite value during initial checks.

5. Confirm dual-arm order

There are two different swap controls:

FORCE_SWAP_ARMS changes how the two policy action blocks are interpreted.
COMMAND_ARM_ORDER changes the blocks immediately before they are sent to the physical arms.

Normal physical command order is:

export COMMAND_ARM_ORDER='left-right'
export COMMAND_SWAP_MODE='absolute'

Choose FORCE_SWAP_ARMS from the checkpoint's training action order:

# Policy output is [left 7D, right 7D]
export FORCE_SWAP_ARMS='false'

# Policy output is [right 7D, left 7D]
export FORCE_SWAP_ARMS='true'

Do not use COMMAND_ARM_ORDER=right-left merely because the policy order is reversed. FORCE_SWAP_ARMS already converts between policy order and physical left/right order. The command-level swap is a separate diagnostic/compatibility override.

The RTC and TE launchers currently have different FORCE_SWAP_ARMS defaults. Set it explicitly for both methods rather than relying on those defaults.

6. Run the client in dry-run mode

Dry-run still connects to the robot, captures cameras and state, contacts the policy server, and computes final commands, but it does not call robot.send_action().

RTC dry-run:

cd /path/to/openpi-jax-rtc-te

export POLICY_PORT='8011'
export ACTIONS_PER_INFERENCE='25'
export PREFETCH_THRESHOLD='4'
export RTC_DELAY_STEPS='3'
export RTC_EXECUTION_HORIZON='25'
export RTC_SOFT_MASK_DECAY='0.6'
export RTC_GUIDANCE_SCALE='1.0'
export FORCE_SWAP_ARMS='false'
export ENABLE_ARM='false'
export DRY_RUN='true'

bash rtc/start_jax_rtc_dual_arm_client.sh \
  --debug-action-details \
  --print-command-counts

TE dry-run:

cd /path/to/openpi-jax-rtc-te

export POLICY_PORT='8012'
export ACTIONS_PER_INFERENCE='50'
export ENSEMBLE_INFERENCE_INTERVAL='4'
export ENSEMBLE_DECAY='0.25'
export ENSEMBLE_MAX_CHUNKS='4'
export FORCE_SWAP_ARMS='false'
export ENABLE_ARM='false'
export DRY_RUN='true'

bash temporal_ensemble/start_jax_temporal_ensemble_dual_arm_client.sh

The RTC debug flags print observation, model action, delta, final action, and optional motor-command counts. The TE client logs its final 14D action and the aligned chunk indices/weights used at each step. Before enabling motion, confirm that neither arm is unexpectedly all zeros, verify that left/right action blocks follow the intended physical arms, and check for NaN or infinite values.

7. Start real-hardware execution

Only after the server validation and client dry-run match the intended setup:

RTC:

cd /path/to/openpi-jax-rtc-te

export STEPS='0'
export ENABLE_ARM='true'
export DRY_RUN='false'
export DISABLE_ARM_ON_EXIT='false'

bash rtc/start_jax_rtc_dual_arm_client.sh

Temporal Ensemble:

cd /path/to/openpi-jax-rtc-te

export STEPS='0'
export ENABLE_ARM='true'
export DRY_RUN='false'
export DISABLE_ARM_ON_EXIT='false'

bash temporal_ensemble/start_jax_temporal_ensemble_dual_arm_client.sh

The launchers intentionally keep action scaling, additional smoothing, command deadbands, and client-side delta limiting disabled or effectively open. They do not add conservative motion filters. Real-hardware behavior therefore depends directly on the policy outputs, RTC/TE controller, robot driver, and physical controller limits.

Press Ctrl+C to stop. If DISABLE_ARM_ON_EXIT=true, the client also requests arm disable during shutdown after the configured settle period.

RTC parameter behavior

The intended RTC setup uses a model horizon of 50 actions while returning a 25-action executable prefix:

full_actions: retains the full model trajectory for the next RTC request.
actions: contains the executable prefix returned to the client.
RTC_DELAY_STEPS: estimates how many immediate actions must remain fixed while a new inference is in flight.
RTC_EXECUTION_HORIZON: limits how many non-expired returned actions are queued for execution.
RTC_SOFT_MASK_DECAY: sets the first soft-overlap weight and its exponential decay across the remaining overlap.
RTC_GUIDANCE_SCALE: controls the RTC-guided sampling contribution.

Server and client values for ACTIONS_PER_INFERENCE, delay, execution horizon, soft-mask decay, and guidance scale should match. The client logs skip, hard, soft, free, queued, and full_prior at chunk switches; use these fields to confirm that the full prior is retained and the soft region is not empty.

The default deployment values are:

model horizon:             50
ACTIONS_PER_INFERENCE:     25
FPS:                       30
PREFETCH_THRESHOLD:        4
RTC_DELAY_STEPS:           3
RTC_EXECUTION_HORIZON:     25
RTC_SOFT_MASK_DECAY:       0.6
RTC_GUIDANCE_SCALE:        1.0

Do not set the server to 25 model-horizon actions if the checkpoint natively predicts 50. RTC needs the full prior trajectory to form a non-empty soft overlap even though only 25 actions are exposed for execution.

Temporal Ensemble parameter behavior

The TE client aligns every action using:

action_index = control_step - request_step

Only predictions targeting the current absolute control step are combined. Older predictions receive lower weight according to the configured exponential decay.

The default deployment values are:

ACTIONS_PER_INFERENCE:       50
FPS:                         30
ENSEMBLE_INFERENCE_INTERVAL: 4
ENSEMBLE_DECAY:              0.25
ENSEMBLE_MAX_CHUNKS:         4

A smaller inference interval creates more overlap and more server load. A larger decay favors newer predictions more strongly. Increasing ENSEMBLE_MAX_CHUNKS retains more historical plans, which may smooth local noise but can also increase hesitation when old and new chunks represent different task stages.

Troubleshooting

`Missing required path: python3`

Set the Python variable to an absolute executable path:

export OPENPI_PYTHON_BIN='/absolute/path/to/python3.11'
export CLIENT_PYTHON_BIN='/absolute/path/to/python3.11'

Client cannot connect to the server

Check that:

VALIDATE_ONLY=false on the server.
Server uses POLICY_HOST=0.0.0.0 when accepting remote connections.
Client and server use the same port.
The GPU server firewall allows that port.
No older server process is still occupying the port.

Checkpoint validation fails

Verify the exact checkpoint directory, required Orbax metadata files, Python 3.11 environment, CONFIG_NAME, and matching NORM_STATS_PATH. A checkpoint from another action representation or dataset cannot be repaired by changing only the path.

Camera initialization fails

Confirm that all four serial numbers are visible on the Piper computer and unique. If using ROBOT_CAMERAS_JSON, each camera entry must include a numeric serial_number and compatible FPS/resolution fields.

Left and right arms are reversed

First keep COMMAND_ARM_ORDER=left-right. Then toggle only FORCE_SWAP_ARMS and inspect dry-run action logs. Use command-level swapping only when you intentionally need to cross the physical output blocks.

RTC soft region is empty

Confirm that the server returns the complete model horizon as full_actions, that the executable prefix is shorter than the full horizon, and that server/client RTC parameters match. The client chunk-switch log should contain nonzero soft steps.

TE appears smooth but hesitates at task transitions

TE may be averaging chunks that encode different semantic plans. Reduce the number/age of retained chunks, increase decay so newer plans dominate, or compare against RTC. Do not stack ordinary TE on top of RTC.

Verification and publication notes

The source tree has been sanitized: machine paths, device serial numbers, hostnames, dataset identifiers, and task prompts use placeholders. CHECKSUMS.sha256 covers the published source files, launchers, configuration templates, and interface documentation.

To verify them:

cd /path/to/openpi-jax-rtc-te
shasum -a 256 -c CHECKSUMS.sha256

Before publishing or redistributing this repository:

Review the licenses of OpenPI, LeRobot, Piper SDK, RealSense, and all code retained in references/.
Add an appropriate repository license and upstream attribution.
Pin tested dependency versions or commits.
Do not commit checkpoints, datasets, robot videos, norm statistics, SSH keys, credentials, or real device serial numbers.
Treat syntax checks, synthetic validation, and dry-run as limited evidence; they do not prove safe real-hardware behavior.

Name		Name	Last commit message	Last commit date
configs		configs
policy		policy
references		references
rtc		rtc
temporal_ensemble		temporal_ensemble
CHECKSUMS.sha256		CHECKSUMS.sha256
DEPENDENCIES.md		DEPENDENCIES.md
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Piper OpenPI JAX: RTC and Temporal Ensemble

中文说明

仓库结构

RTC 与 TE 的区别

动作与双臂假设

相机配置：当前是四相机 client、三相机模型

环境要求

GPU 推理服务器

Piper 客户端

主要环境变量

Server 端

Client 端

真实部署流程

第一步：确认 checkpoint、config 和 norm stats

第二步：先验证 server，不启动网络服务

第三步：启动策略 server

第四步：配置 Piper client

第五步：确认双臂顺序

第六步：先运行 dry-run

第七步：启动真实机械臂执行

RTC 参数说明

TE 参数说明

常见问题

Missing required path: python3

Client 无法连接 server

Checkpoint 验证失败

相机初始化失败

左右臂动作相反

RTC soft 区域为空

TE 平滑但在阶段切换时犹豫

校验与发布说明

English

Repository layout

RTC and Temporal Ensemble compared

Action and robot assumptions

Camera configuration

Requirements

Policy server machine

Piper client machine

Required environment variables

Server variables

Client variables

Deployment workflow

1. Check checkpoint and normalization compatibility

2. Validate the server without opening a network service

3. Start the selected policy server

4. Configure the Piper client

5. Confirm dual-arm order

6. Run the client in dry-run mode

7. Start real-hardware execution

RTC parameter behavior

Temporal Ensemble parameter behavior

Troubleshooting

Missing required path: python3

Client cannot connect to the server

Checkpoint validation fails

Camera initialization fails

Left and right arms are reversed

RTC soft region is empty

TE appears smooth but hesitates at task transitions

Verification and publication notes

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`Missing required path: python3`

`Missing required path: python3`