Genesis SAC training on ROCm fails at inference IPC event reuse tick 32

Problem

PR #2523 后，在 AMD MI300X / ROCm 上验证 g1_walk_flat + SAC + Genesis：

make sync-rocm 会把 pyproject.toml 切到 ROCm profile，但该 profile 没有 genesis extra，且 Make target 只安装 mujoco extra，因此 Genesis 会被移出环境，需要手动 uv pip install genesis-world==1.3.3。
手动补装 Genesis 后，3 iteration smoke 可以完成，但不能代表训练支持。
60 iteration 运行在第 29 个 iteration / coordination tick 32 稳定失败：

torch.AcceleratorError: CUDA error: invalid argument
Search for `hipErrorInvalidValue` ...
unilab.rl.runtime.ipc.inference_ring.CudaRingTransport.wait_observation_ready

run_summary.json: status=failed, completed_iterations=29, collector.exitcode=0, learner 在 training/learner_inference 失败。
显式 training.collector_tensor_device=cpu 不是可用 fallback：Genesis probe 返回 CUDA observations，启动即 fail closed。

Isolation

不依赖 UniLab env、Genesis、SAC 的最小跨进程复现也在第 32 次 event record/wait 失败：

import multiprocessing as mp
import sys
import torch

COUNT = 100

def child(handle, ready, done):
    torch.cuda.set_device(0)
    event = torch.cuda.Event.from_ipc_handle(torch.device("cuda:0"), handle)
    stream = torch.cuda.current_stream("cuda:0")
    for _ in range(COUNT):
        ready.acquire()
        event.record(stream)
        done.release()
    torch.cuda.synchronize()

def main():
    ctx = mp.get_context("spawn")
    event = torch.cuda.Event(interprocess=True)
    ready, done = ctx.Semaphore(0), ctx.Semaphore(0)
    process = ctx.Process(target=child, args=(event.ipc_handle(), ready, done))
    process.start()
    stream = torch.cuda.current_stream("cuda:0")
    try:
        for tick in range(COUNT):
            ready.release()
            done.acquire()
            event.wait(stream)
            torch.cuda.synchronize()
    except BaseException as exc:
        print("FAIL_AT", tick, repr(exc))
        process.terminate()
        process.join()
        return 1
    return process.exitcode or 0

if __name__ == "__main__":
    sys.exit(main())

SharedInferenceRing(capacity=1) 复现在 tick 32；capacity=16 复现在 tick 512，符合每个 reused interprocess event 第 32 次使用失效。该问题与 Uni-SuperMimic #3 的 TF32/rocBLAS 精度问题无关：UniLab ROCm 运行时 torch.backends.cuda.matmul.allow_tf32=False、float32_matmul_precision=highest，TF32 resolver 在 HIP 上 fail closed。

Warm collector/physics本身是健康的：Genesis physics 2048 env 约 281k steps/s，production collector 2048 env active 约 31.8k steps/s。当前是 learner-coupled inference IPC 稳定性缺口，不是吞吐或精度问题。

Requested behavior

make sync-rocm 提供完整 Genesis ROCm 训练安装路径，不能要求手动补装包。
默认 g1_walk_flat + SAC + Genesis 在 HIP/ROCm 上要么使用可长期运行的同步/IPC机制，要么在启动前明确 fail closed 并说明当前不支持；不能在第 32 tick 崩溃。
不用增大 inference_slot_capacity 作为临时规避；capacity 16 仍会在 512 tick 崩溃。
NVIDIA CUDA 行为不回退。
ROCm collector benchmark provenance 不应调用只适用于 NVIDIA 的 nvidia-smi；当前测量完成后因缺少 driver receipt 退出 1，无法写出 JSON。

Acceptance criteria

make sync-rocm 后无需手动安装 genesis-world。
MI300X 上默认 SAC/Genesis 训练超过 reused-event 失效阈值（至少 >512 coordination ticks，或按默认 capacity 覆盖更长窗口）后仍正常运行，或启动前给出明确 unsupported 诊断。
增加覆盖 HIP inference-ring event reuse 或替代 transport 的测试。
修复/更新 ROCm benchmark fixation，使 JSON/CSV artifact 能写出。
复跑 NVIDIA SAC/Genesis 回归。

Environment

UniLab: ce1a3296 (PR #2523 merge)
GPU: AMD Instinct MI300X VF
Torch: 2.14.0+rocm7.2 / HIP 7.2.53211
Genesis: 1.3.3

Related: unilabsim/unisim#471, PR #2523, Uni-SuperMimic #3.

Reproduction route on amd:/home/unilab/unilabsim/UniLab:

make sync-rocm
uv pip install --python .venv/bin/python genesis-world==1.3.3
ROCR_VISIBLE_DEVICES=0 timeout 600 .venv/bin/train \
  --algo sac --task g1_walk_flat --sim genesis \
  algo.num_envs=64 algo.batch_size=256 algo.replay_buffer_n=4 \
  algo.learning_starts=1 algo.updates_per_step=1 algo.max_iterations=60 \
  algo.save_interval=1000 algo.algo_params.use_compile=false \
  training.no_play=true training.export_onnx=false \
  training.log_dir=/tmp/unilab-rocm-genesis-warm

Artifacts: /tmp/unilab-rocm-genesis-warm/run_summary.json, /tmp/unilab-rocm-genesis-warm.log; minimal repro scripts: /tmp/raw_rocm_ipc_event.py, /tmp/unilab_rocm_ipc_repro.py.

Root cause found in the bundled ROCm/HIP runtime, not UniLab event ownership.

The PyTorch ROCm wheel bundles libamdhip64.so.7.2.70201-7.2.53211.70201-81.el8.x86_64.debug. Its exported IPC event handles begin with ASCII /hip_<pid>_<counter>, matching ROCm CLR’s emulated shared-memory IPC event implementation.

In rocm-7.2 CLR source:

#define IPC_SIGNALS_PER_EVENT 32
...
uint32_t signal[IPC_SIGNALS_PER_EVENT];

query(), synchronize(), and record enqueue apply % IPC_SIGNALS_PER_EVENT, but IPCEvent::streamWait() does not:

hipError_t IPCEvent::streamWait(hip::Stream* stream, uint flags) {
  int offset = ipc_evt_.ipc_shmem_->read_index;  // missing modulo
  ...
  &signal[offset]
}

Thus the 33rd re-record/stream-wait uses read_index == 32 and passes &signal[32], one element beyond the registered 32-element signal array; hipStreamWaitEvent returns hipErrorInvalidValue.

This exactly explains the deterministic behavior:

1 event: fails at tick 32 (0-based), the 33rd use.
2 events: fails at tick 64, when the first event reaches its 33rd use.
ring capacity 16: fails at tick 512, when each event reaches its 33rd use.
Event.synchronize() and Event.query() keep working because their implementations take modulo.
Fresh consumer streams do not help; it is per-event indexing.

Upstream fix: ROCm/CLR commit a44f100faaa9 (“fix(clr): Adding %IPC_SIGNALS_PER_EVENT in IPCEventEmulated::streamWait”), which changes exactly this line to read_index % IPC_SIGNALS_PER_EVENT. The bundled 7.2 runtime predates that fix.

Compatibility patch: #2533. MI300X validation crossed tick 32 and completed 128 iterations. At num_envs=2048, SAC/Genesis completed 264,192 env steps with tail throughput 34,690.9 env steps/s and normal shutdown.

Navigation Menu

Repository navigation

Genesis SAC training on ROCm fails at inference IPC event reuse tick 32 #2530

Problem

Isolation

Requested behavior

Acceptance criteria

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Genesis SAC training on ROCm fails at inference IPC event reuse tick 32 #2530

Description

Problem

Isolation

Requested behavior

Acceptance criteria

Environment

Activity

TATP-233 commented on Oct 26, 2026

TATP-233 commented on Oct 26, 2026

TATP-233 commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions