Repository navigation
fix(envs): restore mjwarp collector Backend Step timing and surface CUDA MPS status - #2497
Merged
Merged
Conversation
…iming For an event-backed asynchronous backend step (mjwarp), step_tensor only launches device work; the physics wait surfaces at the first host read of the step outputs, which is the autoreset done-mask gate. That window was measured inside reset_done_ms, so the collector Env Step breakdown showed real device work as Reset Done while Backend Step collapsed to launch overhead and disappeared under the terminal materiality filter. Measure the gate separately and add it to step_core_ms/step_core_cpu_ms. Issue #2495.
Render the rank-local cuda_process_sharing runtime-manifest evidence as a CUDA MPS row: on for a validated request, ambient for a transparent environment-level daemon, off otherwise. This closes the terminal-side observation gap from #2479 where environment-level MPS silently changed benchmark behavior without any run-level signal. Issue #2495.
The first host read of an event-backed asynchronous backend step (mjwarp) lands inside ManagerBasedRlEnv.update_state: the termination phase window, already measured as update_state_queue_drain_ms, is where the host drains the queued device work. Move that window from update_state_ms into step_core_ms via the new TorchEnv._collect_step_core_drain_ms protocol so the collector Env Step breakdown shows the physics wait as Backend Step (2048-env probe: step_core 0.05 -> 5.52 ms, update_state 8.5 -> 3.8 ms). Only receipt-publishing backends re-attribute; synchronous producers drain nothing and keep their existing attribution. Issue #2495.
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TorchEnv.step:修复事件型异步后端(mjwarp)的 Env Step 计时归因,解决 collectorEnv Step分解缺Backend Step行的 bug。mjwarp 的step_tensor只发射 device work;物理等待发生在 step 输出之后第一个 host 读取处:ManagerBasedRlEnv.update_state内 termination 阶段的 queue drain(仓库已测为update_state_queue_drain_ms,注释本就声明其"drain pre-update GPU work",但此前step_core_ms从未包含它)——经新增保护协议TorchEnv._collect_step_core_drain_ms从update_state_ms重归入step_core_ms;reset_done_ms拆出归入step_core_ms。仅
produces_completion_receipts(mjwarp)的后端做重归因;同步后端(mujoco/motrix/genesis)drain 为零,归因不变;各 phase 合计与env_step_total_ms不变。OffPolicyLogger:System 列新增CUDA MPS行,消费既有runtime_manifest["cuda_process_sharing"]证据——验证通过的显式请求显示on,环境级透明 daemon 显示ambient,否则off。补上 Work: HOST_BRIDGE CUDA 载体吞吐归因收口——环境级 MPS 观测缺口、旋钮合并与 action 路径根本修复 #2479 观测缺口的终端侧。Backend Step归因说明。行为影响:仅计时归因与终端显示变化,不改变 step/reset 语义与执行路径。
Linked Work
mainValidation
make test-allpassed on the final local head before this PR was created or updatedCommands actually run:
Remote CI route:
main: current-head CI(286d8091)all pass — https://github.com/Motphys/UniLab/actions/runs/39716808956Impact
Artifacts
Checklist