Skip to content

Add TimingShim for rollout loops positronic does not run #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vertix wants to merge 5 commits into from
Closed

Add TimingShim for rollout loops positronic does not run #494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
edb9801
Add `TimingShim` for rollout loops positronic does not run
vertix Jul 22, 2026
bb59874
Clear stale `gpu_dmon.log` on pass start even with GPU sampling off
vertix Jul 22, 2026
5cbecf8
Flush shim timings per-episode and clear stale host log
vertix Jul 22, 2026
5f0145e
Capture shim episode wall and seal timestamp together
vertix Jul 22, 2026
0b0f215
Mark shim episode dirs unfinished until writes finish
vertix Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions positronic/eval_timing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ def record_env_phases(physics_s: float, render_s: float, server_s: float) -> Non
timer.add_env_phases(physics_s, render_s, server_s)


def _start_gpu_sampler(out_dir: Path) -> subprocess.Popen | None:
def start_gpu_sampler(out_dir: Path) -> subprocess.Popen | None:
"""Background ``nvidia-smi dmon`` writing this box's util+memory to ``gpu_dmon.log``.

``None`` when no ``nvidia-smi`` is on PATH (a CPU dev box) — GPU telemetry is then simply absent, not
Expand All @@ -237,7 +237,7 @@ def _start_gpu_sampler(out_dir: Path) -> subprocess.Popen | None:
log_path = out_dir / GPU_LOG_FILENAME
log_path.unlink(missing_ok=True)
if shutil.which('nvidia-smi') is None:
logger.info('EvalTimer: no nvidia-smi on PATH; skipping GPU sampling')
logger.info('no nvidia-smi on PATH; skipping GPU sampling')
return None
# Sample only the GPU this eval runs on — the first CUDA-visible device, else device 0. Left unpinned,
# dmon logs every visible GPU and ``_parse_dmon`` would average idle/unrelated devices into the numbers.
Expand All @@ -263,7 +263,7 @@ def bind(out_dir: Path) -> Iterator[EvalTimer]:
"""Bind a fresh timer (and a GPU sampler) for the enclosed run, then flush ``timing.jsonl`` on exit."""
timer = EvalTimer(out_dir)
token = _ACTIVE.set(timer)
sampler = _start_gpu_sampler(out_dir)
sampler = start_gpu_sampler(out_dir)
try:
yield timer
finally:
Expand Down
Loading
Loading