Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 45 additions & 1 deletion .claude/skills/remote-training/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: remote-training
description: Manages remote training infrastructure on Nebius Serverless (Jobs for convert/train, Endpoints for inference). Use for dataset conversion, training jobs, serving checkpoints, and end-to-end pipeline validation.
description: Manages the convert/train/serve pipeline on Nebius Serverless (Jobs + Endpoints) for H100 work, and local inference serving on desktop/notebook GPUs via Docker contexts. Use for dataset conversion, training jobs, serving checkpoints (remote or local), and end-to-end pipeline validation.
---

# Remote Training Infrastructure (Nebius Serverless)
Expand Down Expand Up @@ -233,6 +233,50 @@ uv run python -m positronic.cfg.eval sim \
# http://localhost:5001
```

## Serving locally (desktop / notebook)

Nebius Serverless is for H100-class work. For local inference on a consumer GPU
(LeRobot/ACT/SmolVLA, or GR00T inference), serve via Docker contexts instead —
no Nebius, no per-hour compute cost. The contexts and services still live in the
repo; **`docker/CONTEXTS.md` + `docker/docker-compose.yml` are the source of
truth** for which machine/GPU/service to use (`desktop` = RTX 3060 12GB,
`notebook` = RTX 4060 8GB). OpenPI/DreamZero and GR00T *training* still need
H100 (use the Nebius pipeline above).

Run from `docker/`. Set `CACHE_ROOT=/home/vertix` when targeting a remote
context from a Mac (the `${HOME}` volume path differs). `--service-ports`
exposes the WebSocket API on port 8000. Servers take a subcommand: `serve` for
a custom `--checkpoints_dir`, or a named preset (`phail`, `sim_stack`, …) —
check the vendor's `server.py` for available presets.

```bash
# Named preset (desktop)
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always \
--service-ports lerobot-0_3_3-server sim_stack

# Custom checkpoint
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always \
--service-ports lerobot-server serve --checkpoints_dir=<ckpt-dir>

# GR00T inference — codec subcommand required
CACHE_ROOT=/home/vertix docker --context notebook compose run --rm --pull always \
--service-ports groot-server ee_rot6d --checkpoints_dir=<ckpt-dir>
```

Run detached with `-d` for a background server; `docker --context <ctx> ps` /
`logs <id>` / `stop <id>` to manage it. Point the client at the context's
hostname:

```bash
uv run positronic-inference sim \
--policy=.remote --policy.host=desktop --policy.port=8000 \
--output_dir=<...>
```

Gotchas: each GR00T server uses ~6GB, so only one at a time on a 12GB GPU;
on a port conflict, `docker --context <ctx> ps -a | grep -E "server"` then
`stop` the stale container.

## End-to-End Validation

`e2e.sh` runs the whole pipeline for one vendor (convert → train 200 steps → serve
Expand Down
Loading