diff --git a/.claude/skills/remote-training/SKILL.md b/.claude/skills/remote-training/SKILL.md index fd29303af..5f6fe6d77 100644 --- a/.claude/skills/remote-training/SKILL.md +++ b/.claude/skills/remote-training/SKILL.md @@ -1,245 +1,317 @@ --- name: remote-training -description: Manages remote training infrastructure on Nebius VMs. Use for building/pushing Docker images, starting/stopping VM machines (train, train2, train3), running training jobs, dataset generation, and starting inference servers. +description: Manages remote training infrastructure on Nebius Serverless (Jobs for convert/train, Endpoints for inference). Use for dataset conversion, training jobs, serving checkpoints, and end-to-end pipeline validation. --- -# Remote Training Infrastructure +# Remote Training Infrastructure (Nebius Serverless) -This skill manages the Positronic training infrastructure on Nebius GPU VMs. It covers Docker image management, VM lifecycle, training jobs, dataset generation, and inference server deployment. +This skill runs the Positronic convert → train → serve pipeline on +[Nebius Serverless](https://docs.nebius.com/serverless): **Jobs** for batch work +(dataset conversion, training) and **Endpoints** for HTTP inference servers. +Compute is provisioned per job/endpoint and released automatically when it +finishes, so there is no idle compute cost. -## S3 Convention +All operations go through the wrapper scripts in `workflows/nebius/`. Run them from +the repo root. The full reference is `workflows/nebius/README.md`. -``` -s3://interim/{dataset}/{vendor}/{codec}/ — converted LeRobot datasets -s3://checkpoints/{dataset}/{vendor}/{codec_or_experiment}/ — training output -s3://inference/{dataset}/{date_or_exp}/{vendor}/ — inference eval results -``` +## Vendor tokens -Every run writes `run_metadata_*.yaml` with the full CLI command — read it to reconstruct the pipeline. +Every script takes a `` positional that selects the container image and `uv` +extra. Supported: `lerobot_0_3_3` (ACT), `lerobot` (SmolVLA), `openpi`, `gr00t`. -### Current Artifacts +| Vendor | Image | Train/serve hardware | +|--------|-------|----------------------| +| `lerobot_0_3_3` (ACT) | `positro/positronic:latest` | H100 | +| `lerobot` (SmolVLA) | `positro/positronic:latest` | H100 | +| `openpi` | `positro/openpi:latest` | H100 | +| `gr00t` | `positro/gr00t:latest` | H100 | -**sim_stack** (automated testing — `@positronic.cfg.ds.phail.sim_stack_cubes`): +Conversion always runs on CPU (`cpu-e2`, `8vcpu-32gb`); train/serve on +`gpu-h100-sxm` (`1gpu-16vcpu-200gb`). `openpi`/`gr00t` re-use the `lerobot_0_3_3` +converter with their own codec namespace. -| Vendor | Codec | Interim | Latest Checkpoint | -|--------|-------|---------|-------------------| -| groot | `ee_rot6d` | `s3://interim/sim_stack/groot/ee_rot6d/` | `s3://checkpoints/sim_stack/groot/ee_rot6d/230226/` | -| lerobot (0.4.x) | `ee` | `s3://interim/sim_stack/lerobot_04/ee/` | `s3://checkpoints/sim_stack/lerobot_04/smolvla_150k/` | -| lerobot (0.3.3) | `ee` | `s3://interim/sim_stack/lerobot/ee/` | `s3://checkpoints/sim_stack/lerobot/230226-ee/` | -| openpi | `ee` | `s3://interim/sim_stack/openpi/ee/` | `s3://checkpoints/sim_stack/openpi/ee/pi05_positronic_lowmem/230226/` | +## S3 Convention -**phail_unified** (production — `@positronic.cfg.ds.phail.phail_unified`): +``` +s3://interim/{dataset}/{vendor}/{codec}/ — converted LeRobot datasets +s3://checkpoints/{dataset}/{vendor}/{codec_or_experiment}/ — training output +s3://inference/{dataset}/{date_or_exp}/{vendor}/ — inference eval results +``` -| Vendor | Codec | Interim | Latest Checkpoint | -|--------|-------|---------|-------------------| -| smolvla (0.4.x) | `ee` | `s3://interim/phail_unified/smolvla/ee/` | `s3://checkpoints/phail_unified/smolvla/150315_ft_150k/` | -| groot | `ee_rot6d` | `s3://interim/phail_unified/groot/ee_rot6d/` | — | -| lerobot (0.4.x) / SmolVLA | `ee` | — | `s3://checkpoints/phail_unified/smolvla/170316_ee/` | -| lerobot (0.3.3) | `ee` | `s3://interim/phail_unified/lerobot/ee/` | — | -| openpi | `ee` | `s3://interim/phail_unified/openpi/ee/` | — | +Every run writes a `run_metadata_*.yaml` into its output directory capturing the +full CLI command plus a snapshot of the code state (`*.py`/`*.toml`). See +[Analysing a past run](#analysing-a-past-run) to reconstruct what produced a +given checkpoint. -## Two LeRobot Versions +### Where current paths live -Positronic ships two LeRobot integrations because other vendors (GR00T, OpenPI) depend on the 0.3.3 dataset format: +Concrete dataset/checkpoint S3 paths are intentionally **not** listed here — they +rotate as new runs land and any list goes stale. The source of truth is the +config in the codebase: -| | LeRobot 0.4.x | LeRobot 0.3.3 | -|---|---|---| -| **Convert** | `lerobot-convert` | `lerobot-0_3_3-convert` | -| **Train** | `lerobot-train` | `lerobot-0_3_3-train` | -| **Serve** | `lerobot-server` | `lerobot-0_3_3-server` | -| **Codecs** | `@positronic.vendors.lerobot.codecs.*` | `@positronic.vendors.lerobot_0_3_3.codecs.*` | -| **GPU** | Desktop (consumer GPU) | Desktop (consumer GPU) | +- **Datasets**: `positronic/cfg/ds/` (e.g. `@positronic.cfg.ds.phail.sim_stack_cubes`, + `@positronic.cfg.ds.phail.phail_unified`). +- **Checkpoints**: the per-vendor server presets in + `positronic/vendors//server.py` — the named configs (e.g. `phail`, + `sim_stack`) set `checkpoints_dir=` to the path currently in use. Read those + for the live values rather than relying on a checkpoint path memorized + anywhere. -`lerobot-0_3_3-convert` is also used for GR00T and OpenPI dataset conversion. +To discover what physically exists, `aws s3 ls` under the convention above; to +learn what produced a given checkpoint, read its `run_metadata_*.yaml` (see +[Analysing a past run](#analysing-a-past-run)). -## Machines +## Docker Images -| Context | GPU | Use Case | -|---------|-----|----------| -| `desktop` | RTX 3060 (12GB) | Dataset generation, lerobot training/inference, GR00T inference | -| `notebook` | RTX 4060 Laptop (8GB) | Light tasks, testing, dataset generation | -| `vm-train` | H100 (80GB) | GR00T/OpenPI training and inference | -| `vm-train2` | H100 (80GB) | GR00T/OpenPI training and inference | -| `vm-train3` | H100 (80GB) | GR00T/OpenPI training and inference | +Serverless jobs/endpoints pull `positro/:${NEBIUS_IMAGE_TAG:-latest}` from the +registry — they do **not** mount local source, so any code change needs a rebuild + push +before it takes effect remotely. -Only GR00T and OpenPI training/inference require H100. Everything else runs on `desktop`. +**Tag gotcha:** locally `make push-*` pushes `:` and `:` but **not** `:latest` +(that only happens under `CI`). So after a code change, either: +- `cd docker && CI=1 make push-` — updates `:latest` (what the workflow pulls by default), or +- `cd docker && make push- IMAGE_TAG=` then run the workflow with + `NEBIUS_IMAGE_TAG=` — tests a branch build without clobbering `:latest`. -### VM Management +Plain `make push-` with no `CI`/`IMAGE_TAG`/`NEBIUS_IMAGE_TAG` leaves serverless +running the **old** `:latest` image — the change silently won't take effect. -**IMPORTANT**: Always check if a VM is running a job before using it. +| Image | Source | Used For | +|-------|--------|----------| +| `positro/positronic` | `positronic/docker/` | Conversion, lerobot / SmolVLA train+serve | +| `positro/openpi` | `positronic/docker/` (depends on `positro/openpi-base`) | OpenPI train+serve, openpi stats | +| `positro/gr00t` | `positronic/docker/` (depends on `positro/gr00t-base`) | GR00T train+serve | ```bash -# Check connectivity and running containers -ssh -o ConnectTimeout=5 vertix@vm-train 'echo connected' 2>&1 -docker --context vm-train ps 2>/dev/null - -# Start a stopped VM -../internal/scripts/start.sh train # or train2, train3 +cd docker +make push-training # positro/positronic +make push-openpi # positro/openpi (rebuild positro/openpi-base first if ../openpi changed) +make push-groot # positro/gr00t (rebuild positro/gr00t-base first if ../gr00t changed) +make push # all images ``` -## Docker Images +Cross-repo base rebuilds: `cd ../openpi/docker && make push` (or `../gr00t/docker`), +then `cd ../positronic/docker && make push-openpi`. See `docker/CONTEXTS.md`. -| Image | Source | Used For | -|-------|--------|----------| -| `positro/positronic` | `positronic/docker/` | Dataset conversion, lerobot training/inference | -| `positro/gr00t` | `positronic/docker/` (depends on `positro/gr00t-base`) | GR00T training and inference | -| `positro/openpi` | `positronic/docker/` (depends on `positro/openpi-base`) | OpenPI training and inference | -| `positro/dreamzero` | `positronic/docker/` (depends on `positro/dreamzero-base`) | DreamZero inference | +## One-time setup -Images are tagged by branch name. `make push` in `docker/` auto-derives the tag from the current git branch. +The pipeline reads credentials from Nebius MysteryBox secrets and uses a shared +filesystem for `uv`/HF/openpi caches. This is already provisioned for the +Positronic-internal project. To (re)create it for a different project, follow +"One-time setup" in `workflows/nebius/README.md` (four MysteryBox secrets + +one `network_ssd` filesystem). -```bash -cd /home/vertix/dev/positronic/docker -make push-training # Just positro/positronic -make push-groot # positro/gr00t (rebuild base first if ../gr00t changed) -make push-openpi # positro/openpi (rebuild base first if ../openpi changed) -make push # All images -``` +Defaults point at the Positronic-internal project; override via env when needed: -For cross-repo base image rebuilds: `cd ../gr00t/docker && make push` then `cd ../positronic/docker && make push-groot`. +| Variable | Default | Purpose | +|---|---|---| +| `NEBIUS_PARENT_ID` | `project-e00f38wexevrr52b8j` | Project to create jobs/endpoints in | +| `NEBIUS_SUBNET_ID` | `vpcsubnet-e00pk1j1x6hjmr4m92` | VPC subnet | +| `WANDB_SECRET` | `positronic-serverless-wandb-api-key` | MysteryBox name for WandB key. Set empty to disable wandb. | +| `NEBIUS_CACHE_FS` | `computefilesystem-e00f6jyfr5wkawyrab` | Shared cache filesystem **ID** (mounted RW at `/cache`) | ## Pipeline -All commands run from `docker/` directory. Use `CACHE_ROOT=/home/vertix` when targeting remote Docker contexts from Mac. - ### 1. Convert Dataset +`convert.sh` runs the right converter + codec for the vendor as a CPU Job. For +`openpi` it **blocks until convert finishes, then chains a stats job** and prints +the `--stats_path` to use for training. + ```bash -# GR00T / OpenPI / LeRobot 0.3.3 — use lerobot-0_3_3-convert -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always lerobot-0_3_3-convert convert \ +bash workflows/nebius/convert.sh lerobot_0_3_3 \ --dataset.dataset=@positronic.cfg.ds.phail.sim_stack_cubes \ - --dataset.codec=@positronic.vendors.gr00t.codecs.ee_rot6d \ - --output_dir=s3://interim/sim_stack/groot/ee_rot6d/ + --dataset.codec=@positronic.vendors.lerobot_0_3_3.codecs.ee \ + --output_dir=s3://interim/sim_stack/lerobot/ee/ -# LeRobot 0.4.x (SmolVLA) — use lerobot-convert -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always lerobot-convert convert \ +bash workflows/nebius/convert.sh openpi \ --dataset.dataset=@positronic.cfg.ds.phail.sim_stack_cubes \ - --dataset.codec=@positronic.vendors.lerobot.codecs.ee \ - --output_dir=s3://interim/sim_stack/lerobot_04/ee/ + --dataset.codec=@positronic.vendors.openpi.codecs.ee \ + --output_dir=s3://interim/sim_stack/openpi/ee/ +# → also submits openpi-stats-* ; note the printed --stats_path=<...>/stats/assets/ ``` -**Default codecs**: groot `ee_rot6d`, lerobot `ee`, openpi `ee`. +Default codecs: gr00t `ee_rot6d`, lerobot `ee`, openpi `ee`, +lerobot_0_3_3 `ee`. ### 2. Train -Each vendor has a training script at `positronic/vendors/{vendor}/train.py` with usage examples in its docstring. Read the script's docstring for available subcommands and parameters. +`train.sh` runs `python -m positronic.vendors..train` as an H100 Job. The +dataset bucket is mounted read-only via Mountpoint-S3 at `/mnt/input` for +`lerobot_0_3_3`; other vendors stream via `pos3` from the `s3://` path directly. +`--output_dir` / `--output_path` stays an `s3://` URL (handled by `pos3`). -General pattern: +Read each vendor's `positronic/vendors//train.py` docstring for its exact +flags. `--resume=true` resumes an interrupted run. ```bash -# From docker/ directory -[CACHE_ROOT=/home/vertix] docker [--context ] compose run --rm --pull always \ - [subcommand] --input_path= --exp_name= --output_dir= ... +# ACT / SmolVLA +bash workflows/nebius/train.sh lerobot_0_3_3 \ + --input_path=s3://interim/sim_stack/lerobot/ee/ \ + --exp_name=act_sim_stack_v1 \ + --output_dir=s3://checkpoints/sim_stack/lerobot/ \ + --num_train_steps=50000 --save_freq=10000 + +# OpenPI — needs --stats_path from the convert step's chained stats job +bash workflows/nebius/train.sh openpi \ + --input_path=s3://interim/sim_stack/openpi/ee/ \ + --stats_path=s3://interim/sim_stack/openpi/stats/assets/ \ + --output_path=s3://checkpoints/sim_stack/openpi/ \ + --exp_name=pi_sim_stack_v1 \ + --num_train_steps=30000 +# openpi checkpoint lands at /pi05_positronic_lowmem// ``` -| Vendor | Docker service | Machine | Script (read docstring for usage) | -|--------|---------------|---------|-----------------------------------| -| LeRobot 0.4.x (SmolVLA) | `lerobot-train` | desktop | `positronic/vendors/lerobot/train.py` | -| LeRobot 0.3.3 (ACT) | `lerobot-0_3_3-train` | desktop | `positronic/vendors/lerobot_0_3_3/train.py` | -| GR00T | `groot-train` | H100 | `positronic/vendors/gr00t/train.py` | -| OpenPI | `openpi-train` (needs `openpi-stats` first) | H100 | `positronic/vendors/openpi/train.py` | +The first job after a dependency change pays the full `uv`/HF cold-download +(~10 min); later jobs reuse `/cache` and start faster. + +### 3. Serve a Checkpoint + +`serve.sh [server args...]` creates a public +Endpoint on H100 port 8000, blocks until a public IP is allocated, and prints a +banner containing `Endpoint URL: http://` (the endpoint IP), the endpoint +ID/name, and the teardown command. The container then takes ~10–15 min more +to `uv sync` and load the model. + +```bash +# Named preset — checkpoint path comes from the vendor's server.py config +# (e.g. `demo`, `sim_stack`, `phail`). These are the source of truth; prefer them. +bash workflows/nebius/serve.sh lerobot_0_3_3 my-act-demo demo + +# Explicit checkpoint dir (subcommand differs per vendor). Get from the +# vendor's server.py preset or `aws s3 ls` under the S3 convention — not memorized. +bash workflows/nebius/serve.sh lerobot_0_3_3 act-server serve \ + --checkpoints_dir= -**Resume any training**: add `--resume=true` to the same command. +bash workflows/nebius/serve.sh openpi pi-server serve \ + --checkpoints_dir= -### 3. Start Inference Server +bash workflows/nebius/serve.sh gr00t groot-server ee_rot6d \ + --checkpoints_dir= +``` -All servers use subcommands: `serve` for custom checkpoints, or named presets like `phail`, `sim_stack`. +Sanity-check once warm: ```bash -# LeRobot 0.4.x SmolVLA — preset (desktop) -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-server \ - phail - -# LeRobot 0.4.x SmolVLA — custom checkpoint (desktop) -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-server \ - --checkpoints_dir=s3://checkpoints/sim_stack/lerobot_04/smolvla_150k/ - -# LeRobot 0.3.3 ACT (desktop) — requires subcommand -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-0_3_3-server \ - serve \ - --checkpoints_dir=s3://checkpoints/sim_stack/lerobot/230226-ee/ - -# GR00T (desktop or H100) — codec subcommand required -CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports groot-server \ - ee_rot6d \ - --checkpoints_dir=s3://checkpoints/sim_stack/groot/ee_rot6d/230226/ - -# OpenPI (H100) — requires subcommand -docker --context vm-train compose run --rm --pull always --service-ports openpi-server \ - serve \ - --checkpoints_dir=s3://checkpoints/sim_stack/openpi/ee/pi05_positronic_lowmem/230226/ +curl http://:8000/api/v1/models # → {"models": [""]} ``` -All servers expose WebSocket API on port 8000. Available presets per server: +Tear down (releases compute + public IP): -| Server | Presets | -|--------|---------| -| `lerobot-server` | `serve`, `phail` | -| `lerobot-0_3_3-server` | `serve`, `phail`, `sim_stack` | -| `groot-server` | `serve`, `ee`, `ee_rot6d`, `phail`, `sim_stack`, ... | -| `openpi-server` | `serve`, `phail`, `sim_stack` | +```bash +bash workflows/nebius/stop.sh my-act-demo +``` + +To pause without releasing the static IP: `nebius ai endpoint stop ` +(`start` resumes). ### 4. Run Inference Client +`serve.sh` prints the endpoint IP in its banner. To read it again later (by +endpoint name): + ```bash -# With GUI -uv run positronic-inference sim \ - --policy=.remote --policy.host=desktop --policy.port=8000 \ - --driver.show_gui +nebius ai endpoint list --parent-id "$NEBIUS_PARENT_ID" --format json \ + | jq -r --arg n "" \ + '.items[] | select(.metadata.name==$n) | .status.public_endpoints[0]' +# → the public IP; the endpoint serves on port 8000 +``` + +Point the `positronic-inference` CLI at the endpoint IP: -# Headless -MUJOCO_GL=egl uv run positronic-inference sim \ - --policy=.remote --policy.host=desktop --policy.port=8000 \ - --driver.show_gui=False --driver.simulation_time=10 +```bash +uv run positronic-inference sim \ + --policy=.remote --policy.host= --policy.port=8000 \ + --output_dir=s3://inference/sim_stack_validation/// ``` -## Sim Eval End-to-End +View results locally (top-level dir compares multiple runs): ```bash -# 1. Start server in background (-d flag) -# Check the server script's cfn.cli() to determine if a subcommand is needed (see table above) -CACHE_ROOT=/home/vertix docker --context compose run -d --rm --pull always --service-ports \ - [subcommand] --checkpoints_dir= - -# Wait for ready -docker --context logs --tail 5 -# Look for: "Uvicorn running on http://0.0.0.0:8000" - -# 2. Run sim episodes -CACHE_ROOT=/home/vertix docker --context compose run --rm --pull always positronic-inference \ - sim --policy=.remote --policy.host= --policy.port=8000 \ - --driver.num_iterations=50 --driver.simulation_time=30 \ - --output_dir=s3://inference/sim_stack_validation// - -# 3. View results (locally) — pass top-level dir to compare multiple runs uv run python -m positronic.cfg.eval sim \ --dataset.base.path=s3://inference/sim_stack_validation/ --reset_cache --https -# Opens http://localhost:5001 +# http://localhost:5001 +``` + +## End-to-End Validation -# 4. Clean up -docker --context stop +`e2e.sh` runs the whole pipeline for one vendor (convert → train 200 steps → serve +→ `/api/v1/models` smoke → teardown), polling Nebius and printing a per-stage +status line. ~$2–5 per run. Use it to verify a vendor still works after an +image/dependency/script change. + +```bash +bash workflows/nebius/e2e.sh openpi + +# All four — seed the cache with one first, then fan out warm +bash workflows/nebius/e2e.sh lerobot_0_3_3 +for v in lerobot openpi gr00t; do bash workflows/nebius/e2e.sh "$v" & done; wait +``` + +Override via env: `E2E_S3_BASE`, `E2E_EXP_NAME`, `E2E_LOG_ROOT`, `E2E_DATASET`. + +## Analysing a past run + +Each convert/train/serve run writes `run_metadata_*.yaml` into its S3 output +directory. It records the exact CLI command that produced the artifact and a +snapshot of the relevant source files (`*.py`/`*.toml`), so a checkpoint or +dataset can be traced back to the code and arguments that made it — without +guessing. + +```bash +# List the metadata files under a checkpoint/dataset output dir +aws s3 ls s3://checkpoints/sim_stack/openpi/ee/pi05_positronic_lowmem// \ + --recursive | grep run_metadata_ + +# Read one (full command + code snapshot) +aws s3 cp s3://checkpoints/.../run_metadata_YYYYMMDD_HHMMSS.yaml - | less ``` -**Naming**: `s3://inference/sim_stack_validation///` where model_type is `lerobot`, `groot`, `openpi`, or `dreamzero`. +To reproduce a run, copy the command recorded in `run_metadata_*.yaml` and +resubmit it via the matching `workflows/nebius/*.sh` wrapper (set +`NEBIUS_IMAGE_TAG` if the run used a non-`latest` image). + +For inference runs, each episode also has a `static.json` alongside the recorded +data; the eval viewer in [4. Run Inference Client](#4-run-inference-client) +(`positronic.cfg.eval sim --dataset.base.path=…`) renders these for inspection +and side-by-side comparison of multiple runs. -## Monitoring Background Jobs +## Monitoring Jobs & Endpoints ```bash -grep -o '[0-9]*%' /tmp/claude/-home-vertix-dev-positronic/tasks/.output | tail -1 -tail -50 /tmp/claude/-home-vertix-dev-positronic/tasks/.output -grep -i "error\|complete\|finished" /tmp/claude/-home-vertix-dev-positronic/tasks/.output +# Jobs +nebius ai job get # state: PROVISIONING/STARTING/RUNNING/COMPLETED/FAILED +nebius ai job logs --follow +nebius ai job list --parent-id "$NEBIUS_PARENT_ID" --format json | jq '.items[].metadata.name' + +# Endpoints +nebius ai endpoint get +nebius ai endpoint logs --follow # wait for "INFO Started server process" +nebius ai endpoint list --parent-id "$NEBIUS_PARENT_ID" --format json ``` +The `create` call streams the job ID and ready-to-paste follow-up commands. + ## Common Issues -- **CUDA OOM**: Each GR00T server uses ~6GB. On 12GB GPUs (desktop), only one server at a time. -- **Port conflict**: `docker ps -a | grep -E "groot-server|openpi-server"` then `docker stop `. -- **VM unreachable**: `../internal/scripts/start.sh train2` then verify SSH. -- **Headless rendering**: Use `MUJOCO_GL=egl` env var. +- **Job stuck in PROVISIONING/STARTING**: normal — image pull + `uv` resolve. + First run on a cold cache is ~10 min; check `nebius ai job logs --follow`. +- **OpenPI train can't find stats**: pass `--stats_path=/stats/assets/` + exactly as printed by `convert.sh openpi`. Stats must be a *sibling* of the + dataset dir (pos3 forbids upload-inside-download). +- **gr00t with read-only input mount fails**: only `lerobot_0_3_3` uses the RO + Mountpoint-S3 mount; gr00t writes back into the dataset dir, so it streams via + `pos3` from the `s3://` path instead (handled automatically by `train.sh`). +- **Endpoint name collision**: names must be unique in the project. Pick a fresh + name or `stop.sh` the old one. +- **Cold cache + parallel fan-out**: 4 jobs racing to populate `/cache` thrash; + seed with one vendor, then fan out the rest warm (see `e2e.sh` header). +- **Wipe the shared cache**: throwaway `busybox` job mounting the FS — see + "Cleaning the shared cache" in `workflows/nebius/README.md`. ### Nebius Auth (Headless) 1. `nebius --no-browser --auth-timeout 5m iam whoami 2>&1` — extract auth URL 2. User clicks URL, browser redirects to `http://127.0.0.1:PORT/?code=XXX&state=YYY` 3. `curl -s "http://127.0.0.1:PORT/?code=XXX&state=YYY"` on the machine running nebius -4. Auth completes, VM scripts work +4. Auth completes, scripts work diff --git a/positronic/offboard/vendor_server.py b/positronic/offboard/vendor_server.py index a71c40732..1614c9a61 100644 --- a/positronic/offboard/vendor_server.py +++ b/positronic/offboard/vendor_server.py @@ -2,6 +2,7 @@ import asyncio import logging +import time from abc import ABC, abstractmethod from collections.abc import Callable from typing import Any @@ -113,13 +114,24 @@ class VendorServer(ABC): resolve_model(None) → create_policy → reset → warmup """ - def __init__(self, codec: Codec | None, host: str = '0.0.0.0', port: int = 8000, recording_dir: str | None = None): + def __init__( + self, + codec: Codec | None, + host: str = '0.0.0.0', + port: int = 8000, + recording_dir: str | None = None, + idle_timeout_min: float | None = None, + ): self.codec = codec self.host = host self.port = port if recording_dir: self.codec = RecordingCodec(self.codec, pos3.sync(recording_dir)) + self.idle_timeout_min = idle_timeout_min + self._active_sessions = 0 + self._last_activity = time.monotonic() + self.metadata: dict[str, Any] = {} self.app = FastAPI() @@ -168,6 +180,8 @@ async def websocket_endpoint(self, websocket: WebSocket, model_id: str | None = await websocket.accept() logger.info(f'Connected to {websocket.client} requesting {model_id or "default"}') + self._active_sessions += 1 + self._last_activity = time.monotonic() model_handle = None try: model_handle, extra_meta = await self.resolve_model(model_id, websocket) @@ -180,6 +194,7 @@ async def websocket_endpoint(self, websocket: WebSocket, model_id: str | None = try: while True: message = await websocket.receive_bytes() + self._last_activity = time.monotonic() try: raw_obs = deserialise(message) actions = policy.select_action(raw_obs) @@ -198,6 +213,8 @@ async def websocket_endpoint(self, websocket: WebSocket, model_id: str | None = except Exception: logger.debug('Failed to send error to client', exc_info=True) finally: + self._active_sessions = max(0, self._active_sessions - 1) + self._last_activity = time.monotonic() if model_handle is not None: await self.release_policy(model_handle) @@ -207,11 +224,33 @@ async def _startup(self): policy.reset() await self.warmup(policy) + async def _idle_watchdog(self, server: uvicorn.Server): + timeout_s = self.idle_timeout_min * 60 + poll = min(timeout_s, 30) + while not server.should_exit: + await asyncio.sleep(poll) + if self._active_sessions > 0: + continue + idle = time.monotonic() - self._last_activity + if idle >= timeout_s: + logger.warning(f'No activity for {idle:.0f}s (idle timeout {timeout_s:.0f}s); shutting down server') + server.should_exit = True + return + def serve(self): async def _run(): await self._startup() config = uvicorn.Config(self.app, host=self.host, port=self.port, log_level='info') - await uvicorn.Server(config).serve() + server = uvicorn.Server(config) + self._last_activity = time.monotonic() + watchdog = None + if self.idle_timeout_min and self.idle_timeout_min > 0: + watchdog = asyncio.create_task(self._idle_watchdog(server)) + try: + await server.serve() + finally: + if watchdog is not None: + watchdog.cancel() try: asyncio.run(_run()) diff --git a/positronic/vendors/dreamzero/server.py b/positronic/vendors/dreamzero/server.py index eee199810..ac09ce065 100644 --- a/positronic/vendors/dreamzero/server.py +++ b/positronic/vendors/dreamzero/server.py @@ -202,8 +202,11 @@ def __init__( roboarena_port: int = 1234, enable_dit_cache: bool = True, recording_dir: str | None = None, + idle_timeout_min: float | None = None, ): - super().__init__(codec=codec, host=host, port=port, recording_dir=recording_dir) + super().__init__( + codec=codec, host=host, port=port, recording_dir=recording_dir, idle_timeout_min=idle_timeout_min + ) self.model_path = model_path self.dreamzero_venv = Path(dreamzero_venv) self.backbone = backbone @@ -272,6 +275,7 @@ def shutdown_model(self): port=8000, enable_dit_cache=True, recording_dir=None, + idle_timeout_min=None, ) def server( codec: Codec | None, @@ -282,6 +286,7 @@ def server( port: int, enable_dit_cache: bool, recording_dir: str | None, + idle_timeout_min: float | None, ): """Starts the DreamZero inference server.""" with pos3.mirror(): @@ -294,6 +299,7 @@ def server( port=port, enable_dit_cache=enable_dit_cache, recording_dir=recording_dir, + idle_timeout_min=idle_timeout_min, ).serve() diff --git a/positronic/vendors/gr00t/server.py b/positronic/vendors/gr00t/server.py index 123856cd3..e844af450 100644 --- a/positronic/vendors/gr00t/server.py +++ b/positronic/vendors/gr00t/server.py @@ -278,8 +278,11 @@ def __init__( zmq_port: int = 5555, recording_dir: str | None = None, ready_timeout: float = 120.0, + idle_timeout_min: float | None = None, ): - super().__init__(codec=codec, host=host, port=port, recording_dir=recording_dir) + super().__init__( + codec=codec, host=host, port=port, recording_dir=recording_dir, idle_timeout_min=idle_timeout_min + ) self.checkpoints_dir = checkpoints_dir.rstrip('/') self.checkpoint = checkpoint self.modality_config = modality_config @@ -376,6 +379,7 @@ def shutdown_model(self): modality_config='ee', recording_dir=None, ready_timeout=120.0, + idle_timeout_min=None, ) def server( codec: Codec, @@ -386,6 +390,7 @@ def server( modality_config: str, recording_dir: str | None, ready_timeout: float, + idle_timeout_min: float | None, ): """Starts the GR00T inference server with encoding/decoding.""" @@ -399,6 +404,7 @@ def server( port=port, recording_dir=recording_dir, ready_timeout=ready_timeout, + idle_timeout_min=idle_timeout_min, ).serve() diff --git a/positronic/vendors/lerobot/server.py b/positronic/vendors/lerobot/server.py index 857ad1fe9..60afe1a61 100644 --- a/positronic/vendors/lerobot/server.py +++ b/positronic/vendors/lerobot/server.py @@ -33,8 +33,11 @@ def __init__( port: int = 8000, device: str | None = None, recording_dir: str | None = None, + idle_timeout_min: float | None = None, ): - super().__init__(codec=codec, host=host, port=port, recording_dir=recording_dir) + super().__init__( + codec=codec, host=host, port=port, recording_dir=recording_dir, idle_timeout_min=idle_timeout_min + ) self.checkpoints_dir = str(checkpoints_dir).rstrip('/') + '/checkpoints' self.checkpoint = checkpoint self.device = device or _detect_device() @@ -73,10 +76,28 @@ async def release_policy(self, model_handle): await self.policy_manager.release_session() -@cfn.config(codec=lerobot_codecs.ee, checkpoint=None, port=8000, host='0.0.0.0', recording_dir=None) -def main(checkpoints_dir: str, checkpoint: str | None, codec, port: int, host: str, recording_dir: str | None): +@cfn.config( + codec=lerobot_codecs.ee, checkpoint=None, port=8000, host='0.0.0.0', recording_dir=None, idle_timeout_min=None +) +def main( + checkpoints_dir: str, + checkpoint: str | None, + codec, + port: int, + host: str, + recording_dir: str | None, + idle_timeout_min: float | None, +): checkpoints_dir = str(pos3.download(checkpoints_dir)) - InferenceServer(codec, checkpoints_dir, checkpoint, host=host, port=port, recording_dir=recording_dir).serve() + InferenceServer( + codec, + checkpoints_dir, + checkpoint, + host=host, + port=port, + recording_dir=recording_dir, + idle_timeout_min=idle_timeout_min, + ).serve() phail = main.override( diff --git a/positronic/vendors/lerobot_0_3_3/server.py b/positronic/vendors/lerobot_0_3_3/server.py index df6450b16..2782d9361 100644 --- a/positronic/vendors/lerobot_0_3_3/server.py +++ b/positronic/vendors/lerobot_0_3_3/server.py @@ -43,8 +43,11 @@ def __init__( metadata: dict[str, Any] | None = None, device: str | None = None, recording_dir: str | None = None, + idle_timeout_min: float | None = None, ): - super().__init__(codec=codec, host=host, port=port, recording_dir=recording_dir) + super().__init__( + codec=codec, host=host, port=port, recording_dir=recording_dir, idle_timeout_min=idle_timeout_min + ) self.policy_factory = policy_factory self.checkpoints_dir = str(checkpoints_dir).rstrip('/') + '/checkpoints' self.checkpoint = checkpoint @@ -96,7 +99,15 @@ def act(checkpoint_path: str) -> PreTrainedPolicy: return policy -@cfn.config(policy_factory=act, codec=lerobot_codecs.ee, checkpoint=None, port=8000, host='0.0.0.0', recording_dir=None) +@cfn.config( + policy_factory=act, + codec=lerobot_codecs.ee, + checkpoint=None, + port=8000, + host='0.0.0.0', + recording_dir=None, + idle_timeout_min=None, +) def main( policy_factory: Callable[[str], PreTrainedPolicy], checkpoints_dir: str, @@ -105,10 +116,18 @@ def main( port: int, host: str, recording_dir: str | None, + idle_timeout_min: float | None, ): checkpoints_dir = str(pos3.download(checkpoints_dir)) InferenceServer( - policy_factory, codec, checkpoints_dir, checkpoint, host=host, port=port, recording_dir=recording_dir + policy_factory, + codec, + checkpoints_dir, + checkpoint, + host=host, + port=port, + recording_dir=recording_dir, + idle_timeout_min=idle_timeout_min, ).serve() @@ -123,12 +142,16 @@ def main( _DEMO_CHECKPOINT = 's3://positronic-public/checkpoints/sim_stack_cubes/act/' -@cfn.config(policy_factory=act, codec=lerobot_codecs.ee, checkpoint=None, port=8000, host='0.0.0.0') -def demo(policy_factory, checkpoint, codec, port, host): +@cfn.config( + policy_factory=act, codec=lerobot_codecs.ee, checkpoint=None, port=8000, host='0.0.0.0', idle_timeout_min=None +) +def demo(policy_factory, checkpoint, codec, port, host, idle_timeout_min): from positronic.cfg.ds import PUBLIC checkpoints_dir = str(pos3.download(_DEMO_CHECKPOINT, profile=PUBLIC)) - InferenceServer(policy_factory, codec, checkpoints_dir, checkpoint, host=host, port=port).serve() + InferenceServer( + policy_factory, codec, checkpoints_dir, checkpoint, host=host, port=port, idle_timeout_min=idle_timeout_min + ).serve() if __name__ == '__main__': diff --git a/positronic/vendors/openpi/server.py b/positronic/vendors/openpi/server.py index d5f4c5e22..9b0ea26f4 100644 --- a/positronic/vendors/openpi/server.py +++ b/positronic/vendors/openpi/server.py @@ -190,8 +190,11 @@ def __init__( openpi_ws_port: int = 8001, metadata: dict[str, Any] | None = None, recording_dir: str | None = None, + idle_timeout_min: float | None = None, ): - super().__init__(codec=codec, host=host, port=port, recording_dir=recording_dir) + super().__init__( + codec=codec, host=host, port=port, recording_dir=recording_dir, idle_timeout_min=idle_timeout_min + ) self.checkpoints_dir = str(checkpoints_dir).rstrip('/') self.config_name = config_name self.checkpoint = checkpoint @@ -280,6 +283,7 @@ def shutdown_model(self): port=8000, openpi_ws_port=8001, recording_dir=None, + idle_timeout_min=None, ) def server( codec, @@ -290,6 +294,7 @@ def server( port: int, openpi_ws_port: int, recording_dir: str | None, + idle_timeout_min: float | None, ): """OpenPI inference server. @@ -316,6 +321,7 @@ def server( port=port, openpi_ws_port=openpi_ws_port, recording_dir=recording_dir, + idle_timeout_min=idle_timeout_min, ).serve() diff --git a/workflows/nebius/README.md b/workflows/nebius/README.md index 2cb37a45a..1e0ea156c 100644 --- a/workflows/nebius/README.md +++ b/workflows/nebius/README.md @@ -70,7 +70,9 @@ nebius mysterybox secret create \ The names matter — `convert.sh`, `train.sh`, and `serve.sh` reference the secrets by name. If a secret with one of these names already exists, the create call fails; skip it. -Also create one shared filesystem for the dependency caches. Every job and endpoint mounts it +## Shared cache filesystem + +Create one shared filesystem for the dependency caches. Every job and endpoint mounts it read-write at `/cache`; `uv`, HuggingFace, and openpi asset downloads land there and persist across cold starts, so only the first run after a dependency change pays the download cost: @@ -91,38 +93,8 @@ The filesystem is RWX — many jobs/endpoints attach it concurrently. pos3's own (`~/.cache/positronic/s3/`) is deliberately *not* redirected here; it stays on each container's local disk and re-fetches from S3 by design. -### Cleaning the shared cache - -There is no file browser for the filesystem — to inspect or wipe it you mount it in a -throwaway job. Make sure no jobs/endpoints are using the cache first (a wipe while a -warm job reads it will break that job). - -Inspect usage: - -```bash -nebius ai job create --parent-id "$PARENT_ID" --subnet-id "$SUBNET_ID" \ - --name cache-du --image busybox:latest \ - --container-command du --args '-sh /cache /cache/uv /cache/hf /cache/openpi' \ - --platform cpu-e2 --preset 4vcpu-16gb --timeout 1h \ - --volume "$NEBIUS_CACHE_FS:/cache:rw" -# then: nebius ai job logs -``` - -Wipe everything (full reset — the next run repays the cold download): - -```bash -nebius ai job create --parent-id "$PARENT_ID" --subnet-id "$SUBNET_ID" \ - --name cache-wipe --image busybox:latest \ - --container-command find --args '/cache -mindepth 1 -delete' \ - --platform cpu-e2 --preset 4vcpu-16gb --timeout 1h \ - --volume "$NEBIUS_CACHE_FS:/cache:rw" -``` - -To clear only one tool's cache, target its subdir, e.g. `--args '/cache/uv -mindepth 1 --delete'`. Two gotchas: `--volume` needs the filesystem **ID** (not name), and Nebius -space-splits `--args`, so use a no-shell command (`find`/`du`) — a quoted `sh -c "..."` -gets torn apart. Deleting and recreating the filesystem also works but loses the warm -cache for every workflow. +To inspect or wipe this filesystem later, see +[Appendix: Cleaning the shared cache](#appendix-cleaning-the-shared-cache). ## Convert a Positronic dataset @@ -311,6 +283,7 @@ override them** with their own project + subnet IDs: | `NEBIUS_SUBNET_ID` | `vpcsubnet-e00pk1j1x6hjmr4m92` | VPC subnet for the compute instance | | `WANDB_SECRET` | `positronic-serverless-wandb-api-key` | MysteryBox secret name for the WandB key. Set empty (`WANDB_SECRET=`) to skip wandb entirely. | | `NEBIUS_CACHE_FS` | `computefilesystem-e00f6jyfr5wkawyrab` | Shared filesystem **ID** (not name — `--volume` rejects names) mounted RW at `/cache` for the `uv`/HF/openpi caches (`UV_CACHE_DIR`, `HF_HOME`, `OPENPI_DATA_HOME`). Not used by pos3. The default is Positronic-internal; external users must override with their own filesystem ID. | +| `NEBIUS_IMAGE_TAG` | `latest` | Docker image tag the job/endpoint pulls (`positro/:`). `cd docker && make push-* IMAGE_TAG=` pushes that tag unconditionally; set `NEBIUS_IMAGE_TAG=` to run a branch build remotely without clobbering `:latest`. `make push-*` only updates `:latest` when run with `CI` set. Note `convert.sh openpi` chains a stats job on the `positro/openpi` image, so with `NEBIUS_IMAGE_TAG=` you must also have pushed `positro/openpi:` (not just `positro/positronic:`); otherwise leave `NEBIUS_IMAGE_TAG` unset so stats uses `:latest`. | Other operational settings (platform/preset, MysteryBox secret names, S3 endpoint URL, region) are hardcoded — change them by editing the script directly. The vendor positional arg selects @@ -322,3 +295,36 @@ the container image and `uv` extras: | `lerobot` (SmolVLA) | `positro/positronic` | `--extra lerobot` | | `openpi` | `positro/openpi` | `--extra openpi` (serve); none for train/stats | | `gr00t` | `positro/gr00t` | _(none — `/gr00t` is co-installed)_ | + +## Appendix: Cleaning the shared cache + +There is no file browser for the [shared cache filesystem](#shared-cache-filesystem) — to +inspect or wipe it you mount it in a throwaway job. Make sure no jobs/endpoints are using the +cache first (a wipe while a warm job reads it will break that job). + +Inspect usage: + +```bash +nebius ai job create --parent-id "$PARENT_ID" --subnet-id "$SUBNET_ID" \ + --name cache-du --image busybox:latest \ + --container-command du --args '-sh /cache /cache/uv /cache/hf /cache/openpi' \ + --platform cpu-e2 --preset 4vcpu-16gb --timeout 1h \ + --volume "$NEBIUS_CACHE_FS:/cache:rw" +# then: nebius ai job logs +``` + +Wipe everything (full reset — the next run repays the cold download): + +```bash +nebius ai job create --parent-id "$PARENT_ID" --subnet-id "$SUBNET_ID" \ + --name cache-wipe --image busybox:latest \ + --container-command find --args '/cache -mindepth 1 -delete' \ + --platform cpu-e2 --preset 4vcpu-16gb --timeout 1h \ + --volume "$NEBIUS_CACHE_FS:/cache:rw" +``` + +To clear only one tool's cache, target its subdir, e.g. `--args '/cache/uv -mindepth 1 +-delete'`. Two gotchas: `--volume` needs the filesystem **ID** (not name), and Nebius +space-splits `--args`, so use a no-shell command (`find`/`du`) — a quoted `sh -c "..."` +gets torn apart. Deleting and recreating the filesystem also works but loses the warm +cache for every workflow. diff --git a/workflows/nebius/convert.sh b/workflows/nebius/convert.sh index f9a507ab6..1187e3781 100644 --- a/workflows/nebius/convert.sh +++ b/workflows/nebius/convert.sh @@ -18,6 +18,10 @@ SUBNET_ID="${NEBIUS_SUBNET_ID:-vpcsubnet-e00pk1j1x6hjmr4m92}" # Shared filesystem (RWX) holding the uv / HF / openpi caches across cold starts. # pos3's cache stays on local disk (~/.cache/positronic/s3) — never redirected here. CACHE_FS="${NEBIUS_CACHE_FS:-computefilesystem-e00f6jyfr5wkawyrab}" +# Docker image tag pulled by the job. `make push-*` only updates `:latest` under +# CI; locally it pushes `:`/`:`. To convert with a branch build: +# `make push-training IMAGE_TAG=` then run with `NEBIUS_IMAGE_TAG=`. +IMAGE_TAG="${NEBIUS_IMAGE_TAG:-latest}" if [ $# -lt 1 ]; then cat >&2 <<'EOF' @@ -83,7 +87,7 @@ CREATE_OUT=$(nebius ai job create \ --parent-id "$PARENT_ID" \ --subnet-id "$SUBNET_ID" \ --name "$JOB_NAME" \ - --image positro/positronic:latest \ + --image "positro/positronic:${IMAGE_TAG}" \ --container-command uv \ --args "$CONVERT_ARGS" \ --platform cpu-e2 \ @@ -149,7 +153,7 @@ nebius ai job create \ --parent-id "$PARENT_ID" \ --subnet-id "$SUBNET_ID" \ --name "$STATS_JOB_NAME" \ - --image positro/openpi:latest \ + --image "positro/openpi:${IMAGE_TAG}" \ --container-command uv \ --args "$STATS_ARGS" \ --platform cpu-e2 \ diff --git a/workflows/nebius/serve.sh b/workflows/nebius/serve.sh index fda3ba318..045f5d1c2 100644 --- a/workflows/nebius/serve.sh +++ b/workflows/nebius/serve.sh @@ -16,6 +16,10 @@ SUBNET_ID="${NEBIUS_SUBNET_ID:-vpcsubnet-e00pk1j1x6hjmr4m92}" # Shared filesystem (RWX) holding the uv / HF / openpi caches across cold starts. # pos3's cache stays on local disk (~/.cache/positronic/s3) — never redirected here. CACHE_FS="${NEBIUS_CACHE_FS:-computefilesystem-e00f6jyfr5wkawyrab}" +# Docker image tag pulled by the endpoint. `make push-*` only updates `:latest` +# under CI; locally it pushes `:`/`:`. To serve a branch build: +# `make push- IMAGE_TAG=` then run with `NEBIUS_IMAGE_TAG=`. +IMAGE_TAG="${NEBIUS_IMAGE_TAG:-latest}" if [ $# -lt 2 ]; then cat >&2 <<'EOF' @@ -55,17 +59,25 @@ NAME="$2" shift 2 case "$VENDOR" in - lerobot_0_3_3) IMAGE="positro/positronic:latest"; EXTRA="--extra lerobot_0_3_3 " ;; - lerobot) IMAGE="positro/positronic:latest"; EXTRA="--extra lerobot " ;; + lerobot_0_3_3) IMAGE="positro/positronic:${IMAGE_TAG}"; EXTRA="--extra lerobot_0_3_3 " ;; + lerobot) IMAGE="positro/positronic:${IMAGE_TAG}"; EXTRA="--extra lerobot " ;; # openpi.server imports `openpi_client` at module top → needs --extra openpi - openpi) IMAGE="positro/openpi:latest"; EXTRA="--extra openpi " ;; - gr00t) IMAGE="positro/gr00t:latest"; EXTRA="" ;; + openpi) IMAGE="positro/openpi:${IMAGE_TAG}"; EXTRA="--extra openpi " ;; + gr00t) IMAGE="positro/gr00t:${IMAGE_TAG}"; EXTRA="" ;; *) echo "Unknown vendor: '$VENDOR'. Supported: lerobot_0_3_3 | lerobot | openpi | gr00t" >&2 exit 1 ;; esac +# Serverless endpoints have no native idle/scale-to-zero, so opt the server into +# self-shutdown (the base default is no timeout). Override the window with +# NEBIUS_IDLE_TIMEOUT_MIN; skip injection if the caller already passed one. +case " $* " in + *" --idle_timeout_min="*|*" --idle_timeout_min "*) ;; + *) set -- "$@" "--idle_timeout_min=${NEBIUS_IDLE_TIMEOUT_MIN:-20}" ;; +esac + SERVER_ARGS="run --python 3.11 ${EXTRA}python -m positronic.vendors.${VENDOR}.server $*" echo "Creating $VENDOR endpoint '$NAME'..." diff --git a/workflows/nebius/train.sh b/workflows/nebius/train.sh index e8400ed9b..eb081dea8 100644 --- a/workflows/nebius/train.sh +++ b/workflows/nebius/train.sh @@ -20,6 +20,10 @@ WANDB_SECRET="${WANDB_SECRET-positronic-serverless-wandb-api-key}" # Shared filesystem (RWX) holding the uv / HF / openpi caches across cold starts. # pos3's cache stays on local disk (~/.cache/positronic/s3) — never redirected here. CACHE_FS="${NEBIUS_CACHE_FS:-computefilesystem-e00f6jyfr5wkawyrab}" +# Docker image tag pulled by the job. `make push-*` only updates `:latest` under +# CI; locally it pushes `:`/`:`. To test a branch build remotely: +# `make push- IMAGE_TAG=` then run with `NEBIUS_IMAGE_TAG=`. +IMAGE_TAG="${NEBIUS_IMAGE_TAG:-latest}" if [ $# -lt 1 ]; then cat >&2 <<'EOF' @@ -42,10 +46,10 @@ VENDOR="$1" shift case "$VENDOR" in - lerobot_0_3_3) IMAGE="positro/positronic:latest"; EXTRA="--extra lerobot_0_3_3 " ;; - lerobot) IMAGE="positro/positronic:latest"; EXTRA="--extra lerobot " ;; - openpi) IMAGE="positro/openpi:latest"; EXTRA="" ;; - gr00t) IMAGE="positro/gr00t:latest"; EXTRA="" ;; + lerobot_0_3_3) IMAGE="positro/positronic:${IMAGE_TAG}"; EXTRA="--extra lerobot_0_3_3 " ;; + lerobot) IMAGE="positro/positronic:${IMAGE_TAG}"; EXTRA="--extra lerobot " ;; + openpi) IMAGE="positro/openpi:${IMAGE_TAG}"; EXTRA="" ;; + gr00t) IMAGE="positro/gr00t:${IMAGE_TAG}"; EXTRA="" ;; *) echo "Unknown vendor: '$VENDOR'. Supported: lerobot_0_3_3 | lerobot | openpi | gr00t" >&2 exit 1