Skip to content

Sync from GitLab (2026-07-01) #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jarcherNV merged 1 commit into from
Jul 1, 2026
Merged

Sync from GitLab (2026-07-01) #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .agents/skills/optimize-slurm-topology/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
---
name: optimize-slurm-topology
description: Optimize AlpaSim Slurm topology throughput using persistent local Prometheus/Grafana telemetry and run artifacts. Use when tuning service GPU placement, replicas_per_container, runtime.nr_workers, endpoint n_concurrent_rollouts, NRE/physics cache sizes, or Slurm experiment batches for full-duration rollout throughput.
---

# Optimize Slurm Topology

Use this workflow to iteratively improve AlpaSim rollout throughput on Slurm. Optimize for full 20s rollout throughput, not startup-only behavior.

## Inputs

When this skill is invoked, the user must specify a full run command as starting
point for the optimization. This command provides a base topology as starting
point, as well as the target configuration (e.g. driver, sceneset, n_rollouts,
cluster, and any other parameters such as number of cameras, simulation
frequency, etc.).

If the user doesn't specify a full run command, ask them!

## Telemetry and Memory Setup

Use one persistent local Prometheus/Grafana instance for the whole experiment.

1. Start local telemetry once with
`src/tools/scripts/start-prometheus-grafana.sh <file-sd-dir-or-ssh-path>
--grafana-port 3003 --prometheus-port 9093`. Use the non-default ports to
avoid conflicts with user-started telemetry stacks.
2. The `<file-sd-dir-or-ssh-path>` argument is provided by the experiment logs.
For example, the default value on IAD is
`<iad-ssh-alias>:/lustre/fsw/portfolios/av/projects/av_alpamayo_reasoning/data/av_alpamayo_sim/.cache/prometheus/file-sd`
3. Keep the local telemetry stack running until all experiment candidates have
been evaluated. Then stop it with `src/tools/scripts/start-prometheus-grafana.sh stop`
4. Create a repo-local experiment record from `references/experiment-record.md`.
Default path:
`docs/experiments/topology-opt-<driver>-<cluster>-<YYYYMMDD>.md`. This file will be your experiment log and memory. It should contain all necessary information to understand why a topology was tried and what the results were.

## Experiment Loop

0. Start from a known topology and run a baseline experiment.
1. If you already have a baseline, start one or multiple candidate experiments
in parallel (at most 3).
2. The experiments have an initial startup time of a couple of about 5 min
before they appear in Prometheus. After that, they start producing rollouts.
However, because all rollouts are initially started simultenously, there's
significant congestion in the first 10-15 minutes. Wait until you can see
this congestion has cleared and the system has reached a steady state. Use
the 5m `seconds_per_rollout` only as an early diagnostic. Once the full-run
`seconds_per_rollout` has stabilized, typically after 30-45 minutes, use it
as the primary optimization target.
3. Reject candidates that OOM or crash. Analyze the reason for failure and avoid
repeating the same mistake. A typical reason is insufficient GPU memory.
4. Once a candidate reaches steady state, analyze it carefully and document (see
`references/metrics.md` and `references/topology-knobs.md`):
* Its stabilized full-run `seconds_per_rollout`, using the 5m value only to
diagnose recent behavior and confirm that the run remains healthy.
* Its bottlenecks, using
`alpasim:rpc_queue_depth_at_start_latest:max` as the primary bottleneck
signal and `alpasim:rpc_queue_depth_at_start_latest:min` to detect workers
that are starving or receiving uneven load.
* Its used and available resources, including per-GPU utilization, memory
consumption, memory pressure, and memory headroom.
* Opportunities for improvement.
5. Skill improvement reflections: Did you learn something new about the system
that was not yet covered in the skill? This can include, for example:
* How to run experiments or query results.
* Which Prometheus queries are useful.
* How the topology knobs affect throughput and memory.
* Are there additional metrics that we should introduce to better understand
the system?
* Any additional scripts that you wrote to help with the experiments that
would be useful to add to the skill.
* Or anything else that you think is useful to remember for future
experiments.
5. Keep two record sections current (see `references/experiment-record.md`).
These records should include the output of both step 4 and step 5, and should
be updated after every candidate experiment:
- a short Markdown progress document (including table) for humans and quick
parsing;
- a more detailed JSON memory block with fixed inputs, runs, metrics, GPU
utilization, GPU memory consumption and headroom, decisions, and links to
run artifacts.
6. Decide on the next topology changes and go back to step 1.
7. You should stop running experiments as soon as you have enough data to
support your reasoning and decision. It is not required to let them run until
the end. However, do not compare or stop a healthy candidate before its
full-run `seconds_per_rollout` has stabilized, normally 30-45 minutes after
rollout production begins. Note that "steady state" can still contain cyclic
behavior.
8. Stop when you can't make progress over multiple iterations or when you don't
believe there is more enough free resources to improve throughput.

## Supporting documents

* Read `references/experiment-record.md` for how to keep a record of the experiment and its candidates.
* Read `references/metrics.md` for current metric names, PromQL queries, and interpretation.
* Read `references/topology-knobs.md` for guidelines on how to change topology and what to expect from each change.

## Final Reporting

At the end of the optimization, re-read the experiment record and summarize the
results in a DETAILED final report, including:

1. Baseline topology and initial speed (primary stabilized full-run
`seconds_per_rollout`, plus 5m `seconds_per_rollout` for recent-behavior
context), including per-GPU utilization and memory consumption/headroom.
2. All tried topologies, their reasoning, expected effect, measured result,
resource usage, and decision.
3. Best topology found, why it won, remaining bottlenecks or constraints, and
how much GPU utilization and memory headroom remains for further tuning.
4. Any advice on how the skill or the instructions can be improved.
5. Link the repo-local experiment record.
13 changes: 13 additions & 0 deletions .agents/skills/optimize-slurm-topology/agents/openai.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
interface:
display_name: "Optimize Slurm Topology"
short_description: "Tune AlpaSim Slurm topology throughput."
default_prompt: |
Optimize an AlpaSim Slurm topology using persistent local telemetry.
Treat `alpasim:simulation_seconds_per_rollout:avg` as the primary
throughput target once it has stabilized, typically after 30-45 minutes.
Use `alpasim:simulation_seconds_per_rollout:rate5m` only as an early and
recent-behavior diagnostic.
Use the Grafana RPC latency distribution heatmaps and recording rules
`alpasim:driver_drive_rpc_duration_seconds_bucket:rate1m` and
`alpasim:nre_render_rpc_duration_seconds_bucket:rate1m` to distinguish
driver/NRE tail spikes from broad service latency shifts.
162 changes: 162 additions & 0 deletions .agents/skills/optimize-slurm-topology/references/experiment-record.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Experiment Record Template

This file contains templates for both a markdown experiment record and a JSON run memory. The experiment record is intended to be human-readable and editable, while the run memory is intended to be machine-readable and used to resume the optimization process.

Generate these files in the repo for each optimization pass. Default path:

`docs/experiments/topology-opt-<driver>-<cluster>-<YYYYMMDD>.md`
`docs/experiments/topology-opt-<driver>-<cluster>-<YYYYMMDD>.json`

## Json Record

The top-level structure of the JSON record is as follows:

```json
{
"experiment_id": "topology-opt-<driver>-<cluster>-<YYYYMMDD>",
"objective": "minimize stabilized full-run seconds_per_rollout",
"fixed_inputs": {
"deploy": "<cluster>",
"driver": "<driver>",
"base_topology": "topology=<name>",
"scenes": {
"scene_ids": null,
"test_suite_id": "public_2601"
},
"simulation_duration_s": 20,
"rollouts_per_scene": 1,
"git_commit": "<sha>",
"remote_checkout": "<path>",
"telemetry": {
"prometheus_url": "http://localhost:<port>",
"grafana_url": "http://localhost:<port>",
"file_sd_source": "<ssh-or-path>"
},
"slurm": {
"account": "<account>",
"partition": "<partition>",
"gpus_per_node": 8,
"walltime": "04:00:00"
}
},
"current_best": {
"topology": null,
"stabilized_seconds_per_rollout_full_run": null,
"decision_reasoning": null
},
"runs": []
}
```

The run entries in the `runs` array have the following structure:

```json
{
"runs": [
{
"name": "<baseline-or-candidate-name>",
"job_id": "<slurm_job_id>",
"run_dir": "<remote-run-dir>",
"topology": "topology=<name>",
"status": "pending|running|accepted|rejected|failed|stopped",
"hypothesis": "<why this run exists>",
"change": {
"hydra_or_yaml_path": "<old -> new>"
},
"stable_metrics": {
"warmup_excluded_s": null,
"completed_rollouts": null,
"seconds_per_rollout_5m": null,
"seconds_per_rollout_full_run": null,
"rollout_duration_p95_s": null,
"step_duration_p95_s": null,
"rpc_queue_depth_at_start_latest_by_service": {
"sensorsim": {"worker_min": null, "worker_max": null},
"driver": {"worker_min": null, "worker_max": null},
"physics": {"worker_min": null, "worker_max": null},
"controller": {"worker_min": null, "worker_max": null}
},
"rpc_duration_p95_s_by_method": {
"run_controller_and_vehicle": null,
"drive": null,
"submit_egomotion_observation": null,
"submit_image_observation": null,
"submit_route": null,
"ground_intersection": null,
"render_rgb": null
},
"runtime_idle_fraction": null,
"resource_usage": {
"process_cpu_utilization_percent_by_group": {
"<process-group>": {"min": null, "mean": null, "max": null}
},
"gpu_by_index": {
"<gpu-index>": {
"topology_services": [],
"utilization_percent": {
"min": null,
"mean": null,
"max": null
},
"memory_used_gb": {
"min": null,
"mean": null,
"max": null
},
"memory_total_gb": null,
"memory_pressure_percent": {
"mean": null,
"max": null
},
"memory_headroom_gb_min": null
}
},
"host_memory": {
"available_gb_min": null,
"total_gb": null
}
},
"bottleneck": {
"service": null,
"evidence": [],
"confidence": "low|medium|high"
},
"telemetry_quality": {
"missing_metrics": [],
"gaps": [],
"notes": null
}
},
"failures": [],
"decision": {
"outcome": "pending|accept|reject",
"reason": null,
"next": null
},
"skill_reflections": [
{
"topic": "metrics|queries|topology_knobs|run_workflow|failure_mode|other",
"learning": null,
"suggested_skill_update": null
}
]
}
]
}
```


## Markdown Record

Keep a markdown record of the experiment in the repo for human readability. It
should contain a separate entry for each candidate. This entry should include
important findings and decisions on what to try next and why. Also include some
key metrics, including rollouts_per_second, queue-depth (min and max), GPU
utilization and memory and which services are on this GPU and if there is more
headroom (and why it wasn't used).

At the top of the file, also include a table.

| Step | Job | Topology | Change | Observation | Decision | Next | Skill update |
|---:|---|---|---|---|---|---|---|
| 1 | `<job_id>` | `topology=<base>` | baseline | `<stabilized full-run and recent 5m seconds_per_rollout; latest RPC queue worker min/max; per-GPU utilization, memory pressure/headroom; bottleneck>` | baseline | `<next knob>` | `<new learning or none>` |
Loading
Loading