Sync from GitLab (2026-07-12)

.agents/skills/optimize-slurm-topology/references/metrics.md

@@ -17,7 +17,7 @@
 
 Prometheus file-SD targets are written to
 `<run_dir>/prometheus/targets/alpasim.json`. Central file-SD publication uses
-`wizard.telemetry.file_sd_dir` when configured.
+`wizard.prometheus.file_sd_dir` when configured.
 
 ## Query Surface
 
@@ -121,6 +121,10 @@ those raw exporter internals matter.
 | `DCGM_FI_DEV_FB_USED` | `alpasim-dcgm` | `gpu` | GPU framebuffer memory used in MiB. | Above 90% is a hard constraint signal. Reduce cache, concurrency, replicas, or co-location. |
 | `DCGM_FI_DEV_FB_FREE` | `alpasim-dcgm` | `gpu` | Free GPU framebuffer memory in MiB. | Use directly as allocatable memory headroom; add used and reserved memory to derive physical total memory. |
 | `DCGM_FI_DEV_FB_RESERVED` | `alpasim-dcgm` | `gpu` | Driver-reserved GPU framebuffer memory in MiB. | Add to used and free memory to derive total physical framebuffer memory. |
+| `DCGM_FI_PROF_SM_ACTIVE` | `alpasim-dcgm` | `gpu` | Fraction of cycles at least one warp was resident on an SM, averaged over all SMs (0-1). | Real chip usage. High `DCGM_FI_DEV_GPU_UTIL` with low SM active means small or serialized kernels (e.g. co-located processes time-slicing); the GPU has compute headroom. |
+| `DCGM_FI_PROF_SM_OCCUPANCY` | `alpasim-dcgm` | `gpu` | Fraction of resident warps relative to maximum resident warps, averaged over all SMs (0-1). | Distinguishes many-SMs-thin-work from genuinely packed SMs. Low occupancy with high SM active suggests latency-bound kernels. |
+| `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` | `alpasim-dcgm` | `gpu` | Fraction of cycles the tensor core pipe was active (0-1). | Proximity to peak matmul throughput for NN inference workloads; near zero during inference means work is not hitting tensor cores. |
+| `DCGM_FI_PROF_DRAM_ACTIVE` | `alpasim-dcgm` | `gpu` | Fraction of cycles the device memory interface was active (0-1). | High with low SM/tensor activity means memory-bandwidth-bound; co-locating more compute on the GPU will not help. |
 | `node_cpu_seconds_total` | `alpasim-node` | `mode` | Node CPU seconds by mode. | Dashboard converts idle rate to node CPU utilization. If node CPU is saturated, service scaling may not help. |
 | `node_memory_MemAvailable_bytes` | `alpasim-node` | none | Available host memory. | Low available memory indicates node memory pressure. |
 | `node_memory_MemTotal_bytes` | `alpasim-node` | none | Total host memory. | Used with available memory to compute host memory utilization. |

CHANGELOG.md

@@ -2,6 +2,76 @@
 This document lists major updates which change UX and require adaptation.
 It should be sorted by date (more recent on top) and link to MRs which introduce the changes.
 
+## July 2026 public sync updates (12.07.26)
+
+This sync expands the public scene catalog, improves rollout reliability and
+scheduling, and adds new telemetry, evaluation, and Slurm controls.
+
+### Scene catalogs and artifact loading
+
+The public 26.04 catalog is available as the `public_2604` suite, containing
+1,606 scenes after removing known-invalid artifacts. Scene-suite CSV rows now
+pin an exact artifact with `test_suite_id`, `scene_id`, and `uuid`; suite lookup
+no longer selects the newest artifact sharing a scene ID.
+
+Artifacts whose parquet vector map is stored under `fastmap/` are now
+supported. AlpaSim loads it after the preferred `map_data/` and `clipgt/`
+sources, aligns its height to the reconstruction ground mesh, and retains XODR
+as the final fallback.
+
+**Migration**: Add a matching `uuid` column to custom suite CSVs. Every
+`(scene_id, uuid)` pair must exist in the corresponding scene catalog.
+
+### Runtime reliability and scheduling
+
+Failed rollouts are retried up to `runtime.max_rollout_retries` times, with a
+default of two retries. Renderer startup now waits for channel readiness and
+retries known transient NRE initialization failures. Invalid vector maps return
+the structured `ROLLOUT_ERROR_CODE_INVALID_SCENE_MAP` error code.
+
+Scene-affine renderer dispatch now spreads non-affine work across the
+least-loaded renderers and bounds scene replication. New controls include
+`max_renderers_per_scene`, `max_scenes_per_renderer`, and cache refresh timing.
+
+**Migration**: Replace a boolean override such as
+`runtime.scene_affine_dispatch=true` with
+`runtime.scene_affine_dispatch.enabled=true`. Public NRE releases without the
+loaded-scene introspection RPC should leave this disabled.
+
+### Traffic simulation and evaluation
+
+Traffic simulation is explicitly disabled by default through
+`trafficsim=disabled`; select `trafficsim=catk` to enable CATK. CATK now batches
+requests, limits per-request CPU threading, and derives worker counts from the
+selected topology.
+
+Evaluation adds the `open_loop_collision` metric, which checks planned ego
+trajectories against recorded actors over a configurable horizon. Force-GT
+warmup exclusion is now anchored to the actual renderer handover, preventing
+warmup frames from leaking into aggregated metrics. Route generation can be
+disabled with `runtime.simulation_config.route_generator_type=NONE`.
+
+The public Alpamayo 1 driver can derive deterministic per-inference seeds from
+the session seed with `driver.model.force_determinism=true`. The low-speed MPC
+linearization also corrects two erroneous derivative terms.
+
+**Migration**: Custom CATK configs should use
+`catk.loader.prediction_steps` instead of `minimum_future_steps`.
+
+### Telemetry and Slurm execution
+
+Per-process Prometheus exporters can now be scraped by an external Prometheus
+without starting the wizard-managed sidecar. Set
+`wizard.prometheus.start_prometheus=false`; discovery publication and exporter
+lifecycle remain active. Slurm telemetry adds GPU ownership metrics for joining
+AlpaSim processes with DCGM data, and the Grafana dashboard includes the new
+scheduler, GPU, process, and throughput panels.
+
+Slurm deployments can opt into CUDA MPS with `wizard.enable_mps=true` to allow
+co-located GPU services to execute kernels concurrently. This option fails fast
+for non-Slurm run methods. Slurm submission also supports automatic resubmission
+of timed-out jobs.
+
 ## Runtime telemetry with Prometheus and Grafana (30.06.26)
 AlpaSim runs now start Prometheus telemetry by default. The wizard allocates
 metrics ports, starts Prometheus support services, writes scrape configuration,

data/scenes/README.md

@@ -4,11 +4,15 @@ This directory contains public scene and test suite definitions for Alpasim.
 
 ## Files
 
-- `sim_scenes.csv` - Current public scene artifact metadata
-  (uuid, scene_id, NRE version, path, artifact_repository, hf_revision)
-- `sim_suites.csv` - Current public suite-to-scene mappings
-- `sim_scenes_2505.csv` - Legacy public 25.07/25.05 scene artifact metadata
-- `sim_suites_2505.csv` - Legacy public 25.07/25.05 suite-to-scene mappings
+- `sim_scenes.csv` - Public scene artifact metadata for all current releases
+  (26.01 and 26.04)
+- `sim_suites.csv` - Public suite-to-artifact mappings for all current releases
+- `sim_scenes_2505.csv` / `sim_suites_2505.csv` - Legacy public 25.07 release
+  (not loaded by default)
+
+Each suite row includes a readable scene ID and the UUID of the exact artifact.
+By contrast, `scenes.scene_ids=[...]` selects the most recently modified artifact
+for each scene ID.
 
 ### Artifact Repositories
 
@@ -18,10 +22,16 @@ are stored:
 
 ## Available Test Suites
 
-| Suite ID | Scenes | Description |
-|----------|--------|-------------|
-| `public_2601` | 916 | All public NRE scenes from the 26.01 release. Requires sensorsim NRE-GA 26.02 or later. |
-| `public_2507` | 910 | Legacy public NRE scenes from the 25.07 release, hosted on the 25.05 Hugging Face revision. |
+All public scenes are hosted in the
+[nvidia/PhysicalAI-Autonomous-Vehicles-NuRec](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec)
+dataset on Hugging Face; the `hf_revision` column pins each scene to a dataset
+revision.
+
+| Suite ID | Scenes | NRE | HF revision | Description |
+|----------|--------|-----|-------------|-------------|
+| `public_2604` | 1606 | 26.4.x | [26.04](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec/tree/26.04) | Public NRE scenes from the 26.04 release, excluding known-invalid scenes. Mostly new scenarios: only 159 scenes overlap with `public_2601`. |
+| `public_2601` | 913 | 26.1.x | [26.01](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec/tree/26.01) | Public NRE scenes from the 26.01 release, excluding known-invalid scenes. Re-renders scenarios from `public_2507` with newer NRE. Requires sensorsim NRE-GA 26.02 or later. |
+| `public_2507` | 910 | 25.7.x | [25.05](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec/tree/25.05) | Legacy public NRE scenes from the 25.07 release, hosted on the 25.05 Hugging Face revision. |
 
 ## Managing Scenes