Alpasim Operations Guide

This guide covers common operational tasks for tuning, optimizing, and troubleshooting Alpasim.

Performance Tuning

How do I change replica counts and GPU distribution?

The number of service replicas and their GPU assignments are configured in deployment configs located in src/wizard/configs/deploy/:

Local workstation: local.yaml
Wizard-managed OmniDreams renderer: managed_flashdreams.yaml
External OmniDreams renderer: external_video_model.yaml

The default renderer path is NuRec-backed. OmniDreams runs use FlashDreams behind the same renderer endpoint; see VIDEO_MODEL.md for the renderer-specific chunking, camera, and deployment constraints.

Understanding the Configuration

Each service has two key parameters:

services:
  renderer:
    replicas_per_container: 4 # Number of service replicas per container
    gpus: [0, 1, 2, 3] # GPUs to create containers on

How it works:

One container per GPU (or one container total if gpus: null)
Each container runs replicas_per_container service instances
Total replicas = nr_gpus * replicas_per_container

Example:

gpus: [0, 1, 2, 3] --> 4 containers (one per GPU)
replicas_per_container: 4 --> 4 replicas per container
Total: 4 * 4 = 16 service replicas

Balancing Replicas and Concurrent Rollouts

For most services, total simulation throughput capacity is determined by:

Total capacity = nr_gpus * replicas_per_container * n_concurrent_rollouts

where n_concurrent_rollouts is the number of rollouts (simulation episodes) each service replica can process simultaneously. This controls how many scenes can be simulated in parallel.

For NRE-backed renderer deployments, this changed in recent NRE releases: each container may run a single NRE process with internal worker concurrency via --max-workers, instead of multiple replicas per container. In that case, think of effective per-container capacity as:

effective renderer capacity = nr_gpus * max_workers * n_concurrent_rollouts

If replicas_per_container=1 and NRE runs with --max-workers=4, one renderer container can still serve four render workers internally.

All services should have similar total capacity to avoid bottlenecks. Example:

services:
  renderer:
    replicas_per_container: 1
    gpus: [0, 1]

  driver:
    replicas_per_container: 8
    gpus: [2, 3]

  controller:
    replicas_per_container: 16
    gpus: null # CPU-only: 1 container

runtime:
  endpoints:
    renderer:
      n_concurrent_rollouts: 4 # with --max-workers=4: 2 GPUs * 4 workers * 4 concurrent = 32

    driver:
      n_concurrent_rollouts: 2 # 2 GPUs * 8 replicas * 2 concurrent = 32

    controller:
      n_concurrent_rollouts: 2 # 1 CPU * 16 replicas * 2 concurrent = 32

For NRE-backed renderer deployments, tune these together:

services.renderer.replicas_per_container
the NRE container's internal --max-workers
runtime.endpoints.renderer.n_concurrent_rollouts

Recent OSS deploy configs prefer replicas_per_container: 1 plus higher --max-workers.

For OmniDreams/FlashDreams deployments, start with runtime.endpoints.renderer.n_concurrent_rollouts=1 and the +chunking=8frame preset from VIDEO_MODEL.md. Scale only after the renderer server, driver cadence, and GPU memory headroom are known to be stable.

How do I change the model?

By default, the VaVam driver and model are used. The model weights are downloaded using data/download_vavam_assets.sh and stored in data/vavam-driver/.

Using a Different Model

To use a custom model, mount a custom vavam-driver directory:

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    defines.vavam_driver=/path/to/custom/vavam-driver

Default location: data/vavam-driver/ (in repository root) The wizard mounts defines.vavam_driver as /mnt/vavam_driver in the container and the driver loads the model from that path.

Using a Different Driver/Inference Code

To use a custom driver container image:

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    services.driver.image=<your-registry>/<your-driver-image>:<tag>

Your custom image must expose a gRPC endpoint compatible with the driver service interface (see protocol buffer definitions).

For development of driver code within this repository, changes to src/driver/ are automatically mounted into containers at runtime (see Code Changes in TUTORIAL.md).

How do I change inference frequency?

Changing inference frequency is complex and requires coordinating multiple timing parameters.

Understanding the Parameters

The simulator has multiple synchronized "clocks":

Driver inference (control_timestep_us) - How often the model makes decisions
Camera frames (frame_interval_us) - How often cameras capture images
Pose reporting (pose_reporting_interval_us) - How often intermediate poses are reported between control steps (0 = at control_timestep_us rate, the default)
Simulation start - The recorded egomotion start timestamp by default

For correct operation, these must be mathematically aligned.

Step-by-Step Walkthrough

Scenario 1: Simple frequency change (matching camera and inference rates)

To change to 5Hz inference (200ms between decisions):

Set inference frequency (control_timestep_us):

runtime.simulation_config.control_timestep_us=200000  # 200ms = 5Hz

Pose reporting (pose_reporting_interval_us): defaults to 0, which falls back to control_timestep_us. No explicit setting needed unless you want intermediate pose reports.

Match camera frame rate (VaVam default has 1 camera):

runtime.simulation_config.cameras.0.frame_interval_us=200000

For configs with 2 cameras (e.g., +cameras=2cam), also set:

runtime.simulation_config.cameras.1.frame_interval_us=200000

Full command:

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.simulation_config.control_timestep_us=200000 \
    runtime.simulation_config.cameras.0.frame_interval_us=200000

Note: Add cameras.1.frame_interval_us=200000 if using 2-camera configs. pose_reporting_interval_us defaults to 0, falling back to control_timestep_us.

Scenario 2: High-rate camera with lower inference rate

To use 30Hz cameras (33.3ms) but 10Hz inference (100ms):

Camera captures at 30Hz: frame_interval_us=33334 (33.3ms)
Inference runs at 10Hz: control_timestep_us=100002 (must be 3 × 33334)
Subsample frames: driver.inference.Cframes_subsample=3 (use every 3rd frame)
Pose reporting: pose_reporting_interval_us defaults to 0 (falls back to control_timestep_us)

Full command (based on exp/sim/20s_at_30Hz.yaml):

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.simulation_config.control_timestep_us=100002 \
    runtime.simulation_config.cameras.0.frame_interval_us=33334 \
    ++driver.inference.Cframes_subsample=3

Note: Add cameras.1.frame_interval_us=33334 if using 2-camera configs.

Validation

The assert_zero_decision_delay flag (enabled by default in OSS configs) validates timing synchronization at runtime. It checks that:

Camera frames complete exactly at decision time
Pose updates complete exactly at decision time

If misconfigured, the simulator will error with messages like:

Camera camera_front_wide_120fov out of sync with planning.
Last started frame finishes at X which is Y microseconds away from decision time Z.

What it does: At each control step, before calling the driver, the runtime verifies that the last camera frame and pose update completed exactly at now_us (zero delay). This ensures the model receives perfectly synchronized data.

Testing your configuration:

# The flag is true by default, but you can explicitly set it:
runtime.simulation_config.assert_zero_decision_delay=true

Common Frequencies

Based on actual config files in src/wizard/configs/:

Frequency	`control_timestep_us`	Notes
2Hz	500000 (500ms)	VaVam default
5Hz	200000 (200ms)	Example config
10Hz	100000 (100ms)	Base config default
30Hz	33334 (33.3ms)	High frequency

pose_reporting_interval_us defaults to 0 for all frequencies (falls back to control_timestep_us).

Simulation starts from the first USDZ timestamp with valid recorded camera and non-ego actor data. The first policy step is aligned to the first completed camera frames.

See also:

src/runtime/README.md - Zero delay mode for synchronization requirements
src/wizard/configs/driver/vavam_configs.yaml for a 2Hz example

Viewing Results and Metrics

Where are simulation results stored?

After a run completes, results are in wizard.log_dir (e.g., runs/{RUN_DIR}/):

asl/ - Simulation logs (.asl files for debugging)
eval/ - Per-rollout driving quality metrics (metrics_unprocessed.parquet) and videos
aggregate/ - Aggregated results across all rollouts:
- metrics_results.txt - Formatted table of driving scores
- metrics_results.png - Visual summary of driving quality metrics
- metrics_unprocessed.parquet - Combined metrics from all rollouts
- videos/ - Organized by violation types
prometheus/ - Performance profiling data:
- data/ - local Prometheus TSDB for the run
- targets/alpasim.json - generated Prometheus file-SD targets
metrics_plot.png - Performance visualization (CPU/GPU/RPC metrics)
txt-logs/ - Service logs for debugging
wizard-config.yaml - Resolved configuration used for this run

See TUTORIAL.md - Results Structure for detailed breakdown.

Understanding Driving Quality Metrics

The simulation evaluates driving quality across multiple dimensions. Results are in aggregate/metrics_results.txt and visualized in aggregate/metrics_results.png.

Key Metrics

Safety Metrics (binary: 0 = pass, 1 = fail):

collision_at_fault: Driver caused a collision (front/lateral impact)
collision_rear: Rear-end collision (not at fault)
offroad: Vehicle drove off the road

Performance Metrics (continuous):

dist_to_gt_trajectory: Maximum distance from ground truth path (meters)
- Lower is better; indicates how closely the driver follows expected routes
- Aggregated using MAX over time (worst deviation during the drive)
duration_frac_20s: Fraction of 20s drive completed before any failure
- 1.0 = completed full 20s without issues
- <1.0 = failed early (collision, off-road, or excessive deviation)

Distance Between Incidents:

avg_dist_between_incidents: Average km traveled per incident (collision or offroad)
- Higher is better; measures safety over distance
avg_dist_between_incidents_at_fault: Average km traveled per at-fault incident
- Higher is better; excludes rear-end collisions not caused by the driver

Interpreting the Results

The aggregate/metrics_results.txt file shows statistics (mean, std, min, max, quantiles) for each metric across all rollouts. For example:

collision_at_fault: mean=0.05 → 5% of rollouts had at-fault collisions
dist_to_gt_trajectory: mean=2.3 → Average 2.3m deviation from GT path
duration_frac_20s: mean=0.95 → Average 95% of 20s completed

Videos in aggregate/videos/violations/ are organized by failure type for easy review of problematic scenarios.

How do I view performance metrics?

Metrics Plot (Automatically Generated)

After each simulation run, Alpasim automatically generates a comprehensive performance visualization:

Location: runs/{RUN_DIR}/metrics_plot.png

This 3×3 grid plot includes:

Row 1: RPC Performance

RPC Duration histogram - Total time from call start to coroutine resumption
RPC Blocking histogram - Event loop scheduler delay (time from gRPC I/O completion to coroutine resumption)
RPC Queue Depth histogram - Service saturation levels

Row 2: Simulation Timing

Rollout Duration histogram - Total time per rollout
Step Duration histogram - Time per simulation step
Run summary - Shows runtime idle fraction and seconds per rollout

Row 3: Resource Utilization

CPU Utilization boxplots - Per-service CPU usage
GPU Utilization boxplots - GPU compute usage
GPU Memory boxplots - Memory usage with capacity line

Summary header shows:

Async worker idle percentage - How much time runtime spent idle
Sim seconds per rollout - Wallclock time per simulation

Interpreting the Metrics Plot

Identifying Bottlenecks:

High queue depth on a service → Increase replicas_per_container or n_concurrent_rollouts
High RPC duration → Service is slow, consider optimization or scaling
Low GPU utilization (<50%) → Underutilized, can increase load
High GPU utilization (>90%) → May be saturated, check for throttling
Unbalanced service config → Total capacity should match across all services

Performance Indicators:

Low idle percentage (<20%) → Runtime is busy, good utilization
High idle percentage (>80%) → Lots of waiting, check for bottlenecks
Consistent rollout times → Good stability
Wide rollout time variance → Investigate outliers in logs

Simulation Configuration

How do I start the runtime as a server?

Use wizard.run_mode=SERVER to start the normal backing services and keep the runtime alive as a gRPC daemon:

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.run_mode=SERVER \
    wizard.log_dir=runs/{DATETIME}

The wizard writes the resolved client endpoint to generated-runtime-server.yaml in the run directory:

host: localhost
port: 6005

This file is generated before Docker Compose has necessarily finished starting the runtime daemon. Clients should treat host and port as discovery metadata and poll until the runtime gRPC port accepts connections.

After the port accepts connections, clients can call RuntimeService.get_runtime_info() to discover the server's maximum supported concurrent rollouts, available scenes and lightweight scene metadata, service capacity by backing service, and service versions before submitting RuntimeService.simulate() requests.

By default, the runtime server port is allocated from wizard.baseport after the backing service ports. Pin it with wizard.runtime_server_port=<port> when client scripts or firewall rules need a stable port.

Static external drivers use driver_source=external_static with wizard.external_services.driver; driver=manual provides a default localhost:6789 address for the manual-driver workflow. Server clients can also use driver_source=external_dynamic and provide drivers per request with SimulationRequest.available_drivers and n_concurrent_per_driver; those request addresses override the configured driver pool for that RPC only.

To stop the server, call RuntimeService.shut_down(). In managed deployments, Docker Compose or Slurm tears down backing services after the runtime exits.

When running a generated docker-compose.yaml manually for one-shot simulations, start Compose with --exit-code-from runtime-0, for example:

docker compose -f docker-compose.yaml up --exit-code-from runtime-0

The runtime container exits when the simulation is complete, while the backing services are long-running servers. Without --exit-code-from runtime-0, Docker Compose can continue waiting after runtime-0 exits successfully.

How do I enable/disable specific services?

Use runtime.endpoints.<service>.skip to disable services:

# Disable traffic simulation
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.endpoints.trafficsim.skip=true

# Disable physics (log replay mode)
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.endpoints.physics.skip=true \
    runtime.simulation_config.physics_update_mode=NONE \
    runtime.simulation_config.force_gt_duration_us=20000000

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Alpasim Operations Guide

Performance Tuning

How do I change replica counts and GPU distribution?

Understanding the Configuration

Balancing Replicas and Concurrent Rollouts

How do I change the model?

Using a Different Model

Using a Different Driver/Inference Code

How do I change inference frequency?

Understanding the Parameters

Step-by-Step Walkthrough

Validation

Common Frequencies

Viewing Results and Metrics

Where are simulation results stored?

Understanding Driving Quality Metrics

Key Metrics

Interpreting the Results

How do I view performance metrics?

Metrics Plot (Automatically Generated)

Interpreting the Metrics Plot

Simulation Configuration

How do I start the runtime as a server?

How do I enable/disable specific services?

Uh oh!

FilesExpand file tree

OPERATIONS.md

Latest commit

History

OPERATIONS.md

File metadata and controls

Alpasim Operations Guide

Performance Tuning

How do I change replica counts and GPU distribution?

Understanding the Configuration

Balancing Replicas and Concurrent Rollouts

How do I change the model?

Using a Different Model

Using a Different Driver/Inference Code

How do I change inference frequency?

Understanding the Parameters

Step-by-Step Walkthrough

Validation

Common Frequencies

Viewing Results and Metrics

Where are simulation results stored?

Understanding Driving Quality Metrics

Key Metrics

Interpreting the Results

How do I view performance metrics?

Metrics Plot (Automatically Generated)

Interpreting the Metrics Plot

Simulation Configuration

How do I start the runtime as a server?

How do I enable/disable specific services?