nova experiment: compose storm/load into timed workload tests

[nleroy917]

Summary

Add a nova experiment command: a workload-test orchestrator that composes the existing primitives (storm, load, dist or local) into a single timed scenario, links their metrics under one experiment, and visualizes the result as one annotated timeline in Grafana.

The motivating scenario:

1. Drive steady-state read throughput at the cluster.
2. Start a write workload partway through.
3. Stop the write workload.
4. Keep measuring reads to observe recovery.

…with each step independently configurable (qps/wps, filters, indexing on/off, etc.).

The reframe: a timeline of timed, overlapping steps on a shared clock

The scenario above is just steps with a start offset and a duration, sharing one wall clock:

steady_reads (storm)   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■|   start=0    dur=600
write_burst  (load)              |■■■■■■■■|              start=120  dur=180
                                 ↑        ↑
                            writes start  writes stop → recovery = reads
                                          outliving the writes

Key consequences:

• "Stop the write workload" is not a special action — it's write_burst.duration expiring. storm/load already honor duration_s, so steps self-terminate; the orchestrator just sleeps between start offsets. No signal-killing, no stop RPCs. (Decision: duration-based self-terminate for V1. Event-driven hard-stop — "stop when p99 recovers" — is a possible later extension.)
• "Measure recovery" is not a step — it's the read step outlasting the write step.

There are two kinds of timeline entries:

• steps — run a command for a duration (storm / load, dist or local).
• actions — mutate cluster state at an instant (disable/enable indexing, snapshot, drop, recreate). "Indexing off during writes" is an action at the write start + a re-enable action at the write end, not a step.

"Each step configurable (filters, qps, indexing)" is already unlocked by the recent storm filter + qps/wps work — a step is {command, mode, config, overrides, start, duration} where overrides patches the step's config.

Where it lives

Inside supernova, as the 4th top-level verb nova experiment (sibling to embed/load/storm). It needs supernova's config parsing, the metrics schema (for linking), and the CLI surface — a separate tool would duplicate all of that.

The orchestrator is a lightweight control plane, not a new compute layer. It runs on one host (laptop or a small controller): mints IDs, provisions pools, submits jobs, sleeps on the timeline, emits phase events, tears down. Heavy compute stays delegated to the existing -dist machinery. Explicitly not "distributed compute that distributes compute" — the orchestrator never touches a vector.

Orchestration is orthogonal to local/dist

mode is just a field per step, so you mix and match (e.g. distributed reads + local writes):

experiment:
  name: write_contention_recovery
  steps:
    - id: steady_reads
      command: storm
      mode: dist                 # → storm-dist
      start: 0s
      duration: 600s
      config: configs/storm/poshmark.yaml
      overrides: { load: { qps: 75 } }
    - id: write_burst
      command: load
      mode: local                # mix and match
      start: 120s
      duration: 180s
      config: ~/.nova/poshmark.yaml
  actions:
    - { at: 120s, do: disable_indexing }
    - { at: 300s, do: enable_indexing }

Execution model: subprocess via the CLI, uniformly

Drive each step as nova <command>[-dist] <config> subprocesses rather than the in-process SDK.

• The SDK route (import run_storm/run_loader as asyncio tasks) gives clean in-process linking + instant start, but only works for local mode — a dist step is inherently a "dispatch a remote fleet" action and can't run in-process. An SDK orchestrator would need two divergent code paths for local vs dist, fighting the orthogonality.
• Subprocess is one uniform model for both. Uniformity wins.

Solving the dist startup latency (load-dist ≈ 1 min to provision)

Separate provision from start. A dist step's life is provision pool (slow) → ready → submit jobs (fast) → run → teardown. SkyPilot pools already split these. So:

1. Mint experiment_id.
2. Pre-warm all dist pools up front, barrier-wait until every pool is ready. (Decision: pre-warm before t=0 for V1 — precise timeline offsets. Trade-off: all fleets bill for the whole experiment; a per-step pre_warm: false lazy mode is a later option for cost-sensitive steps.)
3. Then start the experiment clock.
4. At each start offset, submit jobs / launch the local subprocess (fast now).
5. Emit a phase event at each transition.
6. Wait for the longest step; tear down.

The minute of provisioning happens before t=0, keeping timeline offsets trustworthy.

Linking metrics: parent experiment_id + the existing events table

Most of this already exists:

• Reuse the NOVA_RUN_ID forwarding pattern: the orchestrator mints experiment_id and forwards NOVA_EXPERIMENT_ID to every child. Each step keeps its own run_id but shares the experiment_id.
• Add one nullable experiment_id column to the runs table (null = standalone run, nothing else changes).
• The events table already exists — the orchestrator writes a phase event at each transition (writes_start, writes_stop, indexing_disabled, …). Grafana renders these as annotation lines on the timeline.

The payoff is the visualization a workload test is actually for: read p99 overlaid with write throughput, vertical annotation lines at writes-start/stop, so you see read latency degrade under write contention and recover afterward. New "Experiment" dashboard with an experiment dropdown overlaying all child runs.

Open trade-offs (to revisit during implementation)

1. Pre-warm cost — pre-warming read + write fleets bills both for the whole run even if writes last 3 min. V1 pre-warms; add per-step pre_warm: false later.
2. Orchestrator location for mode: local steps — a local step measures latency from wherever the orchestrator runs (same laptop→cluster RTT trap as local nova storm). Rule of thumb: local mode is fine for writes (throughput-bound); read-latency steps should be dist + in-region. Needs a loud doc note.
3. Clock skew — the phase annotations use the orchestrator's clock; samples use worker clocks. NTP on EC2 keeps this sub-second, but note it since the whole value is overlaying timelines.
4. Cleanup on failure — a mid-run crash must tear down pools or leak EC2. Needs a finally teardown + a nova experiment teardown <id> escape hatch.
5. Pool reuse — separate pool per dist step (cleaner) vs shared (cheaper). Lean separate.

Phased plan

V1 (first milestone) — minimal, local-only, prove the spine:

• nova experiment <cfg> running a declarative timeline of duration-based steps (mode: local only).
• Two-step scenario (steady reads + write burst) self-terminating on duration.
• experiment_id minted + forwarded (NOVA_EXPERIMENT_ID), runs.experiment_id column added.
• Phase events written to the events table at each transition.

V2 — distributed + pre-warm:

• mode: dist steps; pre-warm all pools before t=0 with a readiness barrier; teardown on completion/failure.
• nova experiment teardown <id>.

V3 — actions + visualization:

• Timed actions (disable/enable indexing, snapshot, recreate).
• "Experiment" Grafana dashboard: experiment dropdown, multi-run overlay, phase annotations from the events table.

Later / maybe:

• Event-driven stops ("stop when p99 recovers below X").
• Per-step pre_warm: false lazy provisioning.
• Pool reuse across steps.

Decisions captured

• Stop semantics: duration-based self-terminate (steps end themselves; recovery = overlapping read step).
• Provisioning: pre-warm all dist pools before t=0 (precise offsets; accept the billing cost in V1).
• Scope: full design above, delivered in the phased milestones (V1 = local-only spine).