feat(sim): profiled replay

[Mrtroll486]

Summary

Pr 1/3 of issue #1039

This PR adds a runnable, deterministic offline serving simulator to pegainfer-sim and provides the versioned timing-profile and step-worker foundations needed by it.

The offline simulator does not depend on GPUs, model weights, HTTP, or wall-clock time. It replays requests through a single logical event loop and produces identical event ordering and reports for the same scenario, workload, timing profile, and seed.

Background

The existing online SimScheduler is useful for validating the OpenAI/vLLM frontend, HTTP streaming, and metrics. It is not suitable for fleet-level scheduling experiments because its results include HTTP, Tokio, operating-system scheduling, and real-time waiting overhead, and because a large HTTP run is not a simulation of a large serving fleet.

This PR adds an independent logical-clock replay module instead of scaling the online simulator with threads or servers. The online simulator continues to cover protocol and integration behavior; the offline simulator focuses on scheduling, routing, and fleet A/B experiments.

Production Invariant

• The same scenario, workload, timing profile, and random seed produce a byte-identical report.
• Every request reaches exactly one terminal state: Completed or Rejected.
• No step exceeds max_num_seqs or max_batched_tokens.
• No request exceeds max_model_len.
• Increasing the logical worker count does not create one OS thread, Tokio task, or HTTP server per worker.
• The existing online simulator protocol remains compatible when profile mode is not explicitly used.

Main Changes

Offline logical-clock simulator

• Represents logical time as integer microseconds.
• Drives the entire fleet from one BinaryHeap event loop.
• Applies stable same-time event ordering:
  1. Worker step completion;
  2. Request arrival;
  3. Worker scheduling;
  4. Monotonic sequence-number tie-break.
• Implements the Waiting -> Prefill -> Decode -> Completed/Rejected lifecycle.
• Supports max_num_seqs, max_batched_tokens, and max_model_len.
• Implements decode-first continuous batching.
• Supports optional per-request prefill chunking.
• Generates one token per active decode request per step.

Fleet and routing

• Workers contain data state only; they do not own tasks, threads, or servers.
• Supports round-robin routing.
• Supports seeded-random routing using a fixed SplitMix64 implementation.
• Replays do not depend on a third-party RNG implementation or version.

Timing and JSON contracts

Adds separate, strict, versioned JSON contracts for:

• Scenario;
• Workload;
• Timing profile;
• Simulation report.

PR1 provides an extensible tagged timing-model enum with a fixed synthetic model:

• Fixed per-step overhead;
• Per-prefill-token cost;
• Per-decode-token cost.

Reports include:

• SHA-256 digests of the exact input bytes;
• Request placement and terminal outcome;
• Queue time, TTFT, ITL, and E2E latency;
• Token timestamps;
• Request and output-token throughput;
• Worker busy time, utilization, and peak running/waiting counts;
• An optional full simulation trace.

Standalone CLI

Adds pegainfer-sim-replay, which runs without starting the HTTP frontend:

cargo run --release -p pegainfer-sim \
  --bin pegainfer-sim-replay -- \
  --scenario pegainfer-sim/examples/offline/scenario.json \
  --workload pegainfer-sim/examples/offline/workload.json \
  --timing-profile pegainfer-sim/examples/offline/timing-profile.json \
  --output /tmp/pegainfer-sim-report.json

The repository includes runnable scenario, workload, and timing-profile examples.

Online simulator foundations and hardening

The preceding commits in this branch also provide:

• Versioned engine timing profiles with strict validation;
• Reusable step-based worker state;
• Profile-driven online scheduling and timing;
• Server CLI profile loading and strict out-of-domain handling;
• An online profile frontend E2E gate;
• Admission checks before completion allocation, preventing oversized max_tokens requests from causing OOM;
• Correct propagation of fallback-token-id in profile mode;
• Error-returning timing conversion instead of runtime panic;
• A distinct rejection type for whole-prefill step-budget failures.

These facilities provide the foundation for future benchmark-driven calibration, while the PR1 offline timing model remains explicitly synthetic.

Acceptance Evidence

• Hand-calculated workloads produce the expected event timestamps.
• Repeated fixed-seed runs produce byte-identical reports.
• Round-robin and random routing produce different placements and latency results for the same workload.
• Sequence capacity, queueing, context rejection, and one-time terminal-state checks pass.
• Every step stays within the configured sequence and token budgets.
• A u32::MAX output request is rejected before allocation.
• A 1024-worker replay completes in one logical event loop.
• The standalone CLI produces a parseable report.
• Existing online frontend, streaming, and metrics regression tests pass.

Verification

cargo fmt --all -- --check

cargo clippy --release \
  -p pegainfer-sim \
  --all-targets -- -D warnings

NO_PROXY=127.0.0.1,localhost \
no_proxy=127.0.0.1,localhost \
cargo test --release \
  -p pegainfer-sim \
  --lib \
  --test frontend_e2e \
  --test offline_replay

cargo test --release \
  -p pegainfer-frontend \
  --lib

Results:

• pegainfer-sim unit tests: 20/20 passed;
• Online frontend E2E tests: 17/17 passed;
• Offline replay tests: 7/7 passed;
• pegainfer-frontend unit tests: 65/65 passed;
• Formatting, clippy, and diff checks passed.

Non-Goals

This PR does not include:

• Fitting timing profiles from real PegaInfer or vLLM benchmarks;
• Claims about real GPU performance or latency accuracy;
• KV-cache identity, capacity, or eviction;
• KV-aware routing or prefix-cache hit/miss modeling;
• Prefill/decode disaggregation;
• Worker failure, retry, or recovery modeling;
• Heterogeneous fleets;
• Speculative-decoding simulation;
• Automatic search for an optimal routing or scheduling policy.

These capabilities are reserved for follow-up calibration and KV-aware routing work.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all tests pass (see CLAUDE.md).

[Mrtroll486]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a91522033c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Keep legacy TPOT independent of batch width

When the CLI runs without --profile, build_runtime still attaches this generated profile, so concurrent requests share one decode step. Multiplying the duration by decode_reqs makes each token interval N * tpot_ms for a batch of N requests, whereas the previous legacy scheduler advanced every ready request after the same fixed TPOT. Consequently, existing vllm bench workloads with concurrency greater than one report artificially increasing TPOT and latency; retain the legacy scheduler for this mode or make its decode-step duration batch-invariant.

Useful? React with 👍 / 👎.

[Mrtroll486]

Solved.

[chatgpt-codex-connector]

Install a logger before promising fallback warnings

In the pegainfer-sim executable, neither main.rs nor pegainfer-frontend installs a logger for the log facade, so this warning is discarded. Whenever the default WarnAndFallback policy encounters a shape outside the measured grid, the run silently switches to parametric timing and the benchmark operator cannot tell that results are no longer profile-backed; initialize logging or surface this warning through the executable's configured stderr/reporting path.

Useful? React with 👍 / 👎.

[Mrtroll486]

Solved.

[Mrtroll486]

2 review suggestions provided by codex has been fixed, regression test added.

[xiaguan]

The overall online worker implementation looks reasonable. I ran the existing library/HTTP tests and exercised the unchanged head with vllm bench serve: all 64 requests completed at each client concurrency of 1, 4, and 8, with queueing appearing above the configured worker capacity. This used a synthetic timing profile, so it validates the serving path rather than prediction accuracy.

The main design question is the JSON contract. Please post a proposal in a comment for discussion before implementing further schema/model changes. We should first establish the minimum useful inputs and how we will obtain their values.

Some thoughts for that discussion:

• Start with the prediction interface and calibration data. What workload description goes in, and what duration comes out? A forward-pass duration is a useful target; request queueing can then be accounted for by the simulator. Explain how measurements will be collected and how prediction error will be checked before committing to a dense three-dimensional table.
• Consider simpler fitted functions. AIConfigurator's forward-pass model has a regression path using prefill token count for prefill, and decode request count plus total KV tokens for decode. Its prefill regression can fit a piecewise-linear curve. These are useful references, not a requirement to reproduce AIC or assume its accuracy transfers here. Discuss whether a separate mixed-workload model is needed.
• Ablate the required fields. Keeping schema_version makes sense. For every other field, identify the behavior it controls or the ambiguity it prevents. scheduler.policy currently has only one implementation; profile_id could be represented by the file name. Model/hardware/backend identity should remain associated with the calibration dataset, but we should decide which metadata must be in the runtime JSON. My worker-level check found identical plans, outputs, and estimated durations after changing descriptive metadata, and after zeroing fallback coefficients in strict mode; that does not establish that dataset identity is dispensable.
• Justify independent knobs and failure behavior. Does a separate per-request max_chunk_tokens add useful behavior beyond consuming the remaining batch token budget? Removing it changes scheduling, so this deserves a concrete workload example. Likewise, explain whether out-of-domain prediction needs a fallback model in the first version or should report missing coverage.

A small proposed JSON example, a purpose for each retained field, and a calibration/validation outline would be enough to start the discussion. We can agree on that before expanding the implementation.

Separately, there is one reproduced correctness issue on e8ae2e18: in apply_outcome, an abort observed after complete_step can retire the ledger entry and remove request metadata while leaving a nonfinished request in WorkerState. A later step then panics on the closed ledger entry and interrupts other requests. A public-engine test submitting 128 requests and concurrently cancelling 127 reproduced this on the first round in two runs. Temporarily removing the request from the worker in that cleanup path made 30 rounds pass with the uncancelled request completing; restoring the original code reproduced the failure. Please keep worker/metadata/ledger cancellation consistent and cover the surviving-request behavior.