Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ No PyPI publication is implied.
gateway; the server has no built-in authentication or TLS.
- [Supported models](#model-support) and [Ouro-Thinking recipe](docs/recipes/ouro_thinking.md)
- [Configuration and compatibility](docs/runtime.md)
- [CUDA startup preparation and its latency tradeoff](docs/runtime.md#startup-execution)
- [Native CUDA build](docs/launching.md#automatic-native-cuda-build)
- [Migration from v0.1.0](CHANGELOG.md)

Expand Down
72 changes: 72 additions & 0 deletions docs/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,78 @@ OMP_NUM_THREADS=1 python -m vllm_rlt.entrypoints.cli --toy --dtype float32 \

For a task-assigned GPU, set `CUDA_VISIBLE_DEVICES` and add `--device cuda --attention-backend triton`. Use `--single-stream` for the single-stream pipeline. `--num-blocks` overrides automatic sizing, or specify `--kv-cache-memory-bytes`; these two overrides are mutually exclusive. CPU auto mode keeps a 256-block default. `StageEngine.memory_plan` records the worker-local sizing decision; it is not an attribute of the public `LLM.engine` client.

## Startup execution

CUDA workers prepare their actual runners after connector initialization and
before sending `ready`. Preparation runs bounded prefill, decode and configured
speculative work with valid scratch KV allocations. It captures the graph shapes
encountered by those workloads and replays representative paths. Scratch requests
never enter the scheduler or publish prefix-cache entries. Streams and submission
queues are drained before request state and KV leases are released; the serving
RNG and request counters are preserved. CPU workers do not execute this workload.

Preparation increases startup time and retains serving workspaces earlier. It
reduces setup work in the first request; steady-state throughput and total cold
process latency require separate measurement for each model and workload.

`StageEngine.prepare_runtime()` is idempotent. Its worker-local `startup_report`
records the covered token lengths and batch sizes, graph keys and replay/fallback
counts, and remaining coverage gaps. The `initialized`, `captured`, `replayed`
and `cleaned` phase snapshots record elapsed time, driver-free memory and PyTorch
allocated/reserved memory, including peaks since the preceding snapshot. The
allocator cache is retained consistently at these boundaries. Times are cumulative
seconds from the start of preparation; memory values are bytes. The first peak
includes earlier engine initialization; subsequent peaks cover the interval since
the preceding snapshot. Driver-free values are boundary samples, not a continuous
device high-water mark. The report is published only after successful cleanup.

Single-request history covers the configured prefill limit and crosses a sliding
window when committed KV permits. Decode batch buckets also cross a KV page.
`decode_workloads` pairs the `batch_size` and `history_tokens` of each completed
recurrent decode traversal. History is the number of preceding tokens, excluding the
decode input itself. Independent maximum batch and history fields do not imply
that their combination was executed. Larger contexts and unreported batch/history
combinations still need an estimated transient allowance, even when the KV pool
could hold them.

Greedy and full-vocabulary top-p sampling use scratch request generators. Block
speculation is warmed with greedy requests; random block speculation retains its
estimated transient allowance and is listed as a coverage gap. Pipeline speculation
supports greedy requests only.
Async multi-stream preparation additionally submits an independent prompt core
and coda before collection, followed by a decode core/coda pair, within the
request/token/KV limits. Its
`concurrent_prepared` and `concurrent` snapshots separate initial shape preparation
from repeated concurrent execution. `concurrent_batches` lists coda row counts;
each is submitted alongside one independent core request. Actual overlap depends
on device execution and host submission cost. These phases drain the serving streams.

This preparation does not replace automatic KV sizing. The report describes the
existing allocated KV pool, including FP8 scale storage when configured, and
sizing still uses the current estimates and reserves. The graph and workspace
allocations remain resident; the KV pool itself is not a transient workspace.
The report is a snapshot of preparation. Repeated `prepare_runtime()` calls return
that snapshot; they do not measure or validate later storage changes. With ordinary
KV tensors, `kv_bytes` is the usable payload including FP8 scales. Future VMM
integration must report physical mapping bytes separately, including alignment
overhead; PyTorch allocator counters do not include those mappings.

Future storage integration must keep the full reserved tensor shape, pointers,
views and strides stable for captured graphs, while exposing only physically
backed blocks to the allocator. Capturing a view limited to the bootstrap blocks
can retain that smaller indexing bound after commitment. Never initialize the
unmapped part of a reservation with whole-tensor operations. Validate replay on
newly committed blocks before publishing readiness; this is a separate step from
the idempotent preparation call. PD/NIXL must retain ordinary storage until the
mapping and registration requirements are supported.

Small explicit pools can reduce warmup coverage. Unseen exact graph shapes retain
bounded lazy capture. Longer contexts and mixed speculative/prompt
overlap retain estimated transient allowances. Pipeline preparation reaches
target verification before releasing its scratch state. The report is a lifecycle
boundary for subsequent memory-planning integration, not a guarantee that every
serving shape or peak allocation has been measured.

## Packed prefill

`ExecutionConfig.packed_prefill` controls packed prefill attention and metadata
Expand Down
7 changes: 7 additions & 0 deletions tests/test_cdb_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -690,11 +690,17 @@ def test_cuda_graph_reserve_reduces_budget_with_fixed_memory_inputs(monkeypatch,
def test_cuda_memory_plan_reclaims_allocator_residue_on_recreation():
import gc

# Earlier tests can leave unreachable engines whose streams retain buffers.
gc.collect()
torch.cuda.synchronize()
torch.cuda.empty_cache()
base = model().to(device="cuda", dtype=torch.bfloat16)
options = dict(
cache_config=CacheConfig(block_size=2, memory_reserve_bytes=0),
scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=3),
attention_backend="triton",
# Isolate planning from per-stream library workspaces created by warmup.
_defer_runtime_preparation=True,
)
first = LLMEngine(base, **options)
budget = first.memory_plan["kv_budget_bytes"]
Expand All @@ -709,5 +715,6 @@ def test_cuda_memory_plan_reclaims_allocator_residue_on_recreation():
second = LLMEngine(base, **options)
assert second.cache_manager.num_blocks == blocks
assert abs(second.memory_plan["kv_budget_bytes"] - budget) < 16 * 1024 * 1024
second.prepare_runtime()
second.add_request("recreated", [1, 2], SamplingParams(max_tokens=2, ignore_eos=True))
assert len(drain(second)["recreated"].token_ids) == 2
4 changes: 3 additions & 1 deletion tests/test_cuda_graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -506,7 +506,9 @@ def __torch_dispatch__(self, func, types, args=(), kwargs=None):

keys = torch.full((5, 2, 4, 3, 9), -1, dtype=dtype, device="cuda")
values = torch.full_like(keys, -2)
proxy = _DeviceCache(SimpleNamespace(key_cache=keys, value_cache=values, backend="triton"))
proxy = _DeviceCache(
SimpleNamespace(key_cache=keys, value_cache=values, key_scale=None, backend="triton")
)
batch = SimpleNamespace(
active_rows=None,
write_mask=None,
Expand Down
2 changes: 2 additions & 0 deletions tests/test_input_preparation.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ def test_loop_preparation_reuses_growing_storage_and_only_saves_final_hidden(
exit_config=ExitConfig("ouro"),
scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=8),
attention_backend=backend,
# Exercise lazy buffer growth independently of startup preallocation.
_defer_runtime_preparation=True,
)
runner = engine.model_runner
cpu_prepare, gpu_prepare = runner.prepare_on_cpu, runner.prepare_on_gpu
Expand Down
Loading
Loading