Repository navigation
Description
Motivation
Size the KV pool from the memory available after actual runtime initialization, warmup and CUDA Graph capture, while keeping KV addresses stable. Replace fixed per-graph-pool allowances with measured usage and an explicit allowance for runtime transients.
At the inspected baseline, plan_cache() runs a temporary prelude/recurrent/coda forward, measures its peak, and subtracts estimated execution buffers, metadata and graph headroom. StageEngine then allocates the final KV tensors before constructing ModelRunner. Graph budgeting includes a fixed allowance per pool and a separate estimate for retained speculative logits.
Warmup already exists: _capture() executes the path twice before capture. Some asynchronous speculative graphs are prepared at startup; ordinary graph entries are mainly captured lazily. What is missing is a coordinated startup lifecycle that completes the necessary work before final KV sizing.
This is a design proposal based on source inspection; no capacity or performance improvement has been measured.
Proposed change
1. Separate three allocation decisions
- Storage capacity: how much physical GPU memory backs the engine-wide KV pool.
- Request ownership: which committed KV blocks a request currently owns; allocate these incrementally as needed.
- Logical addressing: metadata maps layer, loop and token identity to stored KV.
This RFC changes startup profiling and storage capacity. It does not require reserving all future output blocks for each request or organizing physical storage into preallocated depth planes. It should work with the independently discussed KV page-layout changes in #147.
GPU virtual-memory pages and token blocks are different units. Mapping alignment must be handled by storage; the request allocator only exposes fully backed, usable KV blocks.
2. Change startup order
load model and select backend
-> reserve a stable virtual address range for KV
-> commit enough physical storage for valid warmup inputs
-> initialize the actual runner, buffers, metadata and enabled connectors
-> warm the configured execution paths
-> capture the planned graph entries and exercise replay
-> measure remaining runtime transient requirements
-> drain all startup work and release temporary requests/tensors
-> determine and commit final KV capacity
-> publish the usable block pool and accept requests
The virtual reservation has a bounded upper limit derived from supported capacity and device constraints. It does not consume that amount of physical VRAM.
Warmup uses scratch requests whose accessed KV addresses are mapped and valid. Masked/inactive rows must never access unmapped storage. Scratch state must not enter prefix caches, consume real-request RNG state, or survive into serving.
After final sizing, verify replay and a representative execution within the measured budget before declaring the engine ready.
3. Preserve captured addresses
Replace direct allocation of the final K/V tensors with storage that can add physical mappings inside a stable virtual range.
- Captured pointers, tensor views and strides remain valid across physical commitment.
- Runtime metadata controls valid block indices; reserved virtual capacity is never treated as committed capacity.
- The logical allocator receives only committed blocks after sizing.
- One storage owner manages mappings and tensor lifetimes. Unmap/free only after dependent device work and graph users have retired.
- Account for these allocations explicitly: driver/VMM allocations may not appear in PyTorch allocator statistics.
This requires a GPU storage implementation, not simply reallocating or resizing a PyTorch tensor after capture. GPU VMM operations belong outside the request execution hot path.
4. Warm the paths enabled by the configuration
Warmup should trigger compilation, CUDA library initialization and actual workspace allocation, and establish a representative memory envelope. It is not a full benchmark suite.
Cover the relevant prefill token limits, decode batches and graph buckets, boundary-KV stages, packed/staged prefill, and speculative verification when enabled. Include context lengths that affect attention scratch usage. A single largest batch is insufficient when shapes select different kernels or workspaces.
For asynchronous/multi-stream execution, exercise supported overlap and buffer lifetimes. Measure the concurrent envelope rather than assuming it is twice a serial forward peak.
Use existing per-loop execution and scheduling boundaries. Warmup and capture must preserve mixed admission semantics.
5. Measure persistent usage and incremental transient demand
Record driver-free memory alongside PyTorch allocated/reserved/peak statistics at phase boundaries. Drain participating streams before each boundary measurement and apply a consistent temporary-object/cache cleanup policy.
Distinguish:
- Persistent: weights, execution buffers, metadata, captured graphs and retained outputs.
- Transient: additional physical memory needed during subsequent execution above that persistent baseline.
- Bootstrap KV: mapped scratch storage, tracked separately from model workspace.
After cleanup, persistent allocations already reduce driver-free memory. Do not subtract graph memory again, or count temporary probe KV as runtime workspace. PyTorch caching and reusable workspace must be accounted for consistently so that the same bytes are not reserved twice.
Final KV capacity consists of retained bootstrap mappings plus additional mappings that fit within:
- measured remaining physical memory;
- measured additional runtime transient needs and a safety margin;
- the remaining configured instance-memory budget;
- supported/useful capacity and mapping alignment.
Keep gpu_memory_utilization as a cap; this RFC does not change its default to 1.0. Preserve explicit num_blocks / kv_cache_memory_bytes as capacity overrides with clear startup failure if the requested capacity cannot coexist with execution. Overrides may skip automatic sizing, but must not disable required capture preparation.
Log startup phase costs, graph entry coverage, persistent usage, transient allowance, safety margin, committed KV bytes and usable block count. Do not consume all post-capture free memory: later eager execution still needs working space.
6. Resolve lazy graph growth before removing graph headroom
For the measured path, define a bounded graph-entry plan from the configured limits and complete its capture before final sizing. Unplanned shapes use the existing eager fallback, whose workspace is included in warmup coverage.
Changing graph shape keys or bucket padding requires output/metadata validation, including retained output lifetimes and shared graph-pool replay ordering. A bucketed shape must not alter logical loop scheduling.
If a path still permits serving-time capture, retain an explicit bounded budget for it and report that coverage gap. Do not claim fully measured capacity or remove all graph headroom while future graph allocations remain unaccounted for.
Implementation boundaries and rollout
- Coordinate startup warmup and capture using the existing execution paths; add phase measurements. Retain the current allocation strategy while this is introduced.
- Add stable-address GPU KV storage, integrate final capacity publication, and validate it with capture/replay.
- Enable measured sizing on supported configurations. Retire fixed per-pool graph estimates where capture coverage is complete.
StageEngine coordinates initialization. Execution components own warmup/capture and device buffers; memory planning calculates capacity; storage owns mappings; the logical KV manager owns block leases and request mappings. Avoid a second scheduler or duplicate request-state model.
Select supported behavior automatically. Keep the existing bounded CPU behavior and an identified estimate-based fallback for unsupported VMM/platform/connector combinations. Validate connector registration/export requirements before selecting the storage path; report the fallback reason. Clean up partial mappings and graphs if startup fails.
Alternatives and expected impact
- Keep fixed graph reserves: simple, but continues to rely on estimates that vary with enabled paths and retained outputs.
- Allocate a small ordinary KV tensor, then replace it after capture: invalidates captured pointers; recapturing against a larger tensor reintroduces the capacity/capture dependency.
- Use every byte free after capture: leaves no allowance for later transient execution.
- Commit physical storage per request during serving: unnecessary for this startup proposal; requests can incrementally lease blocks from a committed pool.
Expected tradeoff: more predictable serving startup and potentially better use of VRAM, at the cost of more startup work and GPU storage complexity. The capacity gain is configuration-dependent and may be small or negative where existing estimates were insufficient.
Validation
- Verify captured pointers/views/strides remain valid through commitment and that previously captured graphs replay using newly committed blocks.
- Compare generated tokens/depths and KV contents against the existing path for boundary-KV and recurrent models, eager/graph, supported synchronous/asynchronous paths, mixed scheduling, and enabled speculation.
- After sizing, run representative high-memory prefill, decode and supported overlaps; verify transient headroom is sufficient.
- Exercise partial-startup failure, cleanup, repeated engine creation, block reuse and supported connector lifetimes.
- Compare baseline/candidate startup duration, committed KV capacity, physical peak usage, first-request latency and steady-state performance under matched settings. Report limitations and untested combinations.
Preemption snapshot/recomputation remains tracked separately in #191.
References
- vLLM extensible KV configuration: stable virtual reservation and physical commitment after warmup/capture.
- vLLM memory sizing: measured remaining memory, transient allowance and safety margin.
Activity
kunkunblueberry commentedon Nov 16, 2026
I'd like to contribute to the full runtime KV memory work across the three related PRs. These parts are tightly coupled, so I think it is important to understand and help implement the end-to-end path rather than treat capacity planning or storage initialization in isolation.
In particular, I'm interested in:
I'm happy to work across all three PRs and help keep the interfaces and lifecycle consistent. We can coordinate the concrete split of implementation work as the design develops.
cecilsd commentedon Nov 16, 2026
I'd like to take the startup warmup and CUDA Graph capture orchestration part of step 1. I'll focus on running the existing warmup paths during StageEngine startup, capturing a bounded set of graph entries, and checking replay before the engine accepts requests. The first change will use the current KV allocation strategy.
I'll coordinate with @kunkunblueberry on where warmup and capture hand off to memory measurement. KV sizing, stable-address storage, and final capacity publication are outside my scope.
XinhaoTheo commentedon Nov 16, 2026
I like to take on the VMM storage backend, reserving stable KV addresses and committing physical memory without invalidating captured graphs.
@kunkunblueberry @cecilsd, would that split work for you? I will focus on the storage backend and its lifecycle, coordinating with your capacity planning and warmup/capture work on the interfaces.
bjf-frz commentedon Nov 17, 2026
Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:
Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.
XinhaoTheo commentedon Nov 17, 2026
Agree
feat(kv): add CUDA VMM KV storage with fixed addresses
cecilsd commentedon Nov 18, 2026
@kunkunblueberry @XinhaoTheo I've opened PR #198 for startup warmup and CUDA Graph capture orchestration, based on
bed3400, following the agreed division of work. It retains the current KV allocation strategy and has no dependency on the other implementation branches. The following is a proposed handoff for us to agree on before integration.The worker now initializes/registers its connector, calls the idempotent
StageEngine.prepare_runtime(), and sendsreadyonly after preparation succeeds. Preparation uses the actual serving runners and physically valid scratch KV leases. Scratch requests do not enter the scheduler or publish prefix-cache entries. The successful return and cleanedstartup_reportprovide the current step-1 handoff; they do not calculate or approve a new KV capacity.Measurement boundary and ownership
empty_cacheis called between phases (allocator_cache_policy="retain").num_blocksandkv_bytesdescribe the currently allocated pool, including FP8 scales. They are not measured-sizing results. Measurement should account for KV separately and avoid subtracting already-resident Graph memory again.Proposed integration sequence
Storage needs to preserve captured pointers, tensor views, and strides across commitment, and publish only fully backed usable blocks. Failure cleanup must drain dependent device work before releasing mappings or owners. The current PR does not implement VMM, final measured sizing, or post-commit validation; those belong to the subsequent integration.
Validation
pre-commit run --all-filespassed. The full CPU suite recorded 1236 passed / 35 skipped.Does this drained/cleaned handoff and report semantics fit your measurement/storage work? We should also agree on how
StageEnginecoordinates validation after final storage commitment, and on connector registration requirements for the mapping lifecycle, before fixing the cross-PR interfaces.XinhaoTheo commentedon Nov 18, 2026
@cecilsd Thanks. This sequence fits my #194, and it already meets the storage requirements:
initial_blocksbacks the scratch blocks at the final addresses.commit()does not change the pointers, views or strides, so the graphs do not need a new capture.commit(blocks)returns the number of fully backed blocks. Publish only this number.commit()raises an error and does not changecommitted_blocks.Two constraints for the integration:
storage.tensors(the full reservation). A graph on a view of only the scratch blocks keeps the old dim-0 bound and cannot write to new blocks. Do not run a whole-tensor operation such asfill_on these views, because the blocks aftercommitted_blockshave no physical memory.committed_bytesgives the KV bytes, and the driver free bytes already exclude them.Connector: #194 does not allocate GPUDirect RDMA memory, and NIXL registers the full KV tensors. Thus, with a PD/NIXL connector, the first integration must use
torch.empty.cecilsd commentedon Nov 19, 2026
@XinhaoTheo Thanks for clarifying the constraints. I tested #198 (
c8868f0) together with #194 (594dc63) through a temporary adapter, using the actual storage implementation, serving runners, kernels, and CUDA Graph paths.Results on an RTX 5090:
The checkpoint runs covered synchronous, asynchronous, staged-prefill, and asynchronous speculative execution, each with BF16 and FP8 KV. Comparisons were within the same execution mode and KV dtype, using both greedy and seeded sampling. VMM reserved 512 blocks, prepared with 128 backed blocks, then committed 512 blocks and prioritized the newly backed range.
An isolated negative control also reproduced the sliced-view constraint: capturing an indexed write on a view limited to the bootstrap blocks retained the old indexing bound after commitment. The integration therefore needs the full
storage.tensors, must publish only the fully backed count returned bycommit(), and must avoid operations on unmapped regions.For accounting,
committed_bytesincludes physical mapping alignment, while driver free memory already excludes those mappings.prepare_runtime()returns a preparation snapshot, so validation on newly committed blocks must run separately before final capacity and readiness are published. The current PD/NIXL requirement to retain ordinary storage is also recorded.No change to #198's startup execution was needed for these exercised paths. I added the handoff clarifications in
060203d(documentation only). The temporary adapter validates the current handoff; final measured sizing, capacity publication, and connector integration still need validation in the production integration.kunkunblueberry commentedon Nov 19, 2026
Thanks @cecilsd — the drained/cleaned report semantics fit how I plan to size capacity, with two clarifications I need on coverage. I'll answer the three interface questions in order, then the storage-side ask for @XinhaoTheo.
1. Handoff: warmup/capture -> measurement -> commitment
The report fields you listed are the ones I want, and I read them this way:
driver_free/totalare boundary samples taken after the drain and after scratch release;num_blocks/kv_bytesdescribe the pool that exists now (KV reported separately, including FP8 scales) and are not a sizing result;allocator_cache_policy="retain"means PyTorch reserved bytes are not available capacity. So the planner will use driver-free as the physical budget and will not subtract resident graph memory a second time, and it will not callempty_cachebetween phases.Two coverage questions decide how much of the transient margin is measured rather than estimated:
If either is out of scope for #198, the planner will keep an explicit estimated margin for that part and record it as a coverage gap rather than claim fully measured sizing. That is a reporting change on my side, not a blocker for #198.
2. Post-commit validation ownership
I propose
StageEngineowns the ordering, and orchestration stays where #198 put it:Three properties for that validation step: it runs on the full
storage.tensors(a graph captured on a view of only the bootstrap blocks keeps the old dim-0 bound, as your negative control shows); it uses neither real-request RNG nor prefix-cache publication; and on failure it drains, drops the owners, and reports the phase, so readiness is never signalled. If a connector is active, capacity still comes from the same planner but the pool istorch.empty, so no commit/validation step is needed.3. Connector registration and fallback recording
Agreed with @XinhaoTheo on
torch.emptyfor the first integration with PD/NIXL. I will record the decision inmemory_plan— storage kind, the fallback reason, reserved/committed bytes and the published block count — so the startup log states why the VMM path was not selected instead of failing silently. Connector KV registration can then be addressed separately, since NIXL registers ranges at construction.Storage-side ask (#194)
Given that
close()was removed at594dc63, the engine has no deterministic way to return KV memory on shutdown or repeated engine creation; RFC #192's validation calls for both. Options I see: an explicit release that the caller may only use after the graph and buffer owners are dropped, or a documented "drop all references, then collect" contract thatStageEngine.close()implements and tests verify. Which do you prefer, @XinhaoTheo? My integration consumescommit()/committed_blocks/committed_bytesand enforces the publish bound inKVCacheManager, so either contract works as long as releasing is observable.For context on my side: #193 currently only factors the budget arithmetic out of
plan_cacheand records committed bytes; extending it into measured sizing against this report is the next step, after which the fixed estimates retire only where capture coverage is complete.XinhaoTheo commentedon Nov 19, 2026
Thanks @kunkunblueberry I prefer the second option: drop all references, then collect.
torch.emptypool. Its memory also returns to the allocator only when no reference remains.weakref.refto each pool, then check that it is dead afterStageEngine.close()andgc.collect().test_last_reference_releases_memory_the_torch_allocator_never_counteduses this method.cecilsd commentedon Nov 19, 2026
@kunkunblueberry @XinhaoTheo I clarified the coverage report in #198 (
4a5b198), without expanding the default startup workload.Coverage and transient margins
decode_workloadslists actual completed(batch_size, history_tokens)pairs; history excludes the current input token. Independent maxima must not be combined into a measured workload.Additional checkpoint validation
Using the pinned Ouro-1.4B BF16 checkpoint on an RTX 5090, I compared the previous #198 head with this follow-up at fixed capacity: 768 KV blocks, 16-token pages, four loops, up to three requests, and a 128-token aggregate budget/chunk. Separate workloads used one 3064-token prompt, three 1016-token prompts, and a 2024/504/504-token cohort, with eight subsequent positions reserved per request. These are capacity-feasible boundaries for this test configuration, not the model's 64K limit.
The normal scheduler did not put all three requests into one speculative round in the equal-length test, so the controlled runner check is reported separately. That check observed about 0.93 MiB incremental allocated peak for greedy versus 27.24 MiB for random speculation, supporting the need to retain the random-path margin. These are workload observations, not universal memory bounds. Startup rechecks showed small changes in both directions; no stable startup speedup is claimed.
Commitment and release
The proposed
StageEngineordering fits the handoff: after final commitment, publish only backed blocks to the internal allocator, validate captured replay and representative execution on newly backed blocks, and only then signal readiness.prepare_runtime()remains an idempotent preparation snapshot; calling it again does not validate new mappings. PD/NIXL should retain ordinary storage for the first integration and record the fallback reason.I also tested the reference-lifetime contract with the actual #194 storage in sync and async execution.
StageEngine.close()currently drains work but retains the runner/cache owners. Dropping the engine and remaining owners released the pools; retaining an external tensor view kept its mapping valid until the view was dropped. Thereforeclose(); gc.collect()alone is not sufficient while those references remain. Final integration should test owner teardown after graph users and connector writers have retired, rather than explicitly unmapping live views.feat(memory): size KV capacity from the post-warmup measurement
kunkunblueberry commentedon Nov 19, 2026
The capacity-planning piece is open as #200 (draft), rebased onto
bed3400and green on the CPU workflow:measured_kv_capacity()consumes the last drained phase of the step-1 report. Driver-free memory is the physical budget, so weights, buffers and resident graph memory are not subtracted twice;total - freeonly bounds the instance undergpu_memory_utilization. Transient and safety are explicit inputs, and the reported coverage gaps travel with the plan, so an estimated margin is never labelled measured.KVCacheManagertakes a capacity source and enforcespublished <= committed <= max; publishing re-reads the storage's committed count, so a request cannot receive an unbacked block. Release stays reference-based, per the contract agreed above.Two things it does not do yet, both declared in the PR: mapped storage currently rejects
kv_cache_dtype != "auto"because the pool has no place for FP8 per-vector scales, andStageEnginedoes not consume the report until @cecilsd's #198 and @XinhaoTheo's #194 land. The agreed ordering and thememory_planrecord of storage kind and fallback reason are my next change, stacked on those two.XinhaoTheo commentedon Nov 20, 2026
@cecilsd Agreed. The teardown test must drop the engine and all owners, not only call
close().@kunkunblueberry #194 can hold the FP8 scales. Add one
PoolSpecfor each scale tensor, for examplePoolSpec(block_shape[:-1], torch.float32), the same layout askey_scaleon main.commit()does not write the new blocks. If a scale must have an initial value before the first KV write, write it after each commit, only on the new blocks.