Skip to content

[RFC]: Size KV storage after runtime warmup and CUDA Graph capture #192

Open

Description

@bjf-frz

Motivation

Size the KV pool from the memory available after actual runtime initialization, warmup and CUDA Graph capture, while keeping KV addresses stable. Replace fixed per-graph-pool allowances with measured usage and an explicit allowance for runtime transients.

At the inspected baseline, plan_cache() runs a temporary prelude/recurrent/coda forward, measures its peak, and subtracts estimated execution buffers, metadata and graph headroom. StageEngine then allocates the final KV tensors before constructing ModelRunner. Graph budgeting includes a fixed allowance per pool and a separate estimate for retained speculative logits.

Warmup already exists: _capture() executes the path twice before capture. Some asynchronous speculative graphs are prepared at startup; ordinary graph entries are mainly captured lazily. What is missing is a coordinated startup lifecycle that completes the necessary work before final KV sizing.

This is a design proposal based on source inspection; no capacity or performance improvement has been measured.

Proposed change

1. Separate three allocation decisions

  • Storage capacity: how much physical GPU memory backs the engine-wide KV pool.
  • Request ownership: which committed KV blocks a request currently owns; allocate these incrementally as needed.
  • Logical addressing: metadata maps layer, loop and token identity to stored KV.

This RFC changes startup profiling and storage capacity. It does not require reserving all future output blocks for each request or organizing physical storage into preallocated depth planes. It should work with the independently discussed KV page-layout changes in #147.

GPU virtual-memory pages and token blocks are different units. Mapping alignment must be handled by storage; the request allocator only exposes fully backed, usable KV blocks.

2. Change startup order

load model and select backend
  -> reserve a stable virtual address range for KV
  -> commit enough physical storage for valid warmup inputs
  -> initialize the actual runner, buffers, metadata and enabled connectors
  -> warm the configured execution paths
  -> capture the planned graph entries and exercise replay
  -> measure remaining runtime transient requirements
  -> drain all startup work and release temporary requests/tensors
  -> determine and commit final KV capacity
  -> publish the usable block pool and accept requests

The virtual reservation has a bounded upper limit derived from supported capacity and device constraints. It does not consume that amount of physical VRAM.

Warmup uses scratch requests whose accessed KV addresses are mapped and valid. Masked/inactive rows must never access unmapped storage. Scratch state must not enter prefix caches, consume real-request RNG state, or survive into serving.

After final sizing, verify replay and a representative execution within the measured budget before declaring the engine ready.

3. Preserve captured addresses

Replace direct allocation of the final K/V tensors with storage that can add physical mappings inside a stable virtual range.

  • Captured pointers, tensor views and strides remain valid across physical commitment.
  • Runtime metadata controls valid block indices; reserved virtual capacity is never treated as committed capacity.
  • The logical allocator receives only committed blocks after sizing.
  • One storage owner manages mappings and tensor lifetimes. Unmap/free only after dependent device work and graph users have retired.
  • Account for these allocations explicitly: driver/VMM allocations may not appear in PyTorch allocator statistics.

This requires a GPU storage implementation, not simply reallocating or resizing a PyTorch tensor after capture. GPU VMM operations belong outside the request execution hot path.

4. Warm the paths enabled by the configuration

Warmup should trigger compilation, CUDA library initialization and actual workspace allocation, and establish a representative memory envelope. It is not a full benchmark suite.

Cover the relevant prefill token limits, decode batches and graph buckets, boundary-KV stages, packed/staged prefill, and speculative verification when enabled. Include context lengths that affect attention scratch usage. A single largest batch is insufficient when shapes select different kernels or workspaces.

For asynchronous/multi-stream execution, exercise supported overlap and buffer lifetimes. Measure the concurrent envelope rather than assuming it is twice a serial forward peak.

Use existing per-loop execution and scheduling boundaries. Warmup and capture must preserve mixed admission semantics.

5. Measure persistent usage and incremental transient demand

Record driver-free memory alongside PyTorch allocated/reserved/peak statistics at phase boundaries. Drain participating streams before each boundary measurement and apply a consistent temporary-object/cache cleanup policy.

Distinguish:

  • Persistent: weights, execution buffers, metadata, captured graphs and retained outputs.
  • Transient: additional physical memory needed during subsequent execution above that persistent baseline.
  • Bootstrap KV: mapped scratch storage, tracked separately from model workspace.

After cleanup, persistent allocations already reduce driver-free memory. Do not subtract graph memory again, or count temporary probe KV as runtime workspace. PyTorch caching and reusable workspace must be accounted for consistently so that the same bytes are not reserved twice.

Final KV capacity consists of retained bootstrap mappings plus additional mappings that fit within:

  1. measured remaining physical memory;
  2. measured additional runtime transient needs and a safety margin;
  3. the remaining configured instance-memory budget;
  4. supported/useful capacity and mapping alignment.

Keep gpu_memory_utilization as a cap; this RFC does not change its default to 1.0. Preserve explicit num_blocks / kv_cache_memory_bytes as capacity overrides with clear startup failure if the requested capacity cannot coexist with execution. Overrides may skip automatic sizing, but must not disable required capture preparation.

Log startup phase costs, graph entry coverage, persistent usage, transient allowance, safety margin, committed KV bytes and usable block count. Do not consume all post-capture free memory: later eager execution still needs working space.

6. Resolve lazy graph growth before removing graph headroom

For the measured path, define a bounded graph-entry plan from the configured limits and complete its capture before final sizing. Unplanned shapes use the existing eager fallback, whose workspace is included in warmup coverage.

Changing graph shape keys or bucket padding requires output/metadata validation, including retained output lifetimes and shared graph-pool replay ordering. A bucketed shape must not alter logical loop scheduling.

If a path still permits serving-time capture, retain an explicit bounded budget for it and report that coverage gap. Do not claim fully measured capacity or remove all graph headroom while future graph allocations remain unaccounted for.

Implementation boundaries and rollout

  1. Coordinate startup warmup and capture using the existing execution paths; add phase measurements. Retain the current allocation strategy while this is introduced.
  2. Add stable-address GPU KV storage, integrate final capacity publication, and validate it with capture/replay.
  3. Enable measured sizing on supported configurations. Retire fixed per-pool graph estimates where capture coverage is complete.

StageEngine coordinates initialization. Execution components own warmup/capture and device buffers; memory planning calculates capacity; storage owns mappings; the logical KV manager owns block leases and request mappings. Avoid a second scheduler or duplicate request-state model.

Select supported behavior automatically. Keep the existing bounded CPU behavior and an identified estimate-based fallback for unsupported VMM/platform/connector combinations. Validate connector registration/export requirements before selecting the storage path; report the fallback reason. Clean up partial mappings and graphs if startup fails.

Alternatives and expected impact

  • Keep fixed graph reserves: simple, but continues to rely on estimates that vary with enabled paths and retained outputs.
  • Allocate a small ordinary KV tensor, then replace it after capture: invalidates captured pointers; recapturing against a larger tensor reintroduces the capacity/capture dependency.
  • Use every byte free after capture: leaves no allowance for later transient execution.
  • Commit physical storage per request during serving: unnecessary for this startup proposal; requests can incrementally lease blocks from a committed pool.

Expected tradeoff: more predictable serving startup and potentially better use of VRAM, at the cost of more startup work and GPU storage complexity. The capacity gain is configuration-dependent and may be small or negative where existing estimates were insufficient.

Validation

  • Verify captured pointers/views/strides remain valid through commitment and that previously captured graphs replay using newly committed blocks.
  • Compare generated tokens/depths and KV contents against the existing path for boundary-KV and recurrent models, eager/graph, supported synchronous/asynchronous paths, mixed scheduling, and enabled speculation.
  • After sizing, run representative high-memory prefill, decode and supported overlaps; verify transient headroom is sufficient.
  • Exercise partial-startup failure, cleanup, repeated engine creation, block reuse and supported connector lifetimes.
  • Compare baseline/candidate startup duration, committed KV capacity, physical peak usage, first-request latency and steady-state performance under matched settings. Report limitations and untested combinations.

Preemption snapshot/recomputation remains tracked separately in #191.

References

Activity

  1. added
    RFCRequests for comments on major architectural changes or design choices
    on Nov 16, 2026
  2. kunkunblueberry commented on Nov 16, 2026

    @kunkunblueberry
    Contributor

    I'd like to contribute to the full runtime KV memory work across the three related PRs. These parts are tightly coupled, so I think it is important to understand and help implement the end-to-end path rather than treat capacity planning or storage initialization in isolation.

    In particular, I'm interested in:

    • Runtime memory measurement after warmup / CUDA Graph capture.
    • KV cache capacity planning based on the measured runtime memory budget.
    • KV storage reservation / initialization and its integration with KVCacheManager and ModelRunner.

    I'm happy to work across all three PRs and help keep the interfaces and lifecycle consistent. We can coordinate the concrete split of implementation work as the design develops.

  3. cecilsd commented on Nov 16, 2026

    @cecilsd
    Contributor

    I'd like to take the startup warmup and CUDA Graph capture orchestration part of step 1. I'll focus on running the existing warmup paths during StageEngine startup, capturing a bounded set of graph entries, and checking replay before the engine accepts requests. The first change will use the current KV allocation strategy.

    I'll coordinate with @kunkunblueberry on where warmup and capture hand off to memory measurement. KV sizing, stable-address storage, and final capacity publication are outside my scope.

  4. XinhaoTheo commented on Nov 16, 2026

    @XinhaoTheo

    I like to take on the VMM storage backend, reserving stable KV addresses and committing physical memory without invalidating captured graphs.

    @kunkunblueberry @cecilsd, would that split work for you? I will focus on the storage backend and its lifecycle, coordinating with your capacity planning and warmup/capture work on the interfaces.

  5. bjf-frz commented on Nov 17, 2026

    @bjf-frz
    CollaboratorAuthor

    Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:

    • @cecilsd: startup warmup and CUDA Graph capture orchestration (step 1), starting with the current KV allocation strategy.
    • @XinhaoTheo: VMM storage backend: stable KV address reservation, physical commitment without invalidating captured graphs, and its lifecycle.
    • @kunkunblueberry: runtime memory measurement after warmup/capture, KV capacity planning from the measured budget, and integration with KVCacheManager / ModelRunner, keeping the interfaces and lifecycle consistent across the three parts.

    Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.

  6. XinhaoTheo commented on Nov 17, 2026

    @XinhaoTheo

    Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:

    • @cecilsd: startup warmup and CUDA Graph capture orchestration (step 1), starting with the current KV allocation strategy.
    • @XinhaoTheo: VMM storage backend: stable KV address reservation, physical commitment without invalidating captured graphs, and its lifecycle.
    • @kunkunblueberry: runtime memory measurement after warmup/capture, KV capacity planning from the measured budget, and integration with KVCacheManager / ModelRunner, keeping the interfaces and lifecycle consistent across the three parts.

    Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.

    Agree

  7. cecilsd commented on Nov 18, 2026

    @cecilsd
    Contributor

    @kunkunblueberry @XinhaoTheo I've opened PR #198 for startup warmup and CUDA Graph capture orchestration, based on bed3400, following the agreed division of work. It retains the current KV allocation strategy and has no dependency on the other implementation branches. The following is a proposed handoff for us to agree on before integration.

    The worker now initializes/registers its connector, calls the idempotent StageEngine.prepare_runtime(), and sends ready only after preparation succeeds. Preparation uses the actual serving runners and physically valid scratch KV leases. Scratch requests do not enter the scheduler or publish prefix-cache entries. The successful return and cleaned startup_report provide the current step-1 handoff; they do not calculate or approve a new KV capacity.

    Measurement boundary and ownership

    • Participating streams and submission queues are drained before each phase measurement and before scratch ownership is released.
    • The report includes cumulative elapsed seconds, driver free/total bytes, PyTorch allocated/reserved/peak bytes, bounded workload coverage, and Graph keys/replay/fallback information. The initial peak includes earlier initialization; subsequent peaks start at the preceding phase reset. Driver-free values are boundary samples, not a continuous physical high-water mark.
    • Temporary requests, KV leases, request-local RNG, and async/speculative state are cleaned up; serving RNG and business counters are preserved. Graphs, execution buffers, and library workspaces remain alive. No empty_cache is called between phases (allocator_cache_policy="retain").
    • num_blocks and kv_bytes describe the currently allocated pool, including FP8 scales. They are not measured-sizing results. Measurement should account for KV separately and avoid subtracting already-resident Graph memory again.
    • Warmup/capture coverage is bounded and gaps are reported. Existing estimated sizing and reserves for future lazy capture remain in place.

    Proposed integration sequence

    physically backed scratch KV at stable addresses
      -> warmup / capture / representative replay
      -> drain and release temporary state
      -> memory measurement and capacity planning
      -> final physical storage commitment
      -> replay and representative execution validation
      -> publish usable blocks and readiness
    

    Storage needs to preserve captured pointers, tensor views, and strides across commitment, and publish only fully backed usable blocks. Failure cleanup must drain dependent device work before releasing mappings or owners. The current PR does not implement VMM, final measured sizing, or post-commit validation; those belong to the subsequent integration.

    Validation

    • Linux pre-commit run --all-files passed. The full CPU suite recorded 1236 passed / 35 skipped.
    • RTX 5090 startup/fixture coverage recorded 80 passed; other applicable regressions recorded 307 passed / 2 skipped. Dual-GPU skips and unsupported FA4 SM120 cases are disclosed in the PR.
    • Full Ouro-1.4B BF16 weights were checked across five execution modes and BF16/FP8 KV. Within each KV dtype, all 150 matched request cases preserved token IDs and exit depths; observed ordinary-sampler logits and valid KV content, including FP8 scales, matched exactly.
    • First-request TTFT fell from about 728 to 141 ms synchronously and 750 to 141 ms asynchronously in the documented workload. Startup increased to about 2.04 s / 3.18 s, and startup plus first completion became slower. No sustained throughput improvement is established. Independent ops-only/full profiles and measurement details are in the PR.

    Does this drained/cleaned handoff and report semantics fit your measurement/storage work? We should also agree on how StageEngine coordinates validation after final storage commitment, and on connector registration requirements for the mapping lifecycle, before fixing the cross-PR interfaces.

  8. XinhaoTheo commented on Nov 18, 2026

    @XinhaoTheo

    @cecilsd Thanks. This sequence fits my #194, and it already meets the storage requirements:

    • initial_blocks backs the scratch blocks at the final addresses.
    • commit() does not change the pointers, views or strides, so the graphs do not need a new capture.
    • commit(blocks) returns the number of fully backed blocks. Publish only this number.
    • The release synchronizes the device before the unmap. A failed commit() raises an error and does not change committed_blocks.

    Two constraints for the integration:

    • Graphs must capture on storage.tensors (the full reservation). A graph on a view of only the scratch blocks keeps the old dim-0 bound and cannot write to new blocks. Do not run a whole-tensor operation such as fill_ on these views, because the blocks after committed_blocks have no physical memory.
    • The PyTorch counters do not include this memory. For the measurement, committed_bytes gives the KV bytes, and the driver free bytes already exclude them.

    Connector: #194 does not allocate GPUDirect RDMA memory, and NIXL registers the full KV tensors. Thus, with a PD/NIXL connector, the first integration must use torch.empty.

  9. cecilsd commented on Nov 19, 2026

    @cecilsd
    Contributor

    @XinhaoTheo Thanks for clarifying the constraints. I tested #198 (c8868f0) together with #194 (594dc63) through a temporary adapter, using the actual storage implementation, serving runners, kernels, and CUDA Graph paths.

    Results on an RTX 5090:

    Check Result
    Additional GPU integration tests 18 passed
    Ouro-1.4B BF16 checkpoint Eight execution-mode/KV-dtype combinations, each with two input cohorts; 64 matched request pairs
    Generated outputs Token IDs and exit depths matched between ordinary and VMM storage
    Valid KV contents Byte-identical within the checked regions, including FP8 scales; the additional cohort checked every prompt position at every storage depth
    Graph reuse after commitment Newly backed blocks were used with unchanged pointers, full tensor shapes, and strides; no new capture was needed for the covered serving shapes

    The checkpoint runs covered synchronous, asynchronous, staged-prefill, and asynchronous speculative execution, each with BF16 and FP8 KV. Comparisons were within the same execution mode and KV dtype, using both greedy and seeded sampling. VMM reserved 512 blocks, prepared with 128 backed blocks, then committed 512 blocks and prioritized the newly backed range.

    An isolated negative control also reproduced the sliced-view constraint: capturing an indexed write on a view limited to the bootstrap blocks retained the old indexing bound after commitment. The integration therefore needs the full storage.tensors, must publish only the fully backed count returned by commit(), and must avoid operations on unmapped regions.

    For accounting, committed_bytes includes physical mapping alignment, while driver free memory already excludes those mappings. prepare_runtime() returns a preparation snapshot, so validation on newly committed blocks must run separately before final capacity and readiness are published. The current PD/NIXL requirement to retain ordinary storage is also recorded.

    No change to #198's startup execution was needed for these exercised paths. I added the handoff clarifications in 060203d (documentation only). The temporary adapter validates the current handoff; final measured sizing, capacity publication, and connector integration still need validation in the production integration.

  10. kunkunblueberry commented on Nov 19, 2026

    @kunkunblueberry
    Contributor

    Thanks @cecilsd — the drained/cleaned report semantics fit how I plan to size capacity, with two clarifications I need on coverage. I'll answer the three interface questions in order, then the storage-side ask for @XinhaoTheo.

    1. Handoff: warmup/capture -> measurement -> commitment

    The report fields you listed are the ones I want, and I read them this way: driver_free/total are boundary samples taken after the drain and after scratch release; num_blocks/kv_bytes describe the pool that exists now (KV reported separately, including FP8 scales) and are not a sizing result; allocator_cache_policy="retain" means PyTorch reserved bytes are not available capacity. So the planner will use driver-free as the physical budget and will not subtract resident graph memory a second time, and it will not call empty_cache between phases.

    Two coverage questions decide how much of the transient margin is measured rather than estimated:

    • Does the bounded workload coverage include the largest context prefill and the largest decode batch, i.e. the shapes that set the transient envelope? A peak observed on a smaller shape understates what later serving needs.
    • For enabled async/multi-stream, does coverage include one supported overlap, or only serial submissions? The RFC asks for the concurrent envelope rather than twice a serial peak.

    If either is out of scope for #198, the planner will keep an explicit estimated margin for that part and record it as a coverage gap rather than claim fully measured sizing. That is a reporting change on my side, not a blocker for #198.

    2. Post-commit validation ownership

    I propose StageEngine owns the ordering, and orchestration stays where #198 put it:

    prepare_runtime()                      # #198, ends with the report
      -> plan capacity (blocks + explicit transient/safety margins)
      -> storage.commit(blocks)            # #194, returns fully backed block count
      -> KVCacheManager.publish(first `committed` blocks)   # published <= committed <= max
      -> replay captured graphs + one representative execution on newly backed blocks
      -> ready
    

    Three properties for that validation step: it runs on the full storage.tensors (a graph captured on a view of only the bootstrap blocks keeps the old dim-0 bound, as your negative control shows); it uses neither real-request RNG nor prefix-cache publication; and on failure it drains, drops the owners, and reports the phase, so readiness is never signalled. If a connector is active, capacity still comes from the same planner but the pool is torch.empty, so no commit/validation step is needed.

    3. Connector registration and fallback recording

    Agreed with @XinhaoTheo on torch.empty for the first integration with PD/NIXL. I will record the decision in memory_plan — storage kind, the fallback reason, reserved/committed bytes and the published block count — so the startup log states why the VMM path was not selected instead of failing silently. Connector KV registration can then be addressed separately, since NIXL registers ranges at construction.

    Storage-side ask (#194)

    Given that close() was removed at 594dc63, the engine has no deterministic way to return KV memory on shutdown or repeated engine creation; RFC #192's validation calls for both. Options I see: an explicit release that the caller may only use after the graph and buffer owners are dropped, or a documented "drop all references, then collect" contract that StageEngine.close() implements and tests verify. Which do you prefer, @XinhaoTheo? My integration consumes commit()/committed_blocks/committed_bytes and enforces the publish bound in KVCacheManager, so either contract works as long as releasing is observable.

    For context on my side: #193 currently only factors the budget arithmetic out of plan_cache and records committed bytes; extending it into measured sizing against this report is the next step, after which the fixed estimates retire only where capture coverage is complete.

  11. XinhaoTheo commented on Nov 19, 2026

    @XinhaoTheo

    Thanks @kunkunblueberry I prefer the second option: drop all references, then collect.

    • This is the same contract as the current torch.empty pool. Its memory also returns to the allocator only when no reference remains.
    • An explicit release brings back the stale-view problem from the feat(kv): add CUDA VMM KV storage with fixed addresses #194 review, because the storage cannot check that the graph and buffer owners are dropped.
    • The release is observable. Hold a weakref.ref to each pool, then check that it is dead after StageEngine.close() and gc.collect(). test_last_reference_releases_memory_the_torch_allocator_never_counted uses this method.
    • The release synchronizes the device before the unmap. Do not replay a captured graph after the release.
  12. cecilsd commented on Nov 19, 2026

    @cecilsd
    Contributor

    @kunkunblueberry @XinhaoTheo I clarified the coverage report in #198 (4a5b198), without expanding the default startup workload.

    Coverage and transient margins

    • Preparation covers the largest decode batch allowed by the request/token limits and backed KV capacity, at the reported bounded history. It does not cover the model's maximum context combined with the maximum batch. The new decode_workloads lists actual completed (batch_size, history_tokens) pairs; history excludes the current input token. Independent maxima must not be combined into a measured workload.
    • Long contexts and unreported batch/history combinations retain an estimated transient allowance. Random block speculation is also explicitly reported as unmeasured: startup exercises greedy speculative rounds and ordinary stochastic sampling, but not stochastic speculative verification. Pipeline speculation remains greedy-only.
    • Async multi-stream preparation submits independent prompt-core/coda and decode-core/coda work before collection, then drains before measurement. With the Ouro checkpoint, an independent device timeline confirmed simultaneous core/coda kernel execution. This establishes a supported overlap, not a complete concurrent-memory envelope; the remaining margins should stay explicit.

    Additional checkpoint validation

    Using the pinned Ouro-1.4B BF16 checkpoint on an RTX 5090, I compared the previous #198 head with this follow-up at fixed capacity: 768 KV blocks, 16-token pages, four loops, up to three requests, and a 128-token aggregate budget/chunk. Separate workloads used one 3064-token prompt, three 1016-token prompts, and a 2024/504/504-token cohort, with eight subsequent positions reserved per request. These are capacity-feasible boundaries for this test configuration, not the model's 64K limit.

    Check Result
    Sync, async, staged prefill and async block speculation; BF16/FP8 KV; greedy and seeded sampling 112 matched request pairs / 672 output tokens
    Tokens, exit depths, finish reasons, observed ordinary-sampler logits and valid prompt KV (including FP8 scales) Identical between the two heads within each mode/KV dtype
    Separate controlled actual-runner speculative round Three long-history requests / nine verification rows; outputs and accepted counts matched for greedy and random sampling with both KV dtypes

    The normal scheduler did not put all three requests into one speculative round in the equal-length test, so the controlled runner check is reported separately. That check observed about 0.93 MiB incremental allocated peak for greedy versus 27.24 MiB for random speculation, supporting the need to retain the random-path margin. These are workload observations, not universal memory bounds. Startup rechecks showed small changes in both directions; no stable startup speedup is claimed.

    Commitment and release

    The proposed StageEngine ordering fits the handoff: after final commitment, publish only backed blocks to the internal allocator, validate captured replay and representative execution on newly backed blocks, and only then signal readiness. prepare_runtime() remains an idempotent preparation snapshot; calling it again does not validate new mappings. PD/NIXL should retain ordinary storage for the first integration and record the fallback reason.

    I also tested the reference-lifetime contract with the actual #194 storage in sync and async execution. StageEngine.close() currently drains work but retains the runner/cache owners. Dropping the engine and remaining owners released the pools; retaining an external tensor view kept its mapping valid until the view was dropped. Therefore close(); gc.collect() alone is not sufficient while those references remain. Final integration should test owner teardown after graph users and connector writers have retired, rather than explicitly unmapping live views.

  13. kunkunblueberry commented on Nov 19, 2026

    @kunkunblueberry
    Contributor

    The capacity-planning piece is open as #200 (draft), rebased onto bed3400 and green on the CPU workflow:

    • measured_kv_capacity() consumes the last drained phase of the step-1 report. Driver-free memory is the physical budget, so weights, buffers and resident graph memory are not subtracted twice; total - free only bounds the instance under gpu_memory_utilization. Transient and safety are explicit inputs, and the reported coverage gaps travel with the plan, so an estimated margin is never labelled measured.
    • KVCacheManager takes a capacity source and enforces published <= committed <= max; publishing re-reads the storage's committed count, so a request cannot receive an unbacked block. Release stays reference-based, per the contract agreed above.

    Two things it does not do yet, both declared in the PR: mapped storage currently rejects kv_cache_dtype != "auto" because the pool has no place for FP8 per-vector scales, and StageEngine does not consume the report until @cecilsd's #198 and @XinhaoTheo's #194 land. The agreed ordering and the memory_plan record of storage kind and fallback reason are my next change, stacked on those two.

  14. XinhaoTheo commented on Nov 20, 2026

    @XinhaoTheo

    @cecilsd Agreed. The teardown test must drop the engine and all owners, not only call close().

    @kunkunblueberry #194 can hold the FP8 scales. Add one PoolSpec for each scale tensor, for example PoolSpec(block_shape[:-1], torch.float32), the same layout as key_scale on main. commit() does not write the new blocks. If a scale must have an initial value before the first KV write, write it after each commit, only on the new blocks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

RFCRequests for comments on major architectural changes or design choices

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions