Conversation
7d46f71 to
40d56ff
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40d56ffdff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if reservation.was_replacement() { | ||
| self.stats.evictions += 1; | ||
| } | ||
| self.state.publish(reservation, kv_guards); |
There was a problem hiding this comment.
When prefix caching is enabled and a near-capacity prompt publishes a snapshot, these guards can retain most of the KV pool after the request finishes. scheduler_loop checks available_blocks() before calling begin_request, so a repeated request that could reuse those blocks is deferred as though it needed a cold allocation; because the only eviction path runs from reserve_prefix after a request has already been admitted and prefilled, an idle engine can defer that request forever. Admission must account for reusable blocks or evict unpinned prefix entries under KV pressure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0e65237 by decoupling snapshot residency from KV residency. Snapshot entries no longer retain KV guards; released KV blocks return to kvbm’s inactive pool, remain reusable for prefix matches, and count toward available_blocks(). Cold admission can therefore reclaim them without first evicting the snapshot. A regression test covers both warm reuse and full-pool cold reclamation.
| .filter(|&v| v > 0) | ||
| .unwrap_or(pegainfer_qwen35::DEFAULT_MAX_PREFILL_TOKENS); | ||
| let handle = pegainfer_qwen35::start_engine_with_capacity_and_policy( | ||
| let handle = pegainfer_qwen35::launch_with_options_and_policy( |
There was a problem hiding this comment.
When a Qwen3.5 benchmark uses non-greedy sampling with --seed other than 42, this new call routes through launch_with_options_and_policy, whose EngineLoadOptions still hardcodes seed: 42; the previous path passed command_seed(&cli). Consequently the advertised seed no longer controls engine sampling for Qwen3.5, breaking reproducibility and making different seed runs use the same sampling stream. Pass the command seed through the launch options or retain the explicit EngineLoadOptions path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is no longer applicable after the frontend consolidation: pegainfer-server/src/bin/bench_serving/main.rs
and its per-command benchmark seed path no longer exist. Qwen3.5 now launches through ModelLine with an
intentionally fixed engine seed of 42, so no user-provided benchmark seed is silently discarded.
|
This pull request has been inactive for 14 days. It will be closed after another 30 days unless there is new activity. |
40d56ff to
ad9d43e
Compare
ad9d43e to
f0a1936
Compare
|
@codex review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0a19368ef
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| for request in plan.requests { | ||
| self.begin_request( | ||
| request.request_id, | ||
| &request.prompt_tokens, | ||
| self.max_position_embeddings | ||
| .saturating_sub(request.prompt_tokens.len()) | ||
| .max(1), | ||
| None, | ||
| false, | ||
| )?; |
There was a problem hiding this comment.
When the public TP executor receives an invalid plan such as an empty prompt or duplicate request IDs, this loop can successfully broadcast RestoreRequest and insert state for earlier items before a later iteration fails. Because validate_prefill_chunks runs only afterward and this error path does not drop already-begun requests, the failed call leaves controller and worker state consuming capacity, and retrying the same ID then fails as a duplicate. Validate the complete plan before this loop or roll back every request begun by the call.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0e65237 by validating the complete prefill layout—including empty prompts, duplicate or existing request IDs, capacity, and context limits—before broadcasting any RestoreRequest. If a failure occurs after an earlier request has already committed, the executor is poisoned rather than exposing partially admitted state. CPU validation and a real TP2 retry regression cover this path.
d3f79b9 to
0e65237
Compare
|
@codex review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e65237eec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let graph_state = model.create_batch_decode_graph_state_with_capacity( | ||
| slots, | ||
| model.kv_buffer().num_blocks(), | ||
| (model.kv_buffer().num_blocks() - 1) as i32, |
There was a problem hiding this comment.
When TP CUDA Graph decode pads a non-bucket batch after the logical pool has allocated its highest page ID, this selects num_blocks() - 1 as the padding page even though BlockPool::build reserves page 0 and leaves the highest page allocatable. On at least the minimum-capacity rank, padding rows then write K/V into a page owned by a real or cached request, silently corrupting its attention state. Pass the common logical pool's actual padding_block_id() to every worker instead of deriving the physical last page.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bea4e1e. The controller now obtains the reserved padding block ID from the common logical KV pool and passes it to every TP worker for CUDA Graph decode.
| /// Reserve KV capacity for the next one-token decode forward. | ||
| pub(crate) fn schedule_decode(&self, request: &mut RequestKv) -> Result<()> { | ||
| request | ||
| .schedule_decode(self.kv.pool()) |
There was a problem hiding this comment.
When a multi-token request's prompt_len + max_tokens - 1 lands exactly on a page boundary under a full KV budget, the unchanged admission calculation reserves only that many pages, but the newly adopted RequestKv::schedule_decode can transiently draw one additional generation page. Such a request is admitted as servable and later fails allocation; in production decode_step responds by terminating the entire active batch. Size lifetime reservations using prompt_len + max_tokens for multi-token requests, as required by the RequestKv peak draw.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bea4e1e. Admission now reserves prompt_len + max_tokens for multi-token requests, covering RequestKv’s transient peak allocation. One-token requests remain prompt-only because they finish directly after prefill.
0e65237 to
bea4e1e
Compare
|
@codex review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bea4e1e2dc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for view in views { | ||
| all_page_indices.extend_from_slice(view.page_indices()); | ||
| indptr.push(all_page_indices.len() as i32); |
There was a problem hiding this comment.
When several concurrent requests restore the same cached prefix, concatenating every KvView can produce more page-index entries than the fixed page_indices_d capacity of max_total_pages + max_batch_size: shared physical pages count once toward the pool capacity but once per request in this vector. The scheduler can legitimately admit this state because subsequent requests reuse the already-active prefix and need only distinct suffix pages; for example, three requests can each reference a 200-page sequence while using only 400 unique pages. Once the combined views exceed the buffer, memcpy_htod fails and the scheduler terminates the decode batch, so this metadata buffer must be sized for per-row references rather than unique physical pages.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 309eca7. Decode page-index metadata is now sized by per-request references (max_batch × max pages per context) rather than unique physical KV pages, covering shared-prefix reuse across concurrent requests.
|
@codex review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 309eca7b70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| let request_pages = pages_needed(max_kv_tokens(prompt_len, max_tokens), page_size); | ||
| let request_pages = request_lifetime_pages(prompt_len, max_tokens, page_size); |
There was a problem hiding this comment.
When concurrent requests share a long cached prefix and the pool cannot cover each request's full logical lifetime independently, this debits request_pages for every request before begin_request attaches the reusable prefix. Unlike the resolved idle-cache case, available_blocks() counts only reset and inactive pages, so once the first warm request promotes the shared blocks to active, later requests are deferred even though attaching those same active blocks would require only suffix/output pages. This unnecessarily serializes shared-prefix traffic under KV pressure; admission should probe and credit the jointly reusable pages.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 17b38c8. Admission now subtracts reusable active prefix pages from each request’s KV requirement. After the first request activates a cached prefix, waiting requests sharing it are reconsidered immediately instead of being serialized.
|
@codex review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17b38c84f9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Retain registered KV only when prefix caching is enabled. | ||
| if !self.enabled() { | ||
| request.mark_blocks_reset_on_release(); | ||
| } | ||
| applied?; |
There was a problem hiding this comment.
When prefix caching is disabled (the default), this marks only blocks that were assigned before apply_prefill; the apply then registers the newly completed blocks, leaving them unmarked. The single-GPU scheduler retires requests by dropping their state because drop_prefill_state and drop_active_state are no-ops, so the final block registered by a request enters the inactive prefix cache rather than the reset pool. With distinct traffic these useless entries eventually fill the pool and force LRU/registry eviction work despite caching being disabled; mark blocks after a successful apply or explicitly release single-GPU requests through release_request (the same ordering affects apply_decode).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in d4e2e1b. Newly registered KV blocks are now marked after successful prefill and decode applies, so disabling prefix caching returns them to the reset pool on release.
|
@codex review please |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Hi @FeathBow, this PR is ready for review. Could you please take a look when you have time? Thanks! |
FeathBow
left a comment
FeathBow
left a comment
There was a problem hiding this comment.
Btw for some tests, with one slot and no subsequent eviction, removing the duplicate-path timestamp update would leave every assertion passing, so the claimed LRU-refresh behavior is currently untested. Besides, Please remove the leftover new_requests capacity calculation now that prefill no longer creates worker requests? It's a redundant scan. And please remove the unused things, e.g. active/prefilling arguments from both available_pages methods and their callers?
Some tests we could fold this into the existing CLI validation cases. This only parses arguments and validates launch options; it does not exercise prefix caching on either TP configuration.
| pub(crate) fn apply_decode(&self, request: &mut RequestKv, token: u32) -> Result<()> { | ||
| request.apply_decode(token, self.kv.pool())?; | ||
| if !self.enabled() { | ||
| request.mark_blocks_reset_on_release(); |
There was a problem hiding this comment.
Could we ensure that decode registers only blocks whose KV has actually been computed? The delegated SchedulableSequence::apply_decode appends the newly sampled token before registering completed token blocks. With a 254-token prompt and two output tokens, the request ends at KV position 255 but registers the 256-token prefix. If a snapshot for that prefix survives eviction of its original KV, a later request can pair the retained snapshot with this newly registered, incomplete page and restore to 256, reading an unwritten final KV entry. Both single-GPU and TP serving use this path. Please defer registration of the dangling-token block until its KV is complete.
There was a problem hiding this comment.
Fixed by reversing the operations in SchedulableSequence::apply_decode: it now registers KV completed by the current forward first, then appends the newly sampled token. Therefore, a newly crossed block remains unregistered until the next forward computes that token’s KV. Tests were updated to cover this deferred registration.
| } | ||
|
|
||
| #[test] | ||
| fn released_prefix_kv_is_reclaimable_without_snapshot_eviction() { |
There was a problem hiding this comment.
Could this regression exercise Qwen35PrefixCache itself? The test currently creates a BlockPool and a separate SnapshotCache, so it still passes if the joint coordinator starts retaining KV guards again. The allocation assertions mainly verify the dependency's reclaim behavior, while the final lookup checks an unrelated map. Please cover reclamation through the production coordinator, or remove this test's claim to cover the joint-cache regression.
There was a problem hiding this comment.
I removed the misleading reclamation test, added real LRU eviction coverage, removed redundant capacity scans and arguments, and folded the CLI case into existing validation.
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
Signed-off-by: wangke <364517893@qq.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9119897069
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self.model | ||
| .config() | ||
| .max_position_embeddings | ||
| .saturating_sub(req.prompt_tokens.len()), |
There was a problem hiding this comment.
When the direct single-GPU Qwen35Executor receives a prompt whose length equals max_position_embeddings, this computes max_output_tokens as zero. The prefill forward itself accepts that end position, but apply_prefill(Some(first_token)) subsequently calls RequestSequence::append_token, which asserts that an output token remains and panics. The TP executor already rejects this exact case; add the equivalent validation here so this public API returns an error instead of aborting.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. The single-GPU executor now rejects prompts that leave no output-token capacity before mutating state, matching the TP validation path.
Signed-off-by: wangke <364517893@qq.com>
9119897 to
bd17753
Compare
|
@codex review, thanks |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd17753d90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| let request_pages = pages_needed(max_kv_tokens(prompt_len, max_tokens), page_size); | ||
| let request_pages = request_lifetime_pages(prompt_len, max_tokens, page_size); |
There was a problem hiding this comment.
When a GenerateRequest has max_tokens == 0, this admission path treats it like a one-token request and admits it. Both single-GPU and TP final-prefill paths then pass a sampled token to RequestKv::apply_prefill, whose underlying RequestSequence::append_token asserts because its output-token limit is already zero, panicking the Qwen3.5 scheduler thread. Handle zero-token requests as an immediate completion or reject them before materializing KV state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. Qwen3.5 admission now rejects max_tokens == 0 before materializing KV state.
Signed-off-by: wangke <364517893@qq.com>
|
@codex review please |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
| } | ||
| }; | ||
|
|
||
| // The forward computed KV for the existing dangling token. Register |
There was a problem hiding this comment.
SchedulableSequence::apply_decode is reached through RequestKv::apply_decode, whose production callers include qwen3 , glm52, kimi-k2, and pegainfer-kv-store. Deferring registration by one step is a semantic change for all of them, and the only assertions updated are the three inside kvbm.
Two things I'd like recorded before this merges.
First, which consumers did you check? qwen3 has its own prefix cache and most likely benefits from the same fix, but that is a silent behavior change and the description doesn't mention it. The kv-store seal path is the one I'd look at specifically: pad_tail_block is a no-op when kv_position sits on a block boundary, which is now exactly the state where the block that just crossed is still unregistered.
Second, is apply_speculative deliberately left on the old ordering? At scheduled.rs it still appends every accepted token, then registers, then advances kv_position. Its callers are qwen3's dspark path and glm52's MTP path. If the last accepted token's KV is also uncomputed in that step, it has the same asymmetry this commit just fixed for decode; if the accepted contract guarantees computed KV for every token, then it is correct as is. Either answer works, but the two paths in one file now disagree about what "registered" means, so I recommend saying which in a comment.
There was a problem hiding this comment.
Thanks for highlighting the shared impact. I traced the normal decode paths in Qwen3, Qwen3.5, Kimi-K2, and GLM5.2/KV-store. They use the same contract: the forward computes KV for the existing dangling token and returns a newly sampled token whose KV is not yet computed. Registering before appending therefore preserves the intended invariant.
For KV-store sealing, if decode advances kv_position to a page boundary, that page is registered after its final KV row is computed and before the new dangling token is appended to the next page. Therefore, the boundary no-op in pad_tail_block does not skip a completed page.
I also audited apply_speculative. Its result contains computed accepted drafts followed by an uncomputed correction/bonus token, so it has the same underlying issue. Qwen3.5 does not use that path, and changing it would require separate validation speculative decoding, so I am leaving it unchanged in this PR.
I’ve updated the PR description to record the shared apply_decode behavior change, the consumers reviewed, the tests actually run, and why apply_speculative remains out of scope for this PR.
|
|
||
| pub(crate) fn log_prefix_cache_stats(&self) { | ||
| let stats = self.kv_cache.stats(); | ||
| log::info!( |
There was a problem hiding this comment.
log records should to drop the module or model prefix, naming Qwen3.5 specifically, because the text layout already prints each record's target. Only anyhow! and bail! messages keep theirs, since those surface to callers without a target.
Signed-off-by: wangke <364517893@qq.com>
FeathBow
left a comment
FeathBow
left a comment
There was a problem hiding this comment.
Thanks!
The migration defense table is the bigger one. KvState appears in nine qwen35 files on the base commit and in none here, so docs/conventions/migration-defense.md applies, and what it asks for is an audit rather than a paragraph: each old guard, the failure mode it covered, and its successor. Eight fix-up commits on this branch put guards back one at a time; the table's value now is the ones nobody has hit yet.
For the TP pass, half-fixing it also made the two summaries disagree — the TP path now logs prefix cache summary: ranks=… while the single-GPU path logs Qwen3.5 prefix cache summary: …, so a log rule written against one matches only one of the two paths.
To be clear on scope: the crate has plenty of pre-existing prefixed lines (tp_executor.rs:, scheduler/mod.rs, weights.rs), and I am not suggesting this PR touch those. Just the three it adds.
Signed-off-by: wangke <364517893@qq.com>
|
@codex review, temp ignore conflicts |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
3 participants
Description
This PR is a follow-up of #257 and #423. It adds opt-in prefix caching for Qwen3.5 on both single-GPU and tensor-parallel serving.
Qwen3.5 is a hybrid model, so full-attention KV alone is not a valid reusable prefix. A cache hit is reported only when full-attention KV and the complete recurrent/conv state are available at the same token boundary.
Prefix caching remains disabled by default and can be enabled with:
Type of Change
What changed
TokenEvent::Scheduled.cached_tokensreporting for accepted prefix hits.Design
Qwen35PrefixCachecoordinates two independently managed resources:RequestKvandKvCacheManagermanage content-addressed full-attention KV pages.SnapshotCacheandRecurrentStateStoremanage recurrent/conv snapshots.A snapshot entry identifies a token lineage, boundary, and recurrent-state slot; it does not retain KV guards. During lookup, the cache selects the longest boundary for which both matching KV and a snapshot are currently available. A KV-only or snapshot-only candidate is treated as a miss.
This separation preserves joint-hit correctness without allowing cached snapshots to pin the KV pool and block new admissions.
Request flow
begin_request()creates request-local KV state and probes the longest valid joint boundary.cached_tokens.Non-aligned prompt tails are processed normally but do not create snapshots.
Tensor-parallel behavior
Consistency guarantees
KvState migration defense
This PR replaces
KvStatewith theRequestKv/KvViewlifecycle coordinated byQwen35PrefixCache; the table maps each protection in the old path to its successor in the new design.KvStateowned anOwnedPagePermit, releasing request ownership on drop.RequestKvowns kvbm block assignments. Single-GPU scheduler cleanup drops the owning backend state and relies onRequestKvRAII; the direct executor and TP controller callQwen35PrefixCache::release_requestexplicitly. Unapplied reservations are returned byrevert_schedule.KvPoolreserved a dedicatedpadding_page_id()and excluded it from request capacity.BlockPoolreservespadding_block_id(),max_request_blocks()excludes it from admission, and the controller passes the same ID to single-GPU and TP decode buffers.KvState::ensure_capacitybefore advancing state.RequestKv::schedule_prefillandschedule_decodereserve pages before forward and produce immutableKvViews.apply_*commits only after success; failures callrevert_schedule.KvState::seq_len,last_page_len, andadvancedefined KV positions and page extents.RequestKv::kv_positionowns committed progress andKvViewcarries the scheduled extent. Prefill, decode, restore, and snapshot publication validate recurrent positions against them.checked_prefill_end_posand RoPE coverage were validated before entering the prefill chunk loop.prefill.rs; public single-GPU and TP entry points also reject prompts that leave no output position before creating request state.KvViewand recurrent rows; metadata capacity is sized for per-row references, checked before upload, and padded rows use the reserved block.max_request_blocks()andrequest_lifetime_pages(), including kvbm’s transient dangling decode page. Jointly reusable prefix pages are credited without double-charging shared active blocks.KvCacheManagersized to the smallest rank and sends identical page IDs to every worker. Plans are validated before admission; rank responses and restore positions are checked, and post-mutation failures take the fail-closed path.Validation
-D warnings.apply_speculativeremains unchanged because Qwen3.5 does not use it and its terminal/truncated contracts require separate cross-model validation.Performance
Measured on 2026-08-06, using local Qwen3.5-4B weights and idle RTX 4090s. TP1 used GPU 1 with CUDA Graphs enabled; TP2 used GPUs 1/2 with
--cuda-graph=false. The prefix A/B matrix used a 1,024 MiB cache, 2 warmups, and 10 measured iterations.For single-token generation, the table shows TTFT p50 in milliseconds (
cache off -> cache on):For 128 generated tokens, cache reuse primarily reduces prefill/TTFT; decode remains the dominant part of E2E latency:
TP1 serving-load checks also completed:
24.67/94.41/152.25 ms, request throughput of83.11/75.87/69.04 tok/s, and a 2,048-token cache hit for every request.12.03/12.25 ms; mixed-load ITL was12.03/36.12 ms. After initial insertion, 4,096-token warm-prefix injections restored 3,840 tokens with41.42–51.00 msprefill and no warnings.31.84/48.71/93.34/146.05/282.10 msto18.51/21.42/21.34/29.16/25.51 ms.