Repository navigation
Conversation
A streamed turn the client closed during decode was recorded with session_cache_hit False and no cached_tokens, cache_source or ttft_s, so it read as a cache miss even when its prompt had been restored from RAM. The completed prefill event now carries the cache outcome as the request's stats will report it (cache_source, cache_miss_reason, the SSD fields and the served restore record). The server keeps that outcome per request in a small bounded ledger, and the cancellation receipt takes it, with ttft_s measured as in the normal receipt. A request cancelled before its prefill completed records the outcome as null, not as a miss. Normal receipts release the ledger entry.
A cancelled row fell back to the HIT/MISS badge, and with the old receipt always to MISS. It now gets its own badge with the reason and the duration (for example "cancelled · client · 3m 0s"), a tooltip with the raw reason, and its cache outcome beside the badge when the server knew it. The session cache hit rate leaves cancelled requests out.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When a client closes a streamed turn during decode, the server writes a short receipt that always says
session_cache_hit: false, withoutcached_tokens,cache_sourceorttft_s, whatever the cache really did. The dashboard then shows the row as a yellow "MISS" and counts it against the hit rate. In one case, a 92,819-token turn restored 92,636 tokens from RAM and was then closed by the client after 180 s: the request log listed it as a cache miss.How it works, in plain terms
The cache outcome is known as soon as the prompt has been read. The "prefill completed" event now carries that outcome, and the server keeps it per request in a small bounded ledger. When the request is cancelled, its receipt takes the outcome from there. If the request was cancelled before its prompt was read, the cache fields are
null(unknown), notfalse.What changes
session_cache_hit,cached_tokens,cache_source,cache_miss_reason, the SSD fields) andttft_s. Unknown values arenull.cancelled · client · 3m 0s, with its real cache outcome next to it (hit 92,636). Cancelled rows are left out of the hit rate. Receipts from older servers show no outcome rather than a false miss. The served bundle is rebuilt from the frozen lockfile.Tests
ttft_s; a cancel before the prefill completed recordsnull; a normal receipt releases its ledger entry.bun test): the badge and tooltip, the cache hint, older receipts, and the hit rate without cancelled rows.tsc --noEmitpasses.