[DRAFT] feat(selection): cache select() booking inputs so create_reservation replays by id by jeffreywang88 · Pull Request #2 · jeffreywang88/dynamo

jeffreywang88 · 2026-07-08T03:44:04Z

What

POST /select with a selection_id now caches the computed booking inputs — the chosen worker, the normalized sequence hashes, and effective_prefill_tokens — in an in-process SelectionCache keyed by (model, tenant, selection_id). A follow-up POST /reservations carrying that selection_id (plus model_name, tenant_id, and its own reservation_id) replays the cached selection, so the (potentially very long) prompt is not re-sent or re-hashed. worker_id selects the explicit, self-contained form — it wins when both are present and discards any cached selection for the id. A request with neither selection_id nor worker_id is rejected outright.

When it is used

The Ray select-then-reserve flow: Ray calls select, dispatches the request to the returned worker, then books admission with a second call carrying selection_id, reservation_id, and expected_output_tokens. The two calls cannot be fused (select_and_reserve) because Ray decides between them, and under faults a select may never be followed by a reservation. The replay is keyed by selection_id; the booking is registered under reservation_id, so the two ids may differ. The explicit form remains fully supported and takes precedence whenever worker_id is present.

On the cached path only effective_prefill_tokens and expected_output_tokens are honored as overrides; dp_rank and prompt fields are ignored in favor of the cached selection (documented in the API docs, ReservationRequest, and the
Python stubs).

Failure modes and client errors

Every failure returns a specific, actionable message — never an opaque error:

Failure	Status	Error message
Neither `selection_id` nor `worker_id` supplied	`400`	`include selection_id to replay a cached selection, or worker_id with the prompt for a self-contained booking`
No cached entry (never selected, expired, already booked, wrong model/tenant, or a different selector replica)	`404`	`no pending selection {selection_id} for {key} (expired, already used, or never selected); call select again or book with worker_id and the prompt`
Cached worker deregistered before booking	`404`	`schedulable worker {id} not found for {key}`
No schedulable workers / model entry not ready	`503`	`no schedulable workers for {key}`
`effective_prefill_tokens` override exceeds the cached prompt length	`400`	`... must not exceed isl_tokens`
Duplicate booking for the same reservation id	`409`	`Request {id} already exists (assigned to worker {worker})`
Service shutting down	`503`	`selection service is shutting down`

Failures that occur before the booking lands (the worker/readiness/override rows above) re-insert the cached selection, so the identical replay call can be retried once the condition clears — the client never has to re-send the prompt because of a transient fault.

How memory leaks are avoided

The cache needs no background task and no async runtime (it works identically in the HTTP server and the in-process Python SelectionService):

Consumed on use. A successful booking removes the entry; an explicit booking for the same id discards it.
Evict-on-insert. Every insert sweeps expired entries from the front of an insertion-ordered queue (uniform TTL ⇒ front is oldest), amortized O(1).
TTL. Unclaimed entries (abandoned requests) expire after 120 s and are never returned stale — take re-checks the TTL on read.
Size cap. At most 4096 pending selections per selector; oldest are evicted first, so a burst of abandoned selects cannot grow memory unboundedly between TTL windows.

Related Issues

⚠️ This section is required. Choose one path below and delete the other.

🔗 This PR is linked to an issue:

Closes #XXXX

🚫 This PR is NOT linked to an issue:

Confirmed — no related issue

jeffreywang88 · 2026-07-08T04:29:06Z

+  "selection_id": "select-123",
+  "reservation_id": "request-123",


can selection_id and reservation_id be different? what are they used for respectively?

[claude] Yes, they may differ. selection_id names the cached selection to replay (the id given to select); reservation_id is the id the booking is registered under and what the lifecycle endpoints use. Ray passing the same value for both also works.

reservation_id is the id the booking is registered under and what the lifecycle endpoints use

explain this mechanism

[claude] create_reservation registers the booking in the per-model scheduler under request_id = reservation_id (finalize_reservation calls scheduler.add_request), which tracks its active and prefill load. The lifecycle endpoints (/reservations/{id}/prefill_complete, /output_block, DELETE) look the booking up by that same id to clear prefill load, grow decode blocks, and free it. selection_id never reaches the scheduler; it only keys the cache lookup.

jeffreywang88 · 2026-07-08T04:29:56Z

+- **Single-use**: The first successful booking consumes the entry; a repeat
+  replay returns `404` (`no pending selection`).
+- **Retryable on failure**: A booking that fails before landing (worker no
+  longer schedulable, service not ready) re-arms the entry, so the same call


re-arms the entry

how is this done?

[claude] On a pre-booking failure (entry not ready, cached worker unschedulable, invalid prefill override), reserve_from_cached_selection re-inserts the taken entry into the cache before returning the error, so the identical replay call works once the fault clears. Failures after the scheduler booking landed do not re-arm.

jeffreywang88 · 2026-07-08T04:36:02Z

+- **Retryable on failure**: A booking that fails before landing (worker no
+  longer schedulable, service not ready) re-arms the entry, so the same call
+  can be retried once the condition clears.
+- **Bounded window**: Entries expire after 120 seconds, and each selector


tell me the reason why 120 and 4096 are fixed numbers

[claude] They are backstops, not measured values: the expected select-to-reserve gap is sub-second, so 120s is generous headroom for a decide window plus retries, and 4096 bounds worst-case memory under mass abandonment (about 256MB if every entry held a 128k-token prompt's hashes). If operators need tuning we can promote both to KvRouterConfig, which has precedent (router_ttl_secs).

jeffreywang88 · 2026-07-08T04:36:58Z

+- **Bounded window**: Entries expire after 120 seconds, and each selector
+  retains at most 4096 pending selections, evicting oldest first.
+- **Replica-local**: The cache lives in the selector process that served the
+  `/select`. With multiple selector replicas, route the reservation to the


explain "multiple selector replicas". when can this happen?

[claude] The selector can run as several processes for availability, synchronized via --replica-sync-port/--replica-sync-peers (Peer Planes section). Behind a load balancer, select and /reservations can land on different processes, and the cache is process-local, so the replay only hits on the replica that served the select.

jeffreywang88 · 2026-07-08T04:38:59Z

+- **Overrides**: Only `effective_prefill_tokens` and `expected_output_tokens`
+  are honored; `dp_rank` and prompt fields (`token_ids`, `block_hashes`,
+  `sequence_hashes`, `isl_tokens`, `lora_name`) are ignored in favor of the
+  cached selection. Supplying `worker_id` switches to the explicit form.


if minimal form is used, all field except for expected_output_tokens should be ignored because it's not passed in to select.

confirm this statement: expected_output_tokens isn't passed in to select

[claude] Not quite: select does accept expected_output_tokens (optional on SelectRequest) and the entry caches it; the reservation's value, when present, overrides it. effective_prefill_tokens is also honored because it is an output of select the caller may refine, validated against the cached ISL. Everything else is ignored, as documented.

we should ignore expected_output_tokens and effective_prefill_tokens. the ones passed in from select should be honored.

[claude] Done. The replay is now pure: effective_prefill_tokens, expected_output_tokens, and the prefill-tracking decision are captured at select time and booked as-is; the reservation-side values are ignored (worker and prompt already were). The override-above-ISL 400 no longer exists, and the tests assert the cached values win.

jeffreywang88 · 2026-07-08T04:44:26Z

+The self-contained form carries the worker identity and prompt and needs no
+cached selection; it wins whenever `worker_id` is present. It also discards
+any cached selection for `selection_id` (or, when absent, `reservation_id`),
+so a delayed replay cannot book stale state:


are we preserving the semantics prior to this PR?

[claude] For the explicit form, yes: same fields, normalization, validation, and booking. Two deltas: a request missing worker_id used to fail deserialization with 422 and now returns 400 (no ids) or replays (selection_id present), and an explicit booking discards any cached entry for its id, a no-op for clients that never pass selection_id.

jeffreywang88 · 2026-07-08T04:45:10Z

-  semantics are unchanged; no idempotency ledger is added.
+  cross-replica admission, and the pending-selection cache behind the minimal
+  reservation form is local to the selector that served the `/select`. Use
+  `/select_and_reserve` for atomic local booking.


is the logic for select_and_reserve the same as the state prior to this PR?

[claude] Yes. Caching is gated on !book (cached_inputs in schedule_selection), so select_and_reserve never reads or writes the selection cache; its booking path is unchanged.

jeffreywang88 · 2026-07-08T04:47:09Z

        };
+        // `select` (book == false) with a selection_id caches the booking inputs
+        // so a follow-up `create_reservation` can replay them by that id.
+        // Capture the to-be-moved values before they go to the scheduler.


remove this line of comment

[claude] Removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

…lection_id Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 temporarily deployed to external_collaborator July 8, 2026 03:44 — with GitHub Actions Inactive

github-actions Bot added the documentation Improvements or additions to documentation label Jul 8, 2026

jeffreywang88 force-pushed the token-cache branch from 0951a9c to b7436c7 Compare July 8, 2026 04:27

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 04:27 — with GitHub Actions Failure

jeffreywang88 commented Jul 8, 2026

View reviewed changes

jeffreywang88 force-pushed the token-cache branch from b7436c7 to 90f0c84 Compare July 8, 2026 05:02

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 05:02 — with GitHub Actions Failure

jeffreywang88 force-pushed the token-cache branch from 90f0c84 to 73a2e10 Compare July 8, 2026 16:36

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 16:36 — with GitHub Actions Failure

jeffreywang88 force-pushed the token-cache branch from 73a2e10 to 65455d3 Compare July 8, 2026 17:23

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 17:23 — with GitHub Actions Failure

jeffreywang88 force-pushed the token-cache branch from 65455d3 to 12a13a8 Compare July 8, 2026 17:31

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 17:31 — with GitHub Actions Failure

jeffreywang88 and others added 4 commits July 8, 2026 10:52

feat(selection): add TTL+cap bounded pending-selection cache

98adf8b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

feat(selection): replay cached selections in create_reservation by se…

4598bc2

…lection_id Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

docs(bindings): document reservation replay in SelectionService API

ae16a5d

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

docs(router): document minimal reservation replay form

6588b18

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 force-pushed the token-cache branch from 12a13a8 to 6588b18 Compare July 8, 2026 17:53

jeffreywang88 had a problem deploying to external_collaborator July 8, 2026 17:53 — with GitHub Actions Failure

jeffreywang88 closed this Jul 8, 2026

		"selection_id": "select-123",
		"reservation_id": "request-123",

Conversation

jeffreywang88 commented Jul 8, 2026

What

When it is used

Failure modes and client errors

How memory leaks are avoided

Related Issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant