Bound latch memory + document dynamic-steering trust model#229
Merged
Conversation
feat(steering): worker-registered named vectors + latch-by-reference
…e/steering-trust-hardening # Conflicts: # vllm/v1/worker/steering_model_runner_mixin.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Hardens the per-conversation latched steering stack against two review findings, without adding authentication.
Latch byte bound (
controller.py).SteeringController's latch map was bounded by entry count only, but each latchedRequestSteeringOverridepins full steering vectors (all hooks x layers, float32) on every TP rank — so 1024 entries x unbounded per-entry payload is an unbounded host-memory exposure. Adds amax_latched_bytesknob (default 256 MiB) tracked as a running total plus per-entry bytes computed once at latch time. Eviction stays FIFO/oldest-first until both the count cap and the byte cap fit; a single override larger than the byte cap alone is refused (rate-limited warning, no crash) — the triggering request still steers its own turn, only cross-turn persistence is dropped. Eviction remains a pure function of the latch sequence (rank-deterministic).latched_bytesis exposed in the existingstatus()surface.Trust-model documentation. Adds design §8.3 "Trust model and multi-tenancy": the stack assumes single-tenant / trusted-client deployment;
conversation_idis a global, client-chosen, unauthenticated namespace, so in any multi-client deployment the operator or gateway must namespace ids per client (e.g. per-tenant prefix) to prevent collisions; documents the latch count+byte bounds and the FIFO churn caveat (a flood of fresh ids silently evicts other clients' latches); and records the named-vector registry's shared last-writer-wins semantics and why it is deliberately not behind the steering API key (naming sugar over the already-open inline path, capability-equivalent). Mirrors the namespacing requirement into the user-facingconversation_idfield descriptions in the OpenAI completion + chat protocols.Why
Unbounded latch host-memory growth on every rank, and undocumented single-tenant trust assumptions around a guessable/reusable conversation id and a shared mutable vector-name namespace.
No authentication, registry code, or schema changes.
Sibling-PR conflicts
docs/design/dynamic_steering.mdis also touched by feat/steering-determinism-checksum (expected merge conflict).vllm/v1/capture/controller.pyis not touched by any sibling.