feat(steering): worker-registered named vectors + latch-by-reference#238
Merged
RhizoNymph merged 1 commit intoJul 5, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Declarative
rest_of_conversationsteering gates persist a client-supplied vector payload in server memory indefinitely (the conversation latch). #229 bounded that with a byte-capped FIFO latch map, but two failure modes remain:rest_of_conversationquietly gets no cross-turn steering.Root cause: server-side persistence of client-supplied bytes. The fix is a principle — persistence requires server-resident content — implemented as latch-by-reference to a registered name, not by value.
What changed
1. Worker-replicated named-vector registry. Named probe/steer vectors are promoted from a frontend-only registry to a rank-replicated worker registry (
vllm/v1/worker/steering_vector_registry.py)./v1/steering/vectors/register|unregisternow also broadcast to every worker viaengine.collective_rpc(register_steering_vector_name/unregister_steering_vector_name), mirroring/v1/steering/set's flow. Each side stores a sha256 content digest over the canonical packed serialization. Registration stays dev-mode gated, no auth (deliberate — see §8.3).2. Names flow to workers.
build_steering_gatesstops inflatingNamedVec→inline; a name rides the msgpack channel un-inflated (smaller payload) andresolve_gatesresolves it against the worker registry at admission, carrying the source name+digest onResolvedGate. Unknown name at admission (register/admission race) → graceful skip viaresolve_gates_safe. One resolution point for both steer and probe names. Reverses the §8.2 "worker only ever sees packed bytes — no worker-side registry" decision (persistence semantics require worker-side resolution at bridge time).3. Scope rule.
rest_of_conversation+apply=addrequires a named steer source. Inline on that combination → HTTP 400. Ephemeral scopes (this_token/next_step/rest_of_request) keep inline. The consumer skip-and-warns an inline persisted gate arriving from a non-frontend producer.4. Latch by reference. The latch entry is a tagged union:
ByRefLatch(refs=[name,digest,strength], compose_admitted, source)for declarative persisted latches (~0 bytes against the cap) and the existingRequestSteeringOverride(raw vectors, byte-accounted) for operator-authoredSteeringControllersubclasses. Bridging aByRefLatchre-resolves each name from the worker registry and verifies the digest: a missing name or a digest mismatch (name re-registered mid-conversation) disengages the latch (drop + warn once) rather than steering with silently-changed content.compose_admittedand source are preserved.5. LRU eviction.
_latchedis now an LRUOrderedDict; a bridge refreshes recency (move_to_end), eviction drops the least-recently-used first. #229's oversized-ByValuerefusal path is kept as-is. Deterministic (pure function of the latch/bridge sequence).6. Docs + protocol descriptions updated (
dynamic_steering.md§8.2/§8.3, OpenAI protocol steering fields,vectors_protocol.py).Stacked on #229
This is stacked on #229 (
chore/steering-trust-hardening) because it rebuilds the same_latchcode. Base ischore/steering-trust-hardening; please retarget tofeat/dynamic-steeringafter #229 merges. #229's oversized-ByValuesemantics are unchanged (rebased around, not redesigned).Expected sibling conflicts
api_router.py(different handlers) andsteering_model_runner_mixin.py(different regions) — trivial.Out of scope (follow-ups)
ByValuesemantics.Tests
tests/v1/test_steering_schema.py tests/v1/capture/test_steering_controller.py tests/v1/capture/test_declarative_consumer.py tests/v1/worker/test_steering_declarative_cleanup.py tests/v1/worker/test_steering_dynamic_override.py tests/v1/worker/test_steering_vector_registry.py tests/entrypoints/openai/test_steering_vector_registry.py— 132 passed. Covers: 400 for inline+rest_of_conversation+add; named accepted; unknown name→400; NamedVec un-inflated through build_steering_gates; ephemeral scopes keep inline; worker resolution + graceful skip; ResolvedGate carries name+digest; register/unregister/re-register + digest; ByRef latch + digest-mismatch/missing-name disengage + warn-once; LRU recency; determinism. (Two prior tests asserting the now-removed inlinerest_of_conversationlatch were updated to the named form.)