Skip to content

feat(steering): worker-registered named vectors + latch-by-reference#238

Merged
RhizoNymph merged 1 commit into
chore/steering-trust-hardeningfrom
feat/steering-named-latch
Jul 5, 2026
Merged

feat(steering): worker-registered named vectors + latch-by-reference#238
RhizoNymph merged 1 commit into
chore/steering-trust-hardeningfrom
feat/steering-named-latch

Conversation

@RhizoNymph

Copy link
Copy Markdown
Owner

Problem

Declarative rest_of_conversation steering gates persist a client-supplied vector payload in server memory indefinitely (the conversation latch). #229 bounded that with a byte-capped FIFO latch map, but two failure modes remain:

  • Silent cross-conversation disengage. A byte-cap (or count-cap) eviction drops another conversation's latch mid-conversation with no signal — one client's flood of latches silently un-steers everyone else.
  • Silent scope degradation. An oversized latch is refused, so the conversation that asked for rest_of_conversation quietly gets no cross-turn steering.

Root cause: server-side persistence of client-supplied bytes. The fix is a principle — persistence requires server-resident content — implemented as latch-by-reference to a registered name, not by value.

What changed

1. Worker-replicated named-vector registry. Named probe/steer vectors are promoted from a frontend-only registry to a rank-replicated worker registry (vllm/v1/worker/steering_vector_registry.py). /v1/steering/vectors/register|unregister now also broadcast to every worker via engine.collective_rpc (register_steering_vector_name / unregister_steering_vector_name), mirroring /v1/steering/set's flow. Each side stores a sha256 content digest over the canonical packed serialization. Registration stays dev-mode gated, no auth (deliberate — see §8.3).

2. Names flow to workers. build_steering_gates stops inflating NamedVec→inline; a name rides the msgpack channel un-inflated (smaller payload) and resolve_gates resolves it against the worker registry at admission, carrying the source name+digest on ResolvedGate. Unknown name at admission (register/admission race) → graceful skip via resolve_gates_safe. One resolution point for both steer and probe names. Reverses the §8.2 "worker only ever sees packed bytes — no worker-side registry" decision (persistence semantics require worker-side resolution at bridge time).

3. Scope rule. rest_of_conversation + apply=add requires a named steer source. Inline on that combination → HTTP 400. Ephemeral scopes (this_token/next_step/rest_of_request) keep inline. The consumer skip-and-warns an inline persisted gate arriving from a non-frontend producer.

4. Latch by reference. The latch entry is a tagged union: ByRefLatch(refs=[name,digest,strength], compose_admitted, source) for declarative persisted latches (~0 bytes against the cap) and the existing RequestSteeringOverride (raw vectors, byte-accounted) for operator-authored SteeringController subclasses. Bridging a ByRefLatch re-resolves each name from the worker registry and verifies the digest: a missing name or a digest mismatch (name re-registered mid-conversation) disengages the latch (drop + warn once) rather than steering with silently-changed content. compose_admitted and source are preserved.

5. LRU eviction. _latched is now an LRU OrderedDict; a bridge refreshes recency (move_to_end), eviction drops the least-recently-used first. #229's oversized-ByValue refusal path is kept as-is. Deterministic (pure function of the latch/bridge sequence).

6. Docs + protocol descriptions updated (dynamic_steering.md §8.2/§8.3, OpenAI protocol steering fields, vectors_protocol.py).

Stacked on #229

This is stacked on #229 (chore/steering-trust-hardening) because it rebuilds the same _latch code. Base is chore/steering-trust-hardening; please retarget to feat/dynamic-steering after #229 merges. #229's oversized-ByValue semantics are unchanged (rebased around, not redesigned).

Expected sibling conflicts

Out of scope (follow-ups)

Tests

tests/v1/test_steering_schema.py tests/v1/capture/test_steering_controller.py tests/v1/capture/test_declarative_consumer.py tests/v1/worker/test_steering_declarative_cleanup.py tests/v1/worker/test_steering_dynamic_override.py tests/v1/worker/test_steering_vector_registry.py tests/entrypoints/openai/test_steering_vector_registry.py — 132 passed. Covers: 400 for inline+rest_of_conversation+add; named accepted; unknown name→400; NamedVec un-inflated through build_steering_gates; ephemeral scopes keep inline; worker resolution + graceful skip; ResolvedGate carries name+digest; register/unregister/re-register + digest; ByRef latch + digest-mismatch/missing-name disengage + warn-once; LRU recency; determinism. (Two prior tests asserting the now-removed inline rest_of_conversation latch were updated to the named form.)

@RhizoNymph RhizoNymph merged commit 6e16a7a into chore/steering-trust-hardening Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant