Skip to content

Serve endpoints from a dedicated endpoint discovery service - #9

Closed
epk wants to merge 1 commit into
socketry:mainfrom
epk:strict-endpoint-discovery
Closed

Serve endpoints from a dedicated endpoint discovery service#9
epk wants to merge 1 commit into
socketry:mainfrom
epk:strict-endpoint-discovery

Conversation

@epk

@epk epk commented Aug 6, 2026

Copy link
Copy Markdown

Reduces the published xDS surface to EDS, so the monitor does what its name suggests and nothing more.

Why

Envoy's aggregated discovery stream is a singleton to a single management server. By serving endpoints over ADS, the supervisor claimed that stream, which prevented anything else — Envoy's own bootstrap, or a separate control plane — from using ADS for cluster, listener and route configuration.

Endpoints are also the only resource that actually changes as workers come and go. Everything a Cluster carries is static for a given deployment.

Worth noting: control-plane/envoy.yaml was already running this way. It has no cds_config, so Envoy never subscribed to CDS, and the cluster resources the monitor built were discarded unread.

What changed

Serves envoy.service.endpoint.v3.EndpointDiscoveryService/StreamEndpoints directly. StreamEndpoints reuses the generic discovery messages, so this needs no new protocol buffer definitions, and Async::GRPC::XDS::Service::Stream is transport agnostic and reusable as is.

Out-of-band ORCA is untouched. It is a separate gRPC service sharing the same port, and the worker identity it depends on travels in the endpoint hostname.

Removed along with the cluster resources:

  • The health_checks: option, which only configured them.
  • Generation of the client-side weighted round-robin policy. It is now configured on the cluster, pointing at the monitor's port — every input was already known at configuration time.
  • Scheme and protocol validation. It only constrained cluster configuration the monitor no longer emits. Out-of-band ORCA still rejects Unix socket endpoints, because it cannot identify a worker without a distinct address.

Separately, endpoint assignments are published only when they change. The control plane serves state of the world from its resource map, so skipping an identical assignment avoids a redundant version bump without withholding anything: unchanged clusters still appear in the response when another cluster changes, and a newly connected stream has no cached version so it always receives everything. Both properties have tests.

Breaking

Clusters must now be declared in Envoy, or by another control plane, and eds_config must point at the monitor rather than using ads: {}. The getting-started guide and control-plane/envoy.yaml show both sides. health_checks: is gone.

Notes for review

stream_endpoints is close to a copy of Async::GRPC::XDS::Service#stream_aggregated_resources, differing only in that it pins the resource type, and it reaches into that service's nested Stream. Extracting the shared loop into async-grpc-xds seems like the right end state, but felt out of scope here — happy to do it that way instead if you'd prefer.

A dedicated discovery service implies its own resource type, so clients may omit type_url; the shared Stream drops requests without one. Requests are pinned to ClusterLoadAssignment, which also stops a client using this stream to subscribe to anything else in a shared control plane.

Verification

sus (44 examples), rubocop, and bake decode:index:coverage lib (45/45) all pass locally. I could not run the Docker control-plane suite locally — no Docker available — so the api_config_source change to control-plane/envoy.yaml is only verified against a real Envoy by CI on this PR.

🤖 Generated with Claude Code

Endpoints are the only resource that changes as workers come and go.
Clusters, listeners and routes are static for a given deployment, so the
monitor no longer publishes them and no longer serves ADS.

Envoy's aggregated discovery stream is a singleton to a single management
server. Claiming it to deliver endpoints prevented anything else from
using ADS for cluster, listener and route configuration. Serving
`envoy.service.endpoint.v3.EndpointDiscoveryService` directly leaves that
transport free.

`StreamEndpoints` reuses the generic discovery messages, so this needs no
new protocol buffer definitions, and the existing state-of-the-world
stream is transport agnostic.

Out-of-band ORCA is unchanged: it is a separate gRPC service sharing the
same port, and the worker identity it depends on travels in the endpoint
hostname.

Removed with the cluster resources:

  - The `health_checks:` option, which only configured them.
  - Generation of the client-side weighted round-robin policy. Configure
    it on the cluster, pointing at the monitor's port.
  - Scheme and protocol validation, which only constrained cluster
    configuration the monitor no longer emits. Out-of-band ORCA still
    rejects Unix socket endpoints, because it cannot identify a worker
    without a distinct address.

Endpoint assignments are now published only when they change. The control
plane serves state of the world from its resource map, so this avoids a
redundant version bump without withholding anything from a subscriber.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@samuel-williams-shopify

Copy link
Copy Markdown
Contributor

Thanks for your efforts, I adapted and reworked this into #10 which has been merged and released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants