Serve endpoints from a dedicated endpoint discovery service - #9
Closed
epk wants to merge 1 commit into
Closed
Conversation
Endpoints are the only resource that changes as workers come and go.
Clusters, listeners and routes are static for a given deployment, so the
monitor no longer publishes them and no longer serves ADS.
Envoy's aggregated discovery stream is a singleton to a single management
server. Claiming it to deliver endpoints prevented anything else from
using ADS for cluster, listener and route configuration. Serving
`envoy.service.endpoint.v3.EndpointDiscoveryService` directly leaves that
transport free.
`StreamEndpoints` reuses the generic discovery messages, so this needs no
new protocol buffer definitions, and the existing state-of-the-world
stream is transport agnostic.
Out-of-band ORCA is unchanged: it is a separate gRPC service sharing the
same port, and the worker identity it depends on travels in the endpoint
hostname.
Removed with the cluster resources:
- The `health_checks:` option, which only configured them.
- Generation of the client-side weighted round-robin policy. Configure
it on the cluster, pointing at the monitor's port.
- Scheme and protocol validation, which only constrained cluster
configuration the monitor no longer emits. Out-of-band ORCA still
rejects Unix socket endpoints, because it cannot identify a worker
without a distinct address.
Endpoint assignments are now published only when they change. The control
plane serves state of the world from its resource map, so this avoids a
redundant version bump without withholding anything from a subscriber.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Thanks for your efforts, I adapted and reworked this into #10 which has been merged and released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduces the published xDS surface to EDS, so the monitor does what its name suggests and nothing more.
Why
Envoy's aggregated discovery stream is a singleton to a single management server. By serving endpoints over ADS, the supervisor claimed that stream, which prevented anything else — Envoy's own bootstrap, or a separate control plane — from using ADS for cluster, listener and route configuration.
Endpoints are also the only resource that actually changes as workers come and go. Everything a
Clustercarries is static for a given deployment.Worth noting:
control-plane/envoy.yamlwas already running this way. It has nocds_config, so Envoy never subscribed to CDS, and the cluster resources the monitor built were discarded unread.What changed
Serves
envoy.service.endpoint.v3.EndpointDiscoveryService/StreamEndpointsdirectly.StreamEndpointsreuses the generic discovery messages, so this needs no new protocol buffer definitions, andAsync::GRPC::XDS::Service::Streamis transport agnostic and reusable as is.Out-of-band ORCA is untouched. It is a separate gRPC service sharing the same port, and the worker identity it depends on travels in the endpoint
hostname.Removed along with the cluster resources:
health_checks:option, which only configured them.Separately, endpoint assignments are published only when they change. The control plane serves state of the world from its resource map, so skipping an identical assignment avoids a redundant version bump without withholding anything: unchanged clusters still appear in the response when another cluster changes, and a newly connected stream has no cached version so it always receives everything. Both properties have tests.
Breaking
Clusters must now be declared in Envoy, or by another control plane, and
eds_configmust point at the monitor rather than usingads: {}. The getting-started guide andcontrol-plane/envoy.yamlshow both sides.health_checks:is gone.Notes for review
stream_endpointsis close to a copy ofAsync::GRPC::XDS::Service#stream_aggregated_resources, differing only in that it pins the resource type, and it reaches into that service's nestedStream. Extracting the shared loop intoasync-grpc-xdsseems like the right end state, but felt out of scope here — happy to do it that way instead if you'd prefer.A dedicated discovery service implies its own resource type, so clients may omit
type_url; the sharedStreamdrops requests without one. Requests are pinned toClusterLoadAssignment, which also stops a client using this stream to subscribe to anything else in a shared control plane.Verification
sus(44 examples),rubocop, andbake decode:index:coverage lib(45/45) all pass locally. I could not run the Docker control-plane suite locally — no Docker available — so theapi_config_sourcechange tocontrol-plane/envoy.yamlis only verified against a real Envoy by CI on this PR.🤖 Generated with Claude Code