Serve endpoints from a dedicated endpoint discovery service

async-service-supervisor-envoy.gemspec

@@ -6,7 +6,7 @@ Gem::Specification.new do |spec|
 	spec.name = "async-service-supervisor-envoy"
 	spec.version = Async::Service::Supervisor::Envoy::VERSION
 
-	spec.summary = "Envoy xDS monitor for async-service-supervisor."
+	spec.summary = "Envoy endpoint discovery monitor for async-service-supervisor."
 	spec.authors = ["Samuel Williams"]
 	spec.license = "MIT"
 

context/getting-started.md

@@ -1,6 +1,6 @@
 # Getting Started
 
-This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker endpoints to Envoy using xDS.
+This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker endpoints to Envoy using EDS.
 
 ## Installation
 
@@ -16,10 +16,13 @@ The gem depends on `async-service-supervisor` and `async-grpc-xds`.
 
 `async-service-supervisor-envoy` provides:
 
-  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker endpoints through xDS.
+  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker endpoints through EDS.
+  - {ruby Async::Service::Supervisor::Envoy::EndpointDiscoveryService} - The gRPC endpoint discovery service Envoy subscribes to.
   - {ruby Async::Service::Supervisor::Envoy::Endpoint} - A small value object for endpoint state.
 
-The monitor runs an xDS control plane endpoint. Envoy connects to it using ADS and receives CDS/EDS updates derived from supervisor worker state.
+The monitor serves Envoy's Endpoint Discovery Service. Envoy subscribes to it and receives endpoint assignments derived from supervisor worker state.
+
+Endpoints are the only resource that changes as workers come and go. Clusters, listeners and routes are static for a given deployment, so the monitor does not publish them, and it does not use the aggregated discovery service. Configure them in Envoy directly, or with a separate control plane which is then free to use ADS.
 
 ## Endpoint State
 
@@ -68,6 +71,47 @@ Async::Service::Supervisor::Envoy::Monitor.new(
 
 By default, workers are grouped into clusters by `state[:name]`.
 
+## Envoy Configuration
+
+Declare each cluster in Envoy and point its endpoint configuration at the monitor:
+
+``` yaml
+static_resources:
+  clusters:
+    - name: myservice
+      connect_timeout: 1s
+      type: EDS
+      http2_protocol_options: {}
+      eds_cluster_config:
+        service_name: myservice
+        eds_config:
+          resource_api_version: V3
+          api_config_source:
+            api_type: GRPC
+            transport_api_version: V3
+            grpc_services:
+              - envoy_grpc:
+                  cluster_name: supervisor_eds
+
+    - name: supervisor_eds
+      connect_timeout: 1s
+      type: STRICT_DNS
+      http2_protocol_options: {}
+      load_assignment:
+        cluster_name: supervisor_eds
+        endpoints:
+          - lb_endpoints:
+              - endpoint:
+                  address:
+                    socket_address:
+                      address: 127.0.0.1
+                      port_value: 18000
+```
+
+`service_name` must match the cluster name the monitor publishes. Everything else about the cluster is yours: set `http2_protocol_options` when your workers serve HTTP/2, add a transport socket for TLS upstreams, and declare any active health checks. The monitor does not inspect or constrain them.
+
+If clusters are delivered by a separate control plane rather than declared statically, use the same `eds_cluster_config` block. Do not use `eds_config: {ads: {}}`, which would route the subscription to that control plane instead of the monitor.
+
 ## Custom Mapping
 
 You can customize cluster grouping, endpoint selection, and health with a delegate:
@@ -113,10 +157,29 @@ utilization_monitor = Async::Service::Supervisor::UtilizationMonitor.new(interva
 ]
 ```
 
-The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as ADS. It also configures each discovered cluster to use Envoy's client-side weighted-round-robin policy.
+The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as endpoint discovery.
 
 The first sample establishes a baseline. Subsequent reports contain normalized `cpu_utilization` and `rps_fractional` values for each worker. Reports are removed immediately when a worker disconnects.
 
+Workers are identified by the `hostname` published with each endpoint, which Envoy sends as the request authority when it opens an out-of-band reporting stream. Enabling ORCA therefore publishes one endpoint per worker rather than one per shared listener.
+
+Configure the cluster to use client-side weighted round-robin, pointing at the monitor's port:
+
+``` yaml
+load_balancing_policy:
+  policies:
+    - typed_extension_config:
+        name: envoy.load_balancing_policies.client_side_weighted_round_robin
+        typed_config:
+          "@type": type.googleapis.com/envoy.extensions.load_balancing_policies.client_side_weighted_round_robin.v3.ClientSideWeightedRoundRobin
+          enable_oob_load_report: true
+          oob_reporting_period: 1s
+          oob_reporting_config:
+            port_value: 18000
+```
+
+Set `oob_reporting_period` to the monitor's `interval`. Envoy dials this port on each endpoint's own address, which reaches the monitor because it shares a network namespace with the workers.
+
 Out-of-band ORCA requires:
 
   - Envoy 1.39 or later.

context/index.yaml

@@ -1,12 +1,12 @@
 # Automatically generated context index for Utopia::Project guides.
 # Do not edit then files in this directory directly, instead edit the guides and then run `bake utopia:project:agent:context:update`.
 ---
-description: Envoy xDS monitor for async-service-supervisor.
+description: Envoy endpoint discovery monitor for async-service-supervisor.
 metadata:
   documentation_uri: https://socketry.github.io/async-service-supervisor-envoy/
   source_code_uri: https://github.com/socketry/async-service-supervisor-envoy.git
 files:
 - path: getting-started.md
   title: Getting Started
   description: This guide explains how to use `async-service-supervisor-envoy` to
-    publish supervised worker endpoints to Envoy using xDS.
+    publish supervised worker endpoints to Envoy using EDS.

control-plane/envoy.yaml

@@ -8,14 +8,6 @@ admin:
       address: 0.0.0.0
       port_value: 19000
 
-dynamic_resources:
-  ads_config:
-    api_type: GRPC
-    transport_api_version: V3
-    grpc_services:
-      - envoy_grpc:
-          cluster_name: xds_cluster
-
 static_resources:
   listeners:
     - name: listener_http
@@ -45,20 +37,28 @@ static_resources:
                       "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
 
   clusters:
+    # The cluster is static. Only its endpoints are discovered, from the
+    # supervisor's endpoint discovery service.
     - name: app-http1
       connect_timeout: 1s
       type: EDS
       lb_policy: ROUND_ROBIN
       eds_cluster_config:
         service_name: app-http1
         eds_config:
-          ads: {}
           resource_api_version: V3
-    - name: xds_cluster
+          api_config_source:
+            api_type: GRPC
+            transport_api_version: V3
+            grpc_services:
+              - envoy_grpc:
+                  cluster_name: supervisor_eds
+
+    - name: supervisor_eds
       connect_timeout: 1s
       type: STRICT_DNS
       load_assignment:
-        cluster_name: xds_cluster
+        cluster_name: supervisor_eds
         endpoints:
           - lb_endpoints:
               - endpoint:

control-plane/readme.md

@@ -4,11 +4,13 @@ This scenario exercises the intended Envoy control plane topology:
 
 - Falcon workers register with `Async::Service::Supervisor::Worker`.
 - Each worker publishes `state[:endpoint]` with its name, scheme, supported protocols, and concrete addresses.
-- `Async::Service::Supervisor::Envoy::Monitor` maps supervisor state into xDS endpoint resources.
-- Envoy connects to the supervisor's xDS server and subscribes to endpoint updates.
+- `Async::Service::Supervisor::Envoy::Monitor` maps supervisor state into Envoy endpoint assignments.
+- Envoy connects to the supervisor's endpoint discovery service and subscribes to endpoint updates.
 - Envoy routes HTTP traffic to the supervised Falcon workers.
 
-Envoy initiates the xDS connection. The supervisor does not call Envoy's admin API or mutate Envoy directly. Once Envoy has connected and subscribed, the supervisor streams updates over that connection. This matches the normal xDS control plane model and gives Envoy ownership of reconnects, resource ACK/NACK handling, and sidecar lifecycle.
+The cluster itself is declared statically in `envoy.yaml`; only its endpoints are discovered.
+
+Envoy initiates the connection. The supervisor does not call Envoy's admin API or mutate Envoy directly. Once Envoy has connected and subscribed, the supervisor streams updates over that connection. This matches the normal xDS control plane model and gives Envoy ownership of reconnects, resource ACK/NACK handling, and sidecar lifecycle.
 
 ## Running Tests
 
@@ -26,9 +28,9 @@ $ docker compose -f control-plane/docker-compose.yaml down --remove-orphans
 
 The test verifies the happy path for the desired architecture:
 
-- The supervisor can run an xDS server.
+- The supervisor can run an endpoint discovery server.
 - Supervised Falcon workers can publish endpoints.
-- Envoy can subscribe to those endpoints using ADS-backed EDS.
+- Envoy can subscribe to those endpoints using EDS.
 - Envoy can load balance requests across the supervised workers.
 
 This is a framework for lifecycle testing rather than the complete production story. Follow-up cases should cover worker removal, worker recovery, health changes, and xDS stream reconnects.

control-plane/test/envoy.rb

@@ -47,18 +47,25 @@ def eventually(timeout: 20, interval: 0.5)
 		expect(responses.filter_map{|response| response["x-backend-id"]}.uniq.sort).to be == ["backend-a", "backend-b"]
 	end
 
-	it "loads the xDS cluster from the supervisor monitor" do
+	it "discovers cluster endpoints from the supervisor monitor" do
 		uri = admin_uri + "/clusters?format=json"
 
-		cluster_status = eventually do
+		host_statuses = eventually do
 			if (response = Net::HTTP.get_response(uri)).code.to_i == 200
 				clusters = JSON.parse(response.body)
-				clusters.fetch("cluster_statuses").find do |cluster|
+				cluster_status = clusters.fetch("cluster_statuses").find do |cluster|
 					cluster.fetch("name") == "app-http1"
 				end
+
+				hosts = cluster_status&.fetch("host_statuses", nil)
+				hosts if hosts&.size == 2
 			end
 		end
 
-		expect(cluster_status).not.to be_nil
+		addresses = host_statuses.map do |host|
+			host.fetch("address").fetch("socket_address").fetch("port_value")
+		end
+
+		expect(addresses.sort).to be == [9292, 9293]
 	end
 end

guides/getting-started/readme.md

@@ -1,6 +1,6 @@
 # Getting Started
 
-This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker endpoints to Envoy using xDS.
+This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker endpoints to Envoy using EDS.
 
 ## Installation
 
@@ -16,10 +16,13 @@ The gem depends on `async-service-supervisor` and `async-grpc-xds`.
 
 `async-service-supervisor-envoy` provides:
 
-  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker endpoints through xDS.
+  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker endpoints through EDS.
+  - {ruby Async::Service::Supervisor::Envoy::EndpointDiscoveryService} - The gRPC endpoint discovery service Envoy subscribes to.
   - {ruby Async::Service::Supervisor::Envoy::Endpoint} - A small value object for endpoint state.
 
-The monitor runs an xDS control plane endpoint. Envoy connects to it using ADS and receives CDS/EDS updates derived from supervisor worker state.
+The monitor serves Envoy's Endpoint Discovery Service. Envoy subscribes to it and receives endpoint assignments derived from supervisor worker state.
+
+Endpoints are the only resource that changes as workers come and go. Clusters, listeners and routes are static for a given deployment, so the monitor does not publish them, and it does not use the aggregated discovery service. Configure them in Envoy directly, or with a separate control plane which is then free to use ADS.
 
 ## Endpoint State
 
@@ -68,6 +71,47 @@ Async::Service::Supervisor::Envoy::Monitor.new(
 
 By default, workers are grouped into clusters by `state[:name]`.
 
+## Envoy Configuration
+
+Declare each cluster in Envoy and point its endpoint configuration at the monitor:
+
+``` yaml
+static_resources:
+  clusters:
+    - name: myservice
+      connect_timeout: 1s
+      type: EDS
+      http2_protocol_options: {}
+      eds_cluster_config:
+        service_name: myservice
+        eds_config:
+          resource_api_version: V3
+          api_config_source:
+            api_type: GRPC
+            transport_api_version: V3
+            grpc_services:
+              - envoy_grpc:
+                  cluster_name: supervisor_eds
+
+    - name: supervisor_eds
+      connect_timeout: 1s
+      type: STRICT_DNS
+      http2_protocol_options: {}
+      load_assignment:
+        cluster_name: supervisor_eds
+        endpoints:
+          - lb_endpoints:
+              - endpoint:
+                  address:
+                    socket_address:
+                      address: 127.0.0.1
+                      port_value: 18000
+```
+
+`service_name` must match the cluster name the monitor publishes. Everything else about the cluster is yours: set `http2_protocol_options` when your workers serve HTTP/2, add a transport socket for TLS upstreams, and declare any active health checks. The monitor does not inspect or constrain them.
+
+If clusters are delivered by a separate control plane rather than declared statically, use the same `eds_cluster_config` block. Do not use `eds_config: {ads: {}}`, which would route the subscription to that control plane instead of the monitor.
+
 ## Custom Mapping
 
 You can customize cluster grouping, endpoint selection, and health with a delegate:
@@ -113,10 +157,29 @@ utilization_monitor = Async::Service::Supervisor::UtilizationMonitor.new(interva
 ]
 ```
 
-The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as ADS. It also configures each discovered cluster to use Envoy's client-side weighted-round-robin policy.
+The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as endpoint discovery.
 
 The first sample establishes a baseline. Subsequent reports contain normalized `cpu_utilization` and `rps_fractional` values for each worker. Reports are removed immediately when a worker disconnects.
 
+Workers are identified by the `hostname` published with each endpoint, which Envoy sends as the request authority when it opens an out-of-band reporting stream. Enabling ORCA therefore publishes one endpoint per worker rather than one per shared listener.
+
+Configure the cluster to use client-side weighted round-robin, pointing at the monitor's port:
+
+``` yaml
+load_balancing_policy:
+  policies:
+    - typed_extension_config:
+        name: envoy.load_balancing_policies.client_side_weighted_round_robin
+        typed_config:
+          "@type": type.googleapis.com/envoy.extensions.load_balancing_policies.client_side_weighted_round_robin.v3.ClientSideWeightedRoundRobin
+          enable_oob_load_report: true
+          oob_reporting_period: 1s
+          oob_reporting_config:
+            port_value: 18000
+```
+
+Set `oob_reporting_period` to the monitor's `interval`. Envoy dials this port on each endpoint's own address, which reaches the monitor because it shares a network namespace with the workers.
+
 Out-of-band ORCA requires:
 
   - Envoy 1.39 or later.

lib/async/service/supervisor/envoy.rb

@@ -6,6 +6,7 @@
 require_relative "envoy/version"
 require_relative "envoy/delegate"
 require_relative "envoy/endpoint"
+require_relative "envoy/endpoint_discovery_service"
 require_relative "envoy/monitor"
 require_relative "envoy/orca_service"
 require_relative "envoy/supervised"