Serve dedicated cluster and endpoint discovery services

.dockerignore

@@ -4,3 +4,4 @@
 gems.locked
 pkg
 external
+examples/falcon/*.ipc

.github/workflows/test-control-plane.yaml

@@ -10,9 +10,9 @@ env:
 
 jobs:
   test:
-    name: ${{matrix.ruby}} on ${{matrix.os}}
-    runs-on: ${{matrix.os}}-latest
-    continue-on-error: ${{matrix.experimental}}
+    name: ${{ matrix.ruby }} with ${{ matrix.discovery.name }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}-latest
+    continue-on-error: ${{ matrix.experimental }}
 
     strategy:
       matrix:
@@ -24,6 +24,14 @@ jobs:
           - "3.4"
           - "4.0"
 
+        discovery:
+          - name: CDS and EDS
+            publish_clusters: "true"
+            envoy_config: ./envoy.yaml
+          - name: EDS only
+            publish_clusters: "false"
+            envoy_config: ./envoy-eds.yaml
+
         experimental: [false]
 
     steps:
@@ -32,5 +40,7 @@ jobs:
     - name: Run tests
       timeout-minutes: 10
       env:
-        RUBY_VERSION: ${{matrix.ruby}}
+        RUBY_VERSION: ${{ matrix.ruby }}
+        PUBLISH_CLUSTERS: ${{ matrix.discovery.publish_clusters }}
+        ENVOY_CONFIG: ${{ matrix.discovery.envoy_config }}
       run: docker compose -f control-plane/docker-compose.yaml up --build --exit-code-from tests

async-service-supervisor-envoy.gemspec

@@ -25,7 +25,7 @@ Gem::Specification.new do |spec|
 	spec.required_ruby_version = ">= 3.3"
 
 	spec.add_dependency "async", "~> 2.38"
-	spec.add_dependency "async-grpc-xds", "~> 0.3"
+	spec.add_dependency "async-grpc-xds", "~> 0.4"
 	spec.add_dependency "async-http"
 	spec.add_dependency "async-service-supervisor", "~> 0.20"
 	spec.add_dependency "process-metrics", "~> 0.13"

context/getting-started.md

@@ -1,6 +1,6 @@
 # Getting Started
 
-This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker endpoints to Envoy using xDS.
+This guide explains how to use `async-service-supervisor-envoy` to publish supervised worker clusters and endpoints to Envoy using xDS.
 
 ## Installation
 
@@ -16,10 +16,12 @@ The gem depends on `async-service-supervisor` and `async-grpc-xds`.
 
 `async-service-supervisor-envoy` provides:
 
-  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker endpoints through xDS.
+  - {ruby Async::Service::Supervisor::Envoy::Monitor} - A supervisor monitor that publishes worker clusters through CDS and their endpoints through EDS.
   - {ruby Async::Service::Supervisor::Envoy::Endpoint} - A small value object for endpoint state.
 
-The monitor runs an xDS control plane endpoint. Envoy connects to it using ADS and receives CDS/EDS updates derived from supervisor worker state.
+The monitor always serves a dedicated Endpoint Discovery Service stream. By default it also serves Cluster Discovery Service. It does not claim Envoy's Aggregated Discovery Service, so another control plane can use ADS for listeners, routes, and other configuration.
+
+CDS describes the logical services exposed by supervised workers, including their supported protocol, active health checks, and load-balancing policy. EDS supplies the concrete workers currently available for each service.
 
 ## Endpoint State
 
@@ -62,11 +64,59 @@ Add the monitor to your supervisor environment:
 require "async/service/supervisor/envoy"
 
 Async::Service::Supervisor::Envoy::Monitor.new(
-	bind: "http://127.0.0.1:18000"
+	bind: "http://127.0.0.1:18000",
+	management_cluster: "xds_cluster"
 )
 ```
 
-By default, workers are grouped into clusters by `state[:name]`.
+By default, workers are grouped into clusters by `state[:name]`. When cluster publication is enabled, `management_cluster` must match the static Envoy cluster used to reach the monitor.
+
+If another control plane owns cluster configuration, disable CDS publication while retaining supervisor-owned endpoint discovery:
+
+``` ruby
+Async::Service::Supervisor::Envoy::Monitor.new(
+	bind: "http://127.0.0.1:18000",
+	publish_clusters: false
+)
+```
+
+In this mode, the external control plane or bootstrap configuration must define each cluster, configure it to use the monitor's dedicated EDS service, and supply its protocol, health checks, and load-balancing policy. The cluster's EDS service name must match the worker cluster name published by the monitor.
+
+## Envoy Configuration
+
+Configure Envoy to obtain clusters from the monitor's dedicated CDS service. The monitor configures each discovered cluster to obtain its endpoints from the dedicated EDS service on the same management server:
+
+``` yaml
+dynamic_resources:
+  cds_config:
+    resource_api_version: V3
+    api_config_source:
+      api_type: GRPC
+      transport_api_version: V3
+      grpc_services:
+        - envoy_grpc:
+            cluster_name: xds_cluster
+
+static_resources:
+  clusters:
+    - name: xds_cluster
+      connect_timeout: 1s
+      type: STRICT_DNS
+      http2_protocol_options: {}
+      load_assignment:
+        cluster_name: xds_cluster
+        endpoints:
+          - lb_endpoints:
+              - endpoint:
+                  address:
+                    socket_address:
+                      address: 127.0.0.1
+                      port_value: 18000
+```
+
+The bootstrap cluster name must match the monitor's `management_cluster`. Envoy uses it for independent CDS and EDS gRPC streams; no `ads_config` is required.
+
+If a static route refers to a cluster delivered by CDS, set `validate_clusters: false` on that route configuration. Envoy can then load the route before the cluster arrives and will begin routing once CDS and EDS have warmed it.
 
 ## Custom Mapping
 
@@ -113,10 +163,14 @@ utilization_monitor = Async::Service::Supervisor::UtilizationMonitor.new(interva
 ]
 ```
 
-The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as ADS. It also configures each discovered cluster to use Envoy's client-side weighted-round-robin policy.
+The supervisor utilization monitor manages each worker's shared-memory allocation and registration. The Envoy monitor samples it through `sample_by_worker`, combines each worker's `requests_total` counter with processor usage from `process-metrics`, and serves the resulting ORCA reports from the same HTTP/2 endpoint as CDS and EDS. It also configures each discovered cluster to use Envoy's client-side weighted-round-robin policy.
 
 The first sample establishes a baseline. Subsequent reports contain normalized `cpu_utilization` and `rps_fractional` values for each worker. Reports are removed immediately when a worker disconnects.
 
+Workers are identified by the `hostname` published with each endpoint, which Envoy sends as the request authority when it opens an out-of-band reporting stream. Enabling ORCA therefore publishes one endpoint per worker rather than one per shared listener.
+
+The generated load-balancing policy uses the monitor's bind port and reporting interval. Envoy dials that port on each endpoint's own address, which reaches the monitor because it shares a network namespace with the workers.
+
 Out-of-band ORCA requires:
 
   - Envoy 1.39 or later.

context/index.yaml

@@ -9,4 +9,4 @@ files:
 - path: getting-started.md
   title: Getting Started
   description: This guide explains how to use `async-service-supervisor-envoy` to
-    publish supervised worker endpoints to Envoy using xDS.
+    publish supervised worker clusters and endpoints to Envoy using xDS.

control-plane/backend.rb

@@ -30,7 +30,7 @@
 	]
 end
 
-Sync do |task|
+Sync do
 	supervisor_endpoint = IO::Endpoint::Generic.parse(ENV.fetch("SUPERVISOR_ENDPOINT"))
 	http_endpoint = IO::Endpoint.tcp("0.0.0.0", backend_port)
 
@@ -42,16 +42,18 @@
 		scheme: "http"
 	)
 
+	state = {
+		endpoint: {
+			name: service_name,
+			scheme: :http,
+			protocols: Async::HTTP::Protocol::HTTP1.names,
+			addresses: [{address: backend_address, port: backend_port}],
+		}
+	}
+
 	worker = Async::Service::Supervisor::Worker.new(
 		endpoint: supervisor_endpoint,
-		state: {
-			endpoint: {
-				name: service_name,
-				scheme: :http,
-				protocols: Async::HTTP::Protocol::HTTP1.names,
-				addresses: [{address: backend_address, port: backend_port}],
-			}
-		}
+		state: state
 	)
 
 	worker.run

control-plane/docker-compose.yaml

@@ -10,6 +10,7 @@ services:
       - CONSOLE_OUTPUT=XTerm
       - XDS_BIND=http://0.0.0.0:18000
       - SUPERVISOR_ENDPOINT=tcp://0.0.0.0:12000
+      - PUBLISH_CLUSTERS=${PUBLISH_CLUSTERS:-true}
 
   backend-a:
     build:
@@ -49,7 +50,7 @@ services:
     image: envoyproxy/envoy:v1.32-latest
     command: ["envoy", "-c", "/etc/envoy/envoy.yaml", "--log-level", "info"]
     volumes:
-      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
+      - ${ENVOY_CONFIG:-./envoy.yaml}:/etc/envoy/envoy.yaml:ro
     ports:
       - "10000:10000"
       - "19000:19000"

control-plane/envoy-eds.yaml

@@ -0,0 +1,66 @@
+node:
+  id: async-service-supervisor-envoy-control-plane
+  cluster: async-service-supervisor-envoy-control-plane
+
+admin:
+  address:
+    socket_address:
+      address: 0.0.0.0
+      port_value: 19000
+
+static_resources:
+  listeners:
+    - name: listener_http
+      address:
+        socket_address:
+          address: 0.0.0.0
+          port_value: 10000
+      filter_chains:
+        - filters:
+            - name: envoy.filters.network.http_connection_manager
+              typed_config:
+                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
+                stat_prefix: ingress_http
+                route_config:
+                  name: local_route
+                  virtual_hosts:
+                    - name: backend
+                      domains: ["*"]
+                      routes:
+                        - match:
+                            prefix: "/"
+                          route:
+                            cluster: app-http1
+                http_filters:
+                  - name: envoy.filters.http.router
+                    typed_config:
+                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
+
+  clusters:
+    - name: app-http1
+      connect_timeout: 1s
+      type: EDS
+      eds_cluster_config:
+        service_name: app-http1
+        eds_config:
+          resource_api_version: V3
+          api_config_source:
+            api_type: GRPC
+            transport_api_version: V3
+            grpc_services:
+              - envoy_grpc:
+                  cluster_name: xds_cluster
+
+    - name: xds_cluster
+      connect_timeout: 1s
+      type: STRICT_DNS
+      load_assignment:
+        cluster_name: xds_cluster
+        endpoints:
+          - lb_endpoints:
+              - endpoint:
+                  address:
+                    socket_address:
+                      address: supervisor
+                      port_value: 18000
+      http2_protocol_options: {}

control-plane/envoy.yaml

@@ -9,12 +9,16 @@ admin:
       port_value: 19000
 
 dynamic_resources:
-  ads_config:
-    api_type: GRPC
-    transport_api_version: V3
-    grpc_services:
-      - envoy_grpc:
-          cluster_name: xds_cluster
+  # Clusters come from the supervisor's dedicated CDS stream. ADS remains
+  # unconfigured and is available to another control plane.
+  cds_config:
+    resource_api_version: V3
+    api_config_source:
+      api_type: GRPC
+      transport_api_version: V3
+      grpc_services:
+        - envoy_grpc:
+            cluster_name: xds_cluster
 
 static_resources:
   listeners:
@@ -31,6 +35,9 @@ static_resources:
                 stat_prefix: ingress_http
                 route_config:
                   name: local_route
+                  # The application cluster arrives over CDS after this static
+                  # route is loaded.
+                  validate_clusters: false
                   virtual_hosts:
                     - name: backend
                       domains: ["*"]
@@ -45,15 +52,8 @@ static_resources:
                       "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
 
   clusters:
-    - name: app-http1
-      connect_timeout: 1s
-      type: EDS
-      lb_policy: ROUND_ROBIN
-      eds_cluster_config:
-        service_name: app-http1
-        eds_config:
-          ads: {}
-          resource_api_version: V3
+    # This bootstrap cluster lets Envoy reach the supervisor. Application
+    # clusters arrive over CDS and point their EDS subscriptions back here.
     - name: xds_cluster
       connect_timeout: 1s
       type: STRICT_DNS

control-plane/readme.md

@@ -1,21 +1,31 @@
 # Control Plane Testing
 
-This scenario exercises the intended Envoy control plane topology:
+This scenario exercises both supported Envoy control plane topologies:
 
 - Falcon workers register with `Async::Service::Supervisor::Worker`.
 - Each worker publishes `state[:endpoint]` with its name, scheme, supported protocols, and concrete addresses.
-- `Async::Service::Supervisor::Envoy::Monitor` maps supervisor state into xDS endpoint resources.
-- Envoy connects to the supervisor's xDS server and subscribes to endpoint updates.
+- `Async::Service::Supervisor::Envoy::Monitor` maps supervisor state into Envoy endpoint assignments.
+- Envoy connects to the supervisor's dedicated discovery services.
 - Envoy routes HTTP traffic to the supervised Falcon workers.
 
-Envoy initiates the xDS connection. The supervisor does not call Envoy's admin API or mutate Envoy directly. Once Envoy has connected and subscribed, the supervisor streams updates over that connection. This matches the normal xDS control plane model and gives Envoy ownership of reconnects, resource ACK/NACK handling, and sidecar lifecycle.
+The default configuration enables cluster publication. Only the management cluster is declared statically in `envoy.yaml`; the application cluster is discovered through CDS and its workers are discovered through EDS.
+
+The EDS-only configuration disables cluster publication. Both the management and application clusters are declared statically in `envoy-eds.yaml`, while the application's workers are still discovered through EDS. This represents a deployment where bootstrap configuration or another control plane owns clusters.
+
+Envoy initiates the connection. The supervisor does not call Envoy's admin API or mutate Envoy directly. Once Envoy has connected and subscribed, the supervisor streams updates over that connection. This matches the normal xDS control plane model and gives Envoy ownership of reconnects, resource ACK/NACK handling, and sidecar lifecycle.
 
 ## Running Tests
 
 ``` bash
 $ docker compose -f control-plane/docker-compose.yaml up --build --exit-code-from tests
 ```
 
+To run the EDS-only scenario:
+
+``` bash
+$ PUBLISH_CLUSTERS=false ENVOY_CONFIG=./envoy-eds.yaml docker compose -f control-plane/docker-compose.yaml up --build --exit-code-from tests
+```
+
 To clean up containers and networks:
 
 ``` bash
@@ -24,11 +34,12 @@ $ docker compose -f control-plane/docker-compose.yaml down --remove-orphans
 
 ## What This Proves
 
-The test verifies the happy path for the desired architecture:
+The tests verify the happy path for both architectures:
 
-- The supervisor can run an xDS server.
+- The supervisor can run dedicated CDS and EDS services together.
+- The supervisor can serve EDS without serving or publishing CDS.
 - Supervised Falcon workers can publish endpoints.
-- Envoy can subscribe to those endpoints using ADS-backed EDS.
+- Envoy can subscribe to those endpoints using EDS.
 - Envoy can load balance requests across the supervised workers.
 
 This is a framework for lifecycle testing rather than the complete production story. Follow-up cases should cover worker removal, worker recovery, health changes, and xDS stream reconnects.

control-plane/supervisor.rb

@@ -16,7 +16,8 @@ def endpoint(value)
 Sync do
 	supervisor_endpoint = endpoint(ENV.fetch("SUPERVISOR_ENDPOINT"))
 	monitor = Async::Service::Supervisor::Envoy::Monitor.new(
-		bind: ENV.fetch("XDS_BIND")
+		bind: ENV.fetch("XDS_BIND"),
+		publish_clusters: ENV.fetch("PUBLISH_CLUSTERS", "true") == "true"
 	)
 
 	server = Async::Service::Supervisor::Server.new(