feature: zone aware routing - Option 1

[szuecs]

• feature: zone aware eskip.LBEndpoints
• feature: routesrv add zone aware API endpoint /routes/:zone

[greeshma1196]

Test run on 19/03:
I created an application

zkubectl -n zone-aware-routing get all
NAME                          READY   STATUS    RESTARTS   AGE
pod/app1-v1-cdd55fdd9-4rk87   1/1     Running   0          3h1m
pod/app1-v1-cdd55fdd9-8qjbb   1/1     Running   0          20m
pod/app1-v1-cdd55fdd9-9ndls   1/1     Running   0          3h27m
pod/app1-v1-cdd55fdd9-hwxcl   1/1     Running   0          3h1m
pod/app1-v1-cdd55fdd9-jdrnx   1/1     Running   0          3h27m
pod/app1-v1-cdd55fdd9-mj86h   1/1     Running   0          3h27m
pod/app1-v1-cdd55fdd9-vv454   1/1     Running   0          80s
pod/app1-v1-cdd55fdd9-x9g24   1/1     Running   0          94s
pod/app1-v1-cdd55fdd9-xvnq7   1/1     Running   0          13m

Endpointslices for the above application:

kubectl -n zone-aware-routing get endpointslices app1-v1-cwjdl -oyaml | yq
addressType: IPv6
apiVersion: discovery.k8s.io/v1
endpoints:
  - addresses:
      - 2a05:d024:90:1104:565e::8
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-mj86h
      namespace: zone-aware-routing
      uid: 69ae5fc1-b155-4ced-a2c6-c095965b859d
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1104:565e::d
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-jdrnx
      namespace: zone-aware-routing
      uid: eab3475a-2537-4a51-93ab-c373596d96a8
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1104:565e::b
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-9ndls
      namespace: zone-aware-routing
      uid: 689323c4-d8b0-4c24-917e-f0fcfa666f95
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1103:b425::a
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-3-212.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-4rk87
      namespace: zone-aware-routing
      uid: ec6d33a8-5f86-45e3-b949-895bc0d59cd4
    zone: eu-central-1a
  - addresses:
      - 2a05:d024:90:1103:b425::7
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-3-212.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-hwxcl
      namespace: zone-aware-routing
      uid: a924265d-d2db-4fa3-838d-757128f39b45
    zone: eu-central-1a
  - addresses:
      - 2a05:d024:90:1103:4b45::8
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-3-106.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-xvnq7
      namespace: zone-aware-routing
      uid: f247f345-e105-4d8b-a292-cda577323c2f
    zone: eu-central-1a
  - addresses:
      - 2a05:d024:90:1105:c367::4
    conditions:
      ready: false
      serving: true
      terminating: true
    nodeName: ip-172-31-20-21.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-x9g24
      namespace: zone-aware-routing
      uid: f9e7a920-b8b4-4088-ab47-a0392b235fff
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:2a8::4
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-69.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-vv454
      namespace: zone-aware-routing
      uid: fc10ab35-e0f5-45df-945d-2ee2492eef1e
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:2a8::7
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-69.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-xrhjw
      namespace: zone-aware-routing
      uid: 100b0a9d-cbfd-4759-be8e-1cc99b2d8b3f
    zone: eu-central-1c
kind: EndpointSlice
metadata:
  creationTimestamp: "2026-03-19T12:35:42Z"
  generateName: app1-v1-
  generation: 87
  labels:
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    kubernetes.io/service-name: app1-v1
    stack-version: v1
    stackset: app1
  name: app1-v1-cwjdl
  namespace: zone-aware-routing
  ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: app1-v1
      uid: b94959a9-4d9c-4254-a1ea-9290a809b159
  resourceVersion: "73922010"
  uid: 2c00ea63-e807-4f4d-b76e-c13d235fe9b9
ports:
  - name: port-0-0
    port: 9090
    protocol: TCP

Port-forwarded a routesrv pod, this pod's zone is set to eu-central-1c

Response

curl localhost:9990/routes/eu-central-1c
kube_rg__zone_aware_routing__app1_v1_25rzp__all__0_0: Path("/test") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?|app1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Weight(3) && Method("GET") && Header("X-Forwarded-Proto", "https") -> disableAccessLog(2, 3, 404, 429) -> fifo(2000, 20, "1s") -> oauthTokeninfoAnyKV("realm", "/services") -> unverifiedAuditLog("sub") -> oauthTokeninfoAllScope("uid") -> flowId("reuse") -> forwardToken("X-TokenInfo-Forward", "uid", "scope", "realm") -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}") -> stateBagToTag("auth-user", "client.uid") -> <powerOfRandomNChoices, "http://[2a05:d024:90:1105:2a8::4]:9090", "http://[2a05:d024:90:1105:2a8::7]:9090", "http://[2a05:d024:90:1105:fb28::7]:9090">;

This returns the correct set of Endpoints. But when I got the same set of endpoints when I queried for eu-central-1a and eu-central-1b. This is not the expected behavior. Ideally it should fallback and return all endpoints.

Updated JTBD:
When I deploy skipper with routesrv in a Kubernetes cluster with pods distributed across multiple availability zones / I want to have zone-aware routing that automatically adapts based on endpoint availability / So that requests prefer same-zone backends for low latency when sufficient endpoints exist, but gracefully fall back to cross-zone routing when a zone is under-provisioned

Updated Tasks:

• Remove zone filtering from dataclient - return all endpoints with zone metadata
• Ensure route construction always creates LBEndpoints with zone labels
• Implement threshold-based zone filtering in routesrv when serving routes
• Run tests to verify fallback behavior

[greeshma1196]

Test run on 20/3:

The application has the following:
Zone A - 0 endpoints
Zone B - 3 endpoints
Zone C - 6 endpoints

kubectl -n zone-aware-routing get endpointslices app1-v1-cwjdl -oyaml | yq
addressType: IPv6
apiVersion: discovery.k8s.io/v1
endpoints:
  - addresses:
      - 2a05:d024:90:1104:565e::10
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-lt4bp
      namespace: zone-aware-routing
      uid: 5217d86f-bdd9-43c6-9ed5-b0b9656a5d39
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1104:565e::d
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-p6wvq
      namespace: zone-aware-routing
      uid: 81ae26e3-fd13-46ff-a182-5d31271a9fdf
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1104:565e::b
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-8-220.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-kbk76
      namespace: zone-aware-routing
      uid: f6cc804c-c466-431a-9cac-69610f358e8d
    zone: eu-central-1b
  - addresses:
      - 2a05:d024:90:1105:43ef::c
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-143.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-v7s9p
      namespace: zone-aware-routing
      uid: 37483208-2726-4b46-bc70-24381e2f561c
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:43ef::b
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-143.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-jz99r
      namespace: zone-aware-routing
      uid: ec278009-4007-4210-ac2f-0c2d07c6d7a2
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:43ef::12
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-143.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-ft7lj
      namespace: zone-aware-routing
      uid: eba2cced-17cb-4e5d-bf01-dd197f45ee23
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:43ef::8
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-143.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-c9xb7
      namespace: zone-aware-routing
      uid: 31b139fa-c61b-4842-8358-ce62ec826d11
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:43ef::14
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-18-143.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-2pqjj
      namespace: zone-aware-routing
      uid: bce6bb49-5363-48d0-88d1-35030b4caf12
    zone: eu-central-1c
  - addresses:
      - 2a05:d024:90:1105:bdf5::6
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-172-31-23-22.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: app1-v1-cdd55fdd9-4t9pv
      namespace: zone-aware-routing
      uid: c6f4b7f0-30c3-4ef4-a544-c6957ea933bb
    zone: eu-central-1c
kind: EndpointSlice
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2026-03-20T14:19:07Z"
  creationTimestamp: "2026-03-19T12:35:42Z"
  generateName: app1-v1-
  generation: 201
  labels:
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    kubernetes.io/service-name: app1-v1
    stack-version: v1
    stackset: app1
  name: app1-v1-cwjdl
  namespace: zone-aware-routing
  ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: app1-v1
      uid: b94959a9-4d9c-4254-a1ea-9290a809b159
  resourceVersion: "74745418"
  uid: 2c00ea63-e807-4f4d-b76e-c13d235fe9b9
ports:
  - name: port-0-0
    port: 9090
    protocol: TCP

Results:
/routes/eu-central-1a - returns all 9 endpoints as expected

curl localhost:9990/routes/eu-central-1a 2>/dev/null | grep "app1_v1_25rzp__all__0_0"
kube_rg__zone_aware_routing__app1_v1_25rzp__all__0_0: Path("/test") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?|app1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Weight(3) && Method("GET") && Header("X-Forwarded-Proto", "https") -> disableAccessLog(2, 3, 404, 429) -> fifo(2000, 20, "1s") -> oauthTokeninfoAnyKV("realm", "/services") -> unverifiedAuditLog("sub") -> oauthTokeninfoAllScope("uid") -> flowId("reuse") -> forwardToken("X-TokenInfo-Forward", "uid", "scope", "realm") -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}") -> stateBagToTag("auth-user", "client.uid") -> <powerOfRandomNChoices, "http://[2a05:d024:90:1104:565e::10]:9090", "http://[2a05:d024:90:1104:565e::b]:9090", "http://[2a05:d024:90:1104:565e::d]:9090", "http://[2a05:d024:90:1105:43ef::12]:9090", "http://[2a05:d024:90:1105:43ef::14]:9090", "http://[2a05:d024:90:1105:43ef::8]:9090", "http://[2a05:d024:90:1105:43ef::b]:9090", "http://[2a05:d024:90:1105:43ef::c]:9090", "http://[2a05:d024:90:1105:bdf5::6]:9090">;

/routes/eu-central-1b - returns 3 endpoints as expected

curl localhost:9990/routes/eu-central-1b 2>/dev/null | grep "app1_v1_25rzp__all__0_0"
kube_rg__zone_aware_routing__app1_v1_25rzp__all__0_0: Path("/test") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?|app1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Weight(3) && Method("GET") && Header("X-Forwarded-Proto", "https") -> disableAccessLog(2, 3, 404, 429) -> fifo(2000, 20, "1s") -> oauthTokeninfoAnyKV("realm", "/services") -> unverifiedAuditLog("sub") -> oauthTokeninfoAllScope("uid") -> flowId("reuse") -> forwardToken("X-TokenInfo-Forward", "uid", "scope", "realm") -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}") -> stateBagToTag("auth-user", "client.uid") -> <powerOfRandomNChoices, "http://[2a05:d024:90:1104:565e::10]:9090", "http://[2a05:d024:90:1104:565e::b]:9090", "http://[2a05:d024:90:1104:565e::d]:9090">;

/routes/eu-central-1c - returns 6 endpoints as expected

curl localhost:9990/routes/eu-central-1c 2>/dev/null | grep "app1_v1_25rzp__all__0_0"
kube_rg__zone_aware_routing__app1_v1_25rzp__all__0_0: Path("/test") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?|app1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Host("^(app1-v1[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$") && Weight(3) && Method("GET") && Header("X-Forwarded-Proto", "https") -> disableAccessLog(2, 3, 404, 429) -> fifo(2000, 20, "1s") -> oauthTokeninfoAnyKV("realm", "/services") -> unverifiedAuditLog("sub") -> oauthTokeninfoAllScope("uid") -> flowId("reuse") -> forwardToken("X-TokenInfo-Forward", "uid", "scope", "realm") -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}") -> stateBagToTag("auth-user", "client.uid") -> <powerOfRandomNChoices, "http://[2a05:d024:90:1105:43ef::12]:9090", "http://[2a05:d024:90:1105:43ef::14]:9090", "http://[2a05:d024:90:1105:43ef::8]:9090", "http://[2a05:d024:90:1105:43ef::b]:9090", "http://[2a05:d024:90:1105:43ef::c]:9090", "http://[2a05:d024:90:1105:bdf5::6]:9090">;

Context

Currently dataclient does not do filtering based on zone. There should be two code paths:

• if routesrv is configured then dataclient needs to return all endpointslices
• if skipper directly calls the dataclient it needs to filter based on zone

JTBD

When I configure skipper to use dataclient directly (without routesrv) in a multi-zone Kubernetes cluster / I want the dataclient to filter endpoints by my configured zone with threshold-based fallback / So that my skipper instance gets zone-aware routes without needing routesrv as an intermediary

When I configure routesrv to use dataclient to serve zone-specific routes / I want the dataclient to return all endpoints with zone metadata unfiltered / So that routesrv can perform dynamic per-request zone filtering based on the requested zone

Tasks

• Modify GetEndpointSlicesByService and GetEndpointSlicesByTarget to check zone parameter for filtering mode
• When zone is empty (routesrv mode), return all endpoints with zone metadata
• When zone is set (direct skipper mode), filter by zone with threshold fallback
• Verify dataclient and routesrv modes work correctly

[szuecs]

Kubernetes deprecated endpoints in favor of endpointslices and we used this switch only to enable it when it was more recent.
I think we can get rid of this.
Let's do the cleanup in a different PR.

[szuecs]

some small changes should be enough

[szuecs]

rebased on master

[szuecs]

👍

[MustafaSaber]

I did one run, but I think I need to have another deeper run after discussion.

[greeshma1196]

Somewhere in the merge process this PR got messed up, I do not see changes from this PR here :/
#3927

[ponimas]

unrelated question.
is that situation possible - left[i].Address == right[i].Address && left[i].Zone != right[i].Zone?

[greeshma1196]

I do not think this is possible, because each pod gets assigned a unique IP within a cluster and a pod exists in a zone. Probably this could come if there is stale data, i.e. a pod dies, new one comes and there is some time with stale data. But I need to double check on this.

[szuecs]

Any time there is only one unique IP per pod, stale data would not change it, because then you would also have one unique IP per pod, maybe the pod does not exist, but the IP is unique at the old point of time.

[szuecs]

I added a couple of missing tests that are only somewhat related to routesrv. These basically test that the change does not change valkey endpoints (these tests were completely missing before).

[ponimas]

👍

[szuecs]

👍