Pod connectivity issues when worker nodes placed in different subnets

[fruboes]

Describe the bug

Our installation (UPI) consists of the following types of nodes:

• masters (VMs running on ovirt - created manually)
• workers-vm (also VMs as name suggest)
• workers-bm (BMs)
• bootstrap node (VM)

Network configuration is done statically during the first PXE boot (by setting ip and nameserver kernel parameters; in all subsequent boots, machines are directed to proceed to boot from local disks). Everything works as expected (i.e. installation results with a clean oc get co output) if all nodes (i.e. both VMs and BMs) are placed in the same subnet.

When BMs and VMs are placed in a separate subnets (target setup) pod2pod connectivity is lost for pods running on different types of machines (BM/VM), i.e.:

• pod@worker-vm is not able to connect to pod@worker-bm (and viceversa, similarly for worker-bm/master-vm traffic)
• pod@worker-bm1 properly connects to pod@worker-bm2 (i.e. traffic between different nodes of same class is ok)
• pod@worker-vm1 properly connects to pod@worker-vm2 (also to pod@master-vm)

Most notable result is inability to authenticate via oc login, as router-default pods typically land on bm workers, while by design oauth pods run on master (VM) nodes - resulting with "Unable to connect to the server: EOF" error message when trying to oc login (as consequence - we are not able to complete the installation, as we never get a clean oc get co printout).

As for underlying network connectivity, everything seems to properly pass intermediate gateways/firewalls, which demonstrates the test below. At the same time we see different behavior depending if communication goes to pod from machine of same class (e.g. pod runs on vm worker, traffic comes also from vm worker) or different class. We performed the following:

1. started dbg session on worker-vm1 (oc debug node/worker-vm1), run tcpdump on port 6081 with src=any (i.e. geneve)
2. started dbg session on worker-bm1 (oc debug node/worker-bm1), try pinging IP of a pod that is running on worker-vm1 (ingress-canary) - no ping response (100% packet loss)
3. on the worker-vm1 dbg session we note, that ICMP echo request did reach the host (IP of the target pod visible in geneve packet with type of ICMP echo request), but no reply packet is visible
4. started dbg session on worker-vm2 - try pinging same IP of a pod that is running on worker-vm1 (ingress-canary) - ping runs ok with responses visible
5. this time inside the worker-vm1 dbg session we see both ICMP request and reply in tcpdump output

We did a similar test trying to connect (nc -v IP 6443) to oauth pod (running on master-vm machine). Connection is possible if done from worker-vm* machine, for worker-bm* unfortunately not.

Please let us know if there is anything further we can do to debug/resolve the problem. Having different types of machines placed in separate subnets is a must-have for us.

Version

4.21.0-okd-scos.9, UPI baremetal installation

Reproducibility

100%

Log Bundle

File link download: https://mega.nz/file/WdQRySLB#ksMLw4wihWotYdr8FCYPT4vDdURwmx0E2HF6NR1pFOU
(gpg encrypted, I'll send decrypt password via email on request)

---

ClusterID: 2f5fff89-d7ca-4822-ac5d-757c699d4793
ClientVersion: 4.21.0-okd-scos.9
ClusterVersion: Stable at "4.21.0-okd-scos.9"
ClusterOperators:
clusteroperator/authentication is not available (OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.okdpoz.cis.gov.pl/healthz": EOF) because OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.okdpoz.cis.gov.pl/healthz": EOF
clusteroperator/console is not available (RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.okdpoz.cis.gov.pl returns '503 Service Unavailable') because RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.okdpoz.cis.gov.pl returns '503 Service Unavailable'