Leave the route server peers alone when no router nodes match - #70
Conversation
An empty node list was reconciled like any other: nodesByAZ comes out empty, so every zone's desired address set is empty, and every managed peer on every endpoint is deleted as stale. The selector matching nothing is not rare -- moving the router label during a rollout does it -- and the result was BGP torn down across the cluster and rebuilt from scratch once a node came back. Nothing has asked for the peers to be released at that point, so leave them as they stand. Cleanup already does the releasing, and it runs on deletion, where the intent is unambiguous. The guard sits in ReconcileNodes rather than in reconcileRouteServerPeers, so that a zone losing its last node while other zones still have one keeps deleting the peers it should.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthrough
Possibly related PRs
Suggested reviewers: Merge Risk: ⚪ Minimal · up to The change preserves existing route-server peers when no router nodes match while retaining stale-peer cleanup when only a zone becomes empty; no actionable merge-blocking risk remains after normal checks and review. 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: frobware The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
Pipeline controller notification No second-stage tests were triggered for this PR. This can happen when:
Use |
f90d361
into
openshift:main
An empty router node list is reconciled like any other.
reconcileRouteServerPeersbuildsnodesByAZfrom the node list, so an empty list leaves the desired address set empty for every zone, and the loop that deletes managed peers absent from that set deletes all of them. Every BGP session in the cluster goes down together.The selector matching nothing is not a rare state. Moving the router label during a rollout does it, and so does any window where the labelled nodes are briefly gone. Nothing has asked for the peers to be released at that point, and recovering means recreating every peer and renegotiating every session, so the reconcile now returns early and leaves them as it found them.
Cleanupalready does the releasing, and it runs on deletion, where the intent is unambiguous.The guard sits in
ReconcileNodesrather than inreconcileRouteServerPeers, and that placement is the point. A zone that loses its last node while other zones still have one has genuinely stale peers, and those must still be deleted; it is only the case of no nodes anywhere that is meaningless. Keeping the guard a level up leaves the lower function's "not in the desired set means stale" semantics untouched, which is whyTestReconcilePeers_DeleteStalePeerstill passes unmodified: it passesnilnodes to that function directly and asserts the peer is deleted.GCP has the same defect and the same fix, in #63. This is separate so that it can land on its own rather than behind a GCP review.
Worth knowing: #13 is open in draft and rewrites
reconcileRouteServerPeerssubstantially. It keeps the samedesiredIPs := nodesByAZ[az]shape, so the bug survives that rewrite intact. The two should not conflict textually, since this change is in the calling function, and the guard would still protect the rewritten code.Test plan
Two unit tests, written red first.
TestReconcileNodes_EmptyNodeListLeavesPeersAlonefails before the change withdeleted 1 peers, want 0and passes after.TestReconcileNodes_ShrunkNodeListStillDeletespasses both before and after, and exists so the early return cannot quietly widen: one node inus-east-1aand none inus-east-1bmust still deletepeer-ep-b1.go test ./internal/...passes,make lintreports no issues.This has not been exercised against a live AWS cluster. The equivalent fix in #63 was verified on a live GCP cluster, where stripping the router label from every node left the estate intact and restoring it brought all sessions back with no cloud operations at all.