Wrap createOrUpdate in RetryOnConflict - #76
openshift-merge-bot[bot] merged 1 commit into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review. 📝 WalkthroughWalkthroughThe controller compares existing and desired FRR resource specs and labels. It skips updates when relevant values are unchanged. When an update is required, it refreshes the resource version and retries after conflicts by rereading the current object. Priority: ➖ Normal — Schedule the controller reconciliation fix because it stabilizes conflict handling and prevents no-op update storms across CUDN, FRRConfiguration, and RouteAdvertisements resources. Merge Risk: ⚪ Minimal · up to The FRR update flow now avoids unnecessary writes and retries transient conflicts, with no identified merge-blocking risk. 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
The createOrUpdate helper performed a naive Get→Update without retry-on-conflict and without checking whether the spec actually changed. This caused two problems on live clusters: 1. Optimistic concurrency conflicts: when another controller (e.g. OVN-K CUDN controller) modified the same object between Get and Update, the resourceVersion mismatch caused a conflict error. The reconciler marked itself Degraded, which triggered a new reconcile, creating a self-sustaining Degraded/Ready flap storm. 2. No-op update loop: even when spec and labels were identical, the unconditional Update bumped resourceVersion, which triggered watches on CUDN/RouteAdvertisements/FRRConfiguration objects, which triggered new reconciles in an infinite loop. Fix by: - Wrapping the Update in retry.RetryOnConflict so transient conflicts are retried automatically with exponential backoff. - Adding specUnchanged() to compare spec and labels before updating, skipping the write entirely when nothing changed. Validated on a live OCP cluster with CUDNBgpRouting and CUDNBgpConfig. Co-authored-by: Cursor <cursoragent@cursor.com>
58b5e99 to
a6171a2
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jpinsonneau The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
@jpinsonneau: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
…onflict-retry Wrap createOrUpdate in RetryOnConflict
…onflict-retry Wrap createOrUpdate in RetryOnConflict
Summary
While testing netobserv with bgp-cloud-connector, I hit an infinite reconcile loop caused by two problems in
createOrUpdate:Updatecalls bumpedresourceVersioneven when nothing changed, re-triggering watches in a loop.GetandUpdate, the staleresourceVersioncaused a conflict error. The reconciler marked itself Degraded and requeued, producing a Degraded/Ready flap storm.Problem 1 has since been fixed on main (via
specEqual/specSatisfiedsubset matching).This PR addresses problem 2: the
Updatecall is now wrapped inretry.RetryOnConflict, so transient conflicts are retried immediately with exponential backoff instead of failing the entire reconcile and waiting 30 seconds for the next requeue.Test plan
go build ./...passesgo test ./internal/controller/... -count=1 -shortpasses