Skip to content

Wrap createOrUpdate in RetryOnConflict - #76

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
jpinsonneau:fix/createorupdate-conflict-retry
Sep 9, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
jpinsonneau:fix/createorupdate-conflict-retry

Conversation

@jpinsonneau

@jpinsonneau jpinsonneau commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

While testing netobserv with bgp-cloud-connector, I hit an infinite reconcile loop caused by two problems in createOrUpdate:

  1. No-op update storm — unconditional Update calls bumped resourceVersion even when nothing changed, re-triggering watches in a loop.
  2. Optimistic concurrency conflicts — when another controller (e.g. OVN-K's CUDN controller) modified the same object between Get and Update, the stale resourceVersion caused a conflict error. The reconciler marked itself Degraded and requeued, producing a Degraded/Ready flap storm.

Problem 1 has since been fixed on main (via specEqual/specSatisfied subset matching).

This PR addresses problem 2: the Update call is now wrapped in retry.RetryOnConflict, so transient conflicts are retried immediately with exponential backoff instead of failing the entire reconcile and waiting 30 seconds for the next requeue.

Test plan

  • go build ./... passes
  • go test ./internal/controller/... -count=1 -short passes
  • Deploy on live OCP cluster and verify no conflict errors in operator logs when OVN-K modifies the same CUDN concurrently

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4cdbb350-3167-4e13-8c2f-be9a003e1e7d

📥 Commits

Reviewing files that changed from the base of the PR and between 4c6fe47 and a6171a2.

📒 Files selected for processing (1)
  • internal/controller/frr.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • internal/controller/frr.go

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.


📝 Walkthrough

Walkthrough

The controller compares existing and desired FRR resource specs and labels. It skips updates when relevant values are unchanged. When an update is required, it refreshes the resource version and retries after conflicts by rereading the current object.

Priority: ➖ Normal — Schedule the controller reconciliation fix because it stabilizes conflict handling and prevents no-op update storms across CUDN, FRRConfiguration, and RouteAdvertisements resources.

Merge Risk: ⚪ Minimal · up to a6171

The FRR update flow now avoids unnecessary writes and retries transient conflicts, with no identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS. The pull request changes only internal/controller/frr.go. The diff contains no test files or Ginkgo declarations, and the changed file has no It, Describe, Context, When, or similar te…
Test Structure And Quality ✅ Passed PASS: The pull request changes only internal/controller/frr.go; git diff HEAD^ HEAD shows no test-file changes. The repository’s relevant internal/controller/frr_test.go uses standard Go `testin…
Microshift Test Compatibility ✅ Passed PASS: The pull request changes only internal/controller/frr.go (+15/-11). The parent-to-HEAD diff contains no new Ginkgo tests and no changes under test/e2e. Therefore, the MicroShift Test Compati…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS. The pull request changes only internal/controller/frr.go. The parent-to-HEAD diff contains no new or modified test files and no new Ginkgo declarations. Therefore, the SNO compatibility check …
Topology-Aware Scheduling Compatibility ✅ Passed The patch changes only createOrUpdate reconciliation behavior in internal/controller/frr.go: it adds conflict retries and skips unchanged updates. The diff contains no new or modified affinity, to…
Ote Binary Stdout Contract ✅ Passed PASS. The pull request changes only internal/controller/frr.go. The changed lines add retry.RetryOnConflict and move Kubernetes object reads and updates into the retry callback. They add no stdout…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The pull request changes only internal/controller/frr.go. The exact patch adds no Ginkgo tests and no test files. Therefore, the IPv4 and external-connectivity test conditions do not apply.
No-Weak-Crypto ✅ Passed The pull request changes only internal/controller/frr.go. The added code imports Kubernetes retry utilities and performs object reads, metadata updates, and semantic comparisons. It introduces no MD…
Container-Privileges ✅ Passed PASS: The pull request changes only internal/controller/frr.go. The diff adds retry.RetryOnConflict and changes Kubernetes object update handling. It adds no container or Kubernetes manifest setti…
No-Sensitive-Data-In-Logs ✅ Passed The commit changes only internal/controller/frr.go and adds conflict retry plus no-op update logic. The diff adds no logging calls or logged fields. The changed helper returns Kubernetes errors, whi…
Title check ✅ Passed The title clearly identifies the main change: wrapping createOrUpdate in RetryOnConflict to handle optimistic concurrency conflicts.
Description check ✅ Passed The description directly explains the conflict-retry change, the no-op update problem, affected behavior, and test results.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from daxelrod-rh and frobware August 24, 2026 13:24
@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2026
The createOrUpdate helper performed a naive Get→Update without
retry-on-conflict and without checking whether the spec actually
changed. This caused two problems on live clusters:

1. Optimistic concurrency conflicts: when another controller (e.g.
   OVN-K CUDN controller) modified the same object between Get and
   Update, the resourceVersion mismatch caused a conflict error. The
   reconciler marked itself Degraded, which triggered a new reconcile,
   creating a self-sustaining Degraded/Ready flap storm.

2. No-op update loop: even when spec and labels were identical, the
   unconditional Update bumped resourceVersion, which triggered watches
   on CUDN/RouteAdvertisements/FRRConfiguration objects, which triggered
   new reconciles in an infinite loop.

Fix by:
- Wrapping the Update in retry.RetryOnConflict so transient conflicts
  are retried automatically with exponential backoff.
- Adding specUnchanged() to compare spec and labels before updating,
  skipping the write entirely when nothing changed.

Validated on a live OCP cluster with CUDNBgpRouting and CUDNBgpConfig.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpinsonneau
jpinsonneau force-pushed the fix/createorupdate-conflict-retry branch from 58b5e99 to a6171a2 Compare September 8, 2026 15:24
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 8, 2026
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-ci

openshift-ci Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpinsonneau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 8, 2026
@jpinsonneau jpinsonneau changed the title Fix createOrUpdate conflict loop and no-op update storm Wrap createOrUpdate in RetryOnConflict Sep 8, 2026
@alebedev87

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 9, 2026
@openshift-ci

openshift-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@jpinsonneau: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 2b6ad93 into openshift:main Sep 9, 2026
16 checks passed
gavrielg1 pushed a commit to gavrielg1/bgp-cloud-connector that referenced this pull request Sep 9, 2026
…onflict-retry

Wrap createOrUpdate in RetryOnConflict
gavrielg1 pushed a commit to gavrielg1/bgp-cloud-connector that referenced this pull request Sep 9, 2026
…onflict-retry

Wrap createOrUpdate in RetryOnConflict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants