object modified error after successful nab backup

[weshayutin]

Recreate:

1. On the latest build, create nabsl, nab as non-admin user
2. nab completes successfully

$ velero backup get
NAME                                                             STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION                                         SELECTOR
nacuser1-mysql-nonadmin-2-caee74f3-8691-47f4-9c76-eb026da5c23b   Completed   0        0          2025-03-04 14:25:56 +0000 UTC   29d       nacuser1-nacuser1-db7dfb50-0ae7-4cf3-b993-0dc1f353a042   <none>

apiVersion: oadp.openshift.io/v1alpha1
kind: NonAdminBackup
metadata:
  creationTimestamp: "2025-03-04T14:25:56Z"
  finalizers:
  - nonadminbackup.oadp.openshift.io/finalizer
  generation: 2
  name: mysql-nonadmin-2
  namespace: nacuser1
  resourceVersion: "16771477"
  uid: b2158dcc-b171-4377-9bee-88f9cbc06d1c
spec:
  backupSpec:
    csiSnapshotTimeout: 0s
    hooks: {}
    includedNamespaces:
    - nacuser1
    itemOperationTimeout: 0s
    metadata: {}
    snapshotMoveData: true
    storageLocation: nacuser1
    ttl: 720h0m0s
status:
  conditions:
  - lastTransitionTime: "2025-03-04T14:25:56Z"
    message: backup accepted
    reason: BackupAccepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2025-03-04T14:25:56Z"
    message: Created Velero Backup object
    reason: BackupScheduled
    status: "True"
    type: Queued
  dataMoverDataUploads:
    completed: 4
    total: 4
  fileSystemPodVolumeBackups: {}
  phase: Created
  queueInfo:
    estimatedQueuePosition: 0
  veleroBackup:
    nacuuid: nacuser1-mysql-nonadmin-2-caee74f3-8691-47f4-9c76-eb026da5c23b
    name: nacuser1-mysql-nonadmin-2-caee74f3-8691-47f4-9c76-eb026da5c23b
    namespace: openshift-adp
    spec:
      csiSnapshotTimeout: 10m0s
      defaultVolumesToFsBackup: false
      excludedResources:
      - nonadminbackups
      - nonadminrestores
      - nonadminbackupstoragelocations
      - securitycontextconstraints
      - clusterroles
      - clusterrolebindings
      - priorityclasses
      - customresourcedefinitions
      - virtualmachineclusterinstancetypes
      - virtualmachineclusterpreferences
      hooks: {}
      includedNamespaces:
      - nacuser1
      itemOperationTimeout: 4h0m0s
      metadata: {}
      snapshotMoveData: true
      storageLocation: nacuser1-nacuser1-db7dfb50-0ae7-4cf3-b993-0dc1f353a042
      ttl: 720h0m0s
      volumeSnapshotLocations:
      - dpa-sample-1
    status:
      backupItemOperationsAttempted: 4
      backupItemOperationsCompleted: 4
      completionTimestamp: "2025-03-04T14:29:55Z"
      expiration: "2025-04-03T14:25:56Z"
      formatVersion: 1.1.0
      hookStatus: {}
      phase: Completed
      progress:
        itemsBackedUp: 53
        totalItems: 53
      startTimestamp: "2025-03-04T14:25:56Z"
      version: 1

Check the nac logs

whayutin@thinkdoe:~$ sudo su - 
root@thinkdoe:~# oc login --server=https://api.wdh419020225.migration.redhat.com:6443 -u kubeadmin -p uHEX9-u6zSf-giIhy-s7qne
Login successful.

You have access to 76 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "openshift-adp".
root@thinkdoe:~# oc logs -f pod/non-admin-controller-667b4d8c-ch24k -n openshift-adp
2025-03-04T14:19:11Z	INFO	setup	LogLevel: info 
2025-03-04T14:19:11Z	INFO	setup	starting manager
2025-03-04T14:19:11Z	INFO	controller-runtime.metrics	Starting metrics server
2025-03-04T14:19:11Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2025-03-04T14:19:11Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "source": "kind source: *v1alpha1.NonAdminBackup"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore", "source": "kind source: *v1alpha1.NonAdminRestore"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackupsynchronizer", "source": "&{2m0s}"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadminbackupsynchronizer"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "source": "kind source: *v1.Backup"}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadminbackupsynchronizer", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadmindownloadrequest", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminDownloadRequest", "source": "kind source: *v1alpha1.NonAdminDownloadRequest"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadmindownloadrequest", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminDownloadRequest", "source": "kind source: *v1.DownloadRequest"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadmindownloadrequest", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminDownloadRequest"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore", "source": "kind source: *v1.Restore"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore", "source": "kind source: *v1.PodVolumeRestore"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore", "source": "kind source: *v2alpha1.DataDownload"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadmingarbagecollector", "source": "&{24h0m0s}"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadmingarbagecollector"}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadmingarbagecollector", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "source": "kind source: *v1alpha1.NonAdminBackupStorageLocation"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "source": "kind source: *v1.Backup"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "source": "kind source: *v1.PodVolumeBackup"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "source": "kind source: *v1.BackupStorageLocation"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "source": "kind source: *v2alpha1.DataUpload"}
2025-03-04T14:19:11Z	INFO	Starting EventSource	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "source": "kind source: *v1alpha1.NonAdminBackupStorageLocationRequest"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation"}
2025-03-04T14:19:11Z	INFO	Starting Controller	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup"}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadmindownloadrequest", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminDownloadRequest", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadminrestore", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminRestore", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Starting workers	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "worker count": 1}
2025-03-04T14:19:11Z	INFO	Reconciling NonAdminDownloadRequest	{"controller": "nonadmindownloadrequest", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminDownloadRequest", "NonAdminDownloadRequest": {"name":"example","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "example", "reconcileID": "b8e91c4d-f7c8-4c3b-adba-6a227b33d632"}
2025-03-04T14:19:11Z	ERROR	Failed to create NonAdminBackupStorageLocationRequest	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "02dc7541-e24c-4d64-b1bd-ff781b0c5fa8", "error": "nonadminbackupstoragelocationrequests.oadp.openshift.io \"nacuser1-nacuser1-202e21bc-70aa-4980-9df6-d522dcef9826\" already exists"}
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).createNonAdminRequest
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:580
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).Reconcile
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:19:11Z	ERROR	Reconciler error	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "02dc7541-e24c-4d64-b1bd-ff781b0c5fa8", "error": "nonadminbackupstoragelocationrequests.oadp.openshift.io \"nacuser1-nacuser1-202e21bc-70aa-4980-9df6-d522dcef9826\" already exists"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:20:14Z	ERROR	Failed to update status	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "2bf1439b-2ed6-4e41-aeaa-46fba653740b", "error": "Operation cannot be fulfilled on nonadminbackupstoragelocations.oadp.openshift.io \"nacuser1\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).ensureNonAdminRequest
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:515
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).Reconcile
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:20:14Z	ERROR	Reconciler error	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "2bf1439b-2ed6-4e41-aeaa-46fba653740b", "error": "Operation cannot be fulfilled on nonadminbackupstoragelocations.oadp.openshift.io \"nacuser1\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:21:44Z	ERROR	Failed to remove finalizer	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "8751a530-5cc8-4a8c-bbbc-529e36389f19", "error": "Operation cannot be fulfilled on nonadminbackupstoragelocations.oadp.openshift.io \"nacuser1\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/oadp.openshift.io/nonadminbackupstoragelocations/nacuser1/nacuser1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d8398cf5-4bf5-45bf-b643-1f2d275fbe61, UID in object meta: "}
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).removeNaBSLFinalizerUponVeleroBSLDeletion
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:305
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupStorageLocationReconciler).Reconcile
	/workspace/internal/controller/nonadminbackupstoragelocation_controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:21:44Z	ERROR	Reconciler error	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "8751a530-5cc8-4a8c-bbbc-529e36389f19", "error": "Operation cannot be fulfilled on nonadminbackupstoragelocations.oadp.openshift.io \"nacuser1\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/oadp.openshift.io/nonadminbackupstoragelocations/nacuser1/nacuser1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d8398cf5-4bf5-45bf-b643-1f2d275fbe61, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:24:59Z	INFO	Velero BSL Secret with label not found, creating one	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "27881694-3d06-444f-a7af-44c72446b5fa", "oadpnamespace": "openshift-adp", "UUID": "nacuser1-nacuser1-db7dfb50-0ae7-4cf3-b993-0dc1f353a042"}
2025-03-04T14:24:59Z	INFO	Velero BSL with label not found, creating one	{"controller": "nonadminbackupstoragelocation", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackupStorageLocation", "NonAdminBackupStorageLocation": {"name":"nacuser1","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "nacuser1", "reconcileID": "27881694-3d06-444f-a7af-44c72446b5fa", "oadpnamespace": "openshift-adp", "UUID": "nacuser1-nacuser1-db7dfb50-0ae7-4cf3-b993-0dc1f353a042"}




2025-03-04T14:25:56Z	INFO	VeleroBackup with label not found, creating one	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "NonAdminBackup": {"name":"mysql-nonadmin-2","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "mysql-nonadmin-2", "reconcileID": "3fa10068-54d3-466f-8b5b-0ad8e3d97180", "UUID": "nacuser1-mysql-nonadmin-2-caee74f3-8691-47f4-9c76-eb026da5c23b"}
2025-03-04T14:25:56Z	INFO	VeleroBackup successfully created	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "NonAdminBackup": {"name":"mysql-nonadmin-2","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "mysql-nonadmin-2", "reconcileID": "3fa10068-54d3-466f-8b5b-0ad8e3d97180"}
2025-03-04T14:29:52Z	ERROR	Failed to update NonAdminBackup Status	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "NonAdminBackup": {"name":"mysql-nonadmin-2","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "mysql-nonadmin-2", "reconcileID": "54b98088-7a46-45b9-a4cb-903b44d3c96c", "error": "Operation cannot be fulfilled on nonadminbackups.oadp.openshift.io \"mysql-nonadmin-2\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupReconciler).createVeleroBackupAndSyncWithNonAdminBackup
	/workspace/internal/controller/nonadminbackup_controller.go:802
github.com/migtools/oadp-non-admin/internal/controller.(*NonAdminBackupReconciler).Reconcile
	/workspace/internal/controller/nonadminbackup_controller.go:162
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222
2025-03-04T14:29:52Z	ERROR	Reconciler error	{"controller": "nonadminbackup", "controllerGroup": "oadp.openshift.io", "controllerKind": "NonAdminBackup", "NonAdminBackup": {"name":"mysql-nonadmin-2","namespace":"nacuser1"}, "namespace": "nacuser1", "name": "mysql-nonadmin-2", "reconcileID": "54b98088-7a46-45b9-a4cb-903b44d3c96c", "error": "Operation cannot be fulfilled on nonadminbackups.oadp.openshift.io \"mysql-nonadmin-2\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222

[mpryc]

There are number of k8s API errors from the direct API calls, seems like some stress on the API endpoint?

This one is really suspicious, because it may happen it already exists? however I think it's more of the backoff from the API endpoint:

oadp-non-admin/internal/controller/nonadminbackupstoragelocation_controller.go

Lines 566 to 582 in 0fb24a2

	nonAdminBslRequest := nacv1alpha1.NonAdminBackupStorageLocationRequest{
	ObjectMeta: metav1.ObjectMeta{
	Name: veleroObjectsNACUUID,
	Namespace: r.OADPNamespace,
	Labels: labels,
	Annotations: function.GetNonAdminBackupStorageLocationAnnotations(nabsl.ObjectMeta),
	},
	Spec: nacv1alpha1.NonAdminBackupStorageLocationRequestSpec{
	ApprovalDecision: approvalDecision,
	},
	}
	err = r.Create(ctx, &nonAdminBslRequest)
	if err != nil {
	logger.Error(err, "Failed to create NonAdminBackupStorageLocationRequest")
	return false, err
	}

https://github.com/migtools/oadp-non-admin/blob/master/internal/controller/nonadminbackupstoragelocation_controller.go#L303-L307

https://github.com/migtools/oadp-non-admin/blob/master/internal/controller/nonadminbackup_controller.go#L801

Is it possible to check the API Server Logs ?

[mateusoliveira43]

is this the problem #102 ?

[shubham-pampattiwar]

@mateusoliveira43 Most probably it is some update call on the object.