GPU plugin: fix upgrade errors as of newly added fields, add tests by jgehrcke · Pull Request #1119 · kubernetes-sigs/dra-driver-nvidia-gpu

jgehrcke · 2026-05-11T17:04:25Z

We introduced ShareID and Metadata on kubeletplugin.Device via a library bump in Bug fixes for Passthrough-Support feature #994. This broke upgrades from 25.12.0 for all non-empty checkpoints. As a fix, this patch proposes a lightweight wrapper for us to be able to annotate the offending fields with omitempty. This specific upgrade breakage was caught by the test suite, and I've confirmed that with the patch applied the simple GPU workload upgrade test once again passes.
We introduced ParentPCIBusID and PciBusID in Bug fixes for Passthrough-Support feature #994. This breaks upgrades from 25.12.0 when running a DynamicMIG or VfioDevice workload. This patch proposes to fix that by also adding corresponding omitempty annotations. I also added a new test that covers upgrading under dynamic MIG workload and confirmed that this new test failed without the fix, and succeeds with the patch applied.
To make checkpoint upgrade failures faster to debug, I've enriched the log output showing a useful diff emitted just before the otherwise hard-to-interpret message Error: error creating driver: unable to get checkpoint: checkpoint is corrupted. Below you can see an example for when "parentPCIBusID": "" would break checksum verification.
I also added a test that covers the emit-diff-via-logging logic.
Took liberty to act on Helm chart: keep supporting nvidia-dra-driver-gpu-component label for now #1079, I still don't see a downside and this makes it easy to test the general upgradeability by virtue of covering the downstream-25-12-0-on-NGC->upstream-0-4-0-dev upgrade path.

The new log output for better upgrade failure debuggability:

E0511 14:24:30.306830       1 device_state.go:640] checkpoint failed checksum verification; unified diff (on-disk vs re-marshaled by current binary):
--- on-disk
+++ re-marshaled
@@ -51,7 +51,8 @@
                     "giId": 7,
                     "ciId": 0,
                     "migUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4",
-                    "parentUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4"
+                    "parentUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4",
+                    "parentPCIBusID": ""
                   },
                   "device": {
                     "Requests": [
@@ -127,7 +128,8 @@
                     "giId": 7,
                     "ciId": 0,
                     "migUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4",
-                    "parentUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4"
+                    "parentUUID": "GPU-df52ace4-7143-c242-81d8-819a6b3819b4",
+                    "parentPCIBusID": ""
                   },
                   "device": {
                     "Requests": [
I0511 14:24:30.307024       1 main.go:224] shutdown
Error: error creating driver: unable to get checkpoint: checkpoint is corrupted

netlify · 2026-05-11T17:04:32Z

✅ Deploy Preview for dra-driver-nvidia-gpu ready!

Name	Link
🔨 Latest commit	`aacccea`
🔍 Latest deploy log	https://app.netlify.com/projects/dra-driver-nvidia-gpu/deploys/6a03092b97b19b00085d2a7a
😎 Deploy Preview	https://deploy-preview-1119--dra-driver-nvidia-gpu.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

jgehrcke · 2026-05-11T17:23:01Z

On this branch, all tests passed when manually running make bats-gpu:

27 tests, 0 failures in 232 seconds

shivamerla · 2026-05-11T17:56:16Z

+// rejected by the new binary. Circumvent that by adding the `omitempty`
+// annotations here for `ShareID` and `Metadata` which were added as part of the
+// k8s 1.36 release cycle. Also see issue 1080.
+type CheckpointedDevice kubeletplugin.Device


This fixes the immediate kubeletplugin.Device cases, but the same upgrade fragility still exists for other checkpointed upstream API structs, especially resourceapi.ResourceClaimStatus correct?.

For example, if Kubernetes later adds a non-omitempty field anywhere under ResourceClaimStatus -> AllocationResult -> DeviceAllocationResult -> ..., re-marshaling an old checkpoint can again produce new zero-value JSON and fail checksum verification.

Going forward, we need to avoid checkpointing live upstream API types directly. A more robust pattern would be to define explicit checkpoint-only objects for every persisted type we own, like you did in this case, with stable JSON shape and omitempty on forward-added/optional fields, then convert to/from the current Kubernetes types at the boundary. That makes the checkpoint schema versioned and under our control instead of inheriting upstream serialization changes.

I agree with Shiva and this resonates with my proposal at #1080 (comment)

I am working on a separate PR to decouple upstream structures from the Checkpoint

I agree with Shiva and this resonates with my proposal at #1080 (comment)

@shengnuo I forgot to previously recognize that you actually followed and understood the depth of the technical issue in our discussion in #1080 -- I appreciated that, and your push for a generic solution.

still exists for other checkpointed upstream API structs, especially resourceapi.ResourceClaimStatus correct?

Of course!

We can treat individual problems separately though. That is sometimes better in the overall impact per effort consideration.

What we know but what has to be said: this type of breakage never happens spontaneously, but specifically when we update vendored libraries. Hence, we have the opportunity to detect this in tests, and can respond accordingly.

Also: some types are probably already rather stable, and we shouldn't anticipate (even frequent) change in all DRA-related types.

I don't have a strong opinion here, but maybe we should not rush into preparing for changes in all types yet, for the 0.4.0 release. Because there is no user-facing problem to be solved.

I really do appreciate your work and thinking and exploration @shengnuo for the generic solution in #1120; as something that should grow into the next bigger push.

shivamerla · 2026-05-11T19:56:18Z

@jgehrcke can you squash all commits. We can track changes related to resourceapi.ResourceClaimStatus in a separate issue.

Signed-off-by: Dr. Jan-Philip Gehrcke <jgehrcke@nvidia.com>

tests: work towards reactivating GPU upgrade test tests: log last-stable checkpoint before upgrade tests: add upgrade test with DynMIG workload tests: add test confirming checkpoint diff in log output Signed-off-by: Dr. Jan-Philip Gehrcke <jgehrcke@nvidia.com>

…abiity Add CheckpointedDevice, fix ShareID/Metadata upgrade errors GPU plugin: log checkpoint diff on checksum error GPU plugin: fix upgrade errors for MigLiveTuple and VfioDeviceInfo Signed-off-by: Dr. Jan-Philip Gehrcke <jgehrcke@nvidia.com>

jgehrcke · 2026-05-12T11:05:39Z

Thanks for feedback @shengnuo and @shivamerla.

@jgehrcke can you squash all commits.

I have squashed commits now, ack.

We can track changes related to resourceapi.ResourceClaimStatus in a separate issue.

We should do that, ack.

jgehrcke · 2026-05-12T11:46:03Z

Tested again:

$ make image-build-and-copy-to-nodes
make -f deployments/container/Makefile build
[...]
export/compress took: 1.34 s
-rw------- 1 jgehrcke jgehrcke 78M May 12 11:08 ./dra-driver-dev-img.v9252FB.tar.gz
[...]

$ TEST_CHART_LOCAL=1 make bats-gpu
make -f tests/bats/Makefile tests-gpu
[...]
27 tests, 0 failures in 237 seconds

mock-gpu-e2e failed with a transient error (log):

#8 [internal] load metadata for nvcr.io/nvidia/k8s/container-toolkit:v1.19.0
#8 ERROR: failed to do request: Head "https://nvcr.io/v2/nvidia/k8s/container-toolkit/manifests/v1.19.0": net/http: TLS handshake timeout

shivamerla · 2026-05-12T16:07:15Z

/lgtm
/approve

k8s-ci-robot · 2026-05-12T16:07:24Z

LGTM label has been added.

Details

Git tree hash: e901de21091ca801dfae7e78f813ed39a4a13593

k8s-ci-robot · 2026-05-12T16:07:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jgehrcke, shivamerla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jgehrcke,shivamerla]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

shivamerla · 2026-05-12T16:07:29Z

/retest

shivamerla · 2026-05-12T16:12:45Z

/release-note-none

shivamerla · 2026-05-12T16:16:06Z

/skip-tests

github-project-automation Bot added this to DRA Driver for NVIDIA GPUs May 11, 2026

github-project-automation Bot moved this to Backlog in DRA Driver for NVIDIA GPUs May 11, 2026

k8s-ci-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label May 11, 2026

k8s-ci-robot requested review from dims and guptaNswati May 11, 2026 17:04

jgehrcke mentioned this pull request May 11, 2026

[Bug]: last stable -> current dev upgrade test broken on main #1078

Closed

shivamerla reviewed May 11, 2026

View reviewed changes

chart: re-add legacy component labels (issue kubernetes-sigs#1079)

dbe7047

Signed-off-by: Dr. Jan-Philip Gehrcke <jgehrcke@nvidia.com>

jgehrcke force-pushed the jp/fix-gpu-upgrade-errors-add-test branch from 6871b8d to 683fb55 Compare May 12, 2026 11:01

jgehrcke added 2 commits May 12, 2026 11:02

jgehrcke force-pushed the jp/fix-gpu-upgrade-errors-add-test branch from 683fb55 to aacccea Compare May 12, 2026 11:04

k8s-ci-robot assigned shivamerla May 12, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 12, 2026

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 12, 2026

k8s-ci-robot merged commit fefaeea into kubernetes-sigs:main May 12, 2026
17 of 18 checks passed

github-project-automation Bot moved this from Backlog to Done in DRA Driver for NVIDIA GPUs May 12, 2026

shivamerla mentioned this pull request May 13, 2026

Helm chart: keep supporting nvidia-dra-driver-gpu-component label for now #1079

Closed

shengnuo mentioned this pull request May 13, 2026

Fix upgrade issues around compute domain #1129

Merged

Conversation

jgehrcke commented May 11, 2026

Uh oh!

netlify Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for dra-driver-nvidia-gpu ready!

Uh oh!

jgehrcke commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shivamerla May 11, 2026

Choose a reason for hiding this comment

Uh oh!

shengnuo May 11, 2026

Choose a reason for hiding this comment

Uh oh!

shengnuo May 11, 2026

Choose a reason for hiding this comment

Uh oh!

shengnuo May 11, 2026

Choose a reason for hiding this comment

Uh oh!

jgehrcke May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shivamerla commented May 11, 2026

Uh oh!

jgehrcke commented May 12, 2026

Uh oh!

jgehrcke commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shivamerla commented May 12, 2026

Uh oh!

k8s-ci-robot commented May 12, 2026

Uh oh!

k8s-ci-robot commented May 12, 2026

Uh oh!

shivamerla commented May 12, 2026

Uh oh!

shivamerla commented May 12, 2026

Uh oh!

shivamerla commented May 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented May 11, 2026 •

edited

Loading

jgehrcke commented May 11, 2026 •

edited

Loading

jgehrcke May 12, 2026 •

edited

Loading

jgehrcke commented May 12, 2026 •

edited

Loading