Bring Your Own DRANET Provider (BYODP) by aojea · Pull Request #223 · kubernetes-sigs/dranet

aojea · 2026-06-08T15:43:01Z

What type of PR is this?

/kind feature
/kind documentation

What this PR does / why we need it:

This PR introduces the Bring Your Own DRANET Provider (BYODP) feature, allowing users to provide custom implementations for both hardware discovery and user intent via external webhooks.

Instead of hardcoding bare-metal or CNI-specific logic directly into DRANET, we can now delegate these responsibilities to an external HTTP REST server or Unix domain socket. This ensures DRANET's core runtime remains solid, statically verifiable, and predictable, while still exposing flexibility for third-party extensions.

This PR specifically introduces:

Cloud Providers (Cluster Provider Intent): Webhooks can dictate physical infrastructural (e.g., base MTU, hardware MAC, PCI boundaries).
Profile Providers (User Intent): Webhooks can dictate logical network assignments like IPAM (e.g., allocating an IP address from an external IPAM plugin like whereabouts or from a Cloud Provider).
Webhook Capabilities Discovery: DRANET queries /health on startup to negotiate cloudProvider and profileProvider capability support.
Reference Implementation: Adds a fully functional reference webhook server (cmd/webhook-whereabouts) that wraps the standard CNI whereabouts plugin to deliver dynamic IPAM profiles.

Which issue(s) this PR is related to:

Fixes: #103

Special notes for your reviewer:

To make the review process this PR has 3 commits:

Refactor: Decouples the existing cloud hardware discovery logic out of pkg/inventory into a distinct pkg/cloudprovider/discovery package.
Feature & Docs: Introduces the generic webhook client, ProfileProvider interface integration in the inventory DB, and comprehensive architectural documentation (site/content/docs/contributing/webhook-providers.md).
Example & Tests: Adds the webhook-whereabouts reference module and the associated E2E bats tests verifying the dynamic whereabouts CNI integration.
4

Does this PR introduce a user-facing change?

Add support for Custom Webhook Providers (BYODP) via the `--cloud-provider-hint=webhook`, `--profile-provider=webhook`, and `--webhook-url` flags. This enables users to delegate physical hardware discovery and logical network IPAM assignments to external HTTP servers or Unix domain sockets.

netlify · 2026-06-08T15:54:07Z

✅ Deploy Preview for dranet ready!

Name	Link
🔨 Latest commit	`7f86460`
🔍 Latest deploy log	https://app.netlify.com/projects/dranet/deploys/6a292c901512090008c0708a
😎 Deploy Preview	https://deploy-preview-223--dranet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

aojea · 2026-06-08T16:20:45Z

+func (db *DB) GetProfileConfig(deviceName, profile string, claimUID types.UID) (*apis.NetworkConfig, error) {
+	p := db.getProfileProvider()
+	if p == nil {
+		return nil, fmt.Errorf("current cloud provider does not support dynamic profiles")


This means that the user hasn't configured a dynamic profile provider (e.g. they didn't start the dranet daemon with --profile-provider=webhook or the clooud provider does not implement it, however, if this function is being called, it strictly means that a Pod is attempting to start with a specific Profile explicitly requested in its DRANET Network Config

anson627 · 2026-06-08T17:27:49Z

+
+	var hint discovery.CloudProviderHint
+	// Auto-discover cloud provider if not explicitly set
+	if cloudProviderHint == "" {


I understand the intent, the concern is that the code currently makes explicit cloud-provider failures indistinguishable from no underlay provider requested

I would suggest sth like this:

explicitCloudProvider := cloudProviderHint != "" cloudInst, err = discovery.GetInstanceProperties(ctx, hint, webhookURL) if err != nil { if explicitCloudProvider { klog.Fatalf("failed to initialize cloud provider %q: %v", hint, err) } klog.Infof("failed to initialize auto-discovered cloud provider %q: %v", hint, err) cloudInst = nil }

This keeps the same behavior as before, we do not want to make a breaking change because this will cause that dranets that worked before to start to fail ... also we explicitily indicated in the flag is just a hint because of this

sounds good

anson627 · 2026-06-08T17:37:58Z

+				if err := json.NewDecoder(resp.Body).Decode(&caps); err == nil {
+					p.caps = caps
+				} else {
+					p.caps = Capabilities{CloudProvider: true, ProfileProvider: true}


This returns a provider even when /health failed, because all errors above are swallowed

For explicit webhook use, startup should fail if the capability contract cannot be validated, otherwise the dranet pod can become ready with a misconfigured webhook and fail later during claim preparation

good catch, fixed, added new commits for simplifying reviews

anson627 · 2026-06-08T17:39:55Z

+			if devCfg.Claim.Namespace == claim.Namespace && devCfg.Claim.Name == claim.Name {
+				if devCfg.NetworkInterfaceConfigInPod.Profile != "" {
+					if err := np.netdb.ReleaseProfileConfig(deviceName, devCfg.NetworkInterfaceConfigInPod.Profile, claim.UID); err != nil {
+						klog.Errorf("failed to release profile config for claim %v: %v", claim.NamespacedName, err)


it is better to return the ReleaseProfileConfig error instead of only logging it, otherwise unprepare reports success, deletes the claim state, and kubelet will not retry leaked profile/IPAM cleanup

Everything on pod teardown use to get swallowed because a retry does not imply a retry of the operation and the risk of causing deadlock situations on exit.

This is also how CNI works and is how is also specified

Plugins should generally complete a DEL action without error even if some resources are missing

and how NRI works too, we are trying to document the NRI contract too containerd/nri#286

I understand your concern but is like a defer x.Close() , is on the provider to ensure there is no leaks

This e2e test uses a simple Python webhook to demonstrate the power and flexibility of Dranet's architecture. It shows why we do not need to couple traditional networking constructs (like IPAM) to the core API. At its core, IPAM is just an internal process of assigning an IP to an interface. The interface itself only cares about the addresses, routes, and MTU assigned to it. By abstracting these details behind the Profile and Cloud Provider interfaces, Dranet becomes highly portable. It can seamlessly integrate with any kind of IPAM or environment-specific configuration mechanism out there without having to implement or hardcode them natively. The included webhook script dynamically returns the necessary custom attributes, MTU overrides (via cloud intent), and IP addresses (via user intent profile), which Dranet successfully applies to the interface.

anson627 · 2026-06-09T10:51:58Z

/lgtm
/approve

aojea · 2026-06-09T10:56:43Z

/hold

latest e2e test is not working correctly, will unhold once I'm understanding better the problem

aojea · 2026-06-09T11:10:05Z


  # Wait for the interface to be discovered and validate the custom attribute from webhook
-  sleep 5
-  run kubectl --context kind-dranet-test-cluster get resourceslices --field-selector spec.nodeName="$NODE_NAME" -o jsonpath='{.items[0].spec.devices[?(@.name=="dummy1")].attributes.dra\.net\/webhook-attr.string}'


the hyphen fails the apiserver validation webhook-attr and the ResourceSlice was never updated, I can see how this can happen in prod

anson627 · 2026-06-09T11:27:05Z

these may fix e2e issues:

Update tests/manifests/python_webhook_pod.yaml to use webhook_attr and avoid has on map indexing
Run go mod tidy and include go.mod/go.sum changes for cmd/webhook-whereabouts

aojea · 2026-06-09T14:30:37Z

/hold cancel

tests are passing, covering custom webhooks and wherabouts ipam

This refactor removes the redundant 'profile' string parameter from the GetProfileConfig and ReleaseProfileConfig interface methods across the driver, inventory, and cloudprovider packages, replacing it with the full 'NetworkConfig' struct. Reasoning & Trade-offs: Passing the entire NetworkConfig gives the node-level webhook full context regarding the user and cloud intents. This allows the webhook to behave like a Validating Admission Controller, enabling it to intelligently accept or deny configurations (e.g., rejecting an invalid static IP request) using HTTP status codes. While this approach grants powerful node-level validation, it comes with the trade-off of delayed feedback. Because validation happens asynchronously during the DRA NodePrepareResources phase rather than at the API Server, invalid configurations result in Pods getting stuck in a Pending state and Kubelet repeatedly retrying NodePrepareResources, rather than an immediate API rejection on 'kubectl apply'. Includes updates to test mocks, a new webhook unit test, and expanded documentation explaining these architectural trade-offs.

kanlkan

Thank you for opening this PR. I added some comments for the document.

kanlkan · 2026-06-09T22:57:12Z

+  * **Kubelet Retry Loops**: Standard Kubernetes behavior is to retry failed resource preparations. A persistent denial (like a 400 Bad Request) will cause the Kubelet to continuously retry `NodePrepareResources`, which can generate unnecessary load on the node and webhook server compared to an upfront API rejection.
+


I would like to add one more sentence here.

* **Idempotency**: The kubelet may retry `NodePrepareResources`, so DRANET can call this more than once for the same `(device, claimUID)`. It must return an equivalent result without allocating additional resources (e.g. key the allocation by `claimUID`, as `whereabouts` does via `CNI_CONTAINERID`).

kanlkan · 2026-06-09T23:05:47Z

+* `POST /ReleaseProfileConfig`: Frees stateful resources (e.g., releasing an IP address). Also receives the full `NetworkConfig`. Should return `200 OK` on success or if the resource was already released (idempotency).
+


I would like to add one more sentence here.

* **Best-effort teardown**: A failed `ReleaseProfileConfig` is logged but not retried by DRANET (teardown must not block pod deletion). The provider therefore owns leak reclamation and must be able to garbage-collect orphaned allocations on its own, otherwise resources leak permanently.

This extracts the getDeviceNetworkConfig logic from prepareResourceClaim to allow for exhaustive unit testing of the configuration precedence. A comprehensive table-driven test was added to validate that User configurations properly override Cloud and Profile/Webhook intents, and that Webhook blocking correctly bubbles up errors.

kanlkan

Thank you!
/lgtm

k8s-ci-robot · 2026-06-10T23:53:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anson627, aojea, kanlkan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. kind/documentation Categorizes issue or PR as related to documentation. labels Jun 8, 2026

k8s-ci-robot requested review from MikeZappa87 and anson627 June 8, 2026 15:43

aojea requested a review from kanlkan June 8, 2026 15:43

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 8, 2026

aojea requested a review from gauravkghildiyal June 8, 2026 15:43

aojea commented Jun 8, 2026

View reviewed changes

Comment thread pkg/cloudprovider/discovery/discovery.go

aojea commented Jun 8, 2026

View reviewed changes

Comment thread pkg/cloudprovider/discovery/discovery_test.go Outdated

aojea force-pushed the profiles branch 2 times, most recently from 83a4e42 to 0d437fa Compare June 8, 2026 16:08

aojea commented Jun 8, 2026

View reviewed changes

aojea added 3 commits June 8, 2026 16:27

Refactor cloud discovery into discovery package

9cff053

Implement dynamic, on-demand network configurations via Profiles

ecd40e0

Add webhook-whereabouts example and tests

4a142be

aojea force-pushed the profiles branch from 0d437fa to 4a142be Compare June 8, 2026 16:28

anson627 reviewed Jun 8, 2026

View reviewed changes

aojea force-pushed the profiles branch from ee0262e to fff4c9a Compare June 9, 2026 09:34

aojea added 2 commits June 9, 2026 09:42

webhook return explicit errors and refactor initialization

f276c3e

aojea force-pushed the profiles branch from fff4c9a to a1a4b30 Compare June 9, 2026 09:42

k8s-ci-robot assigned anson627 Jun 9, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2026

aojea force-pushed the profiles branch from 52c26b5 to 3b965d2 Compare June 9, 2026 10:55

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2026

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 9, 2026

aojea force-pushed the profiles branch from 3b965d2 to f616563 Compare June 9, 2026 11:08

aojea commented Jun 9, 2026

View reviewed changes

aojea force-pushed the profiles branch from f616563 to 54bdc8a Compare June 9, 2026 12:08

fix device attribute format on e2e

e235446

aojea force-pushed the profiles branch 2 times, most recently from 4e98645 to f38ee84 Compare June 9, 2026 13:51

tests: whereabouts tests

4b4403c

aojea force-pushed the profiles branch from f38ee84 to 4b4403c Compare June 9, 2026 14:14

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 9, 2026

aojea mentioned this pull request Jun 9, 2026

chore: add startup probe to avoid false unhealthy events #222

Merged

kanlkan reviewed Jun 9, 2026

View reviewed changes

aojea added 2 commits June 10, 2026 09:17

improve webhook documentation

7f86460

kanlkan approved these changes Jun 10, 2026

View reviewed changes

k8s-ci-robot assigned kanlkan Jun 10, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2026

k8s-ci-robot merged commit 1b7c7e5 into kubernetes-sigs:main Jun 10, 2026
13 of 14 checks passed

fmuyassarov mentioned this pull request Jun 11, 2026

feat: plumb EventRecorder and emit failure events #221

Merged

sesame0224 mentioned this pull request Jun 19, 2026

BYODP webhook providers cannot garbage-collect orphaned IP allocations when using whereabouts #234

Open

Bowser1704 mentioned this pull request Jun 29, 2026

Support LACP bonded network interfaces #239

Open

		* Kubelet Retry Loops: Standard Kubernetes behavior is to retry failed resource preparations. A persistent denial (like a 400 Bad Request) will cause the Kubelet to continuously retry `NodePrepareResources`, which can generate unnecessary load on the node and webhook server compared to an upfront API rejection.

		* `POST /ReleaseProfileConfig`: Frees stateful resources (e.g., releasing an IP address). Also receives the full `NetworkConfig`. Should return `200 OK` on success or if the resource was already released (idempotency).

Uh oh!

Conversation

aojea commented Jun 8, 2026

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR is related to:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

Uh oh!

Uh oh!

netlify Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for dranet ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anson627 Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anson627 commented Jun 9, 2026

Uh oh!

aojea commented Jun 9, 2026

Uh oh!

aojea Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anson627 commented Jun 9, 2026

Uh oh!

aojea commented Jun 9, 2026

Uh oh!

kanlkan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kanlkan left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Jun 8, 2026 •

edited

Loading

anson627 Jun 8, 2026 •

edited

Loading

aojea Jun 9, 2026 •

edited

Loading

kanlkan left a comment •

edited

Loading