Retire the eBPF dataplane; userspace AF_XDP dataplane becomes the default and only path

[psaab]

Decision

Retire the eBPF dataplane in favor of the Rust userspace AF_XDP dataplane. The userspace path has had feature parity for months, is the canonical smoke target (loss:xpf-userspace-fw0/fw1), has the active perf/HA development surface (PR #1187 / #1188 / #1206 / #1330 / #1332 / #1341 / #1358 / #1366 all landed against it), and is what every new feature gets built on. The eBPF path is the legacy / regression-only path per CLAUDE.md.

Two dataplanes is twice the maintenance, twice the test surface, twice the documentation. Time to consolidate.

Scope summary

Concrete surface to remove or retire:

Surface	Count	Path
BPF C source (XDP + TC programs + headers)	20 files	bpf/
Go eBPF loader + bpf2go bindings + tests	65 files	pkg/dataplane/*.go (excluding pkg/dataplane/userspace/)
make generate (bpf2go invocation) + related clean rules	1 target	Makefile
Legacy local cluster Make targets (cluster-init/-create/-deploy/-destroy/-ssh/-status/-logs/-start/-stop/-restart)	~10 targets	Makefile (these drive bpfrx-fw0/1 legacy eBPF cluster)
Legacy standalone eBPF reference VM (bpfrx-fw)	1 VM	test/incus/setup.sh
Tests under test/incus/per-stream-diag.sh + cluster-setup.sh that reference bpfrx* cluster names	3 files	test/incus/
Docs that reference eBPF / BPF / XDP / bpf2go / BPF verifier rules	75+ docs	docs/, CLAUDE.md, README*
show system buffers CLI surface (currently reports BPF map utilization)	1 surface	pkg/cli/
Userspace-dp Rust code unchanged	181 files	userspace-dp/src/

Feature-parity audit (Phase 0 — gate)

Before removing anything, refresh docs/userspace-dataplane-gaps.md (dated 2026-03-14) against current master. Confirm every eBPF-dataplane feature has a userspace equivalent. Block retirement on any documented gap that production depends on.

Known-implemented in userspace-dp today per the gaps doc:

• Stateful forwarding, zone + global policies, application matching
• SNAT (interface), DNAT, static NAT, NAT64, NPTv6
• Firewall filters, flow export
• TCP MSS clamping, embedded ICMP NAT reversal
• Configurable session timeouts, VLAN, routes + neighbors

Verify also: screen/IDS, SYN cookie flood protection, IPsec, BGP/OSPF integration with FRR, all 7+ CoS classes, RPM probes, NetFlow v9, dynamic feeds, SNMP, DHCP relay/server, syslog, HA chassis cluster state machine + VRRP + sync.

Phases

Phased to minimize blast radius. Each phase ships as a separate PR with triple-review + smoke against loss:xpf-userspace-fw0/1 canonical cluster.

Phase 0 — feature-parity audit + deprecation announcement

• Refresh docs/userspace-dataplane-gaps.md against current master
• File any gap-issues found; block retirement on production-blocking gaps
• Add deprecation notice to all top-level docs (CLAUDE.md, README*, docs/testing.md, docs/development-workflow.md) — "eBPF dataplane is being retired; new development happens on userspace-dp"
• No code removed yet

Phase 1 — docs migration (the "ALLLLLLLL documentation" pass)

• Audit + update all 75+ docs identified in survey
• Master doc updates:
  • CLAUDE.md (project root + ~/.claude/CLAUDE.md if present) — strip BPF Pipeline section, BPF verifier rules, BPF map serialization rules, byte-order rules, XDP/SR-IOV gotchas, kernel-version constraints
  • docs/engineering-style.md — drop BPF-specific rules
  • README* — point all examples at userspace-dp
• Feature/architecture docs:
  • docs/testing.md, docs/development-workflow.md, docs/test_env.md — userspace-dp only
  • docs/ha-cluster-test-plan.md — drop legacy cluster references
  • docs/feature-gaps.md — single dataplane column
  • All docs/*-plan.md PR plan-docs — leave as historical record (don't rewrite history); add deprecation banner at top
• Test environment docs:
  • test/incus/cluster-setup.sh README/comments — userspace-dp only
• Each removed BPF concept gets a "historical: BPF dataplane retired in #" reference so git-history readers can find context

Phase 2 — test environment consolidation

• Remove legacy make cluster-* targets (bpfrx-fw0/1)
• Remove make test-vm (standalone eBPF reference)
• Update test/incus/cluster-setup.sh to drop bpfrx env support
• Promote loss-cluster-* Make targets to cluster-* (rename to be the canonical, not the alternative)
• Update test/incus/per-stream-diag.sh to userspace-dp only
• Make sure make test-failover + make test-ha-crash + make test-restart-connectivity point at userspace-dp cluster
• Smoke: full Pass A (CoS-off, v4+v6, push+reverse) + Pass B (CoS-on per-class 5200-5211 + echo 6200-6211) on the now-canonical userspace cluster

Phase 3 — build system + Go code removal

• Remove make generate target (no more bpf2go)
• Remove BPF_CFLAGS and BPF object cleanup rules
• Remove pkg/dataplane/*_bpfel.go + *_bpfeb.go (bpf2go-generated)
• Remove pkg/dataplane/loader_ebpf.go + loader_stub.go
• Audit remaining pkg/dataplane/*.go — keep what's shared with userspace (e.g. cpumask.go, types.go, persistent_nat.go if userspace-dp consumes them); remove what's eBPF-only
• Update daemon lifecycle to not invoke the eBPF loader
• go build ./... clean; go test ./... passes
• Smoke matrix again

Phase 4 — BPF source removal

• rm -rf bpf/
• Final pass: grep -ri "bpf/" . returns nothing in active code paths (historical PR plan-docs OK)
• CI green; smoke matrix green

Phase 5 — CLI + observability cleanup

• show system buffers — replace BPF-map-utilization output with userspace-dp equivalent (worker queue depths, UMEM frame counts, etc.) OR remove if redundant with show system userspace
• Any operational command output that still claims "eBPF" gets updated
• Prometheus metrics: remove xpf_bpf_* metrics if any; ensure xpf_userspace_dp_* cover the same operator needs
• CLI tab completion: drop any bpf-prefixed commands

Phase 6 — branding cleanup (optional, can be deferred)

• Rename bpfrx-* references everywhere to xpf-* (already the project name)
• bpfrxd binary references — if not the current binary name, drop them
• This phase is cosmetic; could ship later in a separate cleanup PR

Documentation update list (Phase 1 detailed)

The 75+ docs identified in the survey, grouped by category:

Top-level / canonical:
CLAUDE.md, docs/engineering-style.md, docs/testing.md, docs/development-workflow.md, docs/test_env.md

Architecture / pipeline (most BPF-heavy):
docs/userspace-dataplane-architecture.md, docs/afxdp-packet-processing.md, docs/afxdp-module-split.md, docs/shared-umem-plan.md, docs/xdp-io-uring-userspace-dataplane.md, docs/userspace-xdp-pass-bootstrap-and-ipv6.md, docs/userspace-fabric-redirect-fix.md, docs/userspace-master-merge-20260310.md

Feature/decision docs:
docs/dataplane-decision-dpdk-vs-vpp.md, docs/vpp-dataplane-assessment.md, docs/userspace-dnat-plan.md, docs/userspace-icmp-te-debugging.md, docs/syn-cookie-flood-protection.md, docs/embedded-radvd.md, docs/services-application-identification.md, docs/fabric-cross-chassis-fwd.md, docs/userspace-native-gre-plan.md, docs/sync-protocol.md, docs/fabric-performance-optimizations.md

HA / failover:
docs/ha-cluster-test-plan.md, docs/userspace-ha-validation.md, docs/bug-heartbeat-vrf-rebind-split-brain.md, docs/session-sync-architecture.md

Backlogs + audits:
docs/feature-gaps.md, docs/userspace-dataplane-gaps.md, docs/perf-ranked-backlog.md, docs/authoritative-backlog.md, docs/userspace-performance-plan.md, docs/refactoring-audit.md, docs/bugs.md, docs/phases.md

Fairness / CoS validation (recent active):
docs/fairness-regimes.md

Historical PR plans (deprecation banner, don't rewrite):
docs/pr/881-cpu-windows/plan.md, docs/pr/814-max-interfaces/plan.md + reviews

Risk + rollback

Risks:

1. Hidden feature regression: a corner-case eBPF behavior we didn't audit (e.g. a specific NAT64 edge case, or a specific BPF screen check) doesn't have a userspace equivalent. Mitigation: Phase 0 audit + smoke matrix per phase.
2. External customers: anyone running production on the eBPF dataplane gets bricked. Mitigation: deprecation announcement in Phase 0 with timeline; major version bump on Phase 3+ merge.
3. Performance regression for specific workloads: eBPF native XDP on PF passthrough has measurable advantages over userspace AF_XDP for raw packet rate in some scenarios. Mitigation: document the tradeoff; perf-test gate per phase using fairness-eval + iperf3 12-stream baseline.
4. HA failover: if userspace-dp has any subtle HA gap vs eBPF (RETH virtual MAC, fabric forwarding, sync protocol), retirement exposes it. Mitigation: make test-failover + make test-ha-crash + make test-restart-connectivity smoke per phase merge.

Rollback:
Each phase is a separate PR. Revert is one git revert <phase-PR> away. Phase 4 (BPF source removal) is the point-of-no-return; before that PR merges, verify all prior phases have been stable for at least N days on canonical cluster.

Out of scope

• DPDK worker pipeline (dpdk_worker/, pkg/dataplane/dpdk/) — separate experimental path; retirement is independent. Keep DPDK for now.
• Junos config syntax / compiler — no change; CoS / NAT / etc. configs continue to compile, just to one backend.
• cmd/xpfd daemon binary name — stays xpfd.
• Any post-retirement userspace-dp performance refactors (Refactor: userspace-dp/src/protocol.rs — split by domain (snapshot DTOs vs control protocol vs binding status) #1325-Refactor: collapse repeated 9-10-param pub-fn clusters in session/mod.rs (5 fns) and flowexport.rs (finalize_flow) into context structs #1357 ongoing) — separate workstream; this issue retires eBPF, doesn't refactor userspace.

Smoke gate per phase merge

Required before each phase PR merges:

• Pass A (CoS disabled): v4+v6 × push+reverse × multi-stream -P 12 -R — line rate, 0 retrans
• Pass B (CoS enabled): per-class 5200-5211 + echo 6200-6211 — all 24 measurements pass with 0 retrans on unshaped classes, shaped classes hit configured rate cleanly
• make test-failover — 0 packet loss across reboot
• make test-ha-crash — daemon-stop + force-stop + multi-cycle recovery
• make test-restart-connectivity — 0 packet loss during daemon restart

Refs

• CLAUDE.md describes the dual pipeline and notes "eBPF cluster (bpfrx-fw0/1) is regression-only"
• docs/userspace-dataplane-gaps.md — current feature parity baseline (last updated 2026-03-14)
• docs/feature-gaps.md — vSRX comparison; should be unaffected by dataplane choice
• Recent userspace-dp work this session: PRs Size low-rate CoS iperf buffers for reverse fairness #1316, config: typed leaf-value schema (Phase 1 + schedulers Phase 2) for #1319 #1320, Refactor: userspace-dp/src/bin/fairness-eval.rs — extract 383-LOC main() into orchestrator + phase helpers #1330/Refactor fairness-eval into library modules #1332, Add issue 1321 validation contract gates #1335, Fix post-merge validation contract gaps #1341, Fix surplus mouse-latency matrix fixture selection #1353, Add paced persistent mouse latency probes #1358, Fail mouse-latency cells below valid-rep target #1361, Refactor CoS fixture port grid #1366

[psaab]

Phase 0 feature-parity audit — gap issues filed

Audit walked the BPF dataplane source of truth (bpf/xdp/*.c, bpf/tc/*.c, bpf/headers/*.h, pkg/dataplane/*.go) and the userspace-dp implementation (userspace-dp/src/, pkg/dataplane/userspace/) at current HEAD post-#1366. Existing docs/userspace-dataplane-gaps.md (2026-03-14) baseline confirmed; gaps that are production-relevant filed as separate issues, blocking this umbrella.

Filed as blockers for Phase 4 (BPF source removal):

Issue	Severity	Area	Gate today
#1374	BLOCKER	SYN cookie flood protection (kernel kfunc → no userspace equivalent)	Capability-gate falls back to eBPF
#1375	BLOCKER	Three-color policers (RFC 2697/2698)	Capability-gate falls back to eBPF
#1376	BLOCKER	Port mirroring (forwarding-options port-mirroring)	Capability-gate falls back to eBPF
#1377	BLOCKER	Address-persistent SNAT pool mode silently degrades to round-robin	No gate — correctness regression on userspace today
#1378	BLOCKER	Time-based policy schedulers (schedulers { ... }) not propagated to userspace	No gate — scheduled policies do not activate on userspace today
#1379	BLOCKER	Dataplane events (PolicyDeny / ScreenDrop / FilterLog) absent from userspace event stream	Currently masked by eBPF ringbuf still running
#1380	nice-to-have	show system buffers reads BPF map utilization; no userspace equivalent yet	Phase 5 acknowledged; filing for tracking
#1381	BLOCKER (Phase 3 plumbing)	DataPlane interface is BPF-shaped; userspace Manager embeds eBPF Manager	Architectural — must resolve before rm -rf bpf/

Verified covered (do not re-file)

Cross-checked the gaps doc against HEAD — these are genuinely implemented in userspace-dp at the source level: NPTv6, NAT64 forward+reverse + embedded ICMP, NAT pool mode (modulo address-persistent above), DNAT/SNAT/static NAT, zone+global policies, application matching (incl. multi-term apps), stateful conntrack (incl. TCP RST/FIN closing + configurable timeouts), 11 screen checks (land, syn-fin, no-flag, fin-no-ack, winnuke, ping-death, teardrop, icmp-frag, syn-frag, source-route, plus syn-flood/icmp-flood/udp-flood rate limits + port-scan + ip-sweep + session-limit src/dst), firewall filters (incl. forwarding-class + DSCP rewrite + routing-instance PBR + hit counters), TCP MSS clamping (all-tcp / ipsec-vpn / gre-in / gre-out), VLAN ingress + egress tagging, NetFlow v9, fabric cross-chassis forwarding, session delta sync, allow-dns-reply, allow-embedded-icmp, native GRE encap/decap, slow-path TUN inject.

HA/cluster (RETH virtual MAC, VRRP, fabric overlay, IPsec SA sync, ISSU) is Go-side and unaffected by BPF retirement — verified that pkg/cluster/, pkg/vrrp/, pkg/routing/, pkg/ipsec/, pkg/dhcp/, pkg/dhcpserver/, pkg/frr/, pkg/feeds/, pkg/rpm/, pkg/snmp/ paths don't depend on bpf/ source.

Recommendation

Address #1381 first (interface plumbing) since every other gap-fix depends on having a userspace-only DataPlane surface to write to. Then #1379 (events — affects every other observability surface) and #1377/#1378 (silent correctness regressions today). #1374/#1375/#1376 can land in any order before Phase 4; they have explicit capability gates so Phase 0–2 can ship without them as long as Phase 4 waits.

No docs-rot bugs found in docs/userspace-dataplane-gaps.md — every "Implemented" row was confirmed at HEAD.