Feature gap: port mirroring implemented in eBPF, missing from userspace-dp

[psaab]

Gap

The eBPF dataplane implements forwarding-options port-mirroring by reading a mirror_config map keyed on ingress ifindex and cloning the packet to the configured output interface with optional 1-in-N sampling. The userspace dataplane has zero implementation. Configs with forwarding-options port-mirroring are explicitly rejected by the userspace capability gate.

eBPF implementation (source of truth)

• bpf/headers/xpf_common.h:888 — struct mirror_config { __u32 output_ifindex; __u32 rate; }
• bpf/headers/xpf_maps.h:824-826 — mirror_config BPF map definition
• bpf/xdp/xdp_forward.c:330-340 — XDP-side ingress mirror: look up mirror_config by ingress_key, redirect clone
• bpf/tc/tc_forward.c:73-109 — TC-side mirror with sampling rate, bpf_clone_redirect() to output ifindex
• pkg/dataplane/compiler.go:1506-1547 — compilePortMirroring() populates the BPF map
• pkg/dataplane/maps.go:831 — SetMirrorConfig() writes a mirror entry
• pkg/dataplane/userspace/manager.go:775-777 — capability gate

Userspace-dp gap

Searched: grep -rn 'port.mirror\\|port_mirror\\|PortMirror' userspace-dp/src/ — only generic mirroring-as-prose matches (unrelated to port mirroring), no implementation.

Recommended fix

In userspace-dp/src/afxdp/:

1. Add MirrorConfig { output_ifindex, rate } snapshot + protocol field
2. On packet ingress, after policy/forwarding decision: if mirror rule matches, allocate a copy frame from UMEM, build cloned packet, push to output ifindex TX queue
3. Implement 1-in-N sampling via per-binding counter
4. Surface mirror packet counters in status
5. Remove the cfg.ForwardingOptions.PortMirroring != nil gate in pkg/dataplane/userspace/manager.go:775

Note: UMEM is per-binding queue-bound — mirror to a different physical interface requires either a cross-binding inject (similar to fabric redirect) or attaching a slow-path TUN sink.

Blocker for #1373

This must land before Phase 4 (BPF source removal) of #1373 (retire eBPF dataplane). Currently the userspace capability gate falls back to the eBPF dataplane when this config is set.

---

Refined contract (added 2026-05-17 after triple-review of #1384)

See docs/pr/1373-retire-ebpf-dataplane/plan-1376-port-mirroring.md for the full implementation contract refined through 4 rounds of Claude+Codex+Gemini Pro 3 review. New since the original issue body:

• Risks called out: mirror backpressure (mirror egress drop must not stall primary forwarding), failover handoff (no mirror-clone state sync between peers — new active mirrors per current config), buffer doubling (mirror clone must not double-charge UMEM accounting).
• Required test added: Go deriveUserspaceCapabilities() admits port-mirroring configs only after userspace snapshot + Rust runtime support are wired.

[psaab]

Implementation contract

Acceptance criteria

1. `MirrorConfig { output_ifindex: u32, rate: u32 }` defined in `userspace-dp/src/afxdp/` snapshot + matching Go protocol field.
2. On packet ingress, after policy/forwarding decision, if mirror rule matches the ingress ifindex: allocate clone frame from UMEM, copy packet bytes, push to output ifindex TX queue.
3. 1-in-N sampling via per-binding counter (`rate == 0` means mirror every packet; `rate == N` means 1-in-N).
4. Cross-binding mirror (mirror to different physical interface than ingress) handled via either cross-binding inject (similar to fabric redirect) or slow-path TUN sink. Document which path is taken.
5. Mirror packet counters surfaced in `Status` for operator visibility: `mirrored_packets`, `mirrored_bytes`, `mirror_drops_no_frame`.
6. Capability gate at `pkg/dataplane/userspace/manager.go:775-777` for `cfg.ForwardingOptions.PortMirroring` is removed.

Test plan

• Cargo: `mirror::sampling_rate_correctness` over N×K packets — verify floor(N/rate) packets are mirrored ±1 (off-by-one for boundary inputs).
• Cargo: `mirror::cross_binding_inject` — synthesize a packet on binding-A with mirror destination binding-B's interface; verify packet appears on binding-B's TX queue.
• Cargo: `mirror::out_of_frame_drops_increment_counter` — exhaust UMEM, attempt mirror, assert `mirror_drops_no_frame` increments without panicking.
• Integration: cluster smoke with mirror config + tcpdump on output interface; verify packet count matches sampling rate against ingress count.

Dependencies

#1381 should land first — capability gate removal needs the userspace Manager interface clean. Mirror's cross-binding inject path may also benefit from the same primitives the fabric forwarder uses.

Size estimate

L (1-2 weeks). UMEM is per-binding queue-bound; cross-binding inject adds complexity. The sampling rate logic is small; the integration path is the work.

[psaab]

Plan-review round 1 — PLAN-NEEDS-MAJOR

Reviewer	Verdict
Codex	PLAN-NEEDS-MAJOR (8 substantive findings)
Gemini Pro 3	PLAN-KILL (cross-binding UMEM concern is real but addressable)

Convergent substantive findings

1. Cross-binding UMEM: drop the TUN-sink option. TUN is L3 (slowpath.rs:349); port mirroring needs L2 preservation. Per docs/userspace-capture-plan.md:388, TUN strips Ethernet — wrong primitive. Use cross-binding inject (copy packet bytes into destination binding's UMEM/queued TX path). Matches existing invariant at umem/README.md:27.
2. Sampling counter scope: prefer per-worker/per-binding counters (matches eBPF's per-CPU shape at tc/tc_forward.c:92). If contract requires exact global 1-in-N per rule, hot atomic fetch_add is required and likely too expensive — pick per-binding approximate semantics OR pay the atomic cost explicitly.
3. Existing eBPF supports ONE mirror_config per ingress ifindex (xpf_maps.h:815). SetMirrorConfig overwrites; compiler.go:1506-1547 doesn't define overlap precedence. Plan must reject duplicate inputs OR define deterministic precedence; "all matching rules" is a different data model.
4. Clone allocation fairness: mirror traffic MUST be discardable. Normal RX/TX wins; mirror only consumes frames above a reserve OR from a bounded budget; no-frame increments mirror_drops_no_frame. Plan needs explicit budget.
5. IPv6/full payload: Junos describes port mirroring as copying full packet, distinct from sampling. Plan v2 must explicitly copy L2+L3+L4; check Junos output-rate shape if needed.
6. Counters missing: mirrored_packets, mirrored_bytes, mirror_drops_no_frame, mirror_drops_no_binding, mirror_drops_queue_full. Plumb through Rust → Go → Prometheus + CLI.
7. Tests: cargo for sampling correctness, cross-binding inject, frame exhaustion, missing dst binding, overlap rejection, v4/v6 full-frame preservation. Integration: cluster smoke with tcpdump verification AND primary-forwarding-survives-pressure assertion.
8. Non-existent output_ifindex: compile-time skip + log (matches current eBPF compiler.go:1522); runtime missing binding = drop-and-account, not silent disappear.

Gemini's KILL — overreaches

Gemini claims "cross-binding inject is fundamentally infeasible." But Codex correctly notes: copy packet bytes into the destination binding's UMEM. That's not zero-copy across bindings (which IS impossible) — it's a memory copy on the path that's already not the hot path (mirror is opt-in, low-rate, by design). Gemini's KILL ignores that mirroring is a low-rate diagnostic feature, not a per-packet primitive.

Recommended plan v2

• Cross-binding inject ONLY (drop TUN sink reference)
• Per-binding sampling counter (approximate 1-in-N, matches eBPF per-CPU behavior)
• Deterministic overlap precedence: first-match by ingress_ifindex; document the precedence; reject duplicate ingress_ifindex at commit
• Mirror budget: bounded reserve frame pool per binding; no-frame drops counted; mirror discardable under load
• Full L2+L3+L4 packet copy (no truncation)
• All 5 counters plumbed through Prometheus + CLI
• Test plan expanded to cover all 7 angles above

Size: L (1-2 weeks) if #1381 lands first with a clean cross-binding inject primitive; XL otherwise.

Codex task: `task-mp8rgwbp-tqrnra`. Gemini Pro 3 task: `task-mp8rh6fl-033gkn`.

[psaab]

Update after #1392: keep open, but narrow to runtime mirror delivery. Landed: mirror snapshot DTO/building, duplicate ingress rejection, Rust wire round-trip. Remaining: full-L2 cross-binding clone/inject, sampling in the worker path, mirror backpressure/drop accounting, counters/status, and capability gate removal.