Feature gap: 'show system buffers' CLI reports BPF map utilization; no userspace equivalent

[psaab]

Gap

show system buffers reads BPF map fill / utilization from dp.GetMapStats() and prints per-map type / max / used / usage% / warning thresholds plus session counts. After eBPF removal, this command will return either nothing or a degenerate output. The CLI surface needs a userspace-equivalent that reports operator-relevant resource usage (worker queue depths, UMEM frame counts, sharded session table fill, flow cache occupancy, etc.).

#1373 Phase 5 acknowledges this — filing as a tracked dependency for Phase 0 audit and to make the surface concrete.

eBPF implementation (source of truth)

• pkg/cli/cli_show_system.go:20-65 — showSystemBuffers() calls c.dp.GetMapStats() and renders per-map utilization, warns at ≥80%, critical at ≥90%
• pkg/cli/cli_show_system.go:67-... — showSystemBuffersDetail() for deep dive
• pkg/dataplane/maps.go — GetMapStats() enumerates BPF map fill counts via BPF_MAP_GET_NEXT_KEY walk

Userspace-dp gap

pkg/dataplane/userspace/manager.go does not override GetMapStats(). It inherits the eBPF method via the embedded dataplane.DataPlane (line 58), which today reads the BPF map pins. With eBPF removed:

• session table (shared & per-binding), UMEM frame usage, worker queues, flow cache, sharded neighbor table, TX rings have no GetMapStats-compatible surface
• pkg/cli/cli_show_system.go:36 filters out "Array" / "PerCPUArray" — these are exactly the userspace-side counters that would be relevant

Operators rely on this command to spot near-full session/policy/conntrack maps before they cause drops.

Recommended fix

Either:

1. Rewrite the CLI handler: in pkg/cli/cli_show_system.go::showSystemBuffers(), branch on dp.Mode(). For userspace mode, query the existing status surface (pkg/dataplane/userspace/protocol.go already exposes per-binding stats, session counts, UMEM frame counts, queue depths, sharded session table fill) and render an equivalent table.
2. Or remove the command in favor of show system userspace status (already exists) and add a deprecation warning if that's the consensus.

Per the #1373 Phase 5 plan, option 1 keeps operator muscle memory; the column set just changes from BPF maps to userspace resource pools.

Blocker for #1373

Operator-facing visibility regression at Phase 4 (BPF source removal) unless the CLI is rewritten or replaced by then.

---

Refined contract (added 2026-05-17 after triple-review of #1384 and code review of #1386)

See docs/pr/1373-retire-ebpf-dataplane/plan-1380-userspace-buffers.md for the full implementation contract. New since the original issue body:

• Phase classification corrected to Phase 5 (observability cleanup), NOT Phase 4 (BPF source removal). Feature gap: 'show system buffers' CLI reports BPF map utilization; no userspace equivalent #1380 must land before BPF-map-oriented operator surfaces disappear but does not block the Phase 4 forwarding-source removal gate by itself.
• Show userspace buffers from helper status #1386 (merged) is the buffer-display parity fix — preserves Active sessions footer on userspace mode, wires detail flag properly, per-row fallback for mixed PerBinding/Bindings snapshots. Full userspace-equivalent capacity reporting still owed by Feature gap: 'show system buffers' CLI reports BPF map utilization; no userspace equivalent #1380.

[psaab]

Implementation contract

Acceptance criteria

1. `pkg/cli/cli_show_system.go::showSystemBuffers()` branches on `dp.Mode()`. For userspace mode, the function reads from the existing userspace status surface (`pkg/dataplane/userspace/protocol.go`) and renders a table with operator-relevant fill metrics.
2. Userspace branch columns (suggested): per-worker queue depth (TX pending / TX prepared / shared recycle), per-binding UMEM frame counts (pending fill / spare fill / free TX / pending recycle), session table fill (shared & per-binding), flow cache occupancy, sharded neighbor table fill.
3. Warning thresholds preserved: ≥80% utilization = warning row, ≥90% = critical row.
4. `show system buffers detail` (`showSystemBuffersDetail`) also branches and renders the deeper view for userspace.
5. CLI output for eBPF mode is unchanged — pure additive branch.

Test plan

• Go: `pkg/cli/cli_show_system_test.go` adds case with a mock userspace status containing edge values (0%, 79%, 80%, 89%, 90%, 100% utilization); verify warning/critical thresholds fire at the right rows.
• Go: case where userspace status is missing/zero — render "no userspace data" gracefully, not panic.
• Integration: `cli> show system buffers` against running userspace cluster shows actual queue depths matching what `show system userspace status` reports.

Dependencies

#1381 should land first — the userspace branch needs a clean `ControlPlane` (or equivalent) surface to read from without inheriting BPF-map-shaped methods.

Size estimate

S (1-2 days). Pure CLI plumbing — the underlying status data already exists in `pkg/dataplane/userspace/protocol.go`. The work is rendering + threshold logic + test coverage.

Open question

Per #1373 Phase 5: should this be a single `show system buffers` with branching, or should the userspace mode redirect to `show system userspace status` (which already exists)? Default to branching to preserve operator muscle memory.

[psaab]

Plan-review round 1 — PLAN-NEEDS-MAJOR

Reviewer	Verdict
Codex	PLAN-NEEDS-MAJOR (8 findings; including a compile-time blocker)
Gemini Pro 3	PLAN-KILL (some fabricated rationale; some valid concerns)

Compile-time blocker (Codex)

c.dp.Mode() does not compile against the DataPlane interface. DataPlane (at pkg/dataplane/dataplane.go:101) has GetMapStats() but NOT Mode(). Only the concrete userspace Manager has Mode() (pkg/dataplane/userspace/manager.go:176). Plan must use a type assertion or carry Mode via the existing Status API, not call a non-existent interface method.

Gemini's KILL rationale claims "Mode() locks the mutex during teardown" — that's fabricated; the real bug is the method doesn't exist on the interface. Gemini's verdict was right for the wrong reason.

Other Codex findings

1. Column width: existing output is 78 chars (cli_show_system.go:20). Userspace metrics need a different table — different column set, different denominators.
2. Threshold denominator semantics: GetMapStats() gives used / max = fill ratio. UMEM frames, session-table slots, neighbor cache each have different denominators. Plan must specify which metrics have capacities for the ≥80% warning.
3. Status() errors / torn-down helper: must render "userspace buffer data unavailable", not panic.
4. Per-binding vs aggregate: status is N bindings × queues (protocol.go:681). Recommended: aggregate-max-pressure first row, then bounded per-binding pressure rows. Don't dump flow-worker map by default.
5. Remote CLI + gRPC: cmd/cli/show.go:1528 uses ShowText("buffers"); pkg/grpcapi/server_show.go:1718 duplicates the BPF table. Plan must update BOTH paths or share rendering.
6. Tests missing: no pkg/cli/cli_show_system_test.go exists. Mock-status pattern from pkg/cli/cli_show_security_test.go:13 is the template.
7. Operator clarity: header must say "userspace dataplane buffer/ring pressure", NOT "BPF map fill".

Gemini's valid concerns

• "Hijacking command semantics breaks operator mental model" — fair point. Decide: branch or redirect to existing show system userspace status. Codex notes the existing userspace status command is a broader status surface (cli_show_cluster.go:202), so branching is justified for buffer-specific output.
• "Test mock strategy" — same as Codex, addressed via existing security-test pattern.
• "C-level driver updates" — false; AF_XDP fields are already in protocol.rs/protocol.go. No driver changes needed.

Recommended plan v2

• Pick the dispatch mechanism: extend DataPlane interface with Mode() string OR use *ebpf.Manager / *userspace.Manager type assertion in cli_show_system.go. Recommended: interface extension (cleaner; aligns with Feature gap: DataPlane interface contract is BPF-shaped; userspace Manager embeds eBPF Manager #1381 directions).
• Define exact userspace column schema: queue depth | UMEM frames | session-table fill | flow cache | neighbor cache, each with bounded capacities for thresholding
• Specify "no capacity" rendering: for metrics without bounded capacity (e.g. flow-cache occupancy where eviction is LRU), render value without Usage%
• Aggregate-first row, then per-binding — bound row count
• Update gRPC + cmd/cli paths, not just pkg/cli
• Add cli_show_system_test.go with synthetic-status fixtures
• Header text: "Userspace dataplane buffer pressure" when dp.Mode() == userspace

Size: M (3-5 days), not S. The CLI/gRPC dual-path work + test coverage is more than a 1-2 day plumbing.

Codex task: `task-mp8rh7xq-1uxi9p`. Gemini Pro 3 task: `task-mp8rhil6-pexzef`.

[psaab]

Update after #1386: keep open, but narrow to remaining userspace buffer/status breadth. Landed: CLI/gRPC userspace buffer rendering from helper Status(), active-session footer, detail handling, mixed-version capacity fallback. Remaining: decide whether show system buffers must include session-table fill, flow-cache occupancy, neighbor-cache fill, and broader worker queue pressure beyond AF_XDP UMEM/TX rings.