Feature gap: three-color policers (RFC 2697/2698) implemented in eBPF, missing from userspace-dp

[psaab]

Gap

The eBPF dataplane implements two-rate three-color (RFC 2698, POLICER_MODE_TWO_RATE) and single-rate three-color (RFC 2697, POLICER_MODE_SR3C) policers in xpf_helpers.h:1487+. The userspace dataplane has only single-rate two-color policers. Configs with firewall three-color-policer ... are explicitly rejected by the userspace capability gate.

eBPF implementation (source of truth)

• bpf/headers/xpf_common.h:830-851 — struct policer_config with committed_rate, peak_rate, committed_burst, peak_burst for two-rate three-color
• bpf/headers/xpf_helpers.h:1531 — Two-rate three-color (RFC 2698) evaluation
• bpf/headers/xpf_helpers.h:1557 — Single-rate three-color (RFC 2697): CIR fills C, C overflow fills E
• pkg/dataplane/compiler_filter.go:57-87 — compiles ThreeColorPolicers from config into BPF maps
• pkg/dataplane/types.go:851-852 — PolicerModeTwoRate, PolicerModeSR3C constants
• pkg/dataplane/userspace/manager.go:767-768 — capability gate: if len(cfg.Firewall.ThreeColorPolicers) > 0 { addReason(...) }

Userspace-dp gap

Searched: grep -rn 'three.color\\|three_color\\|ThreeColor' userspace-dp/src/ — zero matches.

userspace-dp/src/filter/ implements only flat rate + burst (single-rate two-color via token_bucket.rs).

Recommended fix

In userspace-dp/src/filter/:

1. Extend PolicerConfig (or add ThreeColorPolicerConfig) with peak_rate / peak_burst / excess_burst fields
2. Implement RFC 2697 srTCM: C/E buckets with CIR fill, C overflow into E
3. Implement RFC 2698 trTCM: C/P buckets, both filled independently at CIR/PIR
4. Plumb three-color action (green/yellow/red) to DSCP-rewrite + discard decision
5. Surface three-color policer snapshots through pkg/dataplane/userspace/protocol.go and policer_state in Rust
6. Remove the cfg.Firewall.ThreeColorPolicers gate in pkg/dataplane/userspace/manager.go:767

Blocker for #1373

This must land before Phase 4 (BPF source removal) of #1373 (retire eBPF dataplane). Currently the userspace capability gate falls back to the eBPF dataplane when this config is set.

---

Refined contract (added 2026-05-17 after triple-review of #1384)

See docs/pr/1373-retire-ebpf-dataplane/plan-1375-three-color-policers.md for the full implementation contract refined through 4 rounds of Claude+Codex+Gemini Pro 3 review. New since the original issue body:

• Risks called out: token overflow/math (refill mul stays in u128 or saturates before packet-size conversion), atomicity (per-policer sharded state must avoid cross-worker false sharing while keeping one logical bucket per identity), color semantics (color-aware mode must never promote yellow/red — one wrong branch turns security into bandwidth grant), counter attribution (green/yellow/red/drop counters survive snapshot rebuilds by stable identity).
• Required test added: Go deriveUserspaceCapabilities() admits three-color policer configs only after userspace snapshot + Rust runtime support are wired, and rejects them before that point.

[psaab]

Implementation contract

Acceptance criteria

1. `userspace-dp/src/filter/policer.rs` (or equivalent) extends the policer model with:
   • srTCM (RFC 2697): `committed_rate`, `committed_burst`, `excess_burst`; C bucket fills at CIR, overflow into E bucket.
   • trTCM (RFC 2698): `committed_rate`, `peak_rate`, `committed_burst`, `peak_burst`; C and P buckets fill independently at CIR/PIR.
2. Three-color action: token availability determines green / yellow / red disposition.
3. Three-color action plumbs to:
   • DSCP rewrite per color (e.g. green → DSCP X, yellow → DSCP Y, red → drop or DSCP Z)
   • Discard decision for red traffic (configurable: drop or count-and-mark)
4. `PolicerConfig` snapshot fields surfaced through `pkg/dataplane/userspace/protocol.go` + matching Rust types.
5. Capability gate at `pkg/dataplane/userspace/manager.go:767-768` for `cfg.Firewall.ThreeColorPolicers` is removed.

Test plan

• Cargo: `policer::srTCM_green_yellow_red_at_thresholds` — drive traffic at `< CIR`, `CIR < r < CIR+EBS`, `> CIR+EBS`; assert color matches at packet boundaries.
• Cargo: `policer::trTCM_independent_CIR_PIR` — verify C and P buckets fill independently; green requires both C and P tokens.
• Cargo: `policer::srTCM_bucket_math_boundary_inputs` — `rate=0` rejects; `rate=u32::MAX` doesn't overflow; `burst=0` behaves degenerately as expected.
• Cargo: `policer::three_color_dscp_rewrite` — verify DSCP rewrite per color.
• Go: `pkg/dataplane/userspace/snapshot_test.go` round-trips three-color policer config.
• Integration: lab traffic generator at controlled rates against userspace cluster with three-color policer config; tcpdump verifies DSCP rewrite + drop semantics.

Dependencies

#1381 should land first — capability gate removal needs the userspace Manager interface clean.

Size estimate

M (4-7 days). Bucket math is well-defined by RFCs; the work is plumbing + tests + DSCP-action wiring.

[psaab]

Plan-review round 1 — PLAN-NEEDS-MAJOR

Reviewer	Verdict
Codex	PLAN-NEEDS-MAJOR (10 concrete gaps)
Gemini Pro 3	PLAN-KILL (claims existing DPDK srTCM is broken; valid bug but separate scope)

Convergent substantive findings

1. Boundary math: use u128 refill math like CoS does, not f64. Existing filter policer at userspace-dp/src/filter/policer.rs:31 uses f64 — inadequate. Define/reject rate=0, burst=0, u32::MAX rate/burst explicitly.
2. srTCM E-bucket semantics must be explicit. RFC 2697: C fills first; E fills only when C is full AND EBS remaining. Existing DPDK implementation at dpdk_worker/policer.c:42-43 has a real bug: c_tokens is clamped before the overflow check, so the overflow path is dead and E permanently 0. Gemini caught this; Codex also flagged. This is a separate bug to file, not part of this plan — but plan must NOT inherit the bad reference.
3. trTCM CIR/PIR must fill independently. RFC 2698 — common bug is coupling P refill to C depletion. Plan v2 must explicitly state independent fill.
4. Color-blind vs color-aware mode: RFC 2697/2698 both define. Junos default per current docs is color-aware (omitting color-blind keyword). Config parses ColorBlind (compiler_firewall.go:74) but snapshot ignores it. Plan must plumb the flag and respect both modes.
5. Compile-time validation missing: must reject rate=0, burst=0, PIR < CIR, PBS < CBS. Current compiler accepts zero/missing three-color fields.
6. Time base: userspace-dp/src/afxdp/neighbor.rs:3 monotonic_nanos, NOT DPDK rte_rdtsc cross-core drift risk.
7. DSCP/action surface incomplete: PolicerSnapshot (protocol.go:327 / protocol.rs:547) only carries two-color fields. Per-color DSCP-rewrite/drop needs explicit snapshot fields and TX-path wiring.
8. Hot-path RMW: ForwardingState is ArcSwap; FxHashMap<String, PolicerState> mutable map is not a production model. Per-rule packed atomic / CAS-shaped or sharded state required. Flow-cache hits MUST still run the policer.
9. Per-color counters missing: green_packets, yellow_packets, red_packets (+ bytes) need status fields + Go protocol + Prometheus.

Recommended plan v2

• u128 token math
• Explicit srTCM overflow: "C filled at CIR; on packet arrival, if C ≥ size, deduct from C; else if E ≥ size, deduct from E; else red. E refills ONLY when C is full and EBS remaining."
• Explicit trTCM: "C fills at CIR independent; P fills at PIR independent; green requires both C and P; yellow requires only P; red otherwise"
• Color-blind + color-aware mode flags
• Compile-time validation: positive rates, PIR ≥ CIR, PBS ≥ CBS
• monotonic_nanos time base
• Per-color DSCP map snapshot field
• Per-rule sharded atomic state
• Per-color counters
• File a separate bug for the DPDK srTCM dead-overflow path (separate from this plan)

Size: L (1-2 weeks), not M. Snapshot + schema + Rust metering + action wiring + cache-hit behavior + status + tests = real work.

Codex task: `task-mp8rgkco-imd0yz`. Gemini Pro 3 task: `task-mp8rguaz-njmebb`.

[psaab]

Update after #1395: keep open, but narrow to runtime three-color policer wiring. Landed: integer srTCM/trTCM core, snapshot schema, validation tests. Remaining: forwarding-path and flow-cache-hit enforcement, stable/sharded policer state, per-color counters/status/CLI/Prometheus, and capability gate removal.