Userspace three-color policer core#1395
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a userspace three-color policer core (RFC 2697 srTCM and RFC 2698 trTCM) as the math/state foundation for issue #1375, while also migrating the legacy two-color token bucket off f64 to scaled u128 integer math. The change is intentionally scoped to the policer module: snapshot plumbing, forwarding-path wiring, flow-cache integration, per-color counters/status, and removal of the Go capability gate are explicitly deferred and called out in both the PR description and the updated README.
Changes:
- Replace
f64-based token math in legacyPolicerStatewithu128tokens scaled byTOKEN_SCALE=1e9and arate_bits_per_sec: u64refill rate. - Add
ThreeColorPolicerStateimplementing srTCM and trTCM with color-aware/color-blind modes,ColorTreatment(DSCP rewrite / drop) per color, and validation errors. - Add focused unit tests covering threshold transitions, srTCM C→E overflow, trTCM independent CIR/PIR refill, color-aware non-promotion, color-blind ignore, u64-boundary inputs, and DSCP rewrite plumbing.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| userspace-dp/src/filter/README.md | Documents the integer token-math switch and the new three-color meter core, plus an explicit list of still-gated follow-on work. |
| userspace-dp/src/filter/policer.rs | Reworks PolicerState to scaled-u128 tokens, adds ThreeColorPolicerState, PacketColor, ColorTreatment, ThreeColorTreatments, ThreeColorDecision, PolicerConfigError, and unit tests. |
Claude round-1 review on
|
Claude round-1 self-correction on
|
Round-1 quad-review consolidated synthesis on
|
| Reviewer | Verdict |
|---|---|
| Claude | MERGE-READY → MERGE-NEEDS-MAJOR (self-corrected) |
| Codex | MERGE-NEEDS-MAJOR |
| Gemini Pro 3 | MERGE-READY |
| Copilot | (no review yet on this HEAD) |
Reviewer disagreement: scope philosophy
Codex and Gemini disagree on the same code. Both verified the math core is correct (u128 multiplication, color-aware preservation, srTCM/trTCM bucket math, DSCP rewrite, runtime PIR<CIR validation). Both verified test coverage of the math core. The disagreement is what counts as merge-ready:
- Gemini's framing: "Core engine satisfies RFC 2697/2698 mathematical rigorousness... gated follow-on wiring tasks explicitly demarcated in README." If the README is honest about what's deferred and the math is correct, this is a valid foundation slice. MERGE-READY.
- Codex's framing: "As a dormant core sketch, the token math is mostly sane. As PR Userspace three-color policer core #1395 satisfying Feature gap: three-color policers (RFC 2697/2698) implemented in eBPF, missing from userspace-dp #1375, it is not merge-ready." Feature gap: three-color policers (RFC 2697/2698) implemented in eBPF, missing from userspace-dp #1375's contract requires Go-side wiring + commit-time validation + counters + atomicity. The PR doesn't move those items.
Codex MAJORs (real gaps if scope = full #1375)
- Not wired Go-side —
manager.go:767still rejects three-color configs;protocol.go:331snapshot has only two-color fields;userspace-dp/src/protocol.rs:548mirrors only those. New core is dormant code with no caller. - Compile-time validation missing —
pkg/config/compiler_firewall.go:64-137parses three-color configs but doesn't validate PIR<CIR / PBS<CBS at commit. Invalid config commits, then fails at dataplane construction. Per Feature gap: three-color policers (RFC 2697/2698) implemented in eBPF, missing from userspace-dp #1375 contract: validate at compile. - No per-color/drop counters — required by Feature gap: three-color policers (RFC 2697/2698) implemented in eBPF, missing from userspace-dp #1375 contract for "stable identity across snapshot rebuilds." Existing
FilterTermCounteris filter-term-keyed, not policer-keyed. - No atomicity story —
ThreeColorPolicerStateis plain&mut self. Multi-worker access requires sharding/CAS. README admits this is required before enablement.
What both reviewers verified correct
- u128 refill math (no overflow at any reasonable rate × elapsed)
- Color-aware never promotes incoming yellow/red (lines 284-301 of policer.rs)
- trTCM independent CIR/PIR bucket refill
- DSCP rewrite per color
- Runtime PIR<CIR rejection at constructor (just not at config compile)
Claude miss pattern
Same as #1394 and #1396: labeled "foundation only" without verifying production caller path. Codex consistently flags this; Gemini accepts the README disclaimers.
My revised read (matching Codex): the scope question matters. If this PR claims to advance #1375, the Go-side wiring + commit validation are the load-bearing pieces and should be in scope. If this PR is explicitly the math core + atomicity-deferred, the README should say "Phase 1 of 4" not just "still gated."
Recommendation
Block on:
- Add PIR<CIR / PBS<CBS validation at Go
commit check(pkg/config/compiler_firewall.go), not just Rust constructor. - Decide on a counter shape (per-color, per-policer-ID) and include it in the foundation slice OR explicitly say "counters in PR #N."
Strongly consider:
- Sharded atomic state plan documented (even if not implemented).
- Saturating-arithmetic wrap-recovery instead of
if now_ns <= last_refill_ns return.
Defer to follow-up PRs (acknowledged by README):
- Go three-color snapshot wire format
- Hot-path stable policer IDs
- Flow-cache policer execution
- Capability gate removal
Codex task: task-mp955bjw-cz8j4y. Gemini Pro 3 task: task-mp9565kp-5zyyy1. Not merging — author's decision.
Round-2 update: schema + validation wired, runtime still fail-closedPushed What changed:
Validation run:
The Rust runs pass with the repository's existing warning set. |
Claude round-2 review on
|
Round-2 review consolidated on
|
| Reviewer | Verdict |
|---|---|
| Claude | MERGE-READY-pending → MERGE-NEEDS-MINOR (aligning with Codex) |
| Codex | MERGE-NEEDS-MINOR |
| Copilot | (no fresh review on this HEAD) |
Round-1 MAJORs status — all closed cleanly
| Round-1 MAJOR | Status |
|---|---|
| Not wired Go-side | ✓ three_color_policers field added; manager.go:767 capability gate stays closed for runtime |
| Commit-time PIR<CIR / PBS<CBS validation | ✓ compiler.go rejects before dataplane construction |
| Per-color/drop counters | ⊝ Acceptable deferral while forwarding stays gated |
| Atomicity story | ⊝ Acceptable deferral while forwarding stays gated |
The "deferred while gated" framing works because the userspace capability gate explicitly rejects three-color configs from reaching the dataplane. Author can wire schema + validation now, defer counters + atomicity until runtime ungating.
Codex r2 MINORs
1. Ambiguous configs not commit-rejected
"'Strict validation' still accepts ambiguous three-color policers with both
single-rateandtwo-rate;compiler_firewall.goprocesses both and letstwo-ratewin. It also does not reject bothcolor-awareandcolor-blind. Not runtime-dangerous while userspace stays gated, but it should be commit-rejected before ungating."
A config with conflicting mode flags should fail commit check, not silently pick one. Fix is small: reject in compiler.go when both flags are set.
2. End-to-end commit test gap
"PIR/CIR and PBS/CBS rejection is tested via
CompileConfig, notconfigstore.CommitCheck/Commit. The commit path does callcompileTree -> CompileConfig, so real commits should fail, but the PR claim would be better pinned with one configstore test."
CompileConfig is called by the commit path, so the rejection should work in practice. One configstore-level test would lock the contract.
Recommendation
Block on: nothing structural — both MINORs are small.
Strongly consider in this PR:
- Reject
single-rate AND two-rateset simultaneously at commit - Reject
color-aware AND color-blindset simultaneously at commit - Add one
configstore.CommitCheck/Committest exercising PIR<CIR rejection end-to-end
Defer to follow-up PRs (acknowledged): per-color counters, sharded atomic state, hot-path stable policer IDs, capability gate removal.
Codex task: task-mp96mmn0-4rvzhn. Not merging — author's decision.
Round-3 fix updatePushed What changed:
Validation:
|
Claude round-3 review on
|
Claude round-3 self-correction on
|
Round-3 review consolidated on
|
| Reviewer | Verdict |
|---|---|
| Claude | MERGE-READY → MERGE-NEEDS-MAJOR (self-corrected) |
| Codex | MERGE-NEEDS-MAJOR |
Codex MAJOR — Hierarchical duplicate-block bypass
set-command path correctly rejects ambiguous configs (verified by Codex). But load merge / load override with duplicate hierarchical blocks bypasses the check.
Worked failure:
three-color-policer bad {
single-rate { ... }
}
three-color-policer bad {
two-rate { ... }
}
parser.go:76 appends both blocks as siblings. compileFirewall at compiler_firewall.go:74 walks them and overwrites same-name ThreeColorPolicers map entries — last write wins. The compiled policer has only TwoRateConfigured=true; the new ambiguity check at compiler.go never sees both flags on one compiled entity.
Same pattern hides duplicate single-rate blocks where one has color-blind and another has color-aware because FindChild("single-rate") only inspects the first.
Round-2 MINOR 2 (configstore E2E) — partial
"
configstore/store_test.goonly addsCommitCheckcoverage for ambiguous color mode. PIR<CIR remains only covered throughCompileConfiginparser_ast_test.go."
Recommendation
Block on:
- Reject duplicate
three-color-policer <name>blocks at parse OR compile, OR accumulate flags across duplicates before the ambiguity check. - Reject duplicate
single-rate/two-ratesiblings within a single policer block.
Strongly consider: add configstore.Commit E2E test for PIR<CIR rejection.
Codex task: task-mp99djst-s2bdia. Not merging — author's decision.
Round-4 blocker fixes pushed:
|
Claude round-4 review on
|
Claude round-4 self-correction on
|
|
@copilot review Round-4 commit 591f4fb. Codex flagged 1 MAJOR: duplicate same-mode siblings still bypassed via FindChild taking first only. Worked failure: two single-rate blocks where second's color-aware silently overrides first's color-blind. Re-review against this specific failure mode at compiler_firewall.go:78. |
Agent-Logs-Url: https://github.com/psaab/xpf/sessions/f42ea8d5-2bfb-4a18-95e8-1b5dbdb6bd80 Co-authored-by: psaab <196946+psaab@users.noreply.github.com>
Agent-Logs-Url: https://github.com/psaab/xpf/sessions/f42ea8d5-2bfb-4a18-95e8-1b5dbdb6bd80 Co-authored-by: psaab <196946+psaab@users.noreply.github.com>
Agent-Logs-Url: https://github.com/psaab/xpf/sessions/f42ea8d5-2bfb-4a18-95e8-1b5dbdb6bd80 Co-authored-by: psaab <196946+psaab@users.noreply.github.com>
Fixed on 3f79822 (plus core fix in ac55aee): |
Round-4 quad-review consolidated synthesis on
|
| Reviewer | Verdict |
|---|---|
| Claude | MERGE-NEEDS-MAJOR (self-corrected from READY-pending after Codex MAJOR) |
| Codex | MERGE-NEEDS-MAJOR |
| Gemini Pro 3 | MERGE-NEEDS-MAJOR |
| Copilot | Re-review requested; still on stale commit d36b0e7d |
Codex + Gemini converge on the same single MAJOR. Block.
MAJOR — Duplicate same-mode siblings silently bypassed (Codex + Gemini)
pkg/config/compiler_firewall.go:78:
if sr := tcpInst.node.FindChild("single-rate"); sr != nil {FindChild only returns the FIRST matching child. Duplicate same-mode siblings in one three-color-policer block are silently dropped.
Worked failure (Codex + Gemini both reproduced):
firewall {
three-color-policer bad {
single-rate { color-blind; committed-information-rate 10m; committed-burst-size 100k; excess-burst-size 200k; }
single-rate { color-aware; } // <-- SILENTLY DROPPED
}
}
Second single-rate is silently ignored. ColorAwareConfigured never flips to true. Strict ambiguity validation at pkg/config/compiler.go:243-247 (pol.ColorBlindConfigured && pol.ColorAwareConfigured) is statically unreachable for sibling conflicts. Policer compiles with hidden color-aware parameters.
Round-3 wins (both reviewers confirm)
✓ Top-level duplicate three-color-policer <name> blocks now accumulated at compiler_firewall.go:69-76 (verified by Gemini quote):
tcp := fw.ThreeColorPolicers[tcpInst.name]
if tcp == nil { tcp = &ThreeColorPolicerConfig{ ... }; fw.ThreeColorPolicers[tcpInst.name] = tcp }✓ Cross-mode siblings (single-rate + two-rate in same parent) are rejected.
Test coverage gap (both reviewers)
pkg/configstore/store_test.go covers:
LoadOverrideambiguous mode across blocks (114-148) ✓- Set-syntax ambiguous color (89-112) ✓
- Set-syntax PIR<CIR (150-172) ✓
Missing:
- ⊝ Hierarchical duplicate same-mode blocks (the worked failure above)
- ⊝ Hierarchical ambiguous color (single-rate{color-blind} + single-rate{color-aware})
Gemini follow-up finding (defer)
then block inside three-color-policer likely uses the same FindChild pattern → sibling then blocks similarly dropped. Recommends a sweep changing semantic-weight FindChild lookups to FindChildren loops. Defer to follow-up issue.
Block on
- Replace
FindChild("single-rate")andFindChild("two-rate")atcompiler_firewall.go:78+ withFindChildreniteration; either accumulate fields (consistent with top-level behavior) or reject duplicates explicitly. - Add
configstore.Committest for hierarchical duplicate same-mode blocks (the Codex worked example). - Add
configstore.Committest for hierarchical ambiguous color.
Round-4 claim trace
- Codex
task-mp9a4wiv-lhmsql— MAJOR verified at file:line - Gemini
task-mp9adl0r-xsi6ut— independently reproduced with full input/output trace - Copilot
d36b0e7d— stale; re-review @copilot requested
Not merging — author's decision.
Addressed on ac55aee (with cleanup in 361f431): |
Round-5 verification on
|
Claude round-5 review on
|
Confirmed on 3f79822 (core fix in ac55aee, cleanup in 361f431): |
Round-5 quad-review consolidated synthesis on
|
| Reviewer | Verdict |
|---|---|
| Claude | MERGE-READY |
| Codex | MERGE-READY |
| Gemini Pro 3 | MERGE-READY |
| Copilot | Stale at d36b0e7d; no r5 review yet — re-trigger if needed |
Triple converge MERGE-READY (Codex + Gemini + Claude).
Round-4 MAJOR — CLOSED
FindChild → FindChildren at pkg/config/compiler_firewall.go:78 and :109. Cross-mode ambiguity validation still fires correctly (Codex traced SingleRateConfigured && TwoRateConfigured through validateThreeColorPolicersStrict). Color ambiguity within same mode (e.g., two single-rate blocks with color-blind + color-aware) now accumulates into both flags → strict validator catches.
Tests verified by all reviewers
TestThreeColorPolicerStrictValidation_HierarchicalDuplicateSameModeSiblingsparses + compiles Codex worked-failure inputTestCommitCheck_RejectsAmbiguousThreeColorPolicerAcrossLoadOverrideSameModeSiblingsexercises fullLoadOverride → CommitCheck → CompileConfiglifecycle
Both production paths covered.
Defer to follow-up (Gemini + Codex agreed)
Systemic FindChild pattern still vulnerable to same silent-drop bug:
three-color-policerthenblock atcompiler_firewall.go:144- Legacy policer
if-exceedingandthenblocks - Filter term
fromandthenblocks
Codex notes Gemini correctly identifies these as "pre-existing compiler weaknesses" outside this PR's targeted scope. Recommended follow-up issue: "Sweep semantically-weighted FindChild → FindChildren across compiler_firewall.go".
Recommendation
Block on: nothing.
Defer: systemic FindChild sweep (separate issue).
Round-5 claim trace
- Codex
task-mp9bdpjj-xtxma1— MERGE-READY, both call sites verified - Gemini Pro 3
task-mp9besv9-h0vof3— MERGE-READY, full ambiguity validation traced + systemic FindChild pattern noted as defer - Copilot — Stale at
d36b0e7d; no inline findings on3f79822a
Not merging — author's decision.
|
@copilot review #1395 stable at 3f79822. Prior Copilot review (d36b0e7) was stale. Codex+Gemini+Claude all returned MERGE-READY at 3f79822:
Please review the current SHA. |
Summary
Tests
Remaining gaps