Bench-test harness for field failure modes (resend amp, wedge, rate-limit accuracy, saturation)

[rolker]

Context

We have no load test for udp_bridge. The 2026-04-21 BizzyBoat
deployment surfaced failure modes — resend amplification (#9),
bridge wedge (#10), Starlink-uplink saturation, and stats-path
stall during Starlink-only ops (#20) — that we currently can't
reproduce off-boat. PR #16's test_connection_rate_limit covers
one narrow correctness invariant; nothing exercises the field
shape: sustained rate, multi-topic, with impairment.

Even a crude bench harness would have caught these. The failures
aren't Starlink-specific subtleties — they're "this code path was
never exercised at sustained rate with any loss." The field was
the first environment to push it hard.

The legacy mininet scripts in udp_bridge/test/mininet/ are ROS 1
and non-functional on Jazzy.

MVP scope: range-degradation scenario

One scenario, shipped first. Linux netns + tc qdisc netem, no
sudo (unshare -rn). Smoke runs under colcon test on every
build; full scenario is pytest-discovered but
@pytest.mark.skipif'd unless UDP_BRIDGE_BENCH_SCENARIOS=1 is
set (standard opt-in pattern).

Topology

Three concurrent veth-pair-connected netns paths from day one:

Path	Role	MVP profile
WiFi	Primary, short-range, ranged	Walked through the trajectory below
Cell	Always-on, low-bandwidth	5 Mbit / 1% loss / 60 ms / 20 ms jitter (constant)
Starlink	Always-on, medium-bandwidth	8 Mbit / 0.4% loss / 40 ms / 30 ms jitter (constant)

Cell and Starlink baselines drawn from the 2026-05-18 pier link
calibration; characteristic, not calibrated. The MVP walks only
the WiFi path through the trajectory below; cell and Starlink
are present and forwarding so cross-path interactions are
observable. Per-path profile walking is a follow-up refinement.

Test topic mix

Pub/sub scripts in the boat netns publish a representative mix.
Each path's Connection topics_list carries a subset, mirroring
how operational traffic is currently split (pre-#19):

Tier	Carried by	Example topics	Aggregate rate
Critical	WiFi + Cell + Starlink (replicated)	heartbeat, mission_state	~0.1 Mbit/s
Telemetry	WiFi + Starlink	odom, nav_status, cmd_vel ack	~0.5 Mbit/s
Bulk	WiFi only	one camera stream, viz markers	up to WiFi link cap

Replication on multiple paths uses today's mechanism (same topic
appearing in multiple topics_list entries — the "Pattern A"
workaround documented in #19). When #19 lands, equivalent
coverage is expressed via topic-priority routing on a single
per-host Connection instead of replicated Connections, and the
mix is restated in those terms.

Mix configured via bizzyboat.yaml-style fragments under
test/bench/configs/.

Trajectory

Drive the WiFi link through a sail-out-and-return trajectory using
an orchestrator script that issues periodic tc qdisc change on
the boat-side veth (cell and Starlink remain at the constant
profiles above). Phases held N seconds each:

Phase	rate	loss	delay	jitter
In-range clean	30 Mbit	0%	5 ms	1 ms
Fringe	10 Mbit	0.5%	15 ms	5 ms
Lossy	3 Mbit	3%	40 ms	15 ms
Critical	500 Kbit	10%	100 ms	30 ms
Over-horizon	drop	100%	—	—
Recovery	mirror trajectory back to in-range clean

Trajectory shape informed by the 2026-04-21 BizzyBoat range
observations and 2026-05-18 pier link calibration; coarse, not
calibrated to specific bags.

Why this scenario as MVP: it exposes resend amplification,
wedge, and rate-limit overshoot together under the conditions
that produced them in the field. It exercises transitions —
especially the over-horizon → recovery edge — which is where the
operationally critical failures live.

Invariants over the run (thresholds documented in the
per-scenario README):

• Recv-Q never climbs monotonically for more than N seconds in
  any phase, including the over-horizon → recovery edge.
• No ERROR-severity log emitted during the over-horizon window
  (graceful disconnect is a mode, not a fault).
• After the over-horizon → in-range edge,
  DataRates.success_bytes_per_second returns to within X% of
  in-range baseline within T seconds.
• On the recovery leg, resend/tx_ok ratio does not exceed the
  instantaneous loss rate by more than F×.
• Forwarding resumes without bridge restart after the over-horizon
  window ends.
• bridge_info and topic_statistics publication rates on both
  sides remain ≥ R% of in-range baseline throughout the
  over-horizon and recovery phases. (Catches Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20-class stalls in
  the stats path while the data plane keeps working.)
• (multi-link) Each path's observed forwarded topics ⊆ its
  configured topics_list. Any unauthorized topic on a path is
  a fault (catches mis-routing).
• (multi-link) During WiFi-path degradation, cell-path and
  Starlink-path success_bytes_per_second for tier-Critical
  (and, for Starlink, tier-Telemetry) topics are unaffected
  within Y%. WiFi saturation must not poison the other paths.
• (multi-link) Operator-side subscribers for tier-Critical
  topics observe no gap longer than G seconds during the
  over-horizon window. Cell/Starlink replicas keep them alive.
• (multi-link, requires Add per-topic priority/class scheduling to the rate-limiter #19) Under WiFi-path saturation,
  drop fraction by tier is monotonic: Bulk drops first, then
  Telemetry, then Critical. Pre-Add per-topic priority/class scheduling to the rate-limiter #19, the MVP measures and reports
  this without failing on it (FIFO drop is the known-current
  behavior; the test documents the gap rather than enforcing the
  fix that doesn't exist yet).

Threshold sourcing

Each placeholder is replaced with a concrete value during
plan-task, drawn from the named bag(s):

• N (Recv-Q climb tolerance) — from Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10 wedge-adjacent bags
  in ~/data/logs/analysis/. Pick N as 2× the longest observed
  transient flush in a healthy recovery, not a wedge.
• X, T (recovery completeness, speed) — from 2026-04-21
  BizzyBoat range bags where the boat returned to range. X is
  the observed DataRates ratio post-recovery vs. pre-event
  baseline; T is the observed convergence time.
• F (resend amplification ceiling) — anchor between the
  2026-05-01 pathological case (resend_dropped_bps ~13 MB/s on
  a 1.0 MB/s VPN, i.e. 13× ceiling) and the 2026-05-18 healthy
  baseline (~30% structural overhead on Starlink/VPN). Pick F
  well below pathological — likely 2× — and document the choice.
• Y (cross-path poisoning tolerance) — no field bag isolates
  this cleanly (2026-05-01 was Starlink-only). Initial Y is
  bootstrapped from the MVP harness itself (cell rate with WiFi
  idle as a reference), and revised once a deployment captures
  WiFi-saturation-with-cell-up-and-running.
• G (Critical-topic gap tolerance during over-horizon) — from
  2026-05-01 bags where heartbeat was carried over Starlink
  alone. G is 2× the observed worst-case inter-arrival in the
  non-degraded portion of that bag.
• R (stats-publication rate tolerance during degraded modes)
  — from 2026-05-19 bags (Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20). R is set such that a sustained
  drop to 0% (as observed in Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20) trips the invariant, while
  brief transients (one missed publication per node) do not.
  Initial value: 50% of baseline averaged over a 10s window.

Plan-task extracts the concrete values from these bags and lands
them in the per-scenario README before implementation begins.

Implementation sketch

• Python orchestrator (test/bench/run_scenario.py) that:
  • Sets up N veth-pair-connected netns paths via unshare -rn
    (MVP wires three: WiFi, cell, Starlink).
  • Applies independent tc qdisc netem per path; orchestrator
    walks WiFi-path parameters per the phase table, holds cell
    and Starlink at constant baselines.
  • Spawns the bridge + scripted pub/sub in each netns.
  • Records bridge_info + recv_q_trace for the run window.
  • Runs assertions on the captured bag.
• Reuse bag_analysis/extractors/udp_bridge.py (in marine_tools)
  for parsing — same code path as on-boat bag analysis.
• Smoke and scenarios registered as pytest tests under
  test/bench/. Smoke (clean link, 10s baseline pub/sub, asserts
  non-zero success_bytes_per_second) runs on every
  colcon test. Full scenario opt-in via
  UDP_BRIDGE_BENCH_SCENARIOS=1. If unshare -rn is unavailable
  in the test environment, both skip with a clear reason rather
  than failing.
• Plan-task verifies unshare -rn + tc qdisc work in the
  workspace's CI runner before merging the MVP.

Target shape: follow-up scenarios

These remain the eventual library; each is filed as its own
follow-up issue after the MVP lands. They are not part of #18:

• Resend amplification reproducer — clean baseline link +
  scripted brief outages. Assert resend/tx_ok ratio approaches
  actual loss rate; rx_duplicate/rx approaches 0 once Resend loop amplifies traffic rather than tracking loss #9's
  debounce + backoff land.
• Wedge reproducer — high-rate publisher + remote subscriber
  killed mid-stream. Assert Recv-Q does NOT monotonically climb
  past N seconds.
• Rate-limit accuracy under load — multiple high-rate
  forwarding subscriptions on one Connection with a known cap.
  Assert achieved success bytes/s is within a band of the cap.
• (Subsumed) Pure saturation behavior is exercised by the
  Critical and Lossy phases of the MVP scenario; a dedicated
  saturation scenario may not be needed.

Out of scope

• ROS 2 port of the legacy mininet .bash / .launch
  scaffolding (netns is simpler; mininet adds an abandoned
  dependency).
• Calibration of phase parameters to specific field bags. This is
  a v2 refinement for tuning fixes; not required for exposing
  bugs. The MVP only needs phase parameters in the right ballpark.
• Continuous benchmarking dashboards / regression tracking.
• Per-path walked impairment profiles. MVP walks only the WiFi
  path; cell and Starlink are constant. Walking all three
  simultaneously (e.g. weather-degraded Starlink, terrain-shadowed
  cell) is a follow-up.

Why now

• Recent field issues (Resend loop amplifies traffic rather than tracking loss #9, Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10, Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20, Starlink saturation) all
  share the property "would have been caught by basic load
  testing." The cost of going to the boat to validate each
  candidate fix for Resend loop amplifies traffic rather than tracking loss #9 is the test gap, expressed in deployment
  hours.
• The fix for Resend loop amplifies traffic rather than tracking loss #9 needs before/after evidence from a deterministic
  setup. The MVP scenario provides that on the recovery leg.

---

Authored-By: Claude Code Agent
Model: Claude Opus 4.7 (1M context)

[rolker]

Review

Scope Assessment

Well-scoped? Yes — single MVP scenario (range trajectory), single
deliverable (test/bench/run_scenario.py + assertions), reusing
existing bag_analysis/extractors/udp_bridge.py. The "Target shape:
follow-up scenarios" section explicitly defers wedge/resend/rate-limit
reproducers to their own issues. That separation is exactly what keeps
this one PR-sized.

Right repo? Yes — bench harness for udp_bridge belongs in
rolker/udp_bridge. No workspace-infra leak.

Dependencies:

• Resend loop amplifies traffic rather than tracking loss #9 (closed 2026-05-19) — the bench's recovery-leg assertion is the
  before/after evidence vehicle the closure note implies; Bench-test harness for field failure modes (resend amp, wedge, rate-limit accuracy, saturation) #18 should
  validate Resend loop amplifies traffic rather than tracking loss #9's fix retrospectively, not block on it.
• Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10 (open, wedge) — bench will reproduce it; Bench-test harness for field failure modes (resend amp, wedge, rate-limit accuracy, saturation) #18 unblocks Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10.
• Add per-topic priority/class scheduling to the rate-limiter #19 (open, priority routing) — the last three multi-link invariants
  depend on it. Issue already flags this; deferral is explicit and
  correct.
• Field import: udp_bridge (2026-05-01) #16 (merged) — test_connection_rate_limit is the narrow correctness
  test the harness complements, not duplicates.

Principle Alignment

Principle	Status	Notes
Test what breaks	OK	Targets the exact field-observed failure modes (resend amp, wedge, rate-limit, saturation) rather than coverage. The "would-have-been-caught" framing is on point.
Only what's needed	OK	netns + unshare -rn over mininet is the minimal viable choice; explicitly rejects ROS 2 port of legacy scaffolding.
Improve incrementally	OK	One scenario as MVP, four follow-ups filed separately.
A change includes its consequences	Watch	MVP should also delete/retire the dead ROS 1 test/mininet/ scripts in the same PR — they actively mislead. Issue notes they're non-functional but doesn't promise removal.
Human control and transparency	OK	Phase table, invariants, and rationale are all explicit.
Capture decisions, not just implementations	Watch	The choice "netns over mininet" and "walk WiFi only, hold cell/Starlink constant" are real design decisions worth a short note in the bench README (not a workspace ADR — too local).
Enforcement over documentation	OK	Manual-run by design; no CI claim to enforce. Appropriate at this stage.
Workspace vs. project separation	OK	Self-contained in project repo.

ADR Applicability

ADR	Triggered	Notes
0002 — Worktree isolation	Yes	Standard; not a concern for the issue itself, just the implementation.
0008 — ROS 2 conventions	Watch	Bench scripts in test/bench/ should follow ROS 2 launch/package conventions where they touch them; pure-Python orchestrator is fine. Worth a glance at where the netns'd bridge instances get launched.
0009 — Python package management	Watch	Orchestrator likely needs python3-pytest + perhaps pyroute2/nsenter wrappers. Declare in package.xml via rosdep keys, not bare pip. tc and unshare come from iproute2 / util-linux — rosdep keys exist; add as test deps.
0001 — ADRs	No	Bench-internal design choices don't rise to workspace-ADR level.

Consequences

Items to include in the MVP PR, not defer:

• Remove or clearly mark-archived the ROS 1 test/mininet/ scripts. The
  issue calls them non-functional but doesn't commit to removal; leaving
  them invites future agents to "fix" them.
• Add a short README at test/bench/README.md documenting:
  • Why netns over mininet (decision capture).
  • How to run without sudo (unshare -rn flow).
  • The invariant thresholds (N, X, T, F, Y) — they're
    parameterized in the issue but a real PR needs concrete values
    with rationale.
• Declare any new Python/system test deps in package.xml (ADR-0009).
• Sanity check: does the bench need to coexist with colcon test, or
  is it strictly manual? The MVP says manual-run; the always-runs smoke
  test blurs this. Clarify whether the smoke runs under colcon test
  (and if so, how it skips when unshare -rn is unavailable in CI).

Recommendations

1. Concretize the invariant thresholds before implementation. N,
   X, T, F, Y are placeholders. The plan-task step should pick
   defaults from the 2026-04-21 / 2026-05-18 calibration data, not at
   PR-review time.
2. Decide the smoke-test placement explicitly. Either it's part of
   colcon test (with a skip-on-no-netns guard) or it's strictly
   manual. Don't ship in an in-between state.
3. Confirm the multi-link MVP is genuinely Day 1. The issue says
   three paths from day one, but only the WiFi path is walked. If
   wiring cell+Starlink veth-pairs adds non-trivial implementation
   weight, consider deferring them to the priority-routing follow-up
   (Add per-topic priority/class scheduling to the rate-limiter #19) where the multi-link invariants actually fire. Surface this
   trade-off in plan-task rather than discovering it mid-implementation.
4. Cross-reference Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20 (stats-timer stall during Starlink-only ops)
   in the issue body — the bench's multi-link topology is the natural
   reproducer for Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20, and it's not currently listed alongside Resend loop amplifies traffic rather than tracking loss #9/Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10/Field import: udp_bridge (2026-05-01) #16/Add per-topic priority/class scheduling to the rate-limiter #19.
5. No PR-test-plan friction expected. Bench harness IS the test plan
   for Resend loop amplifies traffic rather than tracking loss #9/Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10 retrospectively; the PR's own test plan can be short.

---

Authored-By: Claude Code Agent
Model: Claude Opus 4.7 (1M context)