Bench-test harness for field failure modes (netns + tc netem)#27
Open
rolker wants to merge 18 commits into
Open
Bench-test harness for field failure modes (netns + tc netem)#27rolker wants to merge 18 commits into
rolker wants to merge 18 commits into
Conversation
4 tasks
rolker
changed the title
There was a problem hiding this comment.
Pull request overview
This PR adds a Linux user+network-namespace (unshare -Urn) + tc netem bench-test harness to reproduce/guard against udp_bridge field failure modes (range degradation, wedge/non-wedge behavior, cross-path invariants) without requiring root/mininet, and retires the legacy ROS1 mininet scaffolding.
Changes:
- Introduces a bench orchestrator (
run_scenario.py) plus rclpy pub/sub helpers and a three-path config to simulate WiFi/Cell/Starlink impairments. - Adds pytest coverage: always-on smoke test plus opt-in longer scenarios (
range_degradation,subscriber_death) gated byUDP_BRIDGE_BENCH_SCENARIOS=1and capability/dep checks. - Archives the old mininet scripts under
test/mininet/.archived/and repoints docs to the new bench harness.
Reviewed changes
Copilot reviewed 17 out of 28 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| udp_bridge/test/mininet/README.md | Replaces mininet instructions with pointer to bench harness; marks mininet as retired. |
| udp_bridge/test/mininet/.archived/test_mininet_multilink_robot.launch | Archives legacy ROS1 launch file for reference. |
| udp_bridge/test/mininet/.archived/test_mininet_multilink_operator.launch | Archives legacy ROS1 launch file for reference. |
| udp_bridge/test/mininet/.archived/robot.bash | Archives legacy mininet helper script for reference. |
| udp_bridge/test/mininet/.archived/robot_setup.bash | Archives legacy ROS1 env setup script. |
| udp_bridge/test/mininet/.archived/robot_launch.bash | Archives legacy ROS1 launch wrapper. |
| udp_bridge/test/mininet/.archived/README.md | Preserves the previous mininet README content under .archived/. |
| udp_bridge/test/mininet/.archived/operator.bash | Archives legacy mininet helper script for reference. |
| udp_bridge/test/mininet/.archived/operator_setup.bash | Archives legacy ROS1 env setup script. |
| udp_bridge/test/mininet/.archived/operator_launch.bash | Archives legacy ROS1 launch wrapper. |
| udp_bridge/test/mininet/.archived/multilink.py | Archives legacy mininet topology script. |
| udp_bridge/test/bench/test_subscriber_death.py | Adds opt-in system-level “no-wedge” regression guard around stalled subscriber behavior (Zenoh-forced). |
| udp_bridge/test/bench/test_smoke.py | Adds a skip-guarded smoke test that runs under default colcon test. |
| udp_bridge/test/bench/test_range_degradation.py | Adds opt-in range-degradation scenario invariants (single-path + multi-link). |
| udp_bridge/test/bench/sub.py | Adds rclpy subscriber helper with optional count-tracing. |
| udp_bridge/test/bench/run_scenario.py | Adds orchestrator to set up netns/veth/netem, launch bridges, drive scenarios, and emit artifact markers. |
| udp_bridge/test/bench/recv_q_trace.py | Adds unix_time column to Recv-Q tracing to support cross-process correlation. |
| udp_bridge/test/bench/README.md | Documents prerequisites, scenarios, invariants, and threshold sourcing/refinement. |
| udp_bridge/test/bench/pub.py | Adds rclpy publisher helper for the three-tier traffic mix. |
| udp_bridge/test/bench/configs/three_path.yaml | Adds a bench-specific 3-path udp_bridge configuration (WiFi/Cell/Starlink + tier replication). |
| udp_bridge/test/bench/bag_reader.py | Adds a small in-tree rosbag2 reader used by invariants. |
| udp_bridge/test/bench/init.py | Marks bench directory as a package (empty). |
| udp_bridge/package.xml | Adds test dependencies needed for pytest + rclpy + system tools (unshare, ip, tc). |
| udp_bridge/mainpage.dox | Updates docs to point to test/bench/ instead of mininet instructions. |
| udp_bridge/CMakeLists.txt | Registers pytest-based bench tests with ament (smoke + opt-in scenarios). |
| .gitignore | Ignores Python bytecode caches produced by bench tests. |
| .agent/work-plans/issue-18/progress.md | Adds work-plan progress notes for issue #18 implementation. |
| .agent/work-plans/issue-18/plan.md | Adds the recorded plan/approach for the bench harness implementation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+9
to
+13
| Usage: | ||
| run_scenario.py --scenario smoke [--duration-s 10] [--outdir DIR] | ||
| run_scenario.py --scenario full-mix [--duration-s 10] [--outdir DIR] | ||
| run_scenario.py --scenario range_degradation [--hold-s 10] [--outdir DIR] | ||
|
|
Comment on lines
+313
to
+324
| def launch_bridge_logged(node_name: str, params_file: Path, domain: int, | ||
| stderr_log: Path) -> _Child: | ||
| """Like `launch_bridge` but tees the bridge's stderr to `stderr_log`. | ||
|
|
||
| Phase 3 needs per-bridge stderr captured so we can check invariant | ||
| 2 (no ERROR-severity log during the over-horizon window). We can't | ||
| redirect to a file inside the Popen because we still need the | ||
| lifecycle transitions to be issued; the cleanest approach is to | ||
| open the log file in append mode and pass it as `stderr=`. Debug | ||
| mode (UDP_BRIDGE_BENCH_DEBUG=1) additionally tees to the terminal | ||
| via `tee` so the user sees the live output. | ||
| """ |
Comment on lines
+327
to
+337
| child = _spawn( | ||
| [ | ||
| 'ros2', 'run', 'udp_bridge', 'udp_bridge_node', | ||
| '--ros-args', | ||
| '-r', f'__node:={node_name}', | ||
| '--params-file', str(params_file), | ||
| ], | ||
| env=env, | ||
| name=node_name, | ||
| stderr=log_fp, | ||
| ) |
Comment on lines
+195
to
+204
| with artifacts.recvq_csv.open() as f: | ||
| for r in csv.DictReader(f): | ||
| try: | ||
| t = float(r['t_seconds_since_start']) | ||
| q = int(r['recv_q_bytes']) if r['recv_q_bytes'] else 0 | ||
| rows.append((t, q)) | ||
| except (ValueError, KeyError): | ||
| continue | ||
| if not rows: | ||
| pytest.skip('No Recv-Q samples recorded; nothing to assert.') |
Comment on lines
+1
to
+12
| """In-tree ros2 bag reader for the bench harness. | ||
|
|
||
| Two surfaces: | ||
| - SQLite shims (`list_topics`, `count_messages`, `find_bag_db`) used | ||
| by the Phase 1 smoke for cheap structural checks. These only work | ||
| on sqlite3-storage bags. | ||
| - rosbag2_py-backed iterators (`iter_messages`, `read_bridge_infos`, | ||
| `read_topic_statistics_arrays`) used by `test_range_degradation.py` | ||
| for the Phase 3 single-path invariants. These work with whichever | ||
| storage backend the bag uses (mcap by default in Jazzy, sqlite3 if | ||
| the recorder was told to use it). | ||
|
|
Comment on lines
+58
to
+67
| def _make_image(self): | ||
| m = Image() | ||
| m.header.stamp = self.get_clock().now().to_msg() | ||
| m.header.frame_id = 'bench' | ||
| m.height = 1 | ||
| m.width = self._payload_bytes | ||
| m.encoding = 'mono8' | ||
| m.step = self._payload_bytes | ||
| m.data = bytes(self._payload_bytes) | ||
| return m |
Six-phase plan for the netns + tc bench-harness MVP: threshold extraction from named bags (F=2× anchored; N/X/T/G/R/Y queries documented), orchestrator + smoke under colcon test, topic mix with three-tier replication, range-degradation trajectory + assertions, multi-link invariants, mininet retirement, CI runner check. Phase 0 (threshold extraction) gates the rest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pub/sub: rclpy (matches Python orchestrator + bag-analysis tooling). - Bulk tier: synthetic sensor_msgs/Image (no ffmpeg dep). - Bag output: tempfile.mkdtemp respecting TMPDIR, UDP_BRIDGE_BENCH_OUTDIR override, preserve on fail. - package.xml test_depend now lists python3-pytest + sensor_msgs + nav_msgs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Re-root all paths under udp_bridge/ (inner package dir, not repo
root). Affects every test/bench/, test/mininet/, package.xml,
CMakeLists.txt reference.
- Drop cross-repo dep on marine_tools' bag_analysis extractor;
inline a small bag_reader.py in udp_bridge/test/bench/ that
pulls the BridgeInfo / TopicStatistics fields the assertions
need (no rosdep path to the sibling-repo extractor).
- Add rosdep keys util-linux (unshare) and iproute2 (tc) to the
test_depend list — review-issue flagged these and I had dropped
them.
- Phase 5: align archive-vs-replace on archive-with-pointer; also
sweep launch/test_mininet_multilink_{operator,robot}.launch
(same ROS-1-vintage scaffolding, no callers).
- Estimated Scope notes Phase 0 lands as its own reviewable commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Concrete values for the invariants defined in #18: - F = 2× (anchored: 13× pathological 2026-05-01 vs ~30% healthy 2026-05-18; well below pathological) - G = 6 s (2× observed non-degraded heartbeat inter-arrival in 2026-05-01: 18,900 samples, 99.97% under 2s, worst 3.0s) - R = 50% over 10s (1Hz baseline; #20 stall 0% for 77 min trips well below; brief transients tolerated) - N = 10 s, X = 90%, T = 30 s, Y = 10% — starting values from physical reasoning; refinement plan documented in README Side-finding from the 2026-05-19 query (relevant to #20): the boat received the operator's stats topic steadily through the 12:53 WiFi-cut, so both bridges kept exchanging stats; what stalled is the operator-side local subscriber view. The mechanism is operator-side downstream of the bridge, not in the bridge itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scaffolding for the netns + tc bench harness. No orchestrator yet (lands in 1b). Smoke test wiring lands in 1c. - udp_bridge/test/bench/configs/three_path.yaml — bridge config for both sides. Three Connections per side (wifi/cell/starlink) with per-path bandwidth caps from 2026-05-18 calibration. Tier replication via topics_list duplication: Critical on all three, Telemetry on WiFi+Starlink, Bulk on WiFi only. - udp_bridge/test/bench/pub.py — rclpy publisher driven by --tier (critical/telemetry/bulk). Critical uses std_msgs/String at 1Hz; Telemetry uses nav_msgs/Odometry at 10Hz; Bulk uses synthetic sensor_msgs/Image (512 KB) at 5Hz. No ffmpeg dep. - udp_bridge/test/bench/sub.py — rclpy subscriber that counts received messages and prints the count on exit. - udp_bridge/test/bench/bag_reader.py — minimal sqlite reader. Smoke counts messages; Phase 3 will extend for full BridgeInfo / TopicStatistics deserialization. - udp_bridge/test/bench/README.md — adds Prerequisites section documenting the apparmor_restrict_unprivileged_userns=0 one-time host setup (Ubuntu 24.04+ default blocks unshare -Urn). - udp_bridge/package.xml — test_depends for ament_cmake_pytest, python3-pytest, rclpy, nav_msgs, sensor_msgs, iproute2 (tc), util-linux (unshare). - .gitignore — exclude __pycache__/ and *.pyc. Plan update: switch from "two netns connected by veth" to "single user+net namespace with N veth pairs inside" — nested `ip netns add` inside an unprivileged user namespace doesn't work on Ubuntu 24.04+ (mount --make-shared /run/netns fails without CAP_SYS_ADMIN). Single-namespace + per-path veth has the same fidelity for the failure modes we care about. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-file orchestrator that: - Re-execs self under `unshare -Urn` to enter an ephemeral user+net namespace (no sudo; preserved by the AppArmor one-time prerequisite documented in test/bench/README.md). - Inside, brings up lo, creates three veth pairs (wifi/cell/ starlink) with /24 IPs on each end, and applies `tc qdisc netem` per path with the clean-baseline profile from the 2026-05-18 pier calibration. - Launches operator_bridge and boat_bridge (lifecycle nodes) in distinct ROS_DOMAIN_IDs so they communicate only via the bridge protocol, not direct DDS. - Drives the smoke scenario: one Critical-tier publisher on boat, one subscriber on operator, ten seconds, expects sub_count > 0. - Cleans up children on exit; preserves the output directory on failure (per the plan's preserve-on-fail rule). `--scenario range_degradation` is a stub returning exit 2; implementation lands in Phase 3. Also flips exec bits on pub.py / sub.py (run_scenario.py shebangs them too) so they can be invoked directly when debugging interactively, in addition to via `python3 ...`. Smoke test wiring + verification land in 1c. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- test/bench/test_smoke.py: pytest wrapper that invokes run_scenario.py --scenario smoke and asserts SUB_COUNT > 0. Skips with a clear reason if `unshare -Urn` is unavailable (host hasn't applied the AppArmor prerequisite) or if no ROS_DISTRO is sourced. - CMakeLists.txt: register the smoke via ament_add_pytest_test so `colcon test` exercises the bench scaffolding on every build. Timeout 180s. Two fixes uncovered while bringing it up end-to-end: - configs/three_path.yaml: drop empty `topics_list: []` lines. ROS 2 can't infer element type from an empty YAML array, so declare_parameter sees the override as PARAMETER_NOT_SET and throws on get_parameter().as_string_array(). Omitting lets the bridge's default `std::vector<std::string>()` take over. - run_scenario.py: harden cleanup. Children now spawn with `start_new_session=True`; cleanup uses os.killpg so we take down the whole subprocess subtree (ros2 run → udp_bridge_node grandchild). Added a SIGTERM/SIGINT handler so cleanup runs on signal-kill — Python atexit doesn't fire on SIGTERM, so the prior `timeout` outside the orchestrator was leaving orphan bridges chewing CPU. Also re-added `text=True` on spawn (dropped by the _spawn() refactor). Verified end-to-end: smoke runs in ~16s in the netns, pub at 1Hz for 10s → bridge → sub receives 10/10 messages on the WiFi-path Critical-tier topic. No orphan processes after exit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `--scenario full-mix` to run_scenario.py: spawns one publisher
per tier (Critical / Telemetry / Bulk) at the boat side and one
subscriber per destination topic at the operator side; reports
per-tier delivery counts via BENCH_RESULT_<TIER>_{PUB,SUB}_COUNT.
Bumps the Bulk tier defaults in pub.py to the pinned spec
(10 Hz × 480 KB) — was a Phase 1 placeholder (5 Hz × 512 KiB).
At 4.8 MB/s offered vs. the WiFi connection's 4 MB/s budget, the
rate limiter trims Bulk even on a clean link; that's intentional
(Phase 4 drop-by-tier invariant). Phase 2 only confirms each tier
delivers something.
Smoke stays Critical-only as planned; full-mix is a manual CLI
invocation (Phase 3's pytest is what `UDP_BRIDGE_BENCH_SCENARIOS=1`
will gate).
Verified end-to-end on this host: full-mix delivers 10/10 Critical,
99/100 Telemetry, 98-99/100 Bulk over a 10s run; exits 0; no orphan
processes after cleanup.
Refs #18.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `--scenario range_degradation`: the orchestrator walks the WiFi path through the sail-out-and-return trajectory specified in issue #18 (in_range_clean → fringe → lossy → critical → over_horizon → critical → lossy → fringe → in_range_clean), holding each phase for --hold-s seconds (default 10) while the full three-tier mix runs across all three connections. During the walk it captures: - a ros2 bag of `bridge_info` / `topic_statistics` (own + operator-side mirrors of the boat's), - a CSV trace of the operator UDP socket's Recv-Q via the recv_q_trace.py script (moved up from test/mininet/ since Phase 3 is its first non-archive caller), - per-bridge stderr logs (so invariant 2 can scan for ERROR/FATAL during the over_horizon window), - a phase_log.json marking wall-clock phase boundaries. Adds `test_range_degradation.py` (pytest opt-in via UDP_BRIDGE_BENCH_SCENARIOS=1) with five single-path invariants drawn from the issue body: 1. Recv-Q never climbs monotonically for more than N seconds. 2. No ERROR/FATAL log emitted during the over_horizon window. 3. message.success_bytes_per_second recovers to ≥ X% of the pre-event in_range baseline by the final in_range phase. 4. resend/success ratio ≤ F × applied netem loss rate in each lossy phase. 5. bridge_info + topic_statistics publication rates stay ≥ R% of the in-range baseline in every phase (catches #20-class stalls in the stats path). Thresholds (N=10s, X=90%, T=30s, F=2×, R=50%/10s) are pulled from the existing `test/bench/README.md` values table; the test's THRESHOLDS dict mirrors them and the per-phase loss rate is derived programmatically from run_scenario.PHASE_TRAJECTORY so the two can't drift. The issue's sixth single-path invariant ("Forwarding resumes without bridge restart") is subsumed by invariant 3 — the orchestrator has no respawn path, so a dead bridge means post-recovery success_bps is zero, which trips X% on its own. Documented in the test module docstring. Extends `bag_reader.py` with rosbag2_py-backed iterators (`iter_messages`, `read_bridge_infos`, `read_topic_statistics_arrays`) that handle both mcap (Jazzy default) and sqlite3 storage and deserialize via the bag's own type catalog, so type-hash-evolved schemas still read. Verified end-to-end on this host: - Manual `--scenario range_degradation --hold-s 5` produces all expected artifacts (bag + recv_q CSV + phase log + stderr logs). - `UDP_BRIDGE_BENCH_SCENARIOS=1 pytest test_range_degradation.py` passes all 5 invariants in 112 s on a clean run. - `--scenario smoke` regression: still exits 0, 10/10 critical. - No orphan processes after pytest exit. Refs #18. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New opt-in scenario reproducing the issue #10 trigger: freeze (SIGSTOP) the Bulk operator subscriber mid-stream while the boat keeps forwarding Bulk, trace the operator bridge's UDP Recv-Q (port 4200), and count-trace the surviving Critical/Telemetry subscribers across the stall (head-of-line measurement for #29). Forces rmw_zenoh_cpp + a zenoh router, since the back-pressure path does not exist under DDS. sub.py gains an optional --count-trace. Finding (measured): the wedge does NOT reproduce at bench scale under FastDDS, CycloneDDS, OR Zenoh, even at ~58 MB/s Bulk into a frozen consumer — Recv-Q and Send-Q stay flat at 0 and survivors are unaffected. publish() does not block the drain thread at bench scale (under Zenoh it's an async hand-off). So test_subscriber_death is a no-wedge REGRESSION GUARD (asserts the healthy behavior, would catch a reintroduced drain-block under a blocking RMW), not a red→green reproduction. The component-level proof of the #10 fix remains the PublishQueue unit tests. Rationale captured in the test's module docstring.
Move the dead ROS-1-vintage mininet scripts (multilink.py, *.bash) and the two launch/test_mininet_multilink_*.launch files (no callers) under test/mininet/.archived/. Replace test/mininet/README.md with a pointer to the netns-based test/bench/ harness, and repoint mainpage.dox's 'Simulating network issues' section from mininet to the bench. recv_q_trace.py already lives in test/bench/.
Four multi-link invariants on the same range_degradation run: - topic_list_confinement: each forwarded topic advertised only on its configured connections (Bulk→WiFi, Telemetry→WiFi+Starlink, Critical→all3); checked against every bridge_info so a transient leak is caught. - cross_path_non_poisoning (Y=10%): when WiFi goes over-horizon, cell+starlink received rate stays >= (1-Y) of baseline (WiFi failure can't starve them). - critical_gap_over_horizon (G=6s): no Critical inter-arrival gap overlapping an over-horizon window exceeds G — the always-on paths keep Critical flowing. - drop_by_tier_report: measure-and-report only (per-topic priority is #19, not implemented) — records per-tier succ/drop in the critical phase. Adds the republished Critical heartbeat to the recorded bag (for the gap invariant) and generalizes bag_reader's connection accessor (get_connection + topic_connection_ids). All 9 range_degradation invariants pass (112s run).
… Phase 6) README: add the subscriber_death scenario row + section (incl. the no-wedge finding), the four Phase-4 multi-link invariants (6-9), the N-threshold observation (Recv-Q stayed 0 → N=10s is conservative), and a Continuous Integration section (no CI wires these; skip-guards + opt-in gating are the protection; smoke skips cleanly without unprivileged userns).
- recv_q_trace.py: emit a wall-clock 'unix_time' column so consumers correlate against cross-process wall-clock events without subtracting a per-process monotonic axis. test_subscriber_death now slices the post-stall window on unix_time directly. - test_subscriber_death: fail (not skip) when the Recv-Q trace has rows but the tracer never found the operator socket — an unbound socket is a harness failure, not a quiet queue, and must not pass vacuously. - run_scenario: reap the SIGKILLed Bulk victim with wait() so it doesn't linger as a zombie until atexit cleanup. Both subscriber_death invariants still pass under Zenoh (39s).
Comment on lines
+195
to
+205
| with artifacts.recvq_csv.open() as f: | ||
| for r in csv.DictReader(f): | ||
| try: | ||
| t = float(r['t_seconds_since_start']) | ||
| q = int(r['recv_q_bytes']) if r['recv_q_bytes'] else 0 | ||
| rows.append((t, q)) | ||
| except (ValueError, KeyError): | ||
| continue | ||
| if not rows: | ||
| pytest.skip('No Recv-Q samples recorded; nothing to assert.') | ||
|
|
Comment on lines
+315
to
+323
| """Like `launch_bridge` but tees the bridge's stderr to `stderr_log`. | ||
|
|
||
| Phase 3 needs per-bridge stderr captured so we can check invariant | ||
| 2 (no ERROR-severity log during the over-horizon window). We can't | ||
| redirect to a file inside the Popen because we still need the | ||
| lifecycle transitions to be issued; the cleanest approach is to | ||
| open the log file in append mode and pass it as `stderr=`. Debug | ||
| mode (UDP_BRIDGE_BENCH_DEBUG=1) additionally tees to the terminal | ||
| via `tee` so the user sees the live output. |
Comment on lines
+58
to
+67
| def _make_image(self): | ||
| m = Image() | ||
| m.header.stamp = self.get_clock().now().to_msg() | ||
| m.header.frame_id = 'bench' | ||
| m.height = 1 | ||
| m.width = self._payload_bytes | ||
| m.encoding = 'mono8' | ||
| m.step = self._payload_bytes | ||
| m.data = bytes(self._payload_bytes) | ||
| return m |
| <test_depend>nav_msgs</test_depend> | ||
| <test_depend>sensor_msgs</test_depend> | ||
| <test_depend>iproute2</test_depend> | ||
| <test_depend>util-linux</test_depend> |
Comment on lines
+9
to
+13
| Usage: | ||
| run_scenario.py --scenario smoke [--duration-s 10] [--outdir DIR] | ||
| run_scenario.py --scenario full-mix [--duration-s 10] [--outdir DIR] | ||
| run_scenario.py --scenario range_degradation [--hold-s 10] [--outdir DIR] | ||
|
|
…+ rosbag2 deps)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18
Summary
Linux-netns +
tc netembench harness reproducing udp_bridge field failuremodes off the boat — no root, no mininet. Runs both bridges co-resident in one
unprivileged user+net namespace over three simulated paths (WiFi/Cell/Starlink)
with a three-tier traffic mix, and asserts invariants from a recorded bag +
Recv-Q trace. Skips cleanly where unprivileged userns is unavailable.
What's here (Phases 0–6 + an added scenario)
mix, range-degradation trajectory + 5 single-path invariants.
cross-path non-poisoning (Y), critical-gap over-horizon (G), drop-by-tier
(measure-and-report; per-topic priority is Add per-topic priority/class scheduling to the rate-limiter #19). All 9 range_degradation
invariants pass.
test_mininet_*.launchmoved to
test/mininet/.archived/with a pointer README;mainpage.doxrepointed to
test/bench/.gating are the protection (documented). README updated.
subscriber_deathscenario (issue Bridge wedges (reader thread blocked, Recv-Q backup) when remote subscriber dies #10 wedge) — freezes (SIGSTOP) theBulk operator subscriber mid-stream, traces operator Recv-Q + survivor
delivery; forces
rmw_zenoh_cpp+ a Zenoh router.Finding — the #10 wedge does not reproduce at bench scale
Across FastDDS, CycloneDDS, and Zenoh, with clean-kill and freeze triggers,
even at ~58 MB/s Bulk into a frozen consumer, the operator Recv-Q/Send-Q stayed
flat at 0 and survivors were unaffected. At bench scale
publish()doesn'tblock the drain thread (under Zenoh it's an async hand-off). So
test_subscriber_deathis a no-wedge regression guard (asserts the healthybehavior; would catch a reintroduced drain-block under a blocking RMW), not
a red→green reproduction. The component-level proof of the #10 fix is the
PublishQueueunit tests in PR #28. Full rationale in the test docstring andtest/bench/README.md. This PR is independent of #28.Test plan
colcon buildclean;test_bench_smokeregistered (skips without userns).range_degradation— all 9 invariants pass (~112 s, FastDDS).subscriber_death— both guard invariants pass (~39 s, Zenoh); head-of-linemeasurement: survivors unaffected (no cross-topic degradation at bench scale).
UDP_BRIDGE_BENCH_SCENARIOS=1; not run in a defaultcolcon test.Authored-By:
Claude Code AgentModel:
Claude Opus 4.7 (1M context)