Transport rework umbrella: params/profiles, wire protocol, QoS (off-season design)

[rolker]

udp_bridge big-picture transport rework — design / tracking (off-season)

Status: design + tracking issue, not yet actionable. Off-season effort.
Capturing the big picture so it isn't lost; open questions are deliberately
left open pending more thought. No implementation is committed by this issue.

This is a deliberately large-blast-radius rework spanning three coupled axes the
field has been pushing on: how parameters are organized, additions/changes
to the wire protocol, and better QoS handling. A lot of it is already
latent in the tracker (#14, #18, #19, #20, #26, #34, #35); part of the goal is to
decide whether those are one coherent rework or several independent ones.

Cross-cutting decision: flag days are acceptable (for now)

This is still a development system, so we do not need wire-protocol
capability negotiation or mixed-version (boat vs operator) interop. Breaking
envelope changes are fine as long as both ends update together.

• Keep one cheap thing: a single protocol version byte in the
  handshake / BridgeInfo — not for negotiation, just so a mismatched pair
  fails loudly ("boat vN / operator vM, update one") instead of silently
  mis-decoding and looking like packet loss.
• This unlocks the protocol axis: e.g. an optional send-timestamp so packet
  ordering is based on real time rather than the per-remote packet_number
  (a transport counter, not a per-topic semantic). That makes the stale-gate
  (Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34) and the reorder buffer (Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35) correct rather than approximate.

Unifying concept: per-topic "transport profile"

Today a topic's behavior is scattered across topics_list (allowlist), the
topics map (source/queue_size/period), the global drop_stale_packets,
and implicit QoS. A profile lets a topic declare its semantics once and
absorbs several open issues:

• delivery class — latest-wins (heartbeat/video/costmap) vs every-sample
  (sonar, pointclouds) vs multiplexed (like /tf)
• gate behavior — drop-stale on/off (Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34) + reorder window (Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35)
• priority / class for the rate-limiter (Add per-topic priority/class scheduling to the rate-limiter #19)
• QoS intent — reliability + durability the republisher advertises

Workstream 1 — parameters / profiles

Settled in discussion:

• Drop the label indirection. Key topics by the source topic name
  directly; topics_list collapses into the map keys (the keys are the
  allowlist).
• Boundary rule: identity & routing are per-topic; behavior is profile.
  • per-topic (always): source topic (the key), destination rename
  • profile (shared): delivery class, gate + reorder window, priority, QoS intent
  • either (profile default, per-topic override): queue_size, period
• Outbound-only stays — it's a deliberate locality principle (each bridge
  config owns only what it sends), not a bug. The real pain is the silent
  failure when a boat-outbound and operator-inbound pair gets edited on only one
  side. Fix = cross-config validation/discoverability (lint two configs:
  "boat advertises X but no peer subscribes" / "operator requests Y but no peer
  advertises"), not restructuring.

Illustrative schema (not final):

profiles:
  realtime-latest:     # heartbeat, video, costmap window
    delivery: latest_wins
    drop_stale: true
    priority: high
    qos: {reliability: best_effort, durability: volatile}
  survey-stream:       # raw sonar, pointclouds — every sample matters
    delivery: every_sample
    drop_stale: false        # never gate these (the 2026-06-29 lesson)
    priority: low
  gridmap-heavy:       # cube tiles / grids
    delivery: latest_wins
    drop_stale: true
    reorder_window_ms: 20
    default_period: 0.5

connections:
  vpn:
    max_bytes_per_second: 1200000
    advertise:                                    # outbound-only, made explicit
      /sensors/m3/cube_bathymetry/coverage_catalog: {profile: gridmap-heavy, queue_size: 2}
      /sensors/m3/cube_bathymetry/coverage_tiles:   {profile: gridmap-heavy, queue_size: 2, period: 0.5}
      /manda_coverage/coverage_swath:               {profile: gridmap-heavy, destination: /operator/manda_coverage/coverage_swath}

Must verify before committing to "topic name as key": how the bridge
currently ingests this config. ROS 2 parameter names don't love /, and the
YAML→param flattening uses .; slashed keys (/sensors/m3/...) may be rejected
or mangled if the loader uses standard ROS param declaration. If it reads the
structure as an opaque map/YAML, it's free. Also grep that nothing else (stats,
diagnostics, logs) keys off the label today.

Workstream 2 — QoS handling

• Live bug, not just cleanup: incompatible QoS (DURABILITY) -> No messages will be sent on .../cube_bathymetry/coverage_catalog (2026-06-29). The
  bridge carries reliability/durability on MessageInternal but doesn't fully
  resolve durability across the link.
• Proper durability across a UDP bridge means the sender caches the latched
  sample and replays it to a late/reconnecting subscriber — a small protocol
  feature (a "send me the latched value" path). With flag days OK this is now
  tractable, but it's the biggest single chunk and may deserve its own phase.
• QoS intent slots naturally into the profile (above).

Workstream 3 — wire protocol

Candidate changes (flag days assumed), roughly increasing scope:

• protocol version byte (above) — do first, trivially
• optional send-timestamp for real-time ordering (helps Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34/Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35)
• durability / latched-replay support (Workstream 2)
• numeric topic IDs instead of string topic names per packet (bandwidth win
  on constrained links; bigger change)

Relationship to existing issues

• Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34 — stale-gate over-broad → opt-in per latest-wins topic (absorbed: a profile field)
• Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35 — reorder/jitter buffer for out-of-order packets (absorbed: a profile field)
• Add per-topic priority/class scheduling to the rate-limiter #19 — per-topic priority/class scheduling for the rate-limiter (absorbed: a profile field)
• Add lifecycle services: RemoveRemote and Unsubscribe #14 — lifecycle services (RemoveRemote/Unsubscribe) — adjacent (runtime config)
• Bench-test harness for field failure modes (resend amp, wedge, rate-limit accuracy, saturation) #18 — bench-test harness (netns + tc netem) — prerequisite: can't safely
  refactor a field-critical transport without a repeatable rig + regression
  fixtures (the ~677k-drop case, the QoS-DURABILITY stall)
• Stats-timer publication stalls during Starlink-only ops while data-republish keeps working #20 — stats-timer stalls during Starlink-only ops — adjacent
• Wrap deserialize() in try/catch at remaining decode* sites #26 — wrap deserialize() in try/catch at decode sites — adjacent robustness

Rough phasing (not final)

1. Bench harness (Bench-test harness for field failure modes (resend amp, wedge, rate-limit accuracy, saturation) #18) — makes the rest measurable rather than "seems better"
2. Version byte
3. Param/profile model (absorbs Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34/Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35/Add per-topic priority/class scheduling to the rate-limiter #19)
4. QoS durability / latched-replay (own phase, biggest chunk)
5. Heavier wire changes (send-stamp, topic IDs)

Open questions (intentionally unresolved)

• Where do profiles live — defined once and shared by both boat and
  operator configs (a common imported file), or duplicated per-config?
  Shared is nicer but cuts against the strict outbound locality.
• Is QoS durability-replay in scope for this rework, or its own separate
  phase/effort?
• Config ingestion: does the current loader use ROS param declaration
  (slashed keys problematic) or read an opaque map/YAML? Decides whether
  "topic name as key" is config-only or needs an encoding step.
• Send-timestamp vs packet_number — does the timestamp replace the
  per-remote counter for ordering, or coexist with it?
• Numeric topic IDs — worth the bandwidth win for the added complexity,
  or skip?
• Scope boundary — is this genuinely one umbrella, or should params /
  QoS / protocol proceed as separate efforts that share the profile concept?
• Do existing Make the stale-packet gate opt-in per latest-wins topic (it's over-broad) #34/Reorder/jitter buffer for out-of-order packets (hold-then-reorder instead of hard-drop) #35/Add per-topic priority/class scheduling to the rate-limiter #19 get absorbed/closed into this, or stay as
  standalone deliverables under an umbrella?

---

Authored-By: Claude Code Agent
Model: Claude Opus 4.8 (1M context)