Skip to content

fix(reticulum): stop announce storms from dropping inbound LXMF #739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rinchen merged 4 commits into from
Jul 31, 2026
Merged

fix(reticulum): stop announce storms from dropping inbound LXMF #739

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6c95e47
fix(reticulum): stop announce WS storms from dropping inbound LXMF
rinchen Jul 31, 2026
f221c09
feat(diagnostics): warn when announce storms pressure LXMF delivery
rinchen Jul 31, 2026
389e99b
feat(diagnostics): guide users on announce-bus-pressure tips
rinchen Jul 31, 2026
8992a42
refactor(reticulum): serialize announce pressure tests and extract LX…
rinchen Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/diagnostics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -433,11 +433,12 @@ Runtime interface-issue rows from the sidecar latch (`interfaceIssueAlert`) are

Additional runtime rows (refreshed from sidecar status + `reticulumPropagationStore`, not only the config audit poll):

| Condition | Trigger | Severity | Action |
| ------------------------------------ | --------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------- |
| `reticulum/sidecar-unhealthy` | Sidecar `running && healthy === false` for ≥ 60 s (`sidecarUnhealthySince`) | error | **Restart stack** |
| `reticulum/propagation-sync-stuck` | Sync active ≥ ~45 s (`RETICULUM_PROPAGATION_SYNC_STALL_MS`) with progress still Establishing (< 15) | warning | Retry sync; check PN path / announce |
| `reticulum/propagation-sync-failing` | Sync idle with `lastSyncError` (excludes user cancel) and attempt within 1 h | warning | See Network → Propagation / troubleshooting |
| Condition | Trigger | Severity | Action |
| ------------------------------------ | --------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------- |
| `reticulum/sidecar-unhealthy` | Sidecar `running && healthy === false` for ≥ 60 s (`sidecarUnhealthySince`) | error | **Restart stack** |
| `reticulum/announce-bus-pressure` | Recent WS `events_lagged` (skipped ≥ 8) **or** sidecar `announce_ws` storm/overflow within 5 min | warning | Tips under issue + **Open Interfaces** (disable unused hubs; Share instance / announce interval / wait) |
| `reticulum/propagation-sync-stuck` | Sync active ≥ ~45 s (`RETICULUM_PROPAGATION_SYNC_STALL_MS`) with progress still Establishing (< 15) | warning | Retry sync; check PN path / announce |
| `reticulum/propagation-sync-failing` | Sync idle with `lastSyncError` (excludes user cancel) and attempt within 1 h | warning | See Network → Propagation / troubleshooting |

| Issue kind | Typical cause | In-app action |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
Expand Down
24 changes: 12 additions & 12 deletions docs/reticulum-sidecar-ipc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ The Connection tab UI edits a subset: **name** and **mode** for all types; **hos
| ------ | ------------------------------ | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| POST | `/api/v1/lxmf/send` | `{ destination_hash, text, reply_to_hash?, reply_to_id?, reply_preview_text? }` | Live: stamps LXMF `FIELD_REPLY_TO` (0x30) / optional `FIELD_REPLY_QUOTE` (0x31) before sign; `{ ok, delivery_method?, delivery_status?, sent_via?, message? }` or `{ ok: false, error: "no_propagation_node" }`. **`delivery_status` on this response is initial enqueue state only** (`queued` or `sending`) — not delivery confirmation. Stub: `{ ok, sent_via?, message? }` |
| POST | `/api/v1/lxmf/reaction` | `{ destination_hash, target_hash, emoji }` | `{ ok, message? }` |
| GET | `/api/v1/lxmf/recent` | `?since_ts=` (ms, optional), `?limit=` (default 200, max 500) | `{ messages: [] }` — ring buffer of recent **inbound** LXMF payloads for WS lag/reconnect catch-up (not durable across sidecar restart; capped at 200) |
| GET | `/api/v1/lxmf/recent` | `?since_ts=` (ms, optional), `?limit=` (default 200, max 500) | `{ messages: [], ring_len }` — ring buffer of recent **inbound** LXMF payloads for WS lag/reconnect catch-up (not durable across sidecar restart; capped at 200); `ring_len` is current buffer occupancy |
| DELETE | `/api/v1/lxmf/messages/{hash}` | | `{ ok }` |
| GET | `/api/v1/contacts` | | `{ contacts: [] }` — overlays announce/peer/Nomad labels onto nameless or hash-prefix contact `display_name` values (does not overwrite a real name) and may persist fills |
| DELETE | `/api/v1/contacts` | | `{ ok, cleared }` — clears LXMF contacts after demoting them into the peer cache (keeps Peers; does not delete chat messages) |
Expand Down Expand Up @@ -201,16 +201,16 @@ Listener persistence: a successful `POST /api/v1/rncp/listener` stores the confi

### System

| Method | Path | Body / notes | Response |
| ------ | ------------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GET | `/api/v1/diagnostics` | | Reticulum-native health snapshot |
| POST | `/api/v1/system/factory-reset` | | `{ ok }` — Electron UI must call `electronAPI.reticulum.factoryReset` (generic `proxyPost` blocks this path) |
| GET | `/api/v1/voice/status` | | LXST stub status |
| GET | `/api/v1/games/status` | | LRGP stub status |
| GET | `/api/v1/identities` | | `{ identities: […] }` — slots under `config/identities/<id>/` + `active_identity`; working key remains `config/identity`; flat identity migrates to `identities/default/` |
| POST | `/api/v1/identities` | `{ display_name? }` | `{ ok, id, identity }` or `{ ok: false, error }` — `rns-stack` only; stage slot → apply working key → commit pointer last (rollback on failure). Cap 16 slots. Errors: `identity_slot_limit_reached`, `display_name_*`. Emits restart. |
| POST | `/api/v1/identities/switch` | `{ identity_id }` | `{ ok }` or `{ ok: false, error }` — stash, install target → working, reconcile, then commit pointer. Errors: `identity_slot_not_configured`, `identity_not_found` |
| POST | `/api/v1/identities/delete` | `{ identity_id }` | `{ ok }` or `{ ok: false, error }` — refuses `cannot_delete_active_identity` / `cannot_delete_last_identity` |
| Method | Path | Body / notes | Response |
| ------ | ------------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GET | `/api/v1/diagnostics` | | Reticulum-native health snapshot — includes `announce_ws` coalesce pressure (`last_window_ingress` / `unique` / `overflow`, `last_storm_at_ms`, `last_flush_at_ms`). Renderer Diagnostics emits `reticulum/announce-bus-pressure` when lag/storm/overflow is recent. |
| POST | `/api/v1/system/factory-reset` | | `{ ok }` — Electron UI must call `electronAPI.reticulum.factoryReset` (generic `proxyPost` blocks this path) |
| GET | `/api/v1/voice/status` | | LXST stub status |
| GET | `/api/v1/games/status` | | LRGP stub status |
| GET | `/api/v1/identities` | | `{ identities: […] }` — slots under `config/identities/<id>/` + `active_identity`; working key remains `config/identity`; flat identity migrates to `identities/default/` |
| POST | `/api/v1/identities` | `{ display_name? }` | `{ ok, id, identity }` or `{ ok: false, error }` — `rns-stack` only; stage slot → apply working key → commit pointer last (rollback on failure). Cap 16 slots. Errors: `identity_slot_limit_reached`, `display_name_*`. Emits restart. |
| POST | `/api/v1/identities/switch` | `{ identity_id }` | `{ ok }` or `{ ok: false, error }` — stash, install target → working, reconcile, then commit pointer. Errors: `identity_slot_not_configured`, `identity_not_found` |
| POST | `/api/v1/identities/delete` | `{ identity_id }` | `{ ok }` or `{ ok: false, error }` — refuses `cannot_delete_active_identity` / `cannot_delete_last_identity` |

## WebSocket

Expand All @@ -227,7 +227,7 @@ Event types: `lxmf_message`, `lxmf_outbound_status`, `events_lagged` (WS subscri
- **`rrc.disconnected`:** payload `{ hub_dest_hash, reason, will_reconnect? }`. When `will_reconnect` is `false` (or `reason` is `local_disconnect`), the renderer drops that hub session. When `true` (or omitted on older sidecars), the UI shows reconnecting and keeps volatile rooms until WELCOME.

- **`lxmf_outbound_status`:** authoritative outbound delivery updates. Payload: `{ message_hash, status, delivery_method?, to_hash?, sent_via? }` where `status` is `delivered`, `failed`, or intermediate `sending` (egress upgrade or Direct→PN fallback). mesh-client maps `delivered` → UI Completes (`acked`) and persists `delivery_status` (+ `delivery_method` when present) to SQLite; Propagated Completes show **Stored at propagation node**; `failed` → Failed. Do **not** treat `/api/v1/lxmf/send` response `delivery_status` (`queued`/`sending`) as terminal. After Direct failure with a preferred remote PN, the sidecar re-queues once as Propagated and emits `sending` + `delivery_method: "propagated"` before a final `delivered`/`failed`.
- **`announce.received`:** emitted for every LXMF identity announce / path response the sidecar observes (named or nameless). Payload: `{ destination_hash, display_name?, hops }`. Display names update the peer-label cache only — announces do **not** auto-create LXMF contacts. That cache is overlayed onto `GET /api/v1/peers` / topology rows **and** onto nameless/hash-prefix rows from `GET /api/v1/contacts` (`list_contacts` may persist those fills) so path-table and contact refreshes keep announce aliases.
- **`announce.received`:** coalesced WS notify for LXMF identity announces / path responses (named or nameless). Sidecar applies identity-key + display-name cache updates immediately, but emits **at most one** WS frame per coalesce window (500ms normal / 1000ms when >256 distinct destinations are pending) so announce storms stay O(1) bus pressure on large meshes (~100k). Payload is either a single `{ destination_hash, display_name?, hops }` (legacy / one-row flush) or `{ announces: [{ destination_hash, display_name?, hops }, ...] }` (capped at 1024, named preferred; overflow dropped — slow peer poll recovers). Each flush publishes pressure counters under `GET /api/v1/diagnostics` → `announce_ws` (ingress/unique/overflow + storm/flush timestamps) for the Diagnostics `reticulum/announce-bus-pressure` warning. Display names update the peer-label cache only — announces do **not** auto-create LXMF contacts. That cache is overlayed onto `GET /api/v1/peers` / topology rows **and** onto nameless/hash-prefix rows from `GET /api/v1/contacts` (`list_contacts` may persist those fills) so path-table and contact refreshes keep announce aliases.
- **`peers_updated`:** also emitted when the live path table **gains** new destination hashes (maintenance tick). Payload may include `{ added: string[], patches: PeerRow[], count }` (added/patches capped at 1024). Renderer applies patches incrementally, including route-field changes. A full peer dump is used on connect, manual Refresh, restart, safety poll, or a `peers_updated` payload that cannot be applied incrementally: `cleared`, `demoted_from_contacts`, or a single-`hash` probe/path event. Hop/timestamp-only churn does not emit.

`lxmf_message` payload fields include `sender_hash`, `text`, `timestamp`, `message_hash`, optional `direction` (`inbound` / `outbound`), optional `delivery_status` (`sending` on optimistic outbound rows), optional `reply_to_hash` / `reply_preview_text` (from LXMF `FIELD_REPLY_TO` / `FIELD_REPLY_QUOTE`), and transport markers `received_via` / `sent_via`. Outbound `sent_via` is **path-table / PacketTap evidence**, not “any local RNode enabled”: atomic values are `rf`, `ble`, `tcp`, or `network`; multi-egress observes join with `+` (e.g. `rf+tcp`, `ble+network`). Inbound `received_via` uses the path-table interface name **matched to local interface config** (same atoms — so a TCP hub named “RNS Testnet” is `tcp`, not `network`). Never use Meshtastic-style `both` for Reticulum.
Expand Down
3 changes: 2 additions & 1 deletion reticulum-sidecar/src/api/lxmf.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,5 +143,6 @@ pub async fn list_recent_lxmf(
) -> Json<serde_json::Value> {
let limit = q.limit.unwrap_or(200).clamp(1, 500);
let messages = stack.list_recent_inbound_lxmf(q.since_ts, limit);
Json(serde_json::json!({ "messages": messages }))
let ring_len = stack.inbound_lxmf_ring_len();
Json(serde_json::json!({ "messages": messages, "ring_len": ring_len }))
}
Loading