perf: pace COT render loop & cut per-frame allocations; fix JS8Call UDP bridge + KiwiSDR auth

.github/FUNDING.yml

@@ -0,0 +1,15 @@
+# These are supported funding model platforms
+
+github: d3mocide
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+polar: # Replace with a single Polar username
+buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
+thanks_dev: # Replace with a single thanks.dev username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

agent_docs/tasks/2026-07-06-frontend-cots-optimization-js8-kiwi-fixes.md

@@ -0,0 +1,113 @@
+# Frontend COT Rendering Optimization + JS8Call/KiwiSDR Bridge Fixes
+
+## Issue
+
+1. **Rendering**: The tactical map needed to stay smooth with thousands of live
+   COTs. The rAF loop ran uncapped (120/144 Hz on fast displays), recomposed
+   every deck.gl layer each tick, and `buildEntityLayers`/`buildTrailLayers`
+   made 5-6 separate `filter`/`map` passes per frame — each allocating
+   intermediate arrays, plus a fresh path array per trail per frame.
+2. **Pipeline**: `useEntityWorker` serialized every maritime entity (with
+   trails) to `localStorage` inline on the WebSocket message path every 10 s —
+   a multi-ms main-thread hitch at AIS scale, with no size cap (quota risk).
+   The TAK worker flushed decode batches every 10 messages, causing excessive
+   worker→main wakeups during the ~11k-message orbital sweeps.
+3. **JS8Call terminal never worked**: the bridge had the UDP API model
+   inverted. JS8Call (WSJT-X model) binds an ephemeral port and *pushes*
+   events to the configured "UDP Server" (127.0.0.1:2242 per our INI); the
+   bridge instead listened on 2245 (where nothing ever arrives — the INI's
+   `UDPClient2*` keys are not real JS8Call settings) and sent commands to a
+   fixed port 2242 (where nothing listens). Additionally, every outgoing
+   datagram used uppercase JSON keys (`{"TYPE": ...}`) while the JS8Call API
+   requires lowercase (`{"type", "value", "params"}`), `RIG.SET_FREQ` lacked
+   the required `params.DIAL`, and `MODE.SET_SPEED` sent a string instead of
+   the numeric submode.
+4. **KiwiSDR password nodes never connected**: `kiwi_client.py` sent
+   `SET auth t=kiwi pwd=<md5>`, which is not part of the KiwiSDR protocol.
+   The reference kiwiclient sends plaintext `SET auth t=kiwi p=<password>`.
+   The waterfall stream also always authenticated with an empty password.
+
+## Solution
+
+- Adaptive frame pacing in the animation loop: ~30 fps when
+  `entities + satellites > 800`, ~60 fps cap otherwise (skips redundant
+  120/144 Hz ticks). dt accumulates across skipped ticks so interpolation is
+  unaffected.
+- Single-pass dataset derivation in `buildEntityLayers` (integrity halos,
+  altitude stems, tactical halos, selection ring, velocity vectors) and
+  `buildTrailLayers` (trails + gap bridges), plus a `WeakMap` cache for
+  smoothed-trail → path3D conversion keyed on the (update-stable) trail array.
+- Maritime snapshot: capped at 750 most-recent entities, interval 10 s → 30 s,
+  serialization moved to `requestIdleCallback`.
+- TAK worker batch size 10 → 64 (flush interval unchanged at 50 ms).
+- JS8 bridge: binds UDP 2242, records JS8Call's datagram source address as the
+  command reply address, sends correctly-shaped lowercase-key API messages,
+  handles `PING`/`STATION.CALLSIGN`/`STATION.GRID`/`RIG.FREQ`/`MODE.SPEED`
+  responses into a merged `STATION.STATUS` broadcast, reports
+  `js8call_connected` from actual datagram liveness (60 s window), and pulls
+  initial station state on first contact. Removed the bogus `UDPClient2*` INI
+  keys; compose/Dockerfile env updated (`JS8CALL_PORT` → `JS8CALL_UDP_SERVER_PORT`).
+- KiwiSDR auth: plaintext `p=<password>` per reference kiwiclient, applied to
+  both SND and W/F streams.
+
+## Follow-up (same branch): WS frame batching + binary icon attributes
+
+- **Coalesced WebSocket frames**: the broadcast service previously sent one
+  binary frame per Kafka message per client with a 256-deep drop-oldest
+  queue — the ~11k-message orbital sweep could silently drop most of a slow
+  client's data. The client worker now drains its queue and coalesces
+  consecutive proto messages into batch frames (`0xbf 0x02 0xbf` magic +
+  u32le length-prefixed records, each record an unmodified legacy frame),
+  capped at 128 messages / 60 KB per send, alert JSON ordering preserved.
+  Queue deepened to 4096. Single messages still use the legacy frame, and the
+  frontend worker decodes both formats (`workers/batchFraming.ts`).
+- **Binary attributes for the 2D entity icon layer**
+  (`layers/entityIconAttributes.ts`): position/angle/color/size are uploaded
+  as persistent typed arrays filled in one pass per paced frame — deck.gl no
+  longer iterates entity objects for them. Per-frame buffers ping-pong so
+  external-buffer references change with content; colors/sizes refresh only
+  on membership/selection change or a 1 s cadence (entityColor tracks
+  altitude/speed, which drift slowly). Picking is index-based; the overlay
+  hover-miss check now keys on `info.index` so binary picks aren't cleared.
+  Globe mode and the no-cache path keep the object-based layers.
+
+## Changes
+
+- `frontend/src/hooks/useAnimationLoop.ts` — adaptive frame pacing; rAF
+  scheduled at tick start so paced skips keep the loop alive.
+- `frontend/src/layers/buildEntityLayers.ts` — one pass over interpolated
+  entities builds all five per-layer datasets.
+- `frontend/src/layers/buildTrailLayers.ts` — merged trail/gap-bridge pass;
+  `WeakMap` path3D cache.
+- `frontend/src/hooks/useEntityWorker.ts` — sea snapshot cap + idle-time write.
+- `frontend/src/workers/tak.worker.ts` — batch size 64.
+- `js8call/server.py` — UDP server model fixed (bind 2242, reply-address
+  routing), JS8Call API message shapes fixed, station-state merge + liveness.
+- `js8call/kiwi_client.py` — plaintext password auth on SND and W/F streams.
+- `js8call/Dockerfile` — INI cleanup, env var rename.
+- `js8call/tests/test_kiwi_compatibility.py` — auth tests updated to the
+  protocol-correct plaintext form.
+- `docker-compose.yml` — `JS8CALL_PORT` (unused) → `JS8CALL_UDP_SERVER_PORT`.
+
+## Verification
+
+- `cd frontend && pnpm run lint && pnpm run typecheck && pnpm run test` —
+  clean; 278/278 tests pass.
+- `cd js8call && uv tool run ruff check . && uv run python -m pytest` — clean;
+  26/26 tests pass (two tests asserting the incorrect MD5 auth form were
+  updated to assert the reference-kiwiclient plaintext form).
+- JS8Call/KiwiSDR runtime paths require a container rebuild
+  (`docker compose up -d --build sovereign-js8call`) and a live JS8Call/KiwiSDR
+  to exercise end-to-end; protocol behavior was verified against the reference
+  kiwiclient implementation and the JS8Call/WSJT-X UDP API model.
+
+## Benefits
+
+- Entity/trail layer construction does ~5x fewer array allocations per frame
+  and the whole pipeline does bounded work per second regardless of display
+  refresh rate — steadier frame times with thousands of COTs, less GC churn.
+- No more periodic main-thread stalls from maritime cache serialization; the
+  cache can no longer blow the localStorage quota.
+- The JS8 terminal can actually exchange traffic with JS8Call (RX spots,
+  directed messages, TX, freq/speed control), and password-protected KiwiSDR
+  nodes can authenticate.

backend/api/services/broadcast.py

@@ -14,8 +14,66 @@
 logger = logging.getLogger("SovereignWatch.Broadcast")
 
 # Max messages queued per client before we start dropping (oldest dropped first).
-# At ~37s orbital cycles emitting 11k messages, 256 gives ~23ms grace before dropping.
-_CLIENT_QUEUE_SIZE = 256
+# The client worker drains the queue in coalesced batch frames (up to
+# _MAX_BATCH_MSGS per WebSocket send), so even the ~11k-message orbital sweep
+# is flushed in a few dozen sends. The deeper queue absorbs that burst for a
+# slow client instead of silently dropping most of it (~1.7 MB transient
+# worst case at ~150 B/message).
+_CLIENT_QUEUE_SIZE = 4096
+
+# Coalesced binary frame format:
+#   [0:3]  0xbf 0x02 0xbf                     — batch magic
+#   then per record: uint32 little-endian length + payload
+# Each payload is an unmodified single TAK message (with its own
+# 0xbf 0x01 0xbf prefix), so the frontend decodes records with the exact
+# same code path as legacy single-message frames. Single messages are still
+# sent as legacy frames for wire compatibility.
+_BATCH_MAGIC = b"\xbf\x02\xbf"
+_MAX_BATCH_MSGS = 128
+_MAX_BATCH_BYTES = 60_000
+
+
+def coalesce_outgoing(items: list) -> list[tuple[str, bytes]]:
+    """
+    Group an ordered mix of proto messages (bytes) and alert tuples
+    (("alert", json_bytes)) into outgoing WebSocket sends, preserving order.
+
+    Returns a list of ("bytes", frame) / ("text", payload) send instructions.
+    Consecutive proto messages are coalesced into batch frames capped at
+    _MAX_BATCH_BYTES; a lone proto message keeps the legacy single frame.
+    """
+    sends: list[tuple[str, bytes]] = []
+    pending: list[bytes] = []
+    pending_bytes = 0
+
+    def flush() -> None:
+        nonlocal pending, pending_bytes
+        if not pending:
+            return
+        if len(pending) == 1:
+            sends.append(("bytes", pending[0]))
+        else:
+            parts = [_BATCH_MAGIC]
+            for m in pending:
+                parts.append(len(m).to_bytes(4, "little"))
+                parts.append(m)
+            sends.append(("bytes", b"".join(parts)))
+        pending = []
+        pending_bytes = 0
+
+    for item in items:
+        if isinstance(item, tuple):
+            msg_type, data = item
+            if msg_type == "alert":
+                flush()
+                sends.append(("text", data))
+            continue
+        if pending_bytes + len(item) > _MAX_BATCH_BYTES or len(pending) >= _MAX_BATCH_MSGS:
+            flush()
+        pending.append(item)
+        pending_bytes += len(item)
+    flush()
+    return sends
 
 
 class BroadcastManager:
@@ -238,21 +296,32 @@ async def _consume(self):
             self._clients.clear()
 
     async def _client_worker(self, ws: WebSocket, q: asyncio.Queue):
-        """Background task per client: dequeue and send, with a generous timeout."""
+        """
+        Background task per client: drain the queue and send coalesced frames.
+
+        Waiting on the first message then opportunistically draining whatever
+        else is already queued turns per-message sends into a handful of batch
+        frames during dense bursts (orbital sweeps), while sparse traffic
+        still goes out immediately as legacy single frames.
+        """
         try:
             while True:
-                msg = await q.get()
+                first = await q.get()
+                items = [first]
+                while len(items) < _MAX_BATCH_MSGS:
+                    try:
+                        items.append(q.get_nowait())
+                    except asyncio.QueueEmpty:
+                        break
+
                 try:
-                    # Handle both TAK proto (bytes) and alert JSON (tuple)
-                    if isinstance(msg, tuple):
-                        msg_type, data = msg
-                        if msg_type == "alert":
+                    for kind, payload in coalesce_outgoing(items):
+                        if kind == "text":
                             await asyncio.wait_for(
-                                ws.send_text(data.decode("utf-8")), timeout=3.0
+                                ws.send_text(payload.decode("utf-8")), timeout=3.0
                             )
-                    else:
-                        # TAK proto (bytes)
-                        await asyncio.wait_for(ws.send_bytes(msg), timeout=3.0)
+                        else:
+                            await asyncio.wait_for(ws.send_bytes(payload), timeout=3.0)
                 except asyncio.TimeoutError:
                     logger.warning("Client send timed out — disconnecting")
                     break

backend/api/tests/test_broadcast_batching.py

@@ -0,0 +1,76 @@
+"""Tests for the coalesced WebSocket batch framing in services/broadcast.py."""
+
+from services.broadcast import (
+    _BATCH_MAGIC,
+    _MAX_BATCH_BYTES,
+    coalesce_outgoing,
+)
+
+MAGIC = b"\xbf\x01\xbf"
+
+
+def _msg(payload: bytes) -> bytes:
+    return MAGIC + payload
+
+
+def _parse_batch(frame: bytes) -> list[bytes]:
+    assert frame[:3] == _BATCH_MAGIC
+    records = []
+    off = 3
+    while off < len(frame):
+        length = int.from_bytes(frame[off : off + 4], "little")
+        off += 4
+        records.append(frame[off : off + length])
+        off += length
+    return records
+
+
+def test_single_message_stays_legacy_frame():
+    m = _msg(b"hello")
+    sends = coalesce_outgoing([m])
+    assert sends == [("bytes", m)]
+
+
+def test_multiple_messages_coalesce_into_batch_frame():
+    msgs = [_msg(bytes([i]) * 10) for i in range(5)]
+    sends = coalesce_outgoing(list(msgs))
+    assert len(sends) == 1
+    kind, frame = sends[0]
+    assert kind == "bytes"
+    assert _parse_batch(frame) == msgs
+
+
+def test_alert_preserves_ordering_and_splits_batches():
+    a, b, c = _msg(b"a"), _msg(b"b"), _msg(b"c")
+    alert = ("alert", b'{"type":"alert"}')
+    sends = coalesce_outgoing([a, b, alert, c])
+    assert len(sends) == 3
+    kind0, frame0 = sends[0]
+    assert kind0 == "bytes"
+    assert _parse_batch(frame0) == [a, b]
+    assert sends[1] == ("text", b'{"type":"alert"}')
+    assert sends[2] == ("bytes", c)  # lone trailing message → legacy frame
+
+
+def test_batch_respects_byte_cap():
+    big = _msg(b"x" * (_MAX_BATCH_BYTES // 2))
+    sends = coalesce_outgoing([big, big, big])
+    # Three ~30 KB messages cannot fit one 60 KB batch → at least two sends
+    assert len(sends) >= 2
+    reassembled = []
+    for kind, frame in sends:
+        assert kind == "bytes"
+        if frame[:3] == _BATCH_MAGIC:
+            reassembled.extend(_parse_batch(frame))
+        else:
+            reassembled.append(frame)
+    assert reassembled == [big, big, big]
+
+
+def test_only_alerts():
+    alert = ("alert", b"{}")
+    assert coalesce_outgoing([alert, alert]) == [("text", b"{}"), ("text", b"{}")]
+
+
+def test_empty_input():
+    assert coalesce_outgoing([]) == []

docker-compose.yml

@@ -360,7 +360,7 @@ services:
       - KIWI_MODE=${KIWI_MODE:-usb}
       - MY_GRID=${MY_GRID:-CN85}
       - JS8CALL_HOST=0.0.0.0
-      - JS8CALL_PORT=2442
+      - JS8CALL_UDP_SERVER_PORT=2242
       - BRIDGE_PORT=8080
       - ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
       - AUTH_ENABLED=${AUTH_ENABLED:-true}