Record every observed route per transmission and show mixed ADVERTs in Relay Airtime Share

[dborup]

Background

Issue #89 is background only; this PR does not close it.

One content hash can be observed on more than one raw route. MeshCore firmware re-sends a stored advert as ROUTE_TYPE_TRANSPORT_DIRECT (shareContactZeroHop), so the same ADVERT can arrive both flooded and zero-hop.

transmissions.route_type stores only the first route that got inserted. Relay Airtime Share (PR #86) therefore put a transmission on the flood row or the zero-hop row depending on which observation happened to arrive first.

Contract

• A transmission stays one payload/content hash. The content hash still ignores route bits, transport codes and path.

• A new column, transmissions.route_mask, records every raw route value (0–3) observed for that transmission. Bit r stands for route r.

• The mask is monotonic: every observation ORs its bit in, atomically in SQL, and no bit is ever cleared.

• transmissions.route_type is unchanged. It keeps its legacy meaning (the first-inserted route), and every other consumer keeps reading it as before.

• Relay Airtime Share classifies each ADVERT by its mask, and counts it once:

  Routes seen	route_class	Label
  both flood (0/1) and zero-hop (2/3)	mixed	ADVERT (mixed)
  only flood	flood	ADVERT (flood)
  only zero-hop	zero_hop	ADVERT (zero-hop)

• A row whose mask is not known yet (legacy, or not backfilled yet) falls back to the previous route_type classification.

• Payload Type Mix still shows ADVERT as one combined type.

• The historical backfill is a lower bound. It never invents information that was lost.

Schema and migration

• internal/dbschema.Apply (ingestor) only runs ALTER TABLE transmissions ADD COLUMN route_mask INTEGER and writes the marker transmissions_route_mask_v1. There is no default, so existing rows stay NULL, which means "not backfilled". AssertReady (server) requires the column.

• The partial index idx_tx_route_mask_null ON transmissions(id) WHERE route_mask IS NULL and the backfill run together as the async migration tx_route_mask_backfill_v1.

  • The ingestor starts it after the MQTT subscription and ingestBuffer.Ready(), so the IngestBuffer covers the one index build that can't be split into chunks.
  • The index build holds writerMu and is recorded as the writer component route_mask_index.

• The backfill runs in batches of 100 transmissions. Each batch is one write transaction under writerMu, followed by a 50 ms yield.

  • It resumes after a crash or a cancel.
  • If NULL rows appear again after completion (for example, an older ingestor wrote rows after a rollback), the next start runs it again.

• The server stays read-only (mode=ro). It reports route_mask_backfill: {status, remaining} on /api/healthz and in the Relay Airtime Share response:

  Status	Meaning
  pending	The index does not exist yet, or the backfill is not running.
  backfilling	The ingestor is backfilling, or the server has not read every backfilled mask yet.
  complete	No row has route_mask IS NULL and this server has read every backfilled mask. It is never derived from a marker.

• Change log (added after review, see "Live refresh" below): internal/dbschema.Apply also creates route_mask_changes, gated by the marker route_mask_changes_v1. AssertReady requires the table, so the ingestor must be deployed before the server, as for every other schema step.

  CREATE TABLE route_mask_changes (
      id              INTEGER PRIMARY KEY AUTOINCREMENT,
      transmission_id INTEGER NOT NULL,
      route_mask      INTEGER NOT NULL,  -- the transmission's full mask after the change
      created_at      INTEGER NOT NULL   -- ingest time, Unix seconds
  );
  CREATE INDEX idx_route_mask_changes_tx ON route_mask_changes(transmission_id);

  It is a new, empty table: no scan or rewrite of existing data. AUTOINCREMENT keeps ids from being reused after deletes, because servers poll by id.

Lower bound

A legacy row gets two things:

• the bit of its stored route_type;
• the bit of every surviving observations.raw_hex header.

A route variant whose frame no longer exists anywhere is not reconstructed. An example is a zero-hop frame overwritten by the same-observer/same-path upsert. Unparseable or truncated frames, and routes outside 0..3, contribute nothing.

Cold-cache migration measurement

The budget was written down before measuring. The measurement ran on a consistent SQLite backup of the staging database, made with the SQLite backup API and verified by SHA-256.

• Host: an isolated demo host of the same hardware class (2 vCPU / 3.8 GB), container --cpus 2 --memory 1g --network none.
• Data: 496,798 transmissions / 6,526,537 observations / 4.95 GB.
• Cold cache: posix_fadvise(DONTNEED) on that run's own DB file, verified with fincore = 0.

Step	Budget	Measured
Apply (ALTER + marker), including OpenStore	ALTER ≤ 1 s	ALTER 8–15 ms; OpenStore 44.5 ms
Partial index (async, after subscribe)	one writer stop covered by the buffer, ≤ 30 s, no packet loss	16.6–19.5 s cold
Writer hold per batch (100 tx)	p99 ≤ 250 ms, max ≤ 1000 ms	p99 159–161 ms, max 251–316 ms
Concurrent live write during the backfill	p99 ≤ 500 ms, max ≤ 1500 ms	p99 121–126 ms, max 216–263 ms
Total backfill, including yields	≤ 60 min	9.0–9.3 min
WAL max / DB growth	≤ 256 MB / ≤ 64 MB	6.1 MB / 0 bytes
Peak RSS	≤ 256 MB	29–31 MB
Crash mid-batch, then restart	resumes, identical result	identical mask checksum, quick_check ok
Status query, cold	EXISTS ≤ 50 ms, COUNT ≤ 2 s	0.2–0.5 ms / 53–90 ms
observations	read only	read only, no index rebuild

The first design built the index inside Apply, before subscribe. It failed the original start-up budget of ≤ 15 s, which is why the index and the backfill moved to the async migration after subscribe.

On the copy, the final implementation left 0 rows NULL. Its mask checksum matches an independent Python calculation. It found 21 mixed transmissions (8 TXT_MSG, 7 ACK, 6 ADVERT).

Order, redelivery and parallel ingest

• Every permutation of each scenario gives the same mask, and route_type stays the first-inserted route. The scenarios:

  • flood + zero-hop from different observers;
  • zero-hop first;
  • same observer and path, where a zero-hop frame is followed by a 0-hop transport flood (the upsert case);
  • redelivery;
  • three route values;
  • single-route cases.

• The mask survives an ingestor restart.

• 200 fresh payloads, each with both variants racing for the writer, all end with 1010.

• A route outside 0..3 sets no bit, but still writes a known mask of 0.

• PATH seen with and without transport codes keeps both flood bits.

• Live ingest never writes into a legacy NULL row; that row belongs to the backfill.

• A backfill batch running at the same time as live ingest loses no bits, because both hold writerMu.

• An observation row never becomes visible without its route bit. When an observation brings a new bit, three writes run in one SQLite transaction:

  1. the OR update of route_mask;
  2. the observation insert or upsert;
  3. when the mask actually changed, one route_mask_changes row with the resulting mask.

  If any of them, or the commit, fails, the transaction rolls back and nothing becomes visible. The failure is counted once in WriteErrors, and InsertTransmission stays non-fatal as before. A trigger-based test records the mask at the moment each observation row is written, including the upsert. Fault-injection tests make each of the three writes fail, for both a new row and an upsert.

• No change row is written for a new transmission (the server reads its full mask when it loads the row), a repeated route, or a legacy NULL mask (the backfill owns it).

Server and in-memory model

• StoreTx gets routeMask uint8 and routeMaskKnown bool in existing padding. unsafe.Sizeof(StoreTx{}) is still 320, and a test pins it.

• Every load path reads t.route_mask: Load, LoadChunked/loadChunk, IngestNewFromDB and IngestNewObservations. Tests assert that the incremental view equals a cold load.

• Backfilled masks reach a running server. Transmissions loaded while their mask is still NULL are queued once they are visible in the store. That includes the background chunk loader, which queues in its locked merge step. On every poller tick the server re-reads up to 2,000 of them:

  • read-only, in lists of 500 ids by primary key;
  • in ascending id order, the same order the backfill uses;
  • stopping at the first id that is still NULL.

  Rows that are gone or evicted are dropped. Merged masks invalidate the RF cache that holds Relay Airtime Share.

• The refresh is serialised by its own mutex. Load paths append to an unsorted inbox, which the refresh sorts outside s.mu.

• advertRouteClass in relay_airtime_share.go is the only classifier.

Live refresh of same-row upserts

The bug, reproduced deterministically before the fix: the same observer hears the same path on a new route. The observation is upserted in place, so it keeps its id and no new row is created, and the database gets both bits (1001). But the poller only reads new transmission and observation ids. After three production poll ticks the in-memory mask was still 0001 or 1000, and Relay Airtime Share still showed ADVERT (flood) or ADVERT (zero-hop) instead of ADVERT (mixed), until a restart.

The fix: the server reads route_mask_changes on every poller tick. The tick body is extracted into Poller.pollStore, so tests drive the exact production path.

• Reading: rows past the cursor in id order, at most 1,000 per tick, without holding s.mu. Masks for the same transmission are ORed together.
• Applying: under s.mu, and only to transmissions the store already holds. A transmission is never re-created. The Relay Airtime Share cache is invalidated only when a mask actually changed.
• Transmissions not in the store yet (for example still in an unpublished start-up chunk): the change is parked and applied when the transmission is published. It is dropped once the start-up load is over (Load or RunStartupLoad returned, whether it succeeded, failed or loaded only part) and the store has seen past that id.
• Cursor: advanced only after every row of the batch was applied or parked.

Start-up: the watermark MAX(id) is read before Load or LoadChunked reads any transmission.

• Changes logged before the watermark are already in the loaded masks.
• Later ones are applied, idempotently, by the poller.
• Old change rows are never re-read after a restart; the full load from transmissions.route_mask is authoritative.
• If the watermark cannot be read, the server reads the log from the start rather than skipping changes.

Tests cover:

• flood→mixed and zero-hop→mixed after one tick, without a restart;
• a change that adds nothing, which leaves the cache alone;
• several changes across batches;
• evicted and deleted transmissions;
• a change before the watermark, after the watermark (hook after the first row is read, for both Load and LoadChunked), and before the first tick;
• a chunk scanned but not yet published, including one below ids the store already holds;
• a partial start-up load;
• a failed watermark read;
• backfill and a live change converging;
• concurrent ingest and poll, where the live view equals a cold load.

Retention of the change log

• Packet prune: PruneOldPackets deletes a batch's change rows in the same transaction as its transmissions, through idx_route_mask_changes_tx (the query plan is asserted in a test).
• Orphan prune: PruneOrphanRouteMaskChanges removes rows whose transmission is gone by any other path. It runs at start-up and after the daily packet prune. Each writer transaction examines a bounded id range, so its work does not depend on how far apart orphans are. It never deletes a row of an existing transmission, and there is no time-based expiry, so a slow server cannot miss a change of a transmission it still holds.
• Size: at most three rows per transmission, one per route bit it can gain. On the staging copy the realistic count is 22 rows for 496,798 transmissions.

API and frontend

• The JSON changes are additive only:

  • a new route_class value, "mixed";
  • the route_mask_backfill object.

  OpenAPI documents both, plus the route-mask semantics and the fallback.

• Relay Airtime Share renders the mixed row with:

  • data-route-class="mixed";
  • the fixed colour var(--status-purple), whatever its position;
  • the tooltip "Same payload observed on both flood and zero-hop routes; counted once."

  The colour and the tooltip come from route_class, not from the label.

Browser verified: local scratch server, desktop 1280 px and 375 px (details below; not staging).
E2E assertion added: test-e2e-playwright.js:855

Browser validation (local scratch datasets)

The branch server ran against these datasets:

• By route: full, flood only, zero-hop only, mixed only.
• Special cases: legacy, empty.
• Backfill states: pending (index missing) and backfilling.

Every set also had non-ADVERT traffic and two UNK rows. For each set, a script compared every API row with its DOM row:

• label, data-payload-type and data-route-class;
• the cnt % / air % text;
• the tooltip count and score, and the mixed tooltip;
• the colour.

Check	Result
All sets, desktop 1280 px and 375 px	0 mismatches
Mixed colour	resolves to rgb(168, 85, 247)
Empty set	shows "No relay-airtime data in this window."
SPA navigation (packets → analytics → nodes → RF → back → back) and a hard reload	0 mismatches
Console errors, page errors, unhandled rejections	none

Master's frontend, served against the branch API as a negative control, renders the mixed row as a plain row without the purple or the tooltip. That shows the check detects the change, and that the older frontend degrades cleanly.

At 375 px the page has no horizontal scroll. Inside the chart, the fixed 80/180 px grid overflows by 38 px. That is identical with master's frontend: a baseline, not changed here.

The new Playwright test checks API↔DOM identity on the fixture. It then injects a flood / mixed / zero-hop response through page.route and asserts the mixed row's identity, colour and tooltip. Its in-page assertions were dry-run against the scratch server. The full Playwright suite was not run locally, because no Playwright module is installed here; CI runs it.

Start-up schema wait (staging gate blocker)

Problem reproduced

The server and the ingestor start together under the unchanged supervisord config (autorestart=true, startretries=10, startsecs=2). On a database without this PR's change log, the server used to call log.Fatalf on the first missing route_mask_changes. In the isolated staging gate, a 70 s write lock held the ingestor's start-up Apply back. supervisord restarted the server with growing back-off and gave up after about 64.6 s, leaving it FATAL. It stayed stopped even though the ingestor created the table a few seconds later.

Start-up contract

• Wait in-process, bounded. The server checks the schema, then retries with a back-off of 250 ms, growing ×1.5 up to 2 s. The deadline is 120 s after the first check. Nothing about this is configurable.
• Retry only the migration in progress. dbschema.AssertReady returns a typed *dbschema.NotReadyError. Only these are retried (Transient()):
  • route_mask_changes does not exist yet and its marker is not recorded;
  • additionally, transmissions.route_mask is missing, because the same Apply adds it just before the table;
  • a probe failed with SQLite BUSY/LOCKED. This is classified by the result code (Code() & 0xff ∈ {5, 6}), never by error text.
• Fail at once on everything else, with the same log.Fatalf as before:
  • any other missing table or column, including missing items alongside the change log;
  • a change log that exists but is malformed (a column, AUTOINCREMENT, or the index not exactly (transmission_id));
  • the marker route_mask_changes_v1 recorded while the table is gone, since the ingestor never recreates it;
  • the column missing while the table exists;
  • permission, read-only, I/O, corruption, not-a-database, a closed handle, and any unknown error.
• One snapshot. AssertReady runs every probe inside one read transaction. A migration committing mid-check is therefore seen either as not started or as done, never half-way (which would be misread as a permanent defect).
• Atomic change-log migration. The ingestor creates the table, its index and its marker in one transaction.
• Readiness stays red while waiting. The API listener (including /api/healthz) and the packet store load only start after the schema is verified. The optional pprof listener (ENABLE_PPROF, off by default) is unchanged. A process test asserts that the port refuses connections at the first wait log and at a status line.
• Flags refreshed after the wait. The server re-reads its optional-column flags after a successful wait. On a pre-Record every observed route per transmission and show mixed ADVERTs in Relay Airtime Share #93 database, the start-up load therefore reads route_mask and queues NULL rows for the backfill refresh.
• Shutdown. SIGINT or SIGTERM during the wait (including during the final check) ends it at once. The server logs shutdown requested while waiting for the ingestor's schema; exiting and exits 0; this is never reported as a timeout.
• After the deadline. The server exits 1 with schema still not ready after … (N attempts): …. By then it has run far longer than startsecs=2, so supervisord counts it as a started process and restarts it without using up startretries. A slow migration therefore gives repeated 120 s waits, never FATAL.
• Logging is bounded. Logged: once at the first miss (naming the missing items), a status line at most every 10 s, one line on success after waiting, one on timeout. A normal start logs nothing extra.
• The server still never writes. It stays mode=ro, and the ingestor remains the only schema writer. supervisord, the Dockerfile, the entrypoint, workflows and deploy scripts are unchanged.

Measured on the real image and supervisord (local Docker, isolated volume)

Image corescope:pr93-local-final, built from 4db9716f with the repository's own Dockerfile. It runs the unchanged supervisord config and entrypoint, with DISABLE_CADDY=true and --network none, on a copy of the E2E fixture reset to the pre-#93 state (no route_mask column, no change log, no markers). "Hold" means a separate process holds BEGIN IMMEDIATE on the database from before the container starts, so the ingestor's Apply cannot run. Readiness is /api/healthz ready:true, polled once per second.

Scenario	Server ready	Server exits	FATAL	Server log
Normal start ×20	2–3 s	0	0	schema ready after 253–255ms (2 attempts) in all 20
Hold 10 s	11 s	0	0	schema ready after 9.239s (9 attempts)
Hold 80 s (longer than the old 64.6 s FATAL)	83 s	0	0	7 status lines; schema ready after 1m21.552s (45 attempts)
Hold 115 s (just under the deadline)	118 s	0	0	11 status lines; schema ready after 1m55.692s (62 attempts)
Hold 125 s (just over the deadline)	126 s	1 (exit 1 at the deadline)	0	one deadline error, then supervisord restarts it; schema ready after 3.329s (6 attempts)
Hold 250 s	252 s	2 (both at the deadline)	0	two deadline errors, then schema ready after 7.255s (8 attempts)
docker stop during a 60 s hold	—	0	0	shutdown requested while waiting…; stopped: corescope-server (exit status 0) in 1.04 s

• On a normal start, the server loses the race to the ingestor every time: it logs one wait line and is ready about 0.25 s later. That is the race that used to exit.
• The ingestor under a held lock exits with database is locked after its own 5 s busy timeout, and supervisord restarts it. It ran past startsecs each time, so it never reached FATAL either, even in the 250 s run.
• Every run, including the normal ones:
  • route_mask_changes was created once;
  • the MQTT start-up buffer reported write path ready; draining backlog (0 dropped);
  • no panic occurred;
  • listening on appeared exactly once per successful server process.
• For comparison, the old head under a 70 s hold gave gave up: corescope-server entered FATAL state at 64.6 s in the isolated staging gate.

Limitations of the wait

• The server's busy timeout is 0. The modernc driver ignores the server DSN's _busy_timeout=5000 (_journal_mode=WAL likewise; WAL persists in the file from the ingestor). A probe that meets a writer's lock therefore reports BUSY at once, and the wait retries it. This PR does not change the DSN.
• Pre-existing signal window. SIGTERM between the end of the wait and the server's later signal.Notify (store load, a few seconds) takes Go's default action. That window is unchanged by this PR, apart from a sub-millisecond window around the flag re-read. SIGTERM during the wait itself is handled.
• No process test observes the flag-refresh wiring in main. It is covered by a unit test on waitForDBSchema, which runs the real start-up load and queue.
• Retry ≠ repair. A database where the ingestor never runs, or fails its Apply for good, gives a server that restarts every 120 s with a clear log. It does not give FATAL. The ingestor's own logs and startretries still surface the ingestor failure.

Independent review

Fresh reviewers, none of whom wrote the code, reviewed each step. Every blocker and should-fix they reported is fixed test-first in its own commit.

• Review 1:

  • blocker: backfilled masks never reached a running server;
  • should-fix: an observation could become visible before its bit;
  • fixed: the poller refresh and the OR ordering.

• Review 2, blocker: the background chunk loader queued an id before its transmission was visible. Fixed: queued in the merge step.

• Review 3: the route bit and the observation write are now atomic. Approved; its test gap (the observation write failing after the OR) was closed.

• Review 4, design B, no blocker. Its should-fix items were fixed:

  • parked changes could stay forever after a partial background load;
  • a failed watermark read skipped changes;
  • the watermark placement and the settled guard were untested.

  Its nits were also closed: the no-change guard and the orphan-prune batch bound are now tested, and the table DDL is shared with the tests.

• Review 5 (the review-4 fixes): no blocker and no should-fix. Each of those fixes was killed by its own test under mutation, and the new tests passed 50 runs under -race. Both of its nits were applied: the test hook's comment now says it runs under s.mu, and the orphan prune guards against a batch size ≤ 0.

• Remaining pre-existing gap: none for same-row upserts; that live-view limitation is fixed by the change log.

• Review 6 (the start-up wait):

  • blocker: after a wait on a pre-Record every observed route per transmission and show mixed ADVERTs in Relay Airtime Share #93 database, the server kept route_mask switched off for its start-up load;
  • should-fix: a recorded marker without its table was retried;
  • should-fix: a shutdown during the final check was lost.

  Also fixed: probe errors are now structured, not text; the index must be exactly (transmission_id); and the log wording.

• Review 7 (the review-6 fixes): all the fixes were confirmed.

  • One should-fix: AssertReady read the schema across several snapshots, so an Apply committing mid-check could be misread as permanent. The reviewer reproduced this 71 and 104 times in 1,500 runs. It is fixed with one read transaction and a deterministic test that commits between two probes.
  • Its nits are documented under "Limitations of the wait".

  Mutation: all 18 mutants of the wait and the classification are killed.

Mutations, race and performance

• Mutations: 49 of 49 mutants killed, every kill naming a failing test (the runners stop on a build failure instead of counting it). The 18 change-log mutants cover:

  • no change row on a same-row upsert;
  • the change row written outside the transaction;
  • the cursor jumping past rows not yet applied, or ids that can be reused;
  • a change for a not-yet-loaded transmission dropped, or a missing transmission re-created;
  • no cache invalidation;
  • retention or the orphan prune deleting rows of existing transmissions, or the orphan prune without a batch bound;
  • the watermark taken after Load or LoadChunked, lazily, or reset to the end after a read error;
  • the settled guard forced true, or not set by RunStartupLoad;
  • a parked change never applied on publish;
  • a change row logged when the mask did not change.

  The earlier 31 mutants cover:

  • the SQL OR replaced, skipped or clearing bits;
  • the first insert without a bit;
  • wrong flood/direct grouping;
  • mixed classified as flood or zero-hop, or counted twice;
  • a false complete, from a marker or from masks the server has not read;
  • no re-run after completion;
  • a mask dropped in each load path;
  • NULL merged as known;
  • the refresh being a no-op, not queued, not invalidating the cache, treating NULL as resolved, or missing from the poller;
  • the background chunk loader queueing before its transaction is visible, or not queueing at all;
  • LoadChunked not queueing, and queueing without the column;
  • the backfill ignoring headers or setting bits for invalid routes;
  • the index build attributed to the MQTT handler;
  • the mixed label equal to the flood label;
  • Payload Type Mix split by mask;
  • the frontend deriving mixed from the label, or dropping the row identity.

• Race: go test -race -count=10 passes on the route-mask, change-log, backfill, prune, relay, healthz, cache, chunked-load and poller tests in cmd/ingestor and cmd/server. The new focused tests pass 20 times in a row, and no test processes are left behind.

• Change log, measured:

  Measurement	Result
  CREATE TABLE + index, cold, 4.95 GB staging copy (demo host, --cpus 2)	6.6–6.8 ms + 3.2–4.0 ms, DB growth 0 bytes; OpenStore including Apply 39–48 ms
  Ingest, normal (3 alternating local rounds vs b674e1e2)	p50 151–153 µs vs 150–152 µs; writer hold p50 unchanged at ≈0.15 ms
  Ingest, new bit on a new row / via same-row upsert	p50 +~35 µs each (the one extra INSERT)
  Rows written	one per actual mask change (90 in the benchmark, one extra 8 KB page)
  Poll tick with 0 / 1 / 100 / 1,000 change rows (100,000 transmissions in memory)	p50 6 µs / 8 µs / 90 µs / 739 µs; rows read = rows logged; parked max 0
  Orphan prune, realistic (22 rows + 1,000 orphans, cold)	15.8 ms writer hold, 18 ms total
  Orphan prune, theoretical maximum (1.49 M rows + 10,000 orphans, cold)	1,501 bounded transactions: writer hold p50 2.8 ms, p99 36.6 ms, max 53.9 ms (21.6 s in total, lock released between batches). The first version held the lock for up to 22.3 s in one batch and was rewritten to examine a bounded id range per transaction.
  Start-up watermark read at 1.49 M rows, cold	4.4 ms; a poll after it reads 0 rows in 0.08 ms

• Ingest hot path: the only extra work is one guarded UPDATE … SET route_mask = route_mask | ?.

  • It runs only when an observation brings a bit the transmission does not have yet: 21 of 496,798 transmissions (~0.004 %) on the staging copy.
  • The mask is read in the existing lookup by hash.

• Ingest benchmark: 15,000 tx × 21 observers, 1 in 500 mixed, measured before the OR moved ahead of the observation insert. The statements are the same.

  Run	Normal p50 (base → branch)	Mixed-insert p50
  Local, 5 alternating rounds (the Mac was loaded by other work)	150/171/153/168/149 µs → 153/202/202/168/155 µs	+40–90 µs (the extra UPDATE)
  Isolated demo host, container --cpus 2, 2 alternating rounds	3,179/3,121 µs → 3,158/3,132 µs (p99 8,505/8,045 → 8,449/8,605 µs; total 1,081/1,063 s → 1,076/1,077 s)	3,048/3,141 µs → 4,284/4,208 µs (+~1.1 ms, one extra fsync'd UPDATE)

• Server poller refresh: at 500,000 queued transmissions, warm, locally:

  Case	Cost per tick
  Queue empty (the steady state after the backfill)	9 ns
  Waiting for the backfill (one read of 500 ids)	0.5 ms
  Catching up (2,000 ids, merge under s.mu)	2.4–3.3 ms
  Sorting the whole queue once	≈ 1 ms

• Server load: StoreTx stays 320 bytes on both. Load time (3.6–4.0 s) and heap after load (206–213 MB) are equal within noise.

• Relay Airtime Share: cold compute is 7–15 ms (base) vs 17–21 ms (branch) on 15,000 packets; warm results are cached.

Known limitations and follow-ups (not in this PR)

• Recovery after a failed watermark read. The server then reads the change log from id 0. Until the start-up load is over, every change for a transmission not yet in memory stays parked, and each tick walks the parked map under s.mu. With the realistic 22-row log this costs nothing; at the theoretical 1.49 M rows it would be expensive for a while. It only happens after a failed watermark read.
• Deploy order. The server requires route_mask_changes (AssertReady). When both start together, it now waits for the ingestor's migration (see "Start-up schema wait"); the ingestor must still be deployed with or before the server. An older ingestor after a rollback writes no change rows, so a running server misses same-row bits again until its next restart; the database mask stays correct.
• Failed Commit, not blocking. Rollback is deferred right after a successful Begin, and a failed Commit is returned, counted once in WriteErrors and logged. On a failed Commit, database/sql returns the connection to the pool, and modernc.org/sqlite v1.34.5 only checks it for an interrupt. A transaction could therefore stay open only if COMMIT failed without SQLite rolling back. In WAL mode, with the write lock already held and the only foreign key checked immediately, that needs an I/O or disk-full error. It could not be reproduced, and every other ingestor transaction has the same exposure. A possible follow-up is an explicit ROLLBACK after a failed commit.
• Time-on-Air depends on ingest order. The Time-on-Air / frame-length estimate still uses the stored frame, so it depends on which variant was stored. The OpenAPI text notes this. Needs its own follow-up.
• Other route_type consumers. Other consumers of route_type (for example packet list filters and other analytics) keep first-inserted semantics. Whether any of them should move to route_mask needs a separate decision.
• Rollback window. While an older ingestor runs, bits it would have ORed into rows that already have a mask are not recorded. Rolling forward again backfills only NULL rows.
• Status after a crash. After a hard kill during the backfill, the status reads backfilling until the ingestor restarts and resumes. After a transient failure, it reads pending until the next start. It never reads complete early.
• Mixed colour. --status-purple is close to the palette violet used positionally by other rows. The labels distinguish them.
• Pre-existing server write. cmd/server/routes.go still has an INSERT INTO transmissions that fails on the read-only connection. This PR does not add it or change it.

Preflight overrides

• check-async-migrations: ALTER TABLE transmissions ADD COLUMN route_mask INTEGER is a nullable column without a default, so it is metadata-only in SQLite: no row rewrite, 8–15 ms on the 4.95 GB staging copy. It is annotated PREFLIGHT: async=false with that reason. The partial index runs in RunAsyncMigration with an async=true annotation.

Scope and deployment

• This PR has not been deployed to staging or production. It makes no claim about behaviour there; a separate staging review is expected.
• The start-up wait was measured only in local Docker, on an isolated volume, never against live staging. A staging candidate built before 4db9716f does not contain the wait and must be rebuilt from this head.
• The staging database was only read, through the SQLite backup API, into a copy. The migration measurement ran on that copy on the isolated host.

🤖 Generated with Claude Code