perf(nodes): serve /api/nodes?region= from an ingestor-built table (#2101)

[A13xB0]

Fixes #2101.

/api/nodes?region=X decided membership with an inline public_key IN (SELECT DISTINCT from_pubkey FROM transmissions ⋈ observations ⋈ observers …) subquery. It was uncached, unbounded in time, and evaluated twice per request (COUNT(*) and the page). fetchAllNodes() pages at 500, so each Nodes view ran it several times. On the ScotMesh instance (1.4 GB database, 2.48M observation ids, 2 vCPU) one evaluation took 3–20 s depending on the page cache, and one open tab kept the reader pool and both cores busy. #2101 reports the same at larger scale.

The answer is small and changes slowly: 7,333 (node, observer) pairs on that database. This PR derives it once in the ingestor and keeps it, the way neighbor_edges is (#1287, #1339). The server then reads a table the size of nodes × observers instead of scanning the advert history.

This supersedes #2102, which cached the subquery's result in the server. A 28 h soak of #2102 showed it only moved the scan: full rebuilds of every distinct region set, 20–30 s each, landed together every 30 minutes, and a new region combination waited behind them (80 s observed). Nothing of that design is in this diff.

What changes

• internal/dbschema
  • Creates node_advert_observers(public_key, observer_idx, last_seen, last_obs_id): one row per observer that has heard at least one of a node's ADVERTs, WITHOUT ROWID, with an observer_idx index.
  • AssertReady requires the table (bug(db): server caches "scope_name column not present" on startup race with ingestor migration — /api/scope-stats 500s until server restart #1321).
  • NodeAdvertObserversBackfill names the backfill's _async_migrations row, which both binaries read.
• cmd/ingestor/node_advert_observers.go is the table's only writer.
  • Observations are folded in by observations.id, in bounded steps of 50k ids. Each step reads a rowid range, with CROSS JOIN pinning the plan, then upserts in one WriterTx.
  • The first full build runs through RunAsyncMigration, so boot is not blocked and ingest interleaves between steps. After it, a 30 s ticker folds whatever has arrived.
  • The watermark lives in memory and resumes from MAX(last_obs_id) after a restart, so there is no state table: the table is the persistence, as for neighbor_edges.
  • PruneOldPackets drops a pair at the same first_seen cutoff that removes its newest advert. Steps and that prune share a mutex, so a step cannot write back a pair from transmissions a concurrent prune has just removed.
  • ADVERT rows whose from_pubkey is still NULL (legacy rows BackfillFromPubkey has not reached) are recorded under the pubkey that backfill will write, extractPubkeyFromAdvertJSON(decoded_json).
• cmd/server
  • GetNodes uses public_key IN (SELECT nao.public_key FROM node_advert_observers nao JOIN observers obs ON obs.rowid = nao.observer_idx WHERE UPPER(TRIM(obs.iata)) IN (…)) once the backfill row is done.
  • Readiness is looked up at most every 30 s until then, by one request at a time, and then latched, so a server that started first switches over without a restart.
  • Until then, with an ingestor that predates this, or on a v2 schema, the legacy query runs unchanged.

Semantics. A pair exists while at least one ADVERT with a non-empty from_pubkey, still in the database, was observed by that observer. That is the legacy subquery's condition. The region tag is joined from observers at query time, as before, so an observer that changes region is followed immediately. The one change is freshness: a node newly heard in a region appears within one tick (30 s) instead of on the next request. #1879 (attributing by home region) is neither addressed nor made harder: it would change what the pairs table records.

Perf (rule 0).

• Per request, the region filter is an index lookup into node_advert_observers joined to observers. TestRegionFilterPlanAvoidsHistory asserts the plan never touches observations or transmissions.
• Ingest cost is one bounded step per 30 s: a rowid range read plus an upsert of the pairs it found. Steady-state ticks measured 37–40 ms.
• The backfill is linear in observations, in 50k-id steps (worst step 0.34 s read on the 2.48M-id database), with a 50 ms yield between steps.

PREFLIGHT-MIGRATION-SCALE: <30s N=2.48M observation ids — backfill 12.0 s on the production 2-vCPU instance, 3.6 s on a snapshot of it

Measurements

These are from a VACUUM INTO snapshot of the production database, run locally against the real OpenStore, the builder and PruneOldPackets:

backfill	3.6 s, 7,333 pairs, 2.48M observation ids
every region (11) and the all-regions set	identical to the legacy query
after PruneOldPackets(3) (251,559 transmissions, 3,805 pairs)	identical again
region query	0.3–3 ms from the table, 38–677 ms legacy on the same (warm, faster) machine

On the production instance, running this change (built from 342e7c79; the final commit differs only in moving the backfill's name into dbschema and adding tests):

backfill	12.0 s in the background, done; the server switched 7 s later
/api/nodes?limit=500&region=… warm	26–61 ms; unfiltered on the same box 31–57 ms
live parity, API vs legacy SQL	14 of 14 region sets identical: 11 single regions, a 7-region set, a 55-code set and all regions
ticks	tick: 8 pair updates in 37ms; ingestor CPU 16 s over its first 11 minutes

Operator notes

• First start after upgrading runs the backfill in the background. Look for [node-advert-observers] backfill: … pair updates in …, "node_advert_observers_backfill_v1" done, then [nodes] region filter now reads node_advert_observers (#2101) from the server. Until then the region filter behaves exactly as before, slow included.
• The server requires the new table (AssertReady). In the single-container image supervisord restarts the server if it wins the race against the ingestor's dbschema.Apply. Split deployments must upgrade the ingestor with or before the server, as for every dbschema addition.
• Downgrading is safe. Older binaries ignore the table. On the next upgrade the builder resumes from its watermark, and the startup PruneOldPackets clears pairs left by prunes the older version made.

Worth a decision

• Customizer (rule 8). NodeAdvertObserversInterval (30 s), the 50k step size and the 50 ms yield are constants. The interval is the only one an operator might want to tune.
• No request context on GetNodes (bug: Regional /api/nodes queries exhaust the SQLite connection pool #2101 also notes this). The region filter no longer scans history, so I have not threaded the request context through; the builder's steps take a context and stop on shutdown.

Validation

• New tests, 22 in all:
  • cmd/ingestor, 14 tests:
    • parity with the legacy query (padded and lower-case tags, multi-region sets, non-ADVERTs, empty from_pubkey, observations without an observer, untagged observers);
    • NULL from_pubkey rows matching BackfillFromPubkey;
    • steps advancing through batches with no adverts;
    • newest last_seen kept across steps;
    • resume from the table;
    • prune parity;
    • the backfill as an async migration, then ticks, then no re-run on restart;
    • a failed backfill leaving the builder not ready;
    • a failed write leaving the watermark in place and being retried;
    • cancellation between steps;
    • ticks skipped until ready and surviving errors;
    • a prune error surfaced;
    • builder, prune and ingest running concurrently.
  • cmd/server, 7 tests:
    • parity from the table, with a pair the history lacks proving the table is read;
    • legacy query while the backfill row is missing, pending or failed;
    • legacy query without an _async_migrations table;
    • switching without a restart, then latched;
    • v2 schema stays legacy;
    • readiness re-checked at most once per interval under concurrency;
    • query plan avoids the history.
  • internal/dbschema, 1 test: the table and index are created by Apply and required by AssertReady.
• Mutation checks. Each of these, applied to the implementation, makes a named test fail (12 of 12 caught):
  • the watermark advancing only on a match;
  • prune not dropping pairs;
  • the upsert overwriting last_seen;
  • NULL from_pubkey skipped;
  • ready after a failed backfill;
  • non-ADVERTs counted;
  • a failed write advancing the watermark;
  • the server ignoring the backfill status;
  • no latch;
  • re-checking on every request;
  • using the table on v2;
  • dropping the region join.
• Suites. go test -race ./... passes in cmd/server, cmd/ingestor, internal/dbschema and cmd/migrate; the new ingestor tests also pass 20× under -race. gofmt and go vet are clean.
• Static analysis on changed lines:
  • staticcheck -checks all: nothing;
  • SonarQube 26.9 (Go, three modules): 0 issues and 0 security hotspots on changed lines; 0 bugs, vulnerabilities, code smells or duplication in the new files;
  • strict golangci-lint: findings addressed except ones that conflict with this repo's conventions (noctx, unchecked deferred Close, t.Parallel).
• Browser. Headless Chromium (Playwright) against the deployed instance with the region filter set as a visitor sets it:
  • the Nodes page shows 320 rows for EDI and 321 for EDI,GLA, matching the API and the legacy query, with no console or page errors;
  • the first-ever EDI,GLA load took 4.4 s for the whole page, against 19 s with perf(nodes): cache region membership for /api/nodes?region= (#2101) #2102;
  • the Map page does not send region to /api/nodes, so it is unaffected.
• Rebase: on current master (093e320c), one commit.

🤖 Generated with Claude Code