perf(db): pin GetChannels and CountFloodAdvertsForNode to their indexes (issue #100)

[dborup]

Summary

Pins GetChannels to idx_tx_channel_hash and CountFloodAdvertsForNode to idx_transmissions_from_pubkey. Both then get the plan SQLite picks when planner statistics exist, but without running ANALYZE. Decision issue: issue #100. This replaces the ANALYZE port in PR #106, which is parked.

Server only (cmd/server, read-only). No changes to cmd/ingestor, the schema or the workflows.

Problem, and why pinning instead of ANALYZE

• Without sqlite_stat1 (production: neither binary runs ANALYZE), SQLite plans GetChannels on idx_transmissions_payload_type, with a temp B-tree for GROUP BY. When the payload_type index is the newer of the two, the correlated sample subquery also goes through it. It then visits every GRP_TXT row once per channel.
• With statistics it picks idx_tx_channel_hash for the outer scan (in channel_hash order, so no GROUP BY B-tree) and for the subquery (channel_hash=?).
• Staging measurements from PR perf(db): port upstream bounded ANALYZE planner statistics (issue #100) #106 (DB copy, 499k tx, 6.6M obs; earlier evidence, not re-measured here):
  • GetChannels (all): cold 6.76 → 4.42 s, warm 289 → 231 ms.
  • AAR/CPH: cold ~14.1 → ~12.1 s, warm ~1.0 → ~0.8 s.
  • GetEncryptedChannels: plan unchanged.
  • The review of PR perf(db): port upstream bounded ANALYZE planner statistics (issue #100) #106 found no plan change in 79 other static server SELECTs.
• Why not ANALYZE: it held the write lock for ~65 s cold on 5 GB, and again on every restart. Pinning gives the same two plans with no write, and leaves every other query exactly as it is today.

Mechanism

GetChannels: INDEXED BY idx_tx_channel_hash plus a fallback

• channelsSQL(v3, regionPlaceholder, pinned) builds all three variants (no region, v3 region, v2 region). pinned adds INDEXED BY idx_tx_channel_hash to both transmissions references: the outer t and the subquery's t2.
• Why not a unary + (the feat(nodes): Recent Adverts split into flood / zero-hop / mixed with per-class counts #97 technique): idx_tx_channel_hash is partial (WHERE payload_type = 5). SQLite uses a partial index only if the query's WHERE clause contains that predicate, and +t.payload_type = 5 no longer counts. Measured on SQLite 3.46.0 (modernc): the plan becomes SCAN t + SCAN t2 USING INDEX idx_transmissions_first_seen, i.e. full scans (mutation M7 below).
• Fallback: INDEXED BY fails at prepare time, before any row is read, when the index is missing or unusable. GetChannels then runs the unpinned query (the pre-PR SQL). The first fallback per DB handle is logged, and channelsPinFallbacks counts them.
• Why not a schema flag: schemaFlag is monotonic ("true is permanent"), which fits ADD-only columns. An index can be dropped, so a flag alone would still need the error fallback, while the error fallback alone covers missing, dropped and differently-defined indexes. It is the simpler of the two.
• One textual change: in the no-region query 'enc_%%' is now passed through Sprintf like the region variants, so SQLite receives 'enc_%'. As a LIKE pattern, %% and % match the same strings; the equivalence test covers it.

CountFloodAdvertsForNode: +payload_type

• The query is now the const floodAdvertCountSQL with from_pubkey = ? AND +payload_type = ?, the same technique as nodeAdvertScanSQL (feat(nodes): Recent Adverts split into flood / zero-hop / mixed with per-class counts #97). idx_transmissions_from_pubkey is not partial, so the + is safe.
• Without statistics, SQLite takes idx_transmissions_payload_type whenever that index is newer than the from_pubkey index, and then visits every ADVERT in the database.
• For an INTEGER column, +payload_type = ? matches exactly the rows payload_type = ? does.

GetEncryptedChannels: unchanged

LIKE 'enc_%' cannot use the partial index, and even with ANALYZE SQLite keeps idx_transmissions_payload_type on the synthetic data (same finding as the PR #106 review). Nothing to pin.

Plan tests (channels_index_pin_test.go)

pinTestDB builds the production indexes (ingestor + internal/dbschema) in two creation orders: the ingestor's order (payload_type first) and payloadTypeLast. setupTestDB cannot be used for this: it has a non-partial channel_hash index and no payload_type index.

Without statistics (TestChannelsSQL_PlanWithoutStats): 12 cases (v3/v2 × both orders × all / 1 region / 2 regions). Assert: outer SEARCH t USING INDEX idx_tx_channel_hash, subquery SEARCH t2 USING INDEX idx_tx_channel_hash (channel_hash=?), no payload_type index, no TEMP B-TREE FOR GROUP BY.

Master is red in all 12 (run against master's SQL verbatim):

before, ingestor order:  SEARCH t USING INDEX idx_transmissions_payload_type (payload_type=?) | USE TEMP B-TREE FOR GROUP BY
                         | CORRELATED SCALAR SUBQUERY 1 | SEARCH t2 USING INDEX idx_tx_channel_hash (channel_hash=?) | ...
before, payloadTypeLast: SEARCH t USING INDEX idx_transmissions_payload_type (payload_type=?) | USE TEMP B-TREE FOR GROUP BY
                         | CORRELATED SCALAR SUBQUERY 1 | SEARCH t2 USING INDEX idx_transmissions_payload_type (payload_type=?) | ...
after, both orders:      SEARCH t USING INDEX idx_tx_channel_hash (channel_hash>?)
                         | CORRELATED SCALAR SUBQUERY 1 | SEARCH t2 USING INDEX idx_tx_channel_hash (channel_hash=?)
                         | USE TEMP B-TREE FOR ORDER BY | USE TEMP B-TREE FOR ORDER BY

With statistics (TestChannelsSQL_PlanWithStats): 6,000 skewed transmissions (55 % GRP_TXT, ~70 % of them enc_*, 7 plaintext channels), then ANALYZE. In all 12 cases the pinned plan is identical to the plan SQLite chooses for the unpinned query, and it is the idx_tx_channel_hash plan. The pin does not make the plan worse once statistics exist. (Master is green here too, as expected: with statistics it already chooses this plan.)

TestFloodAdvertCountSQL_Plan: both orders × with/without statistics give exactly SEARCH transmissions USING INDEX idx_transmissions_from_pubkey (from_pubkey=?), with no sort. Master is red in payloadTypeLast without statistics (SEARCH transmissions USING INDEX idx_transmissions_payload_type (payload_type=?)).

Result equivalence

• TestGetChannels_PinnedMatchesUnpinned, run for v3 and v2 × both orders.
  • Synthetic data covers mixed regions (AAR, " cph ", CPH, NULL, empty and lowercase IATA), enc_*/ENC_*/encX* hashes, NULL channel_hash and NULL decoded_json, undecodable JSON and JSON without text, channel_hash on other payload types, first_seen ties within and across channels, three timestamp formats, transmissions without observations, and observations of unknown or NULL observers.
  • Raw rows: all columns, order included, of channelsSQL(pinned) equal those of channelsSQL(unpinned) for all/1/2 regions (≥5 channel rows each).
  • GetChannels output: reflect.DeepEqual for 7 region inputs ("", AAR, cph, AAR,CPH, " aar , osl ", sto, ZZZ), first with the index present (0 fallbacks), then after DROP INDEX idx_tx_channel_hash (7 fallbacks, same output).
• TestFloodAdvertCountSQL_MatchesUnpinned: identical rows, with and without the +, for every node in the random data.
• TestGetChannels_MissingIndexFallsBack: setupTestDB has no idx_tx_channel_hash. /api/channels, ?region=SJC and ?includeEncrypted=true answer 200 with #test, and the fallback counter shows the pinned query was tried first.

Mutations (7, all killed)

mutation	killed by
M1 pin removed (hint = "")	PlanWithoutStats, PinnedMatchesUnpinned (fallback counter), MissingIndexFallsBack
M2 wrong index name (idx_transmissions_channel_hash)	PlanWithoutStats, PlanWithStats, PinnedMatchesUnpinned, MissingIndexFallsBack
M3 pin only in the outer query, not in the subquery	PlanWithoutStats (payloadTypeLast: subquery on payload_type)
M4 fallback removed	PinnedMatchesUnpinned, MissingIndexFallsBack (500)
M5 + on the wrong column (+from_pubkey = ? AND payload_type = ?)	FloodAdvertCountSQL_Plan, FloodAdvertCountSQL_MatchesUnpinned
M6 GetChannels runs the unpinned SQL first	PinnedMatchesUnpinned, MissingIndexFallsBack
M7 option A: +payload_type = 5 instead of INDEXED BY	PlanWithoutStats (SCAN t + SCAN t2 USING INDEX idx_transmissions_first_seen), PinnedMatchesUnpinned, MissingIndexFallsBack

Local performance (synthetic, not staging)

Hardware: KVM guest, 4 vCPU Intel Xeon @ 2.10 GHz, 15 GB RAM, virtio disk (ext4), Linux 6.18. SQLite 3.46.0 (modernc.org/sqlite v1.34.5). Opened with the server's DSN (mode=ro, default 2 MB page cache).

Method:

• Cold: echo 3 > /proc/sys/vm/drop_caches inside the guest plus a new connection, median of 3. The host's cache is outside my control.
• Warm: same connection after one warm-up, median of 7.
• All rows are read in both cases.

Synthetic DB: 1.25 GB, no sqlite_stat1, indexes in the ingestor's order.

• 500,000 tx and 6,217,531 obs (~12.4/tx).
• 42 % GRP_TXT (209,929), of which 62 % are enc_* over 60 hashes and 79,842 are plaintext over 12 channels.
• 30 % ADVERT over 2,500 nodes (Zipf-like; the heaviest node has 7,752 adverts).
• 60 observers in 6 IATA regions plus NULL. 120 days.

query	before	after	plan before → after
GetChannels all, cold	541 ms (repeat runs: 678 / 941)	1,295 ms (1,064 / 1,089)	payload_type + GROUP BY B-tree → idx_tx_channel_hash
GetChannels all, warm	486 ms	425 ms
GetChannels AAR, cold	2,628 ms (2,132 / 2,508)	2,883 ms (2,886 / 2,577)	same
GetChannels AAR, warm	1,050 ms	1,086 ms
GetChannels CPH, cold / warm	2,275 / 1,188 ms	2,641 / 1,079 ms	same
GetChannels AAR,CPH, cold / warm	2,323 / 1,331 ms	2,444 / 1,154 ms	same
GetChannels all, if the payload_type index were newer (simulated master plan)	3,023 / 2,627 ms	1,295 / 425 ms	subquery on payload_type → idx_tx_channel_hash
CountFlood heavy node, cold / warm	402 / 24.5 ms	434 / 20.7 ms	from_pubkey → from_pubkey
CountFlood median node, cold / warm	3.6 / 0.1 ms	3.2 / 0.0 ms	from_pubkey → from_pubkey
CountFlood, if the payload_type index were newer (simulated master plan)	2,455 / 192 ms	434 / 21 ms	payload_type → from_pubkey

How to read this:

• Warm: GetChannels without region is ~13 % faster; with region between 13 % faster and 3 % slower.
• Cold, locally: GetChannels without region is slower, 0.54–0.94 s → 1.06–1.30 s. With region, the difference is within the noise (±25 % between repeats).
  • This matches PR perf(db): port upstream bounded ANALYZE planner statistics (issue #100) #106's local measurement of the same plan: 5–25 % slower cold on virtio, while reading ~88 % fewer bytes.
  • My assumption, not verified (blockdev --setra is not permitted here): channel_hash order gives random row lookups where rowid order gets sequential readahead.
  • The staging copy measured the opposite for the same plan (6.76 → 4.42 s). Staging decides; I make no cold-speedup claim from local data.
• What the pin reliably buys: it removes the order-dependent worst cases, 2–9× in the simulated rows above. CountFloodAdvertsForNode is unchanged when the from_pubkey index already wins (ingestor order).
• Complexity: GetChannels visits the same GRP_TXT rows as before, now in index order with no GROUP BY sort, plus one channel_hash=? lookup per channel. No new work in any hot path. Results stay cached for 60 s as before.

Staging test plan

1. Before deploy, read-only on the staging DB: confirm sqlite_stat1 is absent and idx_tx_channel_hash exists (SELECT sql FROM sqlite_master WHERE name='idx_tx_channel_hash'). Record EXPLAIN QUERY PLAN for the channel query with and without region (expect payload_type + GROUP BY B-tree).
2. Measure /api/channels, ?region=AAR, ?region=CPH and ?region=AAR,CPH cold (first call after restart, or a dropped cache on the DB copy) and warm (the second call is served from the 60 s cache; measure after it expires or on the DB copy).
3. Deploy. The log must not contain GetChannels: idx_tx_channel_hash not usable.
4. Repeat step 2 and compare with the PR perf(db): port upstream bounded ANALYZE planner statistics (issue #100) #106 staging figures for the statistics plan (6.76 → 4.42 s cold all; AAR/CPH ~14.1 → ~12.1 s). If cold does not improve on staging either, this PR is not worth merging for GetChannels; the flood pin stands on its own.
5. Node detail /api/nodes/{pubkey} for a heavy advertiser: flood_advert_count_7d unchanged. The query only runs when the upstream #2073 scan does not supply it.
6. Rollback: revert the commit. No data or schema state to undo.

Checks

• go vet ./... in cmd/server: clean.
• gofmt -l on touched files: clean (some untouched files on master are not gofmt-clean; unchanged).
• go test -timeout 20m -race -count=1 ./... in cmd/server: ok (766 s). Includes the read-only invariant; its 4 tests (readonly_invariant_test.go) also pass on their own.
• No new map[string]interface{} (the existing return type of GetChannels is unchanged).
• Not run: browser validation (no UI change); cmd/ingestor tests (untouched).

🤖 Generated with Claude Code

https://claude.ai/code/session_01UxGNmBfShy5AxAovsFnx7V

---

Generated by Claude Code