Skip to content

feat(nodes): Recent Adverts split into flood / zero-hop / mixed with per-class counts - #97

Merged
dborup merged 6 commits into
masterfrom
codex/2073-recent-adverts-by-route
Sep 26, 2026
Merged

dborup merged 6 commits into
masterfrom
codex/2073-recent-adverts-by-route

Conversation

@dborup

@dborup dborup commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

What and why

Port/extension of upstream Kpa-clawbot/CoreScope#2073 (open, unimplemented upstream), including the rename from upstream Kpa-clawbot/CoreScope#2071.

The node page's advert list mixed every advert together, so a node that sends many cheap zero-hop adverts buried its few flood adverts — the ones that cost mesh-wide airtime. This PR:

  • renames "Recent Packets" to Recent Adverts on the full node page and the side panel, with upstream #2071's tooltip (the list is from_pubkey, which only ADVERTs carry);
  • adds tabs All / Flood / Zero-hop / Mixed, a count row (24h and 7d), a route badge per advert in All, and a ?adverts= deep link;
  • adds recentAdvertsByRoute, advertCounts and a per-row route_class to GET /api/nodes/{pubkey} — only with include=advertRoutes (opt-in, see below).

Opt-in: include=advertRoutes

The breakdown scans all of a node's ADVERT rows, so it is computed only when asked for:

  • GET /api/nodes/{pubkey}?include=advertRoutes (comma-separated list, may repeat; unknown values ignored) adds the three fields.
  • Without it the response is exactly master's: same top-level fields (node, recentAdverts), no route_class, no new fields. Neither the privacy lookup, the cache nor the scan runs (a test counts cache lookups and scans: 0). Measured response bytes are identical to master (table below).
  • Who sends it: only fetchNodeDetail in public/nodes.js, which serves both the full node page and the side panel. It builds the URL itself, so a missing node-adverts.js cannot break the fetch or its 404 state.
  • Who does not: packets.js (sender/ctrl/anon-req lookups), live.js, channels.js, route-view.js (one per hop) and the claimed-nodes lookup in nodes.js. A unit test fails if any other file asks for it, if fetchNodeDetail stops asking or starts depending on NodeAdverts, or if the claimed-nodes lookup changes. The E2E asserts that every node-detail request the node page makes carries the opt-in, and that the plain URL stays master-shaped on a live server.
  • The client cache (api(), keyed by path) keeps the two URLs apart. invalidateApiCache('/nodes…') prefixes still clear both.
  • Privacy gates (node blacklist / hidden-name 404, and identityHidden for the observer blacklist and observer names) run before the cache and fail closed. A hidden identity never reaches a lookup or a scan.
  • Documented in OpenAPI (query parameter plus schema descriptions) and docs/api-spec.md.

Design

Classification is the one Relay Airtime Share uses (#89), through the shared pure classifyAdvertRoute(mask, maskKnown, routeType):

class rule
flood route_mask has only routes 0/1
zero_hop only routes 2/3 (firmware Mesh::sendZeroHop sends DIRECT / TRANSPORT_DIRECT with path_len = 0)
mixed both groups — listed and counted only as mixed
unknown no usable route. This is the same bucket Relay Airtime Share's route_class calls legacy (its historical plain ADVERT row). One classifier, two names kept for API compatibility; documented in both endpoints' OpenAPI, docs/api-spec.md and the code.

Without usable mask bits (column absent, row not backfilled yet) it falls back to the first-inserted route_type, exactly as #89 does.

One index-ordered pass. GetNodeAdvertRoutes reads the node's ADVERT rows once, newest id first, straight off idx_transmissions_from_pubkey (no sort, no window function). Go then classifies each row, keeps the first 20 of every class (so 100 zero-hop adverts cannot push 2 flood adverts out), and feeds every row at or after the 7d date floor into the 24h/7d counts (50,000-row cap, truncated). The listed rows are read afterwards in one batch as lean rows (no observations array). A unary +payload_type keeps SQLite on the from_pubkey index. Neither binary runs ANALYZE, and without statistics the choice between the from_pubkey and payload_type indexes otherwise depends on index creation order and SQLite version: the sqlite3 3.54 CLI and a test schema with the indexes in the other order pick payload_type, which would visit every ADVERT in the database. A plan test covers this. The previous ROW_NUMBER() OVER (PARTITION BY class) query is kept as a test-only oracle.

Counts. Distinct adverts (by hash) whose first_seen lies in the last 24h / 7d — the axis flood_advert_count_7d uses, so a late buffered upload counts in the window it was sent in. The date floor, exact parseRelayTS window and dedup are shared with advert_stats.go (advertWindow / advertDateFloor). route_mask_backfill says when classes are still provisional.

flood_advert_count_7d is an external contract (ArcScope advisor): route_type = 1 only. It is counted fresh on every request and never cached:

  • the same pass yields it bit-identically to CountFloodAdvertsForNode at the same instant. The pass evaluates that query's own SQL predicates per row (first_seen >= floor, route_type = 1), applies its own row cap to the qualifying rows newest first, and runs the same countFloodAdverts;
  • the scan's value is used only by the request that ran the scan itself. A cache hit, or a request that waited on a concurrent scan (singleflight), runs CountFloodAdvertsForNode as before;
  • without the opt-in it is exactly as on master.

It still differs from advertCounts["7d"].flood (route_mask based: counts transport flood, never a mixed advert).

Cache (unchanged behaviour): per pubkey, 30 s TTL, invalidated when the node has a newer transmission (debounced to one rescan per 5 s), singleflight, at most 128 entries. Measured heap: ~2.5 KB per lean row. That is ~125 KB for a typical entry (~52 rows), ~200 KB at the 80-row maximum, and ~16 MB for 128 typical entries (≤ ~25 MB); the old comment said "a few KB".

Types. New response parts are named structs. NodeAdvertRow stays map-shaped because the shared transmission scanner is (upstream Kpa-clawbot/CoreScope#1383). map[string]interface{} in touched files: db.go 68 → 66, the others unchanged.

Frontend. One shared component, public/node-adverts.js, renders both copies; tabs reuse initTabBar; colours are existing tokens.

Performance

Scratch database (not in the repo): 1,061,500 transmissions (1,487,434 observations) over 10 days, migrated schema, interleaved ids. Heavy nodes: one with 50,000 adverts, one with 10,500, one with 1,000, plus 20 more with 10,500 each. There are also 2,000 nodes with 200 adverts and 390,000 non-advert rows. Adverts are 40 % route 1, 50 % route 2, 5 % route 0, 5 % route 3; 5 % mixed, 2 % NULL mask.

Hardware: Apple M5 (10 cores, 32 GB), Go 1.26.3 locally (CI uses 1.27.1), modernc SQLite 3.46.0.

Load: the machine was shared with other sessions (iOS simulator, Defender scans) and swapped heavily (7.5–10.7 GB swap in use). Load is reported per run, and "quiet" means 1-min load ≤ 8. Both servers ran with packetStore.retentionHours: 24 to keep RSS (~1 GB each) out of swap; node detail is database-bound, so this does not favour either build.

In-process (node_advert_routes_bench_test.go, 5 interleaved rounds × 20 iterations, load 4.0–4.5, median ms/op):

node new stream (also yields the flood count) old window query CountFloodAdvertsForNode
50,000 adverts 104.0 159.2 63.9
10,500 adverts 24.5 32.7 15.3
1,000 adverts 2.85 3.65 1.43

Before this change a cold opted-in request ran window + flood count, 159 + 64 = 223 ms on the 50k node. Now it runs the stream alone: 104 ms.

HTTP (GET /api/nodes/{pubkey}, master fa5bf4f7 and this branch side by side on the same file, requests alternating between the builds; 2 runs × 20–21 rounds of 31.5 s so every "cold" request is past the 30 s TTL). p50 / p95 in ms, from the rotated second run (12 of 21 rounds quiet; the first run agrees within noise):

request master (plain) this PR, plain this PR, opt-in cold this PR, opt-in cached
50,000 adverts (quiet) 67.1 / 103.0 66.7 / 104.4 106.8 / 127.5 67.2 / 87.0
10,500 adverts (quiet) 17.2 / 28.4 17.5 / 25.6 26.4 / 31.7 17.8 / 24.2
1,000 adverts (quiet) 3.1 / 4.7 3.1 / 5.2 4.6 / 6.0 3.4 / 4.6
20 parallel, same 50k node, wall (quiet) 1,739 / 2,048 1,631 / 2,239 1,606 / 1,847 1,641 / 2,025
20 parallel, 20 different 10.5k nodes, wall (all rounds) 1,346 / 1,904 1,203 / 1,931 852 / 2,157 —
  • Without the parameter the branch equals master within noise, including 20 different heavy nodes in parallel. Cached opt-in equals plain: the flood count is still counted fresh, and that is what a plain request costs too.
  • Cold opt-in cost (accepted by the maintainer): +40 ms on the 50k node, +9 ms at 10.5k and +1.5 ms at 1k on this machine. Scaling by the review's server figures is an estimate, not a measurement: a 4-vCPU linux/amd64 server was ~3× slower on these scans, so expect roughly +100–150 ms there on a 50k-advert node. The review's +442 ms is the old window + flood-count path. Only the node page pays it, once per node per TTL (or per 5 s after a new advert).
  • Independent re-review, measured on that server hardware (Xeon 2.1 GHz, 4 vCPU, Go 1.24.7, on 658d8a08): cold opt-in cost +166 ms HTTP (warm connections) and +155 ms in-process on the 50,000-advert node; +34 ms in-process on the 10,500-advert node. Plain (no include) matched master on that machine too. This is a direct measurement that supersedes the "roughly +100–150 ms" estimate above for the 50k node; the two are in the same range.
  • 20-parallel "different nodes" is noisy and bimodal: whichever batch runs first after the 31 s pause pages the servers back in from swap (0.5 s vs 1.5 s). The batches rotated, and the opt-in batch is not slower than master's plain one. Treat this row as order of magnitude only. In real use only the node page sends the parameter, one node at a time.

Response size (bytes, no gzip):

node master this PR, plain this PR, opt-in
50,000 adverts 24,360 24,360 98,631
10,500 adverts 30,775 30,775 106,996
1,000 adverts 34,770 34,770 111,363

The review's reference figures (4-vCPU linux/amd64, Go 1.24.7, before this change): cold 616 vs 174 ms p50 on the 50k node; in-process 436 (window) / 151 (flood count) / 335 (stream without window) ms/op; 20 heavy nodes in parallel master 1.0–1.2 s vs PR 2.4–2.8 s; response 29 → 80 KB for every caller. The last two no longer apply to non-node-page callers.

Tests

Test-first; the new opt-in tests were red before the code.

  • Opt-in (node_advert_routes_optin_test.go):
    • master-shaped default response, also for near-miss parameters, with 0 cache lookups and 0 scans;
    • list forms of include;
    • privacy checked before the cache;
    • a failed privacy lookup fails closed even when it does not also resolve hidden = true (a canceled request context with a hidden-name prefix configured, distinct from the existing blacklist/observer-blacklist case — re-review N1, see below);
    • flood_advert_count_7d fresh next to a cached breakdown;
    • the scan's flood count handed only to the scanning request.
  • Stream (node_advert_routes_stream_test.go):
    • stream vs window oracle: 1,000 random nodes over three schemas (relaxed with/without route_mask, production schema). They cover mixed, NULL, out-of-range and mistyped masks and route types; every first_seen format including garbage, empty, date-only and window edges; duplicate and missing hashes; non-advert rows; per-class limits 1–5/20; row caps 1–40/50,000. Identical ids, order, classes, counts and truncation for all 1,000, and identical full responses for every 25th. The run hit 339 truncated nodes, 292 at a per-class limit, 785 with unknown and 704 with mixed;
    • flood count vs CountFloodAdvertsForNode: bit-identical for 1,000 random nodes. That includes 211 where the row cap binds, 559 with window/parse/dedup skips and 328 with duplicate hashes;
    • query plan: from_pubkey index, no temp B-tree, no window — even without statistics and with the payload_type index created later.
  • Existing node_advert_routes_test.go / cache tests adapted, and the OpenAPI test now also requires the include parameter.
  • Frontend (test-node-adverts.js, 13 → 19):
    • P3 escaping of decoded text (F3), SNR/RSSI (F5), role colour (F12) and the timestamp fallback;
    • aria-selected and tabindex on every tab at render (F14);
    • the caller rule (a textual/lexical check on source, not a runtime one — re-review N2, see below).
  • E2E (test-issue-2073-recent-adverts-e2e.js, 9 → 10 steps): every node-page detail request opts in, and the plain URL stays master-shaped on the server.
  • test-issue-1206-resize-observer-leak-e2e.js mirrors the node-detail request (route glob, api() in-flight key, error text). It now matches the route on the pathname and uses the opted-in path.

Follow-up from the independent re-review of 658d8a08

  • N1 — fail-closed on a failed privacy lookup, now tested. The gate at handleNodeDetail (cmd/server/routes.go) sets advertRoutes = err == nil && !hidden after s.isIdentityHidden. The existing PrivacyBeforeCache test only exercises hidden = true, err = nil (blacklist/observer-blacklist), which isIdentityHidden resolves without ever running the name lookup that can fail — so the fail-open mutant advertRoutes = !hidden (dropping the err check) agreed with the correct code there and survived the suite. A new test (TestNodeDetail_IncludeAdvertRoutesFailsClosedOnLookupError) forces the name lookup itself to error (a hidden-name prefix configured so the lookup isn't skipped, plus an already-canceled request context) and asserts the response stays master-shaped (200, no route_class / advertCounts / recentAdvertsByRoute, 0 cache lookups, 0 scans). Verified red on the fail-open mutant, green on unmodified code.
  • N2 — caller rule is textual, documented and widened. test-node-adverts.js's "only the node page sends include=advertRoutes" check is a regex over source text, not a runtime assertion. It now matches the bare advertRoutes token (not just the literal include=advertRoutes) in every public/*.js file except nodes.js and node-adverts.js (the shared render component, which only names the opt-in in a comment and makes no api()/fetch call). That change is enough to catch the two mutants the review named — a split literal ('?include=' + 'advertRoutes') and a URLSearchParams-built query (params.set('include', 'advertRoutes')) elsewhere — both verified red when injected into a scratch copy of another file, green on unmodified code. It remains a lexical check: a caller that assembles the string from smaller pieces ('advert' + 'Routes') or from data outside the source would still slip through; closing that gap needs a runtime check (loading the caller files in a vm sandbox with a stub api() and asserting on the URLs actually built), which was judged too heavy to add here given the files and functions involved (packets.js, live.js, channels.js, route-view.js, nodes.js).
  • N3 — server performance figures added above. The independent re-review measured the cold opt-in path directly on 4-vCPU/Xeon 2.1 GHz server hardware (Go 1.24.7): +166 ms HTTP (warm connections) and +155 ms in-process on the 50,000-advert node, +34 ms in-process on the 10,500-advert node, plain unchanged from master. Added next to this PR's own M5 numbers in the Performance section above.
  • N4, N5 — out of scope, tracked as known limitations below. CountFloodAdvertsForNode missing its mutex (issue Decide: port upstream SQLite planner statistics (ANALYZE in the ingestor)? #100) and the row scanner's handling of non-integer route_type (pre-existing on master) are not changed by this PR; see Limitations.

Mutations — each applied temporarily; all compiled, all red, files restored byte-identical (sha256):

mutation caught by
classify on first route_type instead of mask (Go) TestAdvertRouteClassSQL_MatchesGoClassifier, TestAdvertRouteClassFromMask, stream oracle, …
stream ignores the mask (SQL input) TestNodeAdvertRoutes_MixedOnlyInMixed, stream oracle
client-side split of the 20 test-node-adverts.js (3 cases)
split of the first 20 only TestNodeAdvertRoutes_PerClassLimitInSQL, stream oracle
mixed also counted as flood TestNodeAdvertRoutes_MixedOnlyInMixed, TestCountAdvertRoutes_DedupAndClasses
7d window used for 24h TestNodeAdvertRoutes_WindowsAndTimestampFormats, stream oracle
privacy filter removed TestNodeDetail_AdvertRouteFieldsPrivacy, …PrivacyBeforeCache
privacy check fails open on a lookup error (advertRoutes = !hidden) TestNodeDetail_IncludeAdvertRoutesFailsClosedOnLookupError (added for re-review N1)
F3 / F5 (snr) / F5 (rssi) / F12 / timestamp fallback unescaped the matching escaping test each
F14 aria-selected="true" on every tab tab render test
server ignores the parameter (always / never computes) …WithoutIncludeIsMasterShaped / …AdvertRouteFields, …ListForms
route_class read without the opt-in …WithoutIncludeIsMasterShaped
live.js / packets.js / claimed nodes send it; node page stops sending it; fetchNodeDetail depends on NodeAdverts caller rule test (widened for re-review N2)
privacy check after the cache lookup …PrivacyBeforeCache
include not in OpenAPI TestOpenAPI_NodeAdvertRouteSchemas
flood count handed to requests that shared a scan …FloodCountOnlyFromOwnScan
flood count from the class instead of route_type = 1 flood parity, …AdvertRouteFields
flood rows share the count cap flood parity
flood count skipped on a cache hit …FloodAdvertCount7dStaysFresh
unary + removed from payload_type TestNodeAdvertRoutes_StreamQueryPlan

Local runs (branch tip; Go 1.26.3):

  • Go checks:
    • go test -race -timeout 20m ./... in cmd/server: ok (228 s);
    • readonly_invariant_test.go: 4/4;
    • go vet, gofmt on touched files and git diff --check: clean.
  • JS checks:
    • CI JS unit block: 74/74;
    • ESLint: 0 errors;
    • check-css-vars: 0 undefined;
    • XSS gate --diff origin/master: clean.
  • E2E against a CI-prepared fixture (freshen, grouped seed, migrate, 2073 seed):
    • test-issue-2073-recent-adverts-e2e.js: 10/10 three times;
    • test-issue-1147-section-order-e2e.js: 2/2;
    • test-e2e-playwright.js with fail-fast disabled in a scratch copy: 131/135, 3 skipped, 1 failed ("Version info lives on Perf dashboard"). The pass/fail/skip sets are identical on master fa5bf4f7 with its own CI-prepared fixture.
    • the whole CI E2E list (112 commands, no fail-fast) after CI caught the test-issue-2027 regression fixed in 658d8a08. 1206 and 2027 were the only failures specific to this branch; both are fixed (28/28 twice, 25/25). The 10 remaining local failures fail identically on master fa5bf4f7 on this machine (load 25–41: packets/gesture/slide-over timeouts, "Version info"); they passed in CI on d9bb8510.

Commits in this round

Limitations

  • Cold cost on advert-heavy nodes remains for the node page (above); no index added by design (cmd/server is read-only, no migration).
  • Per-class lists are all-time (within the database), counts are windowed. An empty Flood tab means no flood advert on record, not "none in 7 days".
  • recentAdverts for an identity hidden only via the observer blacklist is unchanged (pre-existing); only the new fields are omitted.
  • Perf numbers come from a loaded, swapping laptop (loads and swap given above), not from the review's server. The 20-parallel different-nodes row is order-of-magnitude only.
  • This branch merges master fa5bf4f7; master has since gained fix(server): avoid mutating the cached channels slice on includeEncrypted #98 (channels cache append fix, routes.go +8/−2 in another handler). It merges cleanly and is not merged here.
  • Pre-existing, also on master: the first theme load re-renders the node views once, which drops keyboard focus; the E2E waits for it.
  • (N4, re-review of 658d8a0, out of scope) CountFloodAdvertsForNode still lacks the mutex the scan path (nodeAdvertRoutes) added; tracked separately as issue Decide: port upstream SQLite planner statistics (ANALYZE in the ingestor)? #100, not addressed by this PR.
  • (N5, re-review of 658d8a0, out of scope) The row scanner does not drop non-integer route_type values (pre-existing behaviour already on master, neither introduced nor fixed here), and a column-level optimization for the advert scan may be possible (e.g. narrowing the columns read per row); both noted as possible future follow-ups, not pursued in this PR.

Customizer / config (AGENTS.md rule 8)

Hardcoded for now, candidates for config later: the 24h / 7d windows, the 20-per-class limit, the 50,000-row cap, and the cache TTL (30 s), debounce (5 s) and size (128).

🤖 Generated with Claude Code

https://claude.ai/code/session_01F9yCWp8MvMrMcW2Zs3EgYW

dborup and others added 3 commits September 25, 2026 16:24
…per-class counts

Port/extension of upstream `Kpa-clawbot#2073`, with the
"Recent Adverts" rename and tooltip from upstream `Kpa-clawbot#2071`.

Server (GET /api/nodes/{pubkey}):
- classifyAdvertRoute is the pure rule behind Relay Airtime Share's
  advertRouteClass (#89: route_mask, else first-inserted route_type); a
  SQL CASE mirrors it and a test pins both over every mask x route_type.
- recentAdverts is unchanged (20 rows, id DESC) plus route_class on ADVERT
  rows. New recentAdvertsByRoute holds up to 20 adverts per class, the
  class filtered in SQL (ROW_NUMBER per class) so zero-hop adverts cannot
  push flood adverts out; a mixed advert is only listed as mixed.
- New advertCounts: distinct adverts per class by first_seen over 24h and
  7d, sharing the date floor, parseRelayTS window, hash dedup and row cap
  with advert_stats.go (extracted, not copied), plus the route_mask
  backfill status and a truncated flag.
- flood_advert_count_7d is unchanged (route_type 1 only); the difference
  from advertCounts["7d"].flood (route 0, mixed) is documented and tested.
- New fields and route_class are omitted for hidden identities (#68).
- Per-pubkey cache (30 s TTL, targeted invalidation on a newer
  transmission debounced to 5 s, 128 entries, singleflight): the scan
  costs ~135 ms on a node with 50,000 adverts.
- OpenAPI and docs/api-spec.md updated; new schemas use a typed struct.

Frontend: one shared component (public/node-adverts.js) for the node
page and side panel: counts row, All / Flood / Zero-hop / Mixed tabs via
initTabBar, route badge per advert, empty states, provisional note while
the backfill runs, ?adverts= deep link.

Tests: Go unit/handler/cache tests, vm unit test of the real component,
Playwright E2E on a seeded fixture (new CI seed step after migration).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd a single index-ordered scan (#97 review)

Cloud review of PR #97 (P2-1, P2-2, P3):

- Opt-in: GET /api/nodes/{pubkey} computes recentAdvertsByRoute,
  advertCounts and route_class only with include=advertRoutes. Without
  it the response is master's and neither the privacy lookup, the cache
  nor the scan runs. Only nodes.js fetchNodeDetail (full node page and
  side panel) sends it, via NodeAdverts.detailPath; packets, live,
  channels, route view and claimed-node lookups keep the plain URL.
  Privacy gates run before the cache and fail closed.
- The ROW_NUMBER() window query is replaced by one pass over the node's
  ADVERT rows in idx_transmissions_from_pubkey order; classification,
  the 20-per-class lists, 24h/7d counts, row cap and truncation are done
  in Go. A unary + on payload_type keeps the planner off the
  payload_type index when no ANALYZE statistics exist. The old query
  stays as a test-only oracle.
- The same pass yields flood_advert_count_7d bit-identically to
  CountFloodAdvertsForNode. It is used only by the request that ran the
  scan; cache hits and shared scans count it fresh, and it is never
  cached.
- Cache memory comment now gives measured heap per entry.
- P3: tests for escaping of decoded text, SNR/RSSI, role colour and the
  timestamp fallback, and for aria-selected/tabindex on every tab at
  render. "unknown" vs Relay Airtime Share's "legacy" documented as one
  bucket in OpenAPI, docs/api-spec.md and code.

Tests: 8 new Go tests (opt-in shape, list forms, privacy before cache,
fresh flood count, flood only from own scan, stream vs window oracle
and flood count vs CountFloodAdvertsForNode over 1,000 random nodes
each, query plan); test-node-adverts.js 13 -> 20; 2073 E2E 9 -> 10
steps (node page requests opt in, plain URL master-shaped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dborup

dborup commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner Author

Review feedback addressed (commits df466459 + 658d8a08, on top of merge commit 684c7bb4 that brings in master fa5bf4f7)

  1. P2-1 — cold path cost. The ROW_NUMBER() window query is replaced by one index-ordered pass over the node's ADVERT rows (ORDER BY id DESC on idx_transmissions_from_pubkey, no sort, no window). Classification (classifyAdvertRoute), the 20-per-class lists, the 24h/7d counts, the 50,000-row cap and truncated are done in Go. The old query stays as a test-only oracle: identical ids, order, classes, counts and truncation for 1,000 random nodes across three schemas (mixed/NULL/out-of-range/mistyped masks and route types, every first_seen format, duplicate and missing hashes, small limits and caps), plus identical full responses for every 25th node. The same pass also yields flood_advert_count_7d, bit-identical to CountFloodAdvertsForNode (same SQL predicates route_type = 1 and first_seen >= floor, its own row cap, same dedup and parseRelayTS window; parity test over the same 1,000-node shape). It is used only by the request that ran the scan itself. Cache hits and requests that shared a concurrent scan still count it fresh, and it is never cached. The cache comment now states the measured heap (~2.5 KB per row, ~125 KB per typical entry, ≤ ~25 MB at 128 entries). A unary +payload_type pins the scan to the from_pubkey index: without ANALYZE statistics, some SQLite versions and index orders pick idx_transmissions_payload_type and would visit every ADVERT (plan test).
  2. P2-2 — scope. The breakdown is opt-in: GET /api/nodes/{pubkey}?include=advertRoutes. Without it the response is master's (no recentAdvertsByRoute, no advertCounts, no route_class), and neither the privacy lookup, the cache nor the scan is touched (test counts lookups and scans: 0). Only fetchNodeDetail in nodes.js sends it, which covers the full node page and the side panel. It builds the URL itself: 658d8a08 removed a fetch-time dependency on node-adverts.js that CI caught in test-issue-2027, and adapted test-issue-1206 to the opted-in URL. Packets, live, channels, route view and the claimed-nodes lookups keep the plain URL; a unit test fails if any other file asks for it, and the E2E checks the node page's requests. Privacy gates run before the cache and fail closed. Documented in OpenAPI and docs/api-spec.md.
  3. P3. New unit tests kill F3 (decoded.text), F5 (snr, plus rssi), F12 (roleColor), the timestamp fallback and F14 (aria-selected/tabindex on every tab at render). Each mutant was shown red and restored byte-identically. "unknown" (node detail) vs "legacy" (Relay Airtime Share) is documented as one bucket with two names in OpenAPI (both endpoints), docs/api-spec.md and the code. NodeAdvertRow stays map-shaped because the shared scanner is (refactor: drive map[string]interface{} out of API response layer + internal data (694 occurrences in cmd/) — typed structs, codegen for frontend Kpa-clawbot/CoreScope#1383); no new map[string]interface{} in touched files (db.go 68 → 66).

Numbers, hardware and load are in the updated description.


Independent re-review nits addressed (commit 493a055a, on top of merge commit fb84b5e7 that brings in master 7778d262, tree d0c83f5f — #99, #104, #105, #94)

  1. N1 — fail-closed on a failed privacy lookup, now tested. handleNodeDetail sets advertRoutes = err == nil && !hidden after s.isIdentityHidden. The existing PrivacyBeforeCache test only exercises hidden = true, err = nil (blacklist/observer-blacklist), which resolves without ever running the name lookup that can fail — so the fail-open mutant advertRoutes = !hidden agreed with the correct code there and survived. Added TestNodeDetail_IncludeAdvertRoutesFailsClosedOnLookupError, which forces the name lookup itself to error (a hidden-name prefix configured plus an already-canceled request context) and asserts the response stays master-shaped. Verified red on the fail-open mutant, green on unmodified code.
  2. N2 — caller rule widened, documented as textual. The "only the node page sends include=advertRoutes" check in test-node-adverts.js now matches the bare advertRoutes token (not just the literal include=advertRoutes) in every public/*.js file except nodes.js and node-adverts.js (comment-only, no api()/fetch call). Verified red for both mutants the review named — a split literal and a URLSearchParams-built query in another file — via a temporary injection into a scratch copy, restored clean afterward; green on unmodified code. Documented in the test as a lexical check, not a runtime guarantee (a string assembled from smaller pieces would still slip through — judged too heavy to close with a vm-based runtime check here).
  3. N3 — server performance figures added to the description. The independent re-review's direct measurement on that server hardware (Xeon 2.1 GHz, 4 vCPU, Go 1.24.7): cold opt-in cost +166 ms HTTP (warm connections) and +155 ms in-process on the 50,000-advert node, +34 ms in-process on the 10,500-advert node; plain matched master. Added next to this PR's own M5 numbers.
  4. N4, N5 — out of scope, noted as known limitations in the description. CountFloodAdvertsForNode's missing mutex (issue Decide: port upstream SQLite planner statistics (ANALYZE in the ingestor)? #100) and the row scanner's non-integer route_type handling (pre-existing on master) are unchanged by this PR.

go test -race in cmd/server, node test-node-adverts.js, the CI JS unit block (74/74), ESLint (0 errors) and test-issue-2073-recent-adverts-e2e.js against a freshly migrated + seeded fixture all pass on the pushed commit.


Generated by Claude Code

…lf (#97)

CI on df46645 failed test-issue-2027-my-mesh-node-page-e2e.js: its
mocked-404 step loads nodes.js without node-adverts.js, so
NodeAdverts.detailPath threw before the 404 was handled and the page
showed "Failed to load node" instead of "Node not found". The fetch now
builds '/nodes/{pk}?include=advertRoutes' itself; NodeAdverts.detailPath
is removed. The caller-rule unit test now also fails if fetchNodeDetail
depends on NodeAdverts.

test-issue-1206-resize-observer-leak-e2e.js mirrors the node-detail
request (route glob, api() in-flight key, error text); it now matches the
route on the pathname and uses the opted-in path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dborup and others added 2 commits September 26, 2026 13:58
N1: add TestNodeDetail_IncludeAdvertRoutesFailsClosedOnLookupError.
The existing PrivacyBeforeCache test only exercises isIdentityHidden
returning (true, nil) via the blacklist/observer-blacklist, which the
fail-open mutant `advertRoutes = !hidden` (dropping the `err == nil`
check in handleNodeDetail) happens to agree with. The new test forces
the name lookup itself to error (hidden-name prefix configured plus an
already-canceled request context) so the two formulas diverge: verified
red on the fail-open mutant, green on unmodified code.

N2: widen the "only the node page sends include=advertRoutes" caller
check in test-node-adverts.js from the literal `include=advertRoutes`
to the bare `advertRoutes` token (excluding nodes.js and the comment-only
node-adverts.js). Catches the two mutants the review named — a split
literal and a URLSearchParams-built query in another file — without
touching runtime behaviour; documented in the test as still a lexical
check, not a runtime one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9yCWp8MvMrMcW2Zs3EgYW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant