Skip to content

feat(reach): Reach leaderboard with a shared visible-population rank - #69

Merged
dborup merged 9 commits into
masterfrom
codex/reach-leaderboard
Sep 20, 2026
Merged

feat(reach): Reach leaderboard with a shared visible-population rank#69
dborup merged 9 commits into
masterfrom
codex/reach-leaderboard

Conversation

@dborup

@dborup dborup commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Synced with master (#68 merged)

dborup/CoreScope#68 (Reach privacy fix) is now merged into master. This branch has been updated onto the new master with an ordinary merge commit (no rebase, no force-push), and the leaderboard's visibility check now calls #68's shared identityHidden directly (see below) instead of its own copy of the same rule.

Summary

First version of a Reach leaderboard. It answers: "My node is Rank #X on its Reach page. Who is #1?"

  • New page #/reach-rank. For each ranked node it shows:

    • the placement;
    • the name, falling back to the pubkey;
    • the neighbour count;
    • a link to the node's existing Reach page.

    The page also offers:

    • search on name or pubkey;
    • pages of 50 rows;
    • the snapshot time;
    • the disclaimer "Historical neighbour count — not a measure of radio quality or range".

  • New endpoint GET /api/reach-rank?q=&offset=&limit=.

  • Reach page Rank card changes:

    • a "View leaderboard" link;
    • "Not ranked" instead of #0 / N;
    • "—" when the snapshot can't be read;
    • the card now also shows for nodes that have no reliable path token.

  • Not changed: the main menu and Priority+ are untouched. No schema changes, migrations, data deletion or new dependencies.

Rank definition (one definition, one snapshot)

Rank is an all-time neighbour count. It is not range in km, not traffic volume, and not links in the selected time window.
Valid edge A neighbor_edges row whose endpoints are both MeshCore pubkeys (exactly 64 hex characters, case-insensitive) and differ from each other.
Neighbours Distinct neighbours of a pubkey over valid edges: all-time, i.e. within the ingestor's neighborEdgesMaxAgeDays retention. Case and orientation variants of one pair count once.
Ranked population Pubkeys with ≥ 1 valid edge that have a Reach page: a nodes row, or an observers row with a name. A pubkey is excluded when it is node-blacklisted or observer-blacklisted, or when any of its names starts with a hidden-name prefix: its node name, its observer name, or its inactive_nodes name while it has no named nodes row.
Rank 1 + the number of ranked nodes with strictly more neighbours. Competition ranking: equal counts share a placement (1, 1, 3). Ties are listed in pubkey order.
Total The size of the ranked population.
  • No renumbering. Search and paging return the global placements; nothing is renumbered within a result.
  • Same snapshot on both endpoints. /api/nodes/{pk}/reach and /api/reach-rank read rank, total and neighbour count from the same snapshot and expose its time (rank_snapshot_at / snapshot_at). The E2E test asserts equality for real fixture nodes; the reviewer checked all 75.
  • Neighbour counts include hidden neighbours. Neighbours is a count only, never an identity. This was agreed: the neighbour count is "count every valid edge", and visibility filtering is kept separate.

Changes to the existing Reach numbers

Both points were agreed before implementation.

  1. Visible population. The old snapshot grouped every endpoint in neighbor_edges with no filter, so three kinds of pubkey sat in both the ranking and nodes_with_edges:

    • hidden and blacklisted nodes;
    • pubkeys without a Reach page;
    • the empty pubkey.

    A hidden node above a visible one would have pushed it down, revealing that the hidden node exists.

  2. Legacy-data correction. The CI fixture holds 55 legacy edges with an empty endpoint (55 of 110 rows). The current ingestor no longer writes these rows; they expire with edge retention. Under the old count:

    Those rows are now ignored at computation time; nothing is deleted. The effect on the fixture:

    Before After
    Foreign Traffic analytics + Scopes tab redesign #1 1000💥Repeater G2 10 neighbours 9 real neighbours
    fix(frontend): replace legacy scope confirmation glyphs #2 ESP1 Gilroy Repeater 6 5
    Ranked total (nodes_with_edges) 107 75

    The 31 nodes whose only edges were legacy rows now show "Not ranked" with 0 neighbours. The Reach page's "Neighbours" card uses the same rule, so it changes in the same way.

Performance after the merge

Fixture contract, confirmed live on the migrated, freshened CI e2e fixture: GET /api/reach-rank returns total=75, #1 (1000💥Repeater G2) with 9 neighbours, #2 (ESP1 Gilroy Repeater) with 5. The same node's own GET /api/nodes/{pk}/reach reports neighbor_degree=9, degree_rank=1, nodes_with_edges=75, and rank_snapshot_at byte-identical to the leaderboard's snapshot_at — same snapshot, confirmed live, not just asserted in a test.

Cache-hit cost of the merge integration. BenchmarkNodeReachCacheHitVisibility(Scale) (from #68, unchanged, run before/after on the same file) isolates the added cost of looking up the shared rank view on every Reach cache hit, on top of #68's own visibility check:

Reach cache hit master (d9ed1e16, #68 alone) this PR merged (130f9cc8)
50 links, hidden-name prefix configured 268 µs 343 µs
50 links, no prefix 27.2 µs 37.0 µs
hub (150 links/100 obs), prefix configured 877 µs 795 µs
hub, no prefix 39.3 µs 34.7 µs
large (300 links/2000 obs), prefix configured 899 µs 910 µs
large, no prefix 26.1 µs 26.5 µs

Medians of -count=3 -benchmem, Apple M2 Pro. Measured under heavy contention — this machine was running many other agent sessions concurrently (load average 60–90 throughout, roughly 5–8× normal), so these numbers are directional, not precise. The consistent part across runs: the "no prefix" cases (the cleanest isolation of the new per-request cost, since the visibility name lookup itself is skipped without a configured prefix) show +5–10 µs, matching one added mutex-guarded map lookup (currentReachRankView → v.lookup) per request, independent of report size. The "prefix configured" deltas are noisier and don't move consistently with size, which is what you'd expect if the added constant cost (single digit µs) is being swallowed by run-to-run scheduling noise against a background of hundreds of µs to ms from the name lookup itself.

Concurrent mixed traffic, added for this merge (BenchmarkReachAndRankParallel, 3:1 /api/reach-rank to /api/nodes/{pk}/reach, b.RunParallel at GOMAXPROCS=12), on a warm snapshot/view/cache:

Dataset ns/op allocs/op
small (1k nodes) 20.7 µs 48
medium (5k nodes) 14.4 µs 48
large (20k nodes) 13.2 µs 47

No sign of lock contention on the shared snapshot/view mutex even under 12-way concurrency on this heavily loaded machine — throughput holds (actually improves slightly at larger N, likely just more requests hitting the O(1) unfiltered-page path).

Unchanged from the original benchmarks (snapshot rebuild, view rebuild, warm page/search — see below), because none of those paths were touched by the merge conflict resolution; only the Reach page's cache-hit path (above) and the new parallel benchmark are new measurements for this sync.

Implementation

Backend (cmd/server/reach_rank.go; the snapshot moved out of node_reach.go)

  • One read transaction per load. No per-row lookups. The steps:

    • a plain scan of neighbor_edges, validated and counted in Go;
    • rows the ingestor never writes (upper-case or reversed pairs) are deduplicated with one bulk row-value lookup;
    • bulk reads of nodes, observers and inactive_nodes names.

    Any query, scan or rows.Err() error fails the whole load, so a partial read is never published. Names are read here rather than through getCachedNodesAndPM, because that cache swallows DB errors, which would silently drop every node.

  • Refresh (stale-while-revalidate).

    • An expired snapshot keeps being served, with its own snapshot_at, while one background rebuild (singleflight) refreshes it.
    • Reach cache hits therefore never wait on the aggregate. Only a request that finds no snapshot at all waits.
    • The rebuild runs on a detached context with a 15 s bound.
    • A failed rebuild backs off 15 s, and a panic becomes an error (DoChan would otherwise crash the process).
    • With no snapshot at all, /api/reach-rank returns 500, never an empty leaderboard. It deliberately does not return 503: the SPA's api() treats 503 as warm-up and retries for about a minute.

  • Ranked view. It is built from the snapshot and keyed by the blacklist and hidden-prefix generations. A visibility change re-ranks at once from the cached snapshot, with no DB read. View builds are serialised, so a burst of requests builds one view.

  • Reach response cache (5 min). It stores the report plus its body. Rank fields are applied at serve time from the current view, so each cached body is re-marshalled once per view change. An unreadable snapshot is reported as rank_status: "unavailable" instead of caching 0.

  • Input validation. q is at most 64 characters and must be UTF-8 (else 400). offset must be ≥ 0 (else 400). limit defaults to 50; values above 100 become 100; zero, negative or non-numeric values give 50 (the existing clampLimit).

Frontend. public/reach-rank.js reuses the .nq-* Reach styles, the .nodes-search input and .btn.

  • Escaping. Every node-controlled value is escaped, and the pubkey is URL-encoded in the link.
  • URL state. Search and page live in the hash (replaceState), so Back from a Reach page restores them.
  • Query and page limits. Queries are clipped by code point, so an emoji is never cut in half. ?page= is clamped.
  • Stale responses. A response that arrives after a newer request is dropped.
  • Keyboard focus. Focus is kept when a pager button becomes disabled.
  • Contrast. New links use --link-color, the A11y: contrast + typography pass to fix readability complaints Kpa-clawbot/CoreScope#1668 AA link token.

Registration

  • API docs: /api/reach-rank is added to openapi.go and docs/api-spec.md.
  • a11y gate: /reach-rank is added to the axe gate's ROUTES / REGISTERED_PAGES.
  • deploy.yml: only two test lines are added (node test-reach-rank.js, and test-reach-rank-e2e.js in the E2E list). Triggers, permissions, needs and the fork guards are unchanged.

Screenshots (local, CI e2e fixture DB)

All screenshots come from a local Go server built from this branch, running on the committed e2e fixture (freshened and migrated as in CI), in headless Chrome. They were inspected visually; the files are not attached.

Cache and performance

Complexity.

  • Rebuild: one linear scan of neighbor_edges plus linear reads of the name tables, at most once per 60 s, in the background and shared by all requests.
  • View: O(N log N), rebuilt only when the snapshot or a visibility generation changes.
  • Page: an unfiltered page is an O(1) slice.
  • Search: one O(N) substring pass with no DB access.

Memory. One snapshot and one view per server, both bounded by the node count and replaced rather than grown.

Environment: Apple M2 Pro (12 cores), 32 GB, macOS 26.6.2, Go 1.26.0 (CI uses 1.27.1), modernc.org/sqlite v1.34.5, file-backed DB in a temp dir.

Datasets: synthetic and seeded (reachRankBenchSets in reach_rank_bench_test.go). Each node gets 4–5 random neighbours, and 2 % of nodes also have an observer row:

Dataset Nodes Edges
small 1,000 3,940
medium 5,000 19,928
large 20,000 99,851

Values are medians of -count=5 -benchmem at the original feature head 13f4e61c (unaffected by the merge — see Performance after the merge for what changed).

Benchmark small 1k medium 5k large 20k
Snapshot rebuild, this PR (valid-edge scan + names + view) 5.1 ms · 2.4 MB · 61k allocs 27.6 ms · 11.6 MB · 306k allocs 135 ms · 53.9 MB · 1.46M allocs
Master's degree snapshot (old GROUP BY only; baseline) 5.0 ms · 0.3 MB · 6k allocs 41.9 ms · 1.4 MB · 30k allocs 223 ms · 5.9 MB · 120k allocs
View rebuild after a visibility change (no DB) 0.22 ms 1.2 ms 5.2 ms
Warm GET /api/reach-rank page 1 10.3 µs · 28 allocs 10.0 µs 10.1 µs
Warm deep page (offset=N/2) 10.6 µs 10.4 µs 10.5 µs
Warm name search (q=node-01) 13.7 µs 70.8 µs 213 µs
Warm pubkey search (12 hex characters) 19.0 µs 108 µs 872 µs

What changed in the rebuild.

  • The first version was slower. Expressing the valid-edge rule in SQL (lower/GLOB + DISTINCT) made the rebuild about 4× slower than master: 885 ms at 20k nodes.
  • Validation moved to Go. Doing it on a plain scan makes the rebuild faster than master's old degree-only query, even though it now also reads names.
  • The cost is transient memory. Allocation volume is higher: about 14 per row, mostly from the SQLite driver. It is short-lived and happens once per 60 s. Scanning into sql.RawBytes was measured and gave no gain.

Existing Reach hot path. GET /api/nodes/{pk}/reach on a cache hit (1k dataset), measured with the same benchmark on both trees: master 21.7 µs, this PR 21.5 µs, both at 200 allocs/op. The rank-view lookup adds no measurable cost.

Only local synthetic data was measured. There is no production benchmark, and no live Reach endpoints were scanned.

Tests (original, before the master sync)

All results below are from feature head 13f4e61c (before the master sync), except where noted, on the environment above.

Go

  • cd cmd/server && go test -race ./...: ok (241.8 s).
  • cd cmd/ingestor && go test ./...: ok (88.9 s).
  • go vet clean; gofmt clean on every changed .go file; git diff --check clean.
  • New tests in cmd/server/reach_rank_test.go:
    • Ranking: competition ranking and deterministic order; empty and unranked cases; unknown and empty endpoints.
    • Search and paging: keep global ranks.
    • Valid edges: empty, shortened, non-hex, 65-character, prefix and self-edges, case/orientation variants, upper-case duplicates, NULL endpoints. Plus the committed fixture: Foreign Traffic analytics + Scopes tab redesign #1 has 9 neighbours and is not credited with an empty endpoint.
    • Reach page parity: exactly the pubkeys buildNodeInfoMap knows (including NULL-name observers); name fallback.
    • Visibility: hidden, node-blacklisted, observer-blacklisted, observer-name-hidden and inactive_nodes-hidden identities, in both directions. Checked after the cache is warm, via SetHiddenNamePrefixes / SetNodeBlacklist, including a cached Reach body being re-ranked.
    • Consistency: identical rank, total, neighbours and snapshot time on /reach for every row.
    • Cache: cache hit and expiry; stale-while-revalidate (requests don't wait while the load is held); refresh liveness; singleflight for concurrent cold requests; a canceled waiter; backoff after a failure, including a queued refresh.
    • Failures: a panic becomes an error; a cold DB error returns 500, not an empty list; a failed refresh serves the previous snapshot; a partial read (rows.Err) is never published; no DB configured.
    • Handler: response shape and input validation.
  • The refresh and concurrency tests ran 30× at -cpu 1,2,4 under -race with no failures. The reviewer independently ran 50–300× per test.
  • Benchmarks: cmd/server/reach_rank_bench_test.go (see above).

Frontend and CI gates (registered in deploy.yml)

  • The whole CI unit list: 70/70 pass, including the new test-reach-rank.js. That test runs in a vm sandbox and covers:
    • escaping of every field, including attribute and href breakout;
    • the pubkey fallback;
    • global ranks in the status text;
    • snapshot age;
    • hash state;
    • clipping by code point;
    • the Rank card states (ranked / "Not ranked" / "—" unavailable);
    • the no-token Rank card;
    • the index.html script tag.
  • eslint: 0 errors. The 89 warnings are pre-existing, none in the changed files.
  • XSS-sink gate --diff against master: clean.
  • node --check passes on the changed JS files.
  • The axe self-test and route-coverage tests pass.

Browser (local server on the fixture, headless Chrome through the Playwright library)

  • test-reach-rank-e2e.js: 14/14. It covers:
    • API shape and order;
    • the desktop page matching the API;
    • paging and the hash;
    • search keeping global ranks;
    • pubkey search and the no-match state;
    • snapping back from a past-the-end page;
    • keyboard: search → Tab → Enter opens Reach, the Rank card matches (same snapshot), and Back restores the search;
    • the fixture's Foreign Traffic analytics + Scopes tab redesign #1 showing 9 neighbours, not 10;
    • the Rank card on a no-token node;
    • "View leaderboard";
    • mobile 375×812 with no horizontal scroll;
    • a hostile node name rendered as text (intercepted API);
    • API failure showing an error, not an empty list;
    • no page errors.
  • axe gate on /reach-rank (desktop and mobile × dark and light, 13 rules): 0 violations.
  • Existing test-issue-1630-reach-mobile-e2e.js: 7/7.
  • test-node-reach-coverage-e2e.js: SKIP, because the fixture has client RX coverage disabled, the same as in CI.
  • Not registered in CI and pre-existing: test-node-reach-e2e.js times out locally on the fixture, on master's frontend too.

Tests (fresh, on the merged head 130f9cc8)

Go

  • cd cmd/server && go test -race ./...: ok (481 s). No races.
  • go vet clean; gofmt clean on every changed .go file; git diff --check clean.
  • All 68 Reach/privacy tests from fix(reach): hide blacklisted and hidden identities on the Reach page #68 pass on the merged tree, including the two that needed updating for the new cache-entry shape (see above).
  • All 24 pre-existing reach-rank tests pass unchanged, including the 4 named ones: TestReachRank_OnlyValidEdgesCount, TestReachRank_NullEndpointSkipped, TestReachRank_FixtureLegacyEdgesNotCounted, TestReachRank_ConsistentWithNodeReach.
  • New for this merge:
    • TestNodeReach_HiddenNeighbourCountedNotListedOrRanked — cross-endpoint: a hidden neighbour of a visible node is (a) absent from that node's links, by pubkey or name, (b) absent from the leaderboard's rows and unfindable by any search term (pubkey, pubkey prefix, name, name substring), yet (c) still counted in the visible node's own neighbor_degree; the leaderboard's total reflects only the visible population.
    • TestReachRankVisible_MatchesIdentityHidden — reachRankVisible compared directly against identityHidden across every blacklist/observer-blacklist/hidden-prefix combination and 7 name-slot patterns (missing, empty, plain, hidden, mixed, case variants) — an equivalence probe against the parallel-rule bug fixed during the merge, so it can't silently reappear.
    • TestReachRank_OnlyValidEdgesCount strengthened: added a self-edge that exists only in non-canonical (mixed-case) form, with no literal canonical row for the same pair. The original fixture's two self-edge cases both happened to also exist as an exact-match canonical row, which meant canonicalEdgesExisting's existing-row check accidentally masked a missing self-edge guard (found via mutation testing below) — the new case closes that gap.

Mutation / negative controls (each applied to a scratch copy of the merged code, confirmed to make at least one test fail, then reverted):

Mutation Caught by
pubkeyForm accepts invalid input as valid TestReachRank_OnlyValidEdgesCount
Self-edge skip (la == lb) removed TestReachRank_OnlyValidEdgesCount (only after strengthening — see above; the original fixture did not catch this)
Reverse/duplicate-edge dedup (canonicalEdgesExisting) disabled TestReachRank_OnlyValidEdgesCount
Visibility filter removed from buildReachRankView TestReachRankView_HiddenAndBlacklistedNeverPlaced, TestReachRank_VisibilityChangesAfterWarmCache, TestNodeReach_HiddenNeighbourCountedNotListedOrRanked (3 independent tests)
reachBody's viewID freshness check bypassed (cached rank never refreshes) TestReachRank_VisibilityChangesAfterWarmCache

Frontend / E2E, run locally against a fresh copy of the CI fixture (freshened + seeded + migrated, corescope-server on port 13594):

  • node test-reach-rank.js: pass (unit test, vm sandbox).
  • node test-xss-escape-sinks.js: 34/34 pass.
  • npx eslint public/*.js (eslint@8, matching CI): 0 errors (89 pre-existing warnings, none in the changed files).
  • test-reach-rank-e2e.js: 14/14, including the fixture's 9/not-10 assertion, mobile 375×812, keyboard (search → Tab → Enter → Back), hostile node names, and the API-failure error state.
  • test-issue-1630-reach-mobile-e2e.js (existing Reach mobile coverage): 7/7.
  • test-a11y-axe-1668.js, full route list: 0 violations across 120 cells (2 viewports × 2 themes × 30 routes, including /reach-rank).

Independent review

A separate reviewer agent, which did not write the code, reviewed the branch in four rounds. It read the commits through git and ran its own probes in isolated copies. The focus was the rank contract, visibility, cache and concurrency, failure modes, performance, and regressions in the existing Reach page.

  1. Round 1 (c9b2022f): no blockers. SHOULD-FIX and NIT findings, all addressed:

    • a leaderboard row could link to a Reach page that returns 404 (NULL observer name);
    • cached Reach responses waited on the snapshot rebuild, with no backoff;
    • no Rank card on Reach pages of nodes without a reliable token;
    • emoji query clipping;
    • ?page clamping;
    • collapsing view builds;
    • checking all names against hidden prefixes;
    • test quality and docs.

    Two pre-existing Reach privacy gaps were moved to the prerequisite PR by product-owner decision. The valid-edge contract was also decided then.

  2. Round 2 (20809fdd, 15026d21): no blockers. Addressed:

    • a flaky background-refresh test;
    • the backoff re-check inside the refresh;
    • one read transaction for the snapshot load;
    • NULL endpoints;
    • docs.

  3. Round 3 (95e2cc3f, f3919f13): one BLOCKER. The new "refresh in flight" flag could stick at true and stop all refreshes until a restart; the reviewer reproduced it in stress runs. Fixed by removing the flag (3b1a91a4).

  4. Round 4 (3b1a91a4): no BLOCKER remains. The reviewer's 3000-round stress probe got stuck in 2 of 6 runs before the fix and in 0 of 8 after.

The reviewer also checked on the running fixture server that all 75 leaderboard rows match /api/nodes/<pk>/reach in rank, total, neighbours and snapshot time. 13f4e61c added only a test; the merge to 130f9cc8/fc4ba5ab is reviewed fresh below.

  1. Round 5 (merge to 130f9cc8/fc4ba5ab, after fix(reach): hide blacklisted and hidden identities on the Reach page #68 landed): a fresh reviewer, focused on the six areas above (privacy, SQL/edge dedup, ranking, cache/concurrency, abuse potential, frontend) plus explicitly diffing against both merge parents to isolate what the merge itself changed. No blocker. Traced every path that serves a Reach body (cache hit, singleflight miss, the shared computation closure) through writeReachEntry → reachBody → visibleReach, confirmed hiddenKey/viewID are collision-free freshness keys that force a re-marshal on any mismatch, and confirmed reachRankVisible now delegates to identityHidden (the actual conflict-resolution fix). Re-derived the edge validation/dedup logic against TestReachRank_OnlyValidEdgesCount's strengthened fixture and the concurrency machinery in reach_rank.go, and ran the suite including -race. Three low-severity nits, none privacy- or correctness-affecting, none requiring a fix:
    • offset pagination can show a duplicated or skipped row if the ranked view rebuilds between two page fetches (inherent to offset paging over live-rebuildable data, not a regression);
    • the shared singleflight Reach computation is cancellable by an unrelated concurrent requester's disconnect (availability wrinkle, pre-existing pattern, documented in the code's own comment);
    • an inactive_nodes hidden name can over-hide an observer-only identity that now has a distinct current observer name (already covered by TestReachRank_InactiveNamesOnlyForInactiveNodes; fails safe — over-hides, never leaks).

CI, fresh after the merge (run 35442541721, head fc4ba5ab)

Tested as GitHub's own merge ref 40c0ceb8 (master d9ed1e16 + fc4ba5ab) — the same tree verified locally above. Result: ✅ success, all the way through.

Job Result
Go Build & Test ✅ server (87.9 % coverage), ingestor, channel library, decrypt CLI, JS unit list, eslint, XSS gate
Playwright E2E ✅ 691 checks passed, 0 failed
Build & Publish Docker Image ✅ local image build only; GHCR login and push skipped (fork guard)
Release Artifacts skipped (tags only, fork guard)
Deploy Staging skipped (master push only, fork guard)
Publish Badges & Summary skipped (fork guard)

Confirmed explicitly, not inferred from the green checkmark alone: nothing was deployed to staging, nothing was pushed to GHCR, no release was created, and no badges were committed. Coverage badges were only uploaded as CI run artifacts (an ordinary actions/upload-artifact step inside the Go Build & Test / E2E jobs), which is not the same as the 📝 Publish Badges & Summary job that commits them to the repo — that job shows skipped.

Tests of particular interest, all green on this run:

  • test-channel-color-picker-e2e.js: 9/9, including "ArrowRight cycles focus across swatches" — the test that was flaky on fix(reach): hide blacklisted and hidden identities on the Reach page #68's earlier run is stable here.
  • test-reach-rank-e2e.js: 14/14, including the fixture's 9-not-10 assertion (total=75) and mobile/keyboard coverage.
  • test-issue-1122-packets-filter-ux-e2e.js: 6/6, including "Path column row height stays bounded < 60px" — the test that was time-dependent-flaky on this branch's pre-merge CI run (see the old CI section above) passes cleanly now, consistent with it being a fixture-timing artifact unrelated to this branch's code, as already diagnosed there.
  • test-issue-1122-details-row-clamp-e2e.js: 18/18 (from fix(packets): keep the Details summary on one line so rows stay bounded #67, carried in by the master sync).
  • axe gate, full route list: 0 violations across 120 cells, including /reach-rank.

This fresh run replaces the old CI section below for merge-readiness purposes; the old section is kept for history (it covers the original, pre-sync implementation and its own independent-review rounds).

CI (run 35364106106, head 13f4e61c)

Known limitations (updated for the merge)

  • Visibility is up to 60 s behind the DB for renames, same as before: a node renaming itself into or out of a hidden prefix changes the leaderboard at the next snapshot rebuild (≤ 60 s, refreshed in the background). Blacklist and hidden-prefix config changes apply immediately (the view is keyed by both generations).
  • Reach page cache-hit cost with a hidden-name prefix configured is now the sum of fix(reach): hide blacklisted and hidden identities on the Reach page #68's live name lookup and this PR's rank-view lookup — roughly +5–10 µs from the rank-view lookup alone on top of fix(reach): hide blacklisted and hidden identities on the Reach page #68's existing cost (see Performance). Both are accepted for the same reason fix(reach): hide blacklisted and hidden identities on the Reach page #68 accepted its own cost: immediate, fail-closed privacy and an always-current rank on every serve, cached or not.
  • Old hidden names over-hide in one case (fails safe, inherited from fix(reach): hide blacklisted and hidden identities on the Reach page #68): a pubkey that is now only an observer stays hidden by an old hidden inactive_nodes name.
  • Search is a linear substring pass. Fine at 20k nodes (well under 1 ms); no index.
  • "All-time" means within the ingestor's edge retention. Legacy invalid rows remain in the DB until retention removes them; they are only ignored at read time, never deleted.
  • Until the prerequisite PR merges, master's Reach page itself still shows observer-hidden identities — resolved: fix(reach): hide blacklisted and hidden identities on the Reach page #68 is merged, and the leaderboard now shares its identityHidden rule directly (this merge also removed a parallel copy of that rule found during review — see Synced with master).
  • Pre-existing and not addressed (all reproduced on master's frontend before this PR):
    • leaving a Reach page mid-animation throws Leaflet reading '_leaflet_pos' (the E2E ignores exactly that message);
    • .nq-link on the Reach page is 2.52:1 contrast in the light theme;
    • test-node-reach-e2e.js (not registered in CI) times out locally on the fixture.

Not tested

Staging and production were not tested. All validation used a local server on the committed CI e2e fixture DB and synthetic benchmark data. Nothing was deployed.

🤖 Generated with Claude Code

Dennis Jakobsen and others added 7 commits September 18, 2026 15:44
Dennis Jakobsen @claude Dennis Jakobsen
feat(reach): Reach leaderboard with a shared visible-population rank
c9b2022 
Answers "my node is Rank #X on its Reach page, who is #1?".

Backend (cmd/server/reach_rank.go):
- One neighbour-degree snapshot (60s TTL) now backs both the Reach Rank
  card and the new GET /api/reach-rank. Rebuilds are singleflight-shared,
  run on a detached bounded context, and never publish a failed or
  partial read (rows.Err and scan errors fail the load); a failed refresh
  keeps serving the last complete snapshot with its own timestamp.
- Rank contract: neighbours = distinct neighbours in neighbor_edges;
  ranked population = endpoints with a node/observer record that are not
  node-/observer-blacklisted or hidden by name prefix; competition ranking
  (1, 1, 3), ties in pubkey order. Blacklist / hidden-prefix generations
  re-rank from the cached snapshot without a DB read.
- Previously the rank counted hidden and blacklisted nodes and pubkeys
  without a Reach page (incl. the empty pubkey), so a hidden node could
  push every visible node down and was counted in the total.
- /api/nodes/{pk}/reach applies rank fields at serve time from the same
  view (cached bodies are re-marshalled once per view change) and adds
  rank_status / rank_snapshot_at; an unreadable snapshot marks the rank
  "unavailable" instead of caching a zero rank.
- /api/reach-rank: q (<=64 chars, name/pubkey substring), offset (>=0),
  limit (1-100, default 50); search and paging keep global placements.

Frontend: new #/reach-rank page (search, 50-row pages, snapshot age,
"Historical neighbour count — not a measure of radio quality or range"),
and a "View leaderboard" link plus "Not ranked"/unavailable states on
the Reach Rank card. No nav changes.

Tests: Go unit/handler tests (ranking, visibility after warm cache,
consistency with /reach, cache hit/expiry, singleflight, canceled
waiter, cold/partial/refresh DB errors), benchmarks, a vm unit test,
a Playwright E2E, and /reach-rank in the axe gate; both new JS tests are
registered in deploy.yml.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
fix(reach): count only valid edges; refresh the rank snapshot in the …
20809fd 
…background

Independent review follow-ups for the Reach leaderboard:

- Neighbour count now uses valid edges only: both endpoints must be
  64-hex MeshCore pubkeys (case-insensitive) and differ; case and
  orientation variants of one pair count once. Legacy rows (e.g. the
  fixture's 55 empty-endpoint edges) are filtered at computation time,
  nothing is deleted. Applies to the shared snapshot, so the Reach
  page and the leaderboard stay identical (fixture #1: 9, not 10).
- Pubkeys are "known" exactly as buildNodeInfoMap knows them (a node
  row, or an observer row with id and name), so every placement has a
  Reach page; every node/observer name of a pubkey is checked against
  the hidden prefixes.
- Stale-while-revalidate: an expired snapshot is served while one
  background rebuild refreshes it, so Reach cache hits never wait on
  the aggregate; only a cold start waits. A failed rebuild backs off
  15s instead of re-querying on every request, and a panic in the
  load becomes an error instead of crashing the process (DoChan).
- View rebuilds after a visibility change are serialised so a burst
  builds one view.
- Reach page shows the Neighbours / Rank cards also for nodes without a
  reliable path token; the leaderboard clips queries by code point (no
  split emoji / URIError) and clamps absurd ?page values.
- Tests for all of the above, including the committed fixture; docs
  and OpenAPI describe the valid-edge rule and exact limit semantics.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
perf(reach): validate and count edges in Go instead of per-row SQL fu…
15026d2 
…nctions

The valid-edge rule as SQL (lower/GLOB + DISTINCT) made the snapshot
rebuild ~4x slower than master's plain GROUP BY (885 ms vs 223 ms on
20k nodes / 100k edges). Scan neighbor_edges once and do the checks in
Go: a row already in the form the ingestor writes (lower-case,
node_a < node_b) is unique by the primary key and counted directly;
other valid rows (upper-case or reversed) are canonicalised and
deduplicated against each other and against an existing canonical row
with one bulk row-value lookup. Same results (tests unchanged), and the
rebuild is now faster than master's degree-only query: 5.0 / 27 /
135 ms for 1k / 5k / 20k nodes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
fix(reach): review round 2 — deterministic refresh test, one read sna…
95e2cc3 
…pshot

- Background-refresh test: wait until the held load has started, check
  results on the test goroutine, and wait for the snapshot to publish so
  no goroutine outlives the test (was flaky at -cpu 1 and could race the
  next test's hook swap).
- A queued refresh re-checks the backoff instead of starting a second
  load; only real load attempts record failures.
- Stale requests don't each join the background refresh (in-flight flag).
- The edge scan, the odd-row lookup and the name queries run in one read
  transaction, so they see one database state.
- NULL endpoints are skipped instead of failing the load.
- Hidden-name check also uses a node's inactive_nodes name, but only
  while it has no nodes row (that table keeps a returning node's old
  row, so for an active node the name is stale) — same rule as the
  Reach visibility fix.
- Docs: which requests wait for a snapshot read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
fix(reach): inactive name counts unless the node has a named nodes row
f3919f1 
Match the Reach visibility rule: a hidden node that returns with a
nameless advert (stored as an empty name) keeps its hidden
inactive_nodes name instead of being ranked again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
fix(reach): drop the refresh in-flight flag that could stick
3b1a91a 
Review found that the stale-path in-flight flag could be set by a
request whose DoChan joined an already-finishing refresh after that
refresh had cleared the flag; nothing cleared it again, and the rank
snapshot stopped refreshing until restart. The flag only avoided a
bounded pile-up of joined channels during a slow refresh, so remove it:
stale requests start or join the singleflight refresh directly, and the
in-closure backoff re-check still prevents repeat loads after a failure.

Adds a liveness property test (expired snapshot always refreshes under
concurrent stale requests) and captures the stale pointer before
releasing the held load in the background-refresh test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
test(reach): refresh-liveness check keeps callers hitting the stale s…
13f4e61 
…napshot

Probe shape from review: each round, callers keep calling until the
refresh publishes; the round fails if the expired snapshot is never
replaced. Cheap (≈0.4s under -race); the removed flag's race window is
narrow, so this is a property check rather than a reliable reproduction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dborup dborup mentioned this pull request Sep 18, 2026
Merged
Dennis Jakobsen and others added 2 commits September 19, 2026 14:01
Dennis Jakobsen @claude Dennis Jakobsen
Merge origin/master (PR #68 privacy) into PR #69
130f9cc 
Integrates the merged Reach privacy/visibility work (#68,
identity_visibility.go) with the leaderboard's rank-view cache
(reach_rank.go). Real conflicts in cmd/server/node_reach.go and
docs/api-spec.md — both files touched the response cache and the
handler — resolved by combining both freshness dimensions on the
cache entry:

  - hiddenKey (#68): which identities the served body's links /
    direct_observers omit, from the live visibility check.
  - viewID (#69): which reachRankView the served body's rank fields
    were applied from.

reachBody now re-marshals only when either one moved on since the
entry was computed; both are re-checked on every serve, cached or
not, so neither a blacklist/prefix/rename change nor a rank-view
change waits for the 5-minute cache TTL.

computeNodeReach no longer sets the rank fields itself (unchanged
from #69's design) — the handler applies them from the shared rank
view (applyReachRank) after visibility filtering, so a node's own
NeighborDegree still counts a hidden neighbour's edge (a number, not
an identity) while DegreeRank/NodesWithEdges reflect only the
visible population.

Also fixes a parallel visibility rule found during review:
reachRankVisible in reach_rank.go reimplemented the same
IsBlacklisted/IsObserverBlacklisted/IsNameHidden combination as
identityHidden instead of calling it, so the two endpoints could
have drifted apart on a future change to the shared rule. It now
delegates directly.

Updated two of #69's own tests that asserted the pre-merge cache
API/contract (reachCacheSetBody's new viewID param;
NothingHiddenBodyUnchanged and FilteringLeavesRankFieldsUnchanged
now apply the rank view before comparing, and assert the agreed
'visible population' contract — NodesWithEdges/DegreeRank reflect
only visible nodes, NeighborDegree does not). Added
TestNodeReach_HiddenNeighbourCountedNotListedOrRanked (cross-endpoint:
a hidden neighbour is counted but never listed or ranked) and
TestReachRankVisible_MatchesIdentityHidden (equivalence probe against
identityHidden across blacklist/prefix/name-slot combinations), and
strengthened TestReachRank_OnlyValidEdgesCount with a self-edge that
exists only in non-canonical case form, closing a gap where the
self-edge skip could be removed without any existing test catching it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dennis Jakobsen @claude Dennis Jakobsen
bench(reach): add concurrent mixed-traffic benchmark
fc4ba5a 
BenchmarkReachAndRankParallel measures /api/reach-rank and
/api/nodes/{pk}/reach served concurrently (3:1) on a warm
snapshot/view/cache, at GOMAXPROCS parallelism. Confirms the shared
degree-snapshot/rank-view mutex is not a bottleneck under concurrent
readers — part of the performance verification for the PR #68 sync.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dborup
dborup marked this pull request as ready for review September 20, 2026 06:43
@dborup
dborup merged commit c0e7246 into master Sep 20, 2026
6 checks passed
@dborup dborup mentioned this pull request Oct 2, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dborup