fix(reach): hide blacklisted and hidden identities on the Reach page

[dborup]

Summary

Two privacy gaps on the existing per-node Reach endpoint, GET /api/nodes/{pubkey}/reach. An independent review of the Reach leaderboard work found them; both predate it. This PR fixes them on its own, based on master without any leaderboard changes.

#69 (Reach leaderboard) depends on this PR and is to be merged after it: dborup/CoreScope#69.

Contract

Hidden identity. A pubkey's identity is hidden when any of these holds (identityHidden in cmd/server/identity_visibility.go):

• it is in nodeBlacklist (IsBlacklisted);
• it is in observerBlacklist (IsObserverBlacklisted);
• any of its names starts with a hiddenNamePrefixes entry (IsNameHidden). Its names are:
  • its nodes name;
  • its observers name;
  • its inactive_nodes name, but only while it has no nodes row with a non-empty name. The ingestor never deletes a returning node's inactive_nodes row, so an existing current name supersedes the old one. A nameless return, stored as an empty name, does not.

Rules for /api/nodes/{pubkey}/reach

1. Hidden target. The response is 404 {"error":"Not found"}, byte-identical to an unknown node. This is the existing per-pubkey contract used by node detail, neighbours and the rest.
2. Visible target. Hidden identities are removed from links and direct_observers. bidirectional_links and direct_observers count only what is listed. No other field names another identity: node, window and reliable_tokens describe the target itself.
3. Live names. Names are read from the DB on every request, cached report or not, in one bulk query that covers the target and every listed pubkey. Consequences:
   • Hiding applies on the next request, whether it comes from a blacklist change, a prefix change, or a rename of a node, observer or inactive row.
   • Un-hiding by renaming can take up to the 5-minute cache TTL, for a neighbour and for the target itself (for the target, plus the server's 30 s node cache). The name recorded when the report was computed still counts. That errs on the safe side.
4. Fail closed. If the name lookup fails, the response is 500 reach computation failed, never unfiltered data.
5. Rank/total unchanged. neighbor_degree, degree_rank, nodes_with_edges and relay_observations are counts over the whole graph and name no identity. They are not changed here, and a test asserts they are identical with and without hiding. feat(reach): Reach leaderboard with a shared visible-population rank #69 redefines them for Reach and the leaderboard from one snapshot.

Behaviour before and after

Case	master	this PR
Target in nodeBlacklist	404	404 (unchanged)
Target hidden by node name	404	404 (unchanged)
Target in observerBlacklist	200, full report	404
Target hidden only by its observer name	200, full report	404
Observer-only target with a hidden name (any id case)	200, full report	404
Target renamed hidden after its report was cached	404 (node name, live)	404 (node, observer or inactive name, live)
Hidden or blacklisted neighbour or direct observer of a visible node	listed with pubkey and name	removed; counts recomputed
Neighbour aged out into inactive_nodes with a hidden name	listed by pubkey	removed
Cached report, then a blacklist change	cache purged	cache purged, and filtered on serve
Cached report, then a prefix change or a neighbour rename	served unfiltered up to 5 min	filtered on the next request (same cache entry, no recompute)
DB error during the visibility name lookup	isPubkeyHidden failed open	500, fail closed
neighbor_degree / degree_rank / nodes_with_edges	whole graph	unchanged

Consistency with existing visibility rules

• Everything hidden elsewhere is hidden here. Existing handlers apply the node blacklist and hidden node names to nodes (node list, detail, search, neighbour graph). They apply the observer blacklist and hidden observer names to observers (RX dashboard, observers API).
• Reach applies all of these to every identity it lists, because a Reach link or direct observer can be either a node or an observer.
• Two additions are stricter than elsewhere:
  • the inactive_nodes name, which closes a real leak for aged-out hidden nodes;
  • case-insensitive observer-id matching, because ids arrive raw from the MQTT topic.
• The shared helper identityHidden is the rule the leaderboard in feat(reach): Reach leaderboard with a shared visible-population rank #69 will call.

Implementation

cmd/server/identity_visibility.go

• identityHidden is the rule itself.
• hiddenIdentityNames(ctx, pubkeys) is one bulk query per request that returns only names starting with an enforced prefix, so normally no rows:
  • The prefix test is byte-exact (substr/length on BLOB), equal to strings.HasPrefix in IsNameHidden. It is inlined once per prefix, and Go confirms each returned name with IsNameHidden.
  • nodes and inactive_nodes are joined by primary key in a single pass over the listed pubkeys; "no named nodes row" is COALESCE(n.name,'') = ''.
  • observers is read in one pass, matched case-insensitively.
  • inactive_nodes is probed first, so minimal DBs without it keep working.
  • The query uses the request context.
• isIdentityHidden is the same check for a single pubkey.

cmd/server/config.go

• One lazy hide-prefix helper is now shared by IsNameHidden and ActiveHiddenNamePrefixes, with unchanged behaviour.
• The new EnforcedHiddenNamePrefixes() returns exactly the prefixes IsNameHidden enforces. Without them, the name lookup and all per-identity name work are skipped.

cmd/server/node_reach.go

• The cache entry holds the computed report, and visibleReach filters it on every serve.
• The last body is reused while the set of hidden identities is unchanged (hiddenKey). reachCacheSetBody only attaches a body to the entry it was computed for.
• Blacklists are checked before the cache.
• The target's live names are checked on a cache miss before the scan, and again on every serve.

Tests

All results below are from feature head eb4afecd, the last code commit. The later master merge 6ab3103f changes no file of this PR; see Base, head and CI.

Suites and static checks

• cd cmd/server && go test -race ./...: ok (307 s).
• cd cmd/ingestor && go test ./...: ok (88.7 s).
• go vet ./... clean; gofmt clean on every changed .go file; git diff --check clean.

Targeted tests (cmd/server/node_reach_visibility_test.go)

Test	Covers
HiddenTargets404	Hidden by node name, node-blacklisted, observer-blacklisted, observer-name-only hidden, observer-only hidden, observer-only observer-blacklisted, upper-case id form. Each returns the unknown-node 404 body. Visible node and observer targets return 200.
HiddenNeighboursFiltered	Exact expected links and direct_observers sets and recounted counts. No hidden pubkey or name anywhere in the body, including an inactive-only hidden node and blacklisted identities.
VisibleControlsNotOverfiltered	Over-filtering controls: a name that merely contains 🚫, and a node visible under both its node and observer name, keep being listed and keep their own page.
WarmCacheHonoursVisibilityChanges	On the next request, in both directions: node rename, observer rename (including a mixed-case id), prefix change, blacklist change, and the target renamed after being cached.
CacheHitMissAndInvalidation	A cache hit reuses the report. A prefix change filters the same cached entry at serve time, with no recompute. A blacklist change purges and recomputes.
FilteringLeavesRankFieldsUnchanged	neighbor_degree, degree_rank and nodes_with_edges are identical with and without hiding.
NameLookupFailureFailsClosed	A lookup failure returns 500 without report data.
RecordedNameStillHidesAfterRowDeleted	The name recorded in the report still hides after the row is deleted.
WhitespacePrefixStillLooksUpNames	A whitespace-only prefix still triggers the name lookup.
NoInactiveNodesTable	Works when the DB has no inactive_nodes table.
StaleInactiveNameDoesNotHideReturnedNode	A returned node with a name is not hidden by its old inactive row; a nameless return stays hidden.
NothingHiddenBodyUnchanged	Full deep comparison with the computed report, and no DB access without prefixes.
HiddenIdentityNames_LargeBatchOnlyHidingNames	A 1000+ key batch returns only hiding names.
HiddenIdentityNames_PrefixMatchIsExact	The SQL prefix test matches IsNameHidden: case, multi-byte, exact-length, shorter, not-at-start.
ReachCacheSetBody_StaleEntryUntouched	A body is never attached to a newer cache entry.

Negative controls. Each of these mutations was applied to the final code in a scratch copy. Every one makes at least one of the tests above fail:

Mutation	Tests that fail
Target check removed	HiddenTargets404, WarmCache…, BlacklistMutationBustsCache
List filter removed	7 tests, including HiddenNeighboursFiltered and CacheHitMissAndInvalidation
Cached body reused without re-check	WarmCache…, CacheHitMiss…, WhitespacePrefix…, StaleInactive…
Observer blacklist dropped from the rule	HiddenTargets404, HiddenNeighboursFiltered
Observer name lookup removed	5 tests
inactive_nodes lookup removed	HiddenNeighboursFiltered, LargeBatch…, StaleInactive…
"Named nodes row" guard removed	StaleInactive…
Fail-open on lookup error	NameLookupFailureFailsClosed
Recorded-name check removed	RecordedNameStillHides…
SQL prefix test never matches	7 tests
SQL prefix test always matches	PrefixMatchIsExact, LargeBatch…
at guard removed	ReachCacheSetBody_StaleEntryUntouched

Linux. GitHub CI (ubuntu-latest) is the Linux verification; see CI below. A local Linux container was not practical: the Docker daemon isn't running here, and a Go image would have to be downloaded.

Performance

BenchmarkNodeReachCacheHitVisibility (50 links, 5 direct observers) and …Scale measure a Reach cache hit.

• Scale datasets:
  • hub: 150 links, 100 direct observers, 200 observers, 3000 nodes;
  • large: 300 links, 100 direct observers, 2000 observers, 5000 nodes.
• Common setup: file-backed DB, including inactive_nodes with aged-out rows.
• How measured: Apple M2 Pro, Go 1.26.0, modernc.org/sqlite v1.34.5, load average about 3–4. Values are medians of -count=5 -benchmem, from the same benchmark file run on master and at head eb4afecd.

Reach cache hit	master	this PR	factor
50 links, hidden-name prefix configured	33.6 µs	101.7 µs	×3.0
hub, prefix configured	38.2 µs	458 µs	×12
large, prefix configured	39.4 µs	873 µs	×22
no prefix configured (all three sizes)	13.5–19.2 µs	14.1–24.0 µs	×1.0–1.2

Why it costs more with a prefix. It is the live per-request name lookup, which is what makes a rename apply on the next request instead of after the 5-minute cache TTL. It costs roughly 1 µs per listed identity plus about 0.13 µs per observers row. config.example.json ships hiddenNamePrefixes: ["🚫"], so deployments using it take this path.

What the review-driven optimisation bought. The first version returned every name of every listed identity. At the hub size it took 1.7 ms, and 5.0 ms at 2000 observers. Changes since then:

• the prefix filter moved into SQL;
• the prefix test is inlined per prefix;
• the nodes and inactive_nodes lookups are merged into one pass;
• the no-prefix fast path skips the lookup and per-identity lower-casing.

Query variants (interleaved A/B, same process):

Variant	Hub	Large
Separate inactive branch	518 µs	985 µs
Merged pass (chosen)	460 µs	883 µs

Decision: the cost is accepted. The measured ~102–873 µs per cache hit with a prefix configured is accepted in exchange for immediate, fail-closed privacy. There is no extra name snapshot and no further optimisation in this PR.

Alternative not taken: a short-TTL shared name snapshot would make hits roughly master-speed. Renames would then take effect within that TTL instead of on the next request.

Review

A separate reviewer agent, which did not write the code, reviewed this PR in four rounds. It read the commits through git and ran its own probes and mutations in isolated copies.

1. Rounds 1–2 (be05d38e, 783ca759): no blockers. Fixed:
   • leaks through inactive and mixed-case observers;
   • the whitespace-prefix gate;
   • the recorded-name test;
   • docs.
2. Final review of the whole diff (25986c9c): no BLOCKER. All seven requirements hold, and there is no path that serves unfiltered data. That covers cache hits, singleflight waiters, error paths and body reuse; a concurrent churn probe under -race found 0 leaks.
   • SHOULD-FIX: the cache-hit cost at realistic sizes. Addressed in 9c8352e8, b4de6b27 and eb4afecd; see Performance.
   • SHOULD-FIX: other endpoints still use the weaker rule. Deliberately left as a follow-up.
   • NITs fixed: the at-guard test, request context, deep-compare test, docs, prefix helper.
   • NITs noted: one redundant lookup on a cache miss; non-ASCII observer ids.
3. Confirmation rounds (9c8352e8, then eb4afecd): no BLOCKER.
   • The SQL prefix filter matched IsNameHidden with 0 mismatches over 2880 identities and 32 prefix sets. The inputs covered NULL, empty, whitespace, multi-byte, invalid UTF-8 and NUL, with the merged nodes/inactive_nodes logic.
   • The reviewer ran 5 more mutations of its own, and each one failed tests.

Base, head and CI

Synced with master. master was merged into this branch with an ordinary merge commit; there was no rebase and no force-push.

• Merge commit: 6ab3103f
  • parent 1: the feature head eb4afecd, the last code commit;
  • parent 2: master 8b9b9d60, after the merges of fork PRs 28 and 50.
• Result tree: 10530a0e. It is identical to the merge tree precomputed read-only with git merge-tree before merging.
• No conflicts and no overlapping files. Measured against master, the diff is still exactly the 7 files of this PR, and the patch is byte-identical to 37d82e8a..eb4afecd.
• Master's changes are carried over byte-identical:
  • the RX-coverage icon and rainbow channel names (from PRs 28 and 50);
  • the Packets row clamp from fix(packets): keep the Details summary on one line so rows stay bounded #67, including its test registration.
• Workflows unchanged. The .github/ tree is identical to master's, including triggers, permissions, jobs, needs and fork guards.
• Re-checked locally on the merge commit:
  • gofmt and go vet (server, ingestor) are clean, and so is git diff --check;
  • the 68 Reach/privacy/visibility/blacklist tests pass under -race -count=3;
  • cd cmd/server && go test -race ./... is ok (337 s).

CI run 35437448578: ✅ success. It ran on head 6ab3103f as a pull_request run. The checked-out merge ref 4a4efd72 (master 8b9b9d60 + 6ab3103f) has the same tree, 10530a0e.

Job	Result
Go Build & Test	✅ server (includes the Reach/privacy tests), ingestor, channel library, decrypt CLI, JS unit list
Playwright E2E	✅ 677 checks passed, 0 failed
Build & Publish Docker Image	✅ local image build only; GHCR login and push skipped (fork guard)
Release Artifacts	skipped (tags only, fork guard)
Deploy Staging	skipped (master push only, fork guard)
Publish Badges & Summary	skipped (fork guard)

E2E tests of interest:

• test-channel-color-picker-e2e.js: 9/9, including "ArrowRight cycles focus across swatches".
• test-issue-1630-reach-mobile-e2e.js: 7/7.
• Packets row height:
  • the test-issue-1122-packets-filter-ux-e2e.js suite passes 6/6, including "Path column row height stays bounded < 60px";
  • test-issue-1122-details-row-clamp-e2e.js passes 18/18.
• axe gate: 0 violations over 116 cells.

Side effects. Nothing was deployed, pushed to GHCR, released or committed as badges. Coverage badges were only uploaded as CI artifacts.

Previous run (35424296968, head eb4afecd, against master 31c2aa8e): Go passed. E2E failed on one assertion in the unchanged test-channel-color-picker-e2e.js, "ArrowRight cycles focus across swatches". It was not re-run; this fresh run on the synced head replaces it, and that test passes there.

Known limitations

• Per-request name lookup. With hidden prefixes configured, every Reach request, including cache hits, runs one bulk name query (see Performance).
• Un-hiding by rename can take up to the cache TTL (see contract point 3).
• Old hidden names over-hide in one case (fails safe). A pubkey that is now only an observer stays hidden by an old hidden inactive_nodes name.
• Other endpoints are out of scope; a follow-up is planned.
  • /api/nodes/{pk}/neighbors does not filter its per-neighbour entries, so a visible node's neighbour list can still show hidden or blacklisted neighbours. This was found by reading the code in review.
  • /api/nodes/{pk} and /api/observers/{id} still show observer-blacklisted pubkeys and nodes hidden only by their observer name. A node detail page can therefore offer a Reach link that now returns 404.
  • /api/nodes/{pk}/rx-coverage (opt-in client RX coverage) checks only the node blacklist and node name for its target.
• Nameless adverts un-hide active nodes (ingestor, follow-up). A still-active node whose hidden name is overwritten by a nameless advert becomes visible everywhere.
• Redundant lookup on a cache miss. The requesting call does one extra name lookup (NIT; misses are dominated by the scan).
• Timing. A hidden-by-name target 404s faster than an unknown one, because unknown targets go through the scan. Master behaves the same, and a prober needs the full pubkey.
• Observer blacklist changes need a restart. The observer blacklist is static config; there is no runtime setter.

Not tested

Staging and production were not tested. Validation used local Go tests and a local server on the committed CI e2e fixture DB, plus GitHub CI on Linux (ubuntu-latest). Nothing was deployed.

🤖 Generated with Claude Code