Skip to content

perf(db): resolve packet endpoints at read time - #163

Merged
MrAlders0n merged 1 commit into
devfrom
perf/live-endpoint-resolution
Sep 25, 2026
Merged

MrAlders0n merged 1 commit into
devfrom
perf/live-endpoint-resolution

Conversation

@MrAlders0n

Copy link
Copy Markdown
Member

#128 saved a full snapshot of every endpoint candidate (id, name, key, location) on each observation. In hindsight that was the wrong choice. Ambiguous one-byte hashes match ~7 nodes on average, and the same list got copied for every observer and every packet. On prod it grew to about 22 GB, roughly 60% of packet_observations.

This drops the column and resolves source/destination at read time, same as path hops already do.

  • List pages (including scroll and WS reconnect backfill) resolve all their endpoints in one batch, two queries at most. Rows that used to show nothing now get endpoints too.
  • Packet detail resolves every observation in the same single batch instead of per observation.
  • Live WS events are unchanged.
  • No API shape change.

Tradeoff: old packets show today's node names rather than the ones at capture time. With 3 to 7 day retention that barely matters.

Migration 038 drops resolved_endpoints. Space comes back as old rows age out, or right away with VACUUM FULL packet_observations.

Storing a full candidate snapshot on every observation was the wrong call. The same node lists got copied thousands of times, about 22 GB on prod. Resolve endpoints live like path hops and drop the column.
@MrAlders0n
MrAlders0n requested a review from 446564 as a code owner September 25, 2026 17:44
@MrAlders0n
MrAlders0n merged commit 91b4b45 into dev Sep 25, 2026
3 checks passed
@MrAlders0n
MrAlders0n deleted the perf/live-endpoint-resolution branch September 25, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant