[dborup-agent]

Relates to #411

Problem

Every StoreTx carried two maps for observation dedup from the moment it was created: obsKeys, keyed by a concatenated observerID + "|" + pathJSON string that was allocated per observation, and observerSet for UniqueObserverCount. Most transmissions are heard a handful of times, so these maps and their keys cost about 1 KB per transmission to deduplicate lists that are usually under ten long.

Plan (one change, two commits)

1. Measure first (below): the observation-count distribution and the heap/allocs before any change.
2. Red commit test(store): …: the equivalence tests plus a stub obs_dedup.go that keeps today's eager maps behind the new helpers. They fail only on "a transmission below the threshold holds maps" and on the estimate terms.
3. Green commit perf(store): …: the implementation.

Current master was checked first. The Regions extras are gone (#523/#524). #495 (no SQL under PacketStore.mu), #496 and #511 changed the load paths, but none of them touches the dedup sites. Kpa Kpa-clawbot#2181 was used as inspiration only: our fork additionally has the background fill, the in-memory hash merge (#215) and exact trackedBytes accounting (#113/#164/#202), and all three are handled here.

Measurements before choosing the threshold

test-fixtures/e2e-fixture.db: 499 transmissions and 500 observations, i.e. 1 per transmission (one has 2). It is too small to choose anything from, so the synthetic set below is the basis.

Synthetic 30K set (obs411Data, obs_dedup_411_data_test.go): log-normal rows per transmission (median 4, σ 0.75), 48 64-hex observers, one of 3 paths per row, about 1 % unknown observers. This produces exact repeats and a second path from the same observer naturally.

	value
rows / kept after dedup	159,365 / 154,923
distinct observer/path pairs per tx: mean / p50 / p90 / p99 / p99.9 / max	5.16 / 4 / 10 / 21 / 36 / 75
tx with 1–8 / 9–16 / 17–32 / 33–64 / >64 pairs	25,516 / 3,730 / 704 / 47 / 3

For comparison, Kpa Kpa-clawbot#2181 reported a mean of 6.25 and a max of 40 on a real 7-day window. That is the same shape, and the synthetic set has the heavier tail.

Per-check cost (BenchmarkObsDedup, a check that misses, which is the ingest common case; median of 3). Adversarial means one observer on k equal-length paths that differ only at the end, the worst case for the scan.

k	scan (realistic)	scan (adversarial)	old concat+map lookup
4	19 ns, 0 allocs	36 ns	58–83 ns, 1 alloc (80–96 B)
16	62 ns	108 ns	55–61 ns, 1 alloc
31	129 ns	179 ns	62–82 ns, 1 alloc
≥ 32 (map, struct key)	24 ns	24–33 ns	50–74 ns, 1 alloc

Threshold: obsIndexThreshold = 32.

• Below 32, a check costs at most about 180 ns in the adversarial case. That is 2–3× the old lookup, which allocated on every call.
• Building a transmission up to 31 observations costs a few µs in total.
• On the light synthetic set only about 0.2 % of transmissions (p99.9 = 36) build the maps. With the production distribution (obsPerTx about 18: median 15, p90 35, p99 69) about 13 % build them, and the heap still drops by about 19 %. A threshold of 64 measured about 5 % less heap on top of that; it is a possible follow-up. The threshold stays 32.
• A lower threshold (16) would give maps to about 15 % of transmissions. A higher one (64) would double the worst-case scan for 3 transmissions in 30K.

Change

• obs_dedup.go:
  • lookupObservation(observerID, pathJSON) (dup, knownObserver) answers both questions in one pass: a scan below the threshold, and from the threshold on a map keyed by obsDedupKey{observerID, pathJSON}, which shares the observations' strings, so no key is allocated.
  • addObservation(obs, knownObserver) keeps Observations, ObservationCount, UniqueObserverCount and the maps in step, builds the maps when the list reaches 32, and returns the change in obsIndexBytes.
• All six places that added an observation now go through these two helpers and no longer copy the counter updates: hot Load, scanAndMergeChunk (LoadChunked), loadChunkWhere (background fill), IngestNewFromDB, IngestNewObservations and moveObservations (hash merge).
• The observation identity contract (observer ID + path_json) is unchanged.
• trackedBytes / maxMemoryMB:
  • The perTxMapsBytes (200 B/tx) and per-observation key terms (obsKeyEntryBytes + observer + path) are removed, because nothing allocates them any more.
  • A busy transmission's maps are charged by obsIndexBytes, a function of the map sizes alone: 96 B + 88 B per key + 40 B per observer, measured on Go swiss maps with 32–600 entries. The charge is added as a delta at every insert and credited through txChargedBytes, following the same pattern as resolvedRelayBytes, so eviction stays exact.
• StoreTx stays 320 bytes. The two fields keep their place and are simply nil below the threshold.

Not changed: cmd/server stays read-only, there are no new map[string]interface{}, no frontend or config changes, and no workflow changes.

Equivalence tests (obs_dedup_411_test.go)

One skewed set is used: 600 ordinary transmissions, half of them 3 days old, plus 3 busy ones with 600 distinct rows and 40 exact repeats past the threshold. It goes through five fill paths: hot Load, LoadChunked, background fill (LoadChunked plus loadChunk), ingest (IngestNewFromDB from empty), and upsert (first row via IngestNewFromDB, the rest via IngestNewObservations). Every path is checked against the rows themselves and against hot Load.

Requirement	Test	Red before	Mutants killed
Duplicate detection	PathsKeepTheSameObservationsAndIndexes, ThresholdAndAnswers, TestObsDedupCorrectness	yes (maps below threshold)	M1, M4, M7, M8
Unique observer counts	same, plus HashMergeDedupsAcrossTheThreshold	yes	M2, M9, M10
Same-row route updates (#89 change log)	SameRowRouteUpdateKeepsObservations	yes	M1, M3, M10
Resolved relay indexes (byPathHop, byNode, pathHopResolved, fallbackByNode)	PathsKeepTheSameObservationsAndIndexes, RebuildKeepsIndexesAndCharges	yes	M11
trackedBytes exact + identical across paths	all of the above via acct113Check (now including obsIndexBytes), EstimateHasNoEagerMapTerms	yes	M5, M6
Eviction (retention) back to exactly 0	EvictionCreditsWhatEachPathCharged	yes	M5, M6
maxMemoryMB evicts the same tx on every path	MemoryBudgetEvictsTheSameOnEveryPath	yes	M5
Rebuild (buildPathHopIndex) changes nothing	RebuildKeepsIndexesAndCharges	yes	M4, M6, M7
Hash merge below / across the threshold	HashMergeDedupsAcrossTheThreshold	yes	M1, M3, M5, M8
No allocation per check below threshold	ThresholdAndAnswers (AllocsPerRun == 0)	yes	—

On the red commit, the equivalence halves (counts, keys, relay indexes, trackedBytes across paths) already pass. Master was equivalent across paths, and the red comes from the structural assertions and the estimate terms. The mutants show that the equivalence halves have teeth. All 11 mutants were killed; the details are in the report comment.

Existing tests that pinned the removed estimate terms or read observerSet directly are updated: eviction_test.go, tracked_bytes_accounting_113_test.go and hash_migrate_readonly_215_test.go. obs_dedup_test.go now goes through the helpers and holds BenchmarkObsDedup.

Benchmarks (before = red commit, which runs master's store code; -count=6 -benchtime=5x, benchstat)

benchmark	sec/op	B/op	allocs/op
Load_Skewed30K_411	2.680 s → 2.450 s (−8.6 %, p=0.015)	385.1 Mi → 352.0 Mi (−8.6 %)	9.226 M → 8.918 M (−3.3 %)
Load_30K_Unlimited	1.044 s → 0.946 s (−9.4 %, p=0.004)	150.5 Mi → 134.4 Mi (−10.7 %)	4.796 M → 4.586 M (−4.4 %)
IngestNewFromDB_158	~ (p=0.82)	−2.2 %	−1.8 %
IngestNewObservations_158	~ (p=0.70)	−0.3 %	−1.0 %
TrackedBytes_IngestNewTx_113	~	−4.9 %	−3.0 %
TrackedBytes_IngestNewObs_113	~	−0.5 %	−0.6 %
TrackedBytes_Evict_113	~ (p=0.065)	~	~

Heap after load (Load_Skewed30K_411 heapMiB metric and CORESCOPE_OBS411_MEASURE=30000, 3 runs each): 145.4 MiB → 115.2 MiB (−20.8 %), 5,082 → 4,026 B/tx. trackedBytes went from 149.0 to 122.5 MiB, so tracked/heap moved from 1.03 to 1.06. On the data of TestTrackedBytesTracksTheHeap_113 the ratio moved from 1.06 to 1.13. That test's bound is 0.85–1.15, and it passes. The heap dropped by more than the removed estimate terms, so trackedBytes now over-charges slightly, which means eviction triggers a little earlier, never later. I left storeTxBaseBytes unchanged on purpose, because recalibrating it would change budget enforcement for every transmission. That makes it a candidate follow-up (see below).

Perf justification (AGENTS.md rule 0):

• Below the threshold, a check is O(k) with k < 32, so it is bounded per transmission.
• From the threshold on, a check is O(1).
• There is no work proportional to the number of transmissions, nothing new under the lock beyond what each insert already did, and no new unbounded structures (the maps are freed with their transmission).

Validation (local)

• cd cmd/server && go test ./...: ok
• cd cmd/ingestor && go test ./...: ok
• go test -race on the store/load/eviction/ingest/hash-migrate/route-mask tests: ok, no races
• sh test-all.sh: 259/259
• node test-frontend-helpers.js: 719 passed
• Playwright against a local Go server, using a copy of e2e-fixture.db prepared as in CI (freshen, the Packets page collapse button in the left column of the table opens the dialog. Kpa-clawbot/CoreScope#1486/"Group Data" message type missing from packet view window's "message type" filter Kpa-clawbot/CoreScope#1791 seed, migrate, the four seed SQL files):
  • test-e2e-playwright.js: 154/157 passed, 3 skipped
  • 16 E2Es that touch observation/observer counts or Heard By all pass, including test-issue-76-heard-by-e2e.js, test-issue-165-grouped-hop-warn-e2e.js and test-packet-detail-sender-hash-size-obs-e2e.js
• scripts/check-xss-sinks.sh --diff origin/master: no frontend changes
• Fork guards unchanged: 9 in deploy.yml, 1 in release-fast-path.yml

No staging or production access was used; heap on staging is left to the coordinator.

Follow-up

• Possible recalibration of the trackedBytes base terms. The ratio to the heap is now 1.06–1.13 (it was 1.03–1.06), conservative but closer to the 1.15 bound of TestTrackedBytesTracksTheHeap_113.
• obsIndexThreshold is a code constant, not a config knob. It is a memory/CPU trade-off rather than a user-facing value, so there are no customizer implications.

🤖 Generated with Claude Code

[dborup-agent]

Rapport — CS-pve-agent1 PR#527 #411 — head e0f7eb7

Status: Draft PR implements #411. All requirements are covered by red→green tests with mutants. CI is green on every job with no reruns. Not merged and not ready, by design.

Legend: [T] = test evidence (run locally or in CI), [A] = analysis/measurement, [K] = known limitation or residual.

Commits

Commit	Content
6ccdc336	test (red): equivalence tests plus a stub obs_dedup.go that keeps today's eager maps
e0f7eb77	perf (green): scan below obsIndexThreshold = 32, struct-keyed maps from 32 on, exact obsIndexBytes charge

Requirements

Requirement	Test(s)	Red on 6ccdc336	Green on e0f7eb77	Mutants killed
Measure first: distribution plus heap/allocs before the change	TestObsDedup411_Measure, BenchmarkLoad_Skewed30K_411, BenchmarkObsDedup	n/a	[A] fixture: 499 tx / 500 obs, 1 per tx. Synthetic 30K: mean 5.16, p50 4, p99 21, p99.9 36, max 75. Before: 145.4 MiB heap after load	n/a
Threshold justified by data	BenchmarkObsDedup (k = 4…600, realistic and adversarial)	n/a	[A] 32: about 0.2 % of tx build maps; worst-case scan ≤ ~180 ns; ≥ 32 is a 24 ns map lookup	n/a
Duplicate detection identical on all paths	PathsKeepTheSameObservationsAndIndexes, ThresholdAndAnswers, TestObsDedupCorrectness	[T] fail	[T] pass	M1, M4, M7, M8
Unique observer counts identical	the tests above plus HashMergeDedupsAcrossTheThreshold	[T] fail	[T] pass	M2, M9, M10
Same-row route updates (#89 change log)	SameRowRouteUpdateKeepsObservations (below and past the threshold)	[T] fail	[T] pass	M1, M3, M10
Resolved relay indexes (byPathHop, byNode, pathHopResolved, fallbackByNode)	PathsKeepTheSameObservationsAndIndexes, RebuildKeepsIndexesAndCharges	[T] fail	[T] pass	M11
trackedBytes exact and identical across hot / LoadChunked / background / ingest / upsert	all of the above via acct113Check (now including obsIndexBytes), EstimateHasNoEagerMapTerms	[T] fail	[T] pass	M5, M6
Eviction (retention) back to exactly 0	EvictionCreditsWhatEachPathCharged	[T] fail	[T] pass	M5, M6, M7
maxMemoryMB evicts the same tx on every path	MemoryBudgetEvictsTheSameOnEveryPath	[T] fail	[T] pass	M5, M6
Rebuild (buildPathHopIndex) changes nothing	RebuildKeepsIndexesAndCharges	[T] fail	[T] pass	M4, M6, M7
Hash merge (#215) below and across the threshold	HashMergeDedupsAcrossTheThreshold, TestHashMigrate_MergesDuplicatesInMemory_215	[T] fail	[T] pass	M1, M3, M5, M8
No allocation per check below the threshold	ThresholdAndAnswers (AllocsPerRun == 0)	[T] fail	[T] pass	M3
-race on the store/load/eviction tests	go test -race -run 'ObsDedup|TrackedBytes|Evict|HashMigrate|Load|RouteMask|Ingest|Chunk|Startup'	n/a	[T] ok (530 s), no races	n/a
cmd/server read-only, no new map[string]interface{}	readonly_invariant_test.go in the full suite; grep of the added lines	n/a	[T] pass; [A] 0 added	n/a
No hardcoded colours / XSS gate	scripts/check-xss-sinks.sh --diff origin/master	n/a	[T] no frontend changes	n/a
Fork guards unchanged	grep	n/a	[A] 9 in deploy.yml, 1 in release-fast-path.yml; no workflow changes	n/a

On the red commit the equivalence halves (keys, counts, relay indexes, trackedBytes across paths) already pass, because master was equivalent across paths. The red comes from the structural assertions (no maps below the threshold) and the estimate terms. The mutants show that the equivalence halves have teeth. [A]

Mutants (each applied alone on e0f7eb77, then reverted)

#	Mutant	Result
M1	scan ignores the path	killed (9 tests)
M2	an unknown observer is counted	killed (5)
M3	maps built from the first observation	killed (8)
M4	maps not updated past the threshold	killed (6)
M5	txChargedBytes omits obsIndexBytes	killed (3)
M6	background fill drops the map charge	killed (4)
M7	IngestNewObservations admits duplicates	killed (4)
M8	hash merge keeps shared pairs twice	killed (2)
M9	observerSet built with the empty observer	killed (3)
M10	map lookup ignores observerSet	killed (7)
M11	ingest skips relay-hop indexing for later observations	killed (2)

Benchmarks (before = red commit, which runs master's store code; benchstat, n=6) [T]

Benchmark	sec/op	B/op	allocs/op
Load_Skewed30K_411	−8.6 % (p=0.015)	−8.6 %	−3.3 %
Load_30K_Unlimited	−9.4 % (p=0.004)	−10.7 %	−4.4 %
IngestNewFromDB_158 / IngestNewObservations_158	~	−2.2 % / −0.3 %	−1.8 % / −1.0 %
heap after hot load, 30K skewed		145.4 → 115.2 MiB (−20.8 %)

Local runs [T]

• cmd/server go test ./...: ok
• cmd/ingestor go test ./...: ok
• sh test-all.sh: 259/259
• node test-frontend-helpers.js: 719 passed
• Local Go server with e2e-fixture.db prepared as in CI (freshen, the Packets page collapse button in the left column of the table opens the dialog. Kpa-clawbot/CoreScope#1486/"Group Data" message type missing from packet view window's "message type" filter Kpa-clawbot/CoreScope#1791 seed, migrate, the four seed SQL files):
  • test-e2e-playwright.js: 154/157 passed, 3 skipped
  • 16 E2Es that touch observation or observer counts / Heard By: all pass
• The server was stopped by pid afterwards.

CI per job (head e0f7eb77) [T]

Job	Result
Go Server Tests (1/3, 2/3, 3/3)	pass
Go Ingestor Tests	pass
Go Libraries & CLI	pass
Go Build & Test	pass
JavaScript Unit Tests & Lint	pass
Script & Container Unit Checks	pass
Playwright E2E Tests (1/3, 2/3, 3/3)	pass
Verify E2E Shards & Coverage	pass
Verify Go Test Assignments	pass
Validate Docker Image	pass
Build & Publish Docker Image	pass
Publish Badges & Summary / Release Artifacts / Deploy Staging	skipping (PR)

The known flaky test (#271) did not fail, so no rerun was needed.

Residuals

• [K] tracked/heap ratio. trackedBytes now over-charges slightly: tracked/heap is 1.06 on the skewed set (was 1.03) and 1.13 on the data of TestTrackedBytesTracksTheHeap_113 (was 1.06; the bound is 1.15). This is conservative, because eviction triggers earlier, never later. Recalibrating storeTxBaseBytes was left out on purpose, because it would change budget enforcement for every transmission.
• [K] The fixture DB has only 1 observation per tx, so the threshold rests on the synthetic set plus the upstream-reported real shape. A heap measurement on staging is left to the coordinator.
• [K] obsIndexThreshold is a code constant, so there are no customizer implications.
• [K] perf(store): evaluate on-demand subpath counts instead of permanent indexes #413 had not merged when this branch was cut from 0350322e, so no sync was needed. If it merges first, the overlap in store.go is limited to the six dedup sites and the estimate constants.
• [K] A local Go build cache got corrupted once ("bad reloc size") during the benchmarking. A fresh GOCACHE fixed it; CI builds clean.

[dborup-agent]

Review — CS-pve-agent3 PR#527 — head e0f7eb7

Dom: APPROVE med nits

Legend: [T] = test or benchmark run by me, [A] = analysis or code reading, [K] = known limitation or not verified.

Independent, read-only review. Everything ran on git archive copies in a scratch directory: the PR head, the red commit 6ccdc336, origin/master dcc68979, and the merged tree git merge-tree --write-tree origin/master e0f7eb77… (tree 0c63f5c3, clean). The PR head was e0f7eb77 both before and after the review (git ls-remote). During the review, master moved to e14cac27 (#525/#526: channels frontend, openapi text, one channel test). It does not touch the store code, and the merge with it is still clean [A].

Findings

#	Severity	Finding	Evidence
F1	Should fix (test only)	Mutant MX1 survives. It clears a hash-merge loser's obsKeys/observerSet in finishHashMerge before s.trackedBytes -= s.txChargedBytes(l). The code on the head has the right order: credit at hash_migrate.go:446, clear at :466. No test pins that order, because in TestObsDedup411_HashMergeDedupsAcrossTheThreshold each duplicate row has at most 28 observations, so a loser never holds maps. With a heavier distribution about 13 % of tx hold maps (see Q1), so a loser past the threshold is realistic. A scratch-only extra case {"rows already past", 9} (36 per row) passes on the head and kills MX1: after the merge: trackedBytes = 25947, charges of the live store sum to 22523 (drift 3424), and the drift survives the eviction. Suggest adding that case.	[T]
F2	Nit	The threshold rationale rests on the light synthetic set. The obs_dedup.go comment and the PR body say that "about 0.2 % of transmissions reach 32". With the production mean (obsPerTx 18.1) the share is 13.1 %, and threshold 64 measured another −5 % heap at no measurable Load cost (Q1). 32 is still correct and still saves about 19 %, so this does not block. Either update the comment and the claim, or consider 64 in a follow-up.	[T] [A]
F3	Info	#529 (perf/413-subpath-on-demand @ cb8ad366) conflicts after this PR, in comments only. There are 3 comment hunks in store.go (the Kpa-clawbot#743 constants comment and the estimateStoreTxBytes doc) and 1 comment line in eviction_test.go (TestEstimateStoreTxBytes). Resolved in favour of #527 (plus "removed by #413" on the spTxIndex line), the combined tree passes go vet including the tests. No code hunk conflicts.	[T]
F4	Info	trackedBytes now sits slightly closer to the heap, and in the same direction on both distributions (see Q3). That is not a regression.	[T]

Answers to the review points

Q1. Threshold vs a prod-like distribution. The public production /api/perf?mem=1 aggregates show obsPerTx 18.1 (≈40K tx, ≈724K observations) [T]. They give only the mean, so the tail is assumed [K]. I made the author's generator env-configurable in scratch (median 15, σ 0.7, 128 observers, 3 paths). It gives distinct pairs per tx of mean 18.38, p50 15, p90 35, p99 69, p99.9 113, max 184, on 30K tx with 551,485 kept observations. Heap is from TestObsDedup411_Measure, 3 runs each, identical to ±0.2 MiB [T]:

tree	tx with maps	heap after load	B/tx	trackedBytes	tracked/heap
master dcc68979	100 %	385.1 MiB	13,460	357.0 MiB	0.927
PR (threshold 32)	13.1 % (3,928)	310.4 MiB (−19.4 %)	10,850	298.1 MiB	0.960
threshold 16 (scratch)	46.6 %	335.5 MiB	11,728	325.3 MiB	0.970
threshold 64 (scratch)	1.5 %	294.4 MiB (−23.6 %)	10,290	280.8 MiB	0.954
no maps at all (scratch)	0 %	290.7 MiB	10,160	276.7 MiB	0.952

• BenchmarkLoad_Skewed30K_411 with the same distribution (n=4, 3x): master 7.36 s, PR 7.31 s, 64 7.22 s, no maps 7.30 s. The spread is about ±1 %, so Load is SQLite-bound and the threshold is not measurable there. Allocations: 31.34 M → 30.48 M per Load [T].
• Scan cost without maps (BenchmarkObsDedup): realistic k=31/64/128 is 83/173/336 ns, adversarial 150/288/560 ns, 0 allocs [T].
• So the saving holds under the heavier distribution: −19.4 % heap vs the author's −20.8 % on the light set. But the share of tx with maps is 13 %, not 0.2 % (F2).

Q2. Worst case of the scan on hot paths. lookupObservation has exactly six call sites [A]:

• once per scanned row in Load, scanAndMergeChunk, loadChunkWhere, IngestNewFromDB and IngestNewObservations;
• once per loser observation in moveObservations.

Each call is O(min(k, 31)), and from 32 on it is O(1). The worst case is therefore O(31²) per tx while it is being built, and O(k_loser · 31) per hash merge. No path calls it in a loop over all observations of all tx. buildPathHopIndex (rebuild), eviction and the analytics scans do not call it. Under s.mu (ingest, hash merge) a check costs at most about 150 ns adversarial (Q1 numbers) and never allocates. addObservation is the only append to tx.Observations in cmd/server. No code removes single observations from a live tx or mutates ObserverID/PathJSON after insert (the only reset is the loser cleanup), so the scan and the maps cannot disagree with master's eager maps [A].

Q3. trackedBytes / maxMemoryMB (obsIndexBytes).

• Accuracy: on the heavy set the maps cost 19.8 MiB of real heap (PR 310.4 − no-maps 290.7) and are charged 21.4 MiB (298.1 − 276.7), so obsIndexBytes is about 8 % conservative. At threshold 64 it is 3.7 vs 4.1 MiB [T].
• Exactness: the charge is added as a delta at every insert and credited through txChargedBytes on eviction (store.go:5758/5828/5878) and in the hash merge (credit before clear, F1). The maps only grow while the tx is live [A]. acct113Check plus the PR tests hold this exact on all five fill paths [T].
• Compared with master, tracked/heap rises by about 3 points on both distributions: heavy 0.927 → 0.960, author's light set 1.03 → 1.06. In real-heap terms, eviction therefore triggers at a slightly lower heap per budgeted MB than on master, never higher. In transmissions, the same budget holds about 20 % more tx, which is the point of the change (heavy set: ≈151K → ≈181K tx per 1800 MB) [T] [A]. TestTrackedBytesTracksTheHeap_113 (0.85–1.15) passes [T].

Q4. -race, #495 lock guard, #529.

• go test -race -run 'ObsDedup|TrackedBytes|Evict|HashMigrate|Load|RouteMask|Ingest|Chunk|Startup|NoSQLUnderStoreLock' on the merged tree: ok (418 s), 0 races [T].
• TestNoSQLUnderStoreLock and TestNoSQLUnderStoreLockDetects: PASS. TestServerDBConnIsReadOnly and the readonly invariants: PASS [T].
• perf(store): on-demand subpath counts instead of permanent indexes #529: see F3. git merge-tree of (master + perf(store): avoid eager per-transmission observation dedup maps #527) with cb8ad366 gives 2 files with comment-only conflicts [T].

Standing requirements

• Red → green [T]: on the red commit 6ccdc336, 8 tests fail:

  • PathsKeepTheSameObservationsAndIndexes
  • EvictionCreditsWhatEachPathCharged
  • MemoryBudgetEvictsTheSameOnEveryPath
  • RebuildKeepsIndexesAndCharges
  • HashMergeDedupsAcrossTheThreshold
  • SameRowRouteUpdateKeepsObservations
  • ThresholdAndAnswers
  • EstimateHasNoEagerMapTerms

  All pass on the head. The green commit's test edits are mechanical [A]: the hasObservation → lookupObservation rename, gofmt, and removal of the dropped estimate terms in the fix(store): make trackedBytes accurately enforce maxMemoryMB #113/hash_migrate: server-side DB writes on the read-only handle; failures logged as collisions; ghost duplicates share a hash #215 tests and eviction_test.go. Every acceptance criterion has a red-then-green test, except the F1 branch.

• No behaviour change beyond the purpose [A]:

  • same identity contract (observer ID + path_json);
  • UniqueObserverCount semantics are the same, including the empty observer;
  • same order of mergeObservedPathHashSize and the append on every path;
  • StoreTx layout is the same.

  The removed estimate terms are what changes budget enforcement, and they match what is no longer allocated.

• cmd/server read-only: unchanged. There are 0 added map[string]interface{} lines [T].

• No frontend or CSS files changed. scripts/check-xss-sinks.sh --diff origin/master, run in a detached worktree at the head: "no public/**/*.{js,html} changes to scan", exit 0 [T].

• Fork guards: 9 in deploy.yml and 1 in release-fast-path.yml, on the head, merged and master. No workflow changes [T].

• Closing keywords: none in the PR body or commits [T].

• Commit author and committer: dborup <kontakt@meshview.dk> on both commits [T].

• CI on head e0f7eb77, run attempt 1, no reruns: every job passes [T]:

  • Go Server 1/3, 2/3 and 3/3
  • Ingestor
  • Libraries & CLI
  • Build & Test
  • JS Unit & Lint
  • Script & Container
  • Playwright 1/3, 2/3 and 3/3
  • Verify E2E Shards
  • Verify Go Test Assignments
  • Validate and Build Docker

  Badges, Release and Deploy Staging are skipped, as expected for a PR. The known flakes flaky: #226 Hash Stats adopters sort E2E — server sorts multiByteNodes on packets only, ties come out in random order #256 and flaky: TestResolvedPathBackfill_WriteHoldUnderBudget_188 — wall-clock write-hold budget fails under CI load #267 did not show.

Tests and mutants (merged tree on dcc68979)

• cd cmd/server && go test ./...: ok. cd cmd/ingestor && go test ./...: ok [T].
• sh test-all.sh: 259/259. node test-frontend-helpers.js: 719 passed, 0 failed [T].
• E2E against a local Go server on e2e-fixture.db, prepared as in CI: freshen, the inline Packets page collapse button in the left column of the table opens the dialog. Kpa-clawbot/CoreScope#1486/"Group Data" message type missing from packet view window's "message type" filter Kpa-clawbot/CoreScope#1791 seed, corescope-migrate, then seeds 2073, 199, 245 and 122. The server was stopped by port [T].
  • test-e2e-playwright.js: 153/157 passed, 4 skipped. "Version info lives on Perf dashboard" times out locally, and it fails identically against a master build on the same fixture. It is environmental and unrelated to this PR, so it was skipped in a scratch copy to let the fail-fast suite finish [K].
  • The 17 E2Es that touch observation or observer counts or Heard By all pass:
    • 76-heard-by, 165-grouped-hop-warn, packet-detail-sender-hash-size-obs
    • audio-lab/live-1297
    • 1147, 1281, 1151, 2027, 242, 353, 304, 480, 521, 406
    • node-analytics-coverage, xss-333
• Own mutants, each on a copy of the merged tree, with -run 'ObsDedup|TrackedBytes|Evict|HashMigrate|113|215|164' [T]:

#	Mutant	Result
MX1	loser maps cleared before the loser is credited (finishHashMerge)	survived (F1); killed by the extra 36-per-row case
MX2	observerSet not seeded from the existing observations when the maps are built	killed (7 tests)
MX3	moveObservations drops the addObservation charge delta	killed (HashMergeDedupsAcrossTheThreshold)
MX4	threshold off by one (> instead of >=)	killed (HashMergeDedupsAcrossTheThreshold, ThresholdAndAnswers)

Not verified

• [K] The real observation distribution beyond the mean: the public aggregates give only obsPerTx, so the p99 ≥ 60 tail is modelled.
• [K] Heap on staging or production: no access, by design.
• [K] -race on the full cmd/server package and on cmd/ingestor: only the store-related subset above was run.
• [K] CI on the merged tree, and the merge with e14cac27 beyond git merge-tree (clean, and it touches no store code).
• [K] Browser screenshots: the PR has no UI changes.
• [K] The author's light-set numbers (−20.8 % heap) were not re-run; only the heavy distribution above was measured.

[dborup-agent]

Rapport — CS-pve-agent1 PR#527 runde-2 — head 71a21ee

Legend: [T] = test run by me, [A] = analysis or code reading, [K] = known limitation or not verified.

Review feedback addressed (commit 71a21ee6)

1. F1 (test only): added the case {"rows already past", 9} (about 36 observations per row) to TestObsDedup411_HashMergeDedupsAcrossTheThreshold, so each hash-merge loser holds maps. No production code changed. [T]
2. F2 (nit): corrected the threshold comment in obs_dedup.go (comment only) and the PR description. With the production distribution (obsPerTx about 18) about 13 % of transmissions build maps, not 0.2 %. The threshold stays 32. Both now mention that 64 measured about 5 % less heap on top, as a possible follow-up. [A]
3. F3, F4 (info): no change needed. perf(store): on-demand subpath counts instead of permanent indexes #529 conflicts only in comments and is for its own PR to resolve. [A]

Branch: merged origin/master (e14cac27) with a merge commit a3b0e743, then one commit 71a21ee6. No rebase, amend or force-push. The PR stays draft.

Tests

• Mutant MX1 (clears the loser's obsKeys/observerSet before trackedBytes -= txChargedBytes(l) in finishHashMerge): dies only in the new subcase, after the merge: trackedBytes = 25947, charges of the live store sum to 22523 (drift 3424), and the drift survives the eviction. The other two subcases still pass. [T]
• Head without the mutant: the test passes. [T]
• go test -race -run 'ObsDedup|TrackedBytes|Evict|HashMigrate|Load|RouteMask|Ingest|Chunk|Startup|NoSQLUnderStoreLock' in cmd/server: ok (455 s), 0 races. [T]
• cmd/ingestor go test ./...: ok. sh test-all.sh: 260 passed, 0 failed. [T]
• Not run locally: E2E against a local Go server. No frontend files changed by this PR, so I relied on CI's Playwright jobs. [K]

CI on head 71a21ee (run attempt 1, no reruns)

All pass: Go Build & Test, Go Ingestor Tests, Go Libraries & CLI, Go Server Tests 1/3, 2/3 and 3/3, JavaScript Unit Tests & Lint, Script & Container Unit Checks, Playwright E2E 1/3, 2/3 and 3/3, Validate Docker Image, Build & Publish Docker Image, Verify E2E Shards & Coverage, Verify Go Test Assignments. Skipped by the workflow: Publish Badges & Summary, Release Artifacts, Deploy Staging. No flaky tests (#271, #301) were hit. [T]