You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(store): memory estimate ignores resolved byPathHop entries and the #162 pathHopResolved side map #164
At large stores the actual heap can therefore exceed what the eviction budget thinks it holds. This is a follow-up from Macmini's review of #162, out of that PR's scope.
cmd/server/store.go ~5027, estimateStoreTxBytes: counts perPathHopBytes per raw hop (txGetParsedPath), plus subpaths, maps and indexes. It counts nothing for resolved pubkey keys.
estimateStoreTxBytesTypical (~5051) is used for the cold-load budget and has the same gap.
Size (Macmini, synthetic, 1–5 resolved hashes per transmission)
Transmissions
Side map
Per transmission
100K
6.1–11.6 MB
61–116 B
500K
46–74 MB
92–148 B (map growth step)
Only transmissions with resolved relays get a side-map entry. Each resolved byPathHop entry also costs a slice slot (~8 B, plus amortised growth) that is not counted either. Before #162 these entries grew without bound, about 16 B per extra observation.
Proposed fix
Account the resolved entries incrementally. They arrive after insertion, so recomputing them in estimateStoreTxBytes at eviction time would not match what was added.
When addResolvedPubkeysToPathHopIndex appends k new keys, add k × (slice slot + side-map hash) to trackedBytes. Add a fixed overhead the first time a transmission gets a side-map entry.
Subtract the same amount when the transmission is evicted, using the side-map length, and when a rebuild drops entries (retainResolvedPathHops).
Alternatively, include a per-transmission constant for "typical resolved relays" in both estimate functions. This is simpler, but less accurate.
Keep estimateStoreTxBytesTypical consistent with whichever you choose.
Acceptance criteria
A test shows that trackedBytes rises when resolved entries are added and returns to its previous value after eviction or rebuild, with no drift over repeated observations.
A benchmark or measurement compares trackedBytes with the measured heap (runtime.ReadMemStats) for 100K transmissions with resolved relays, before and after the fix. The gap shrinks.
No new work in hot paths beyond O(1) per appended key. No change to eviction ordering.
Summary
The store's memory estimate (
trackedBytes), which drives memory-based eviction, does not count two things:byPathHop, which have never been counted;pathHopResolvedthat PR fix(store): index a transmission once per relay key in byPathHop (traffic share inflation) #162 adds.At large stores the actual heap can therefore exceed what the eviction budget thinks it holds. This is a follow-up from Macmini's review of #162, out of that PR's scope.
Relates to #158, #162.
Where (#162 head
682cd623)cmd/server/store.go~5027,estimateStoreTxBytes: countsperPathHopBytesper raw hop (txGetParsedPath), plus subpaths, maps and indexes. It counts nothing for resolved pubkey keys.estimateStoreTxBytesTypical(~5051) is used for the cold-load budget and has the same gap.addResolvedPubkeysToPathHopIndexappends resolved entries, and fix(store): index a transmission once per relay key in byPathHop (traffic share inflation) #162 records them inpathHopResolved map[*StoreTx][]uint64. Both are added after the transmission is inserted, once per new relay key over later observations.trackedBytesnever accounts for them.Size (Macmini, synthetic, 1–5 resolved hashes per transmission)
Only transmissions with resolved relays get a side-map entry. Each resolved
byPathHopentry also costs a slice slot (~8 B, plus amortised growth) that is not counted either. Before #162 these entries grew without bound, about 16 B per extra observation.Proposed fix
estimateStoreTxBytesat eviction time would not match what was added.addResolvedPubkeysToPathHopIndexappends k new keys, add k × (slice slot + side-map hash) totrackedBytes. Add a fixed overhead the first time a transmission gets a side-map entry.retainResolvedPathHops).estimateStoreTxBytesTypicalconsistent with whichever you choose.Acceptance criteria
trackedBytesrises when resolved entries are added and returns to its previous value after eviction or rebuild, with no drift over repeated observations.trackedByteswith the measured heap (runtime.ReadMemStats) for 100K transmissions with resolved relays, before and after the fix. The gap shrinks.