Skip to content

fix(store): make trackedBytes accurately enforce maxMemoryMB #113

Closed

Description

@dborup

Upstream source

Evaluate and adapt Kpa-clawbot/CoreScope#2035: correct the in-memory store accounting used by packetStore.maxMemoryMB.

Current fork evidence

Verified against fork master 1bdf088e:

  • StoreTx has no persistent record of how many bytes it was charged.
  • A transmission is charged before pickBestObservation assigns PathJSON, so path-dependent byPathHop and O(path²) subpath costs are omitted.
  • Eviction later recomputes the estimate after the path exists and can subtract more than was originally added, drifting trackedBytes downward.
  • The estimate omits the decoded cache, separately allocated dedup-key strings, and relevant observation strings.
  • docs/go-migration.md still incorrectly says the Go server ignores packetStore.maxMemoryMB.

An underestimated counter can prevent intended eviction and leave the real heap pressed against its runtime/container limit.

Fork-specific adaptation

Do not cherry-pick blindly. The fork pins unsafe.Sizeof(StoreTx{}) to 320 bytes for route-mask padding and has additional indexes/caches. Preserve that layout with a justified design or update the contract deliberately, and recalibrate constants on this branch.

Acceptance criteria

  • Record, directly or equivalently, the exact estimate charged per transmission.
  • Apply only deltas when selected paths or other charged allocations change.
  • Eviction subtracts exactly the amount charged.
  • Include path/subpath indexes, decoded-cache allowance, dedup-key allocations, and relevant observation strings.
  • Deterministic insert → path assignment → recharge → eviction returns the counter to exactly zero.
  • Mixed insert/update/eviction tests show no drift; memory eviction reliably crosses high and reaches low watermarks.
  • Cold-load budgeting and documentation use the revised estimate.
  • Existing route-mask, index, and cache behavior remains intact.

Verification

Run focused and full server tests under -race, static checks, before/after tracked-vs-heap measurements on a read-only production-sized copy, cold-load/eviction measurements at representative limits, and insert/eviction performance checks.

No deploy or live-environment mutation is part of this issue.

Activity

  1. added 2 commits that reference this issue on Oct 4, 2026
  2. dborup commented on Oct 4, 2026

    @dborup
    OwnerAuthor

    Fixed by #198 (merged as 81039e35). trackedBytes now records the exact charge per transmission and charges only deltas when the selected path changes. Eviction subtracts exactly what was charged, and the estimate includes paths, subpaths, the decode cache, dedup keys and observation strings. The tracked/heap ratio went from about 0.5–0.6 to about 1.0 on synthetic 100K and 500K stores.

    Leftovers are tracked in #202: a hash-migration double credit and the uncharged path_json fallback, both pre-existing. Confirming the ratio on real data is part of the next staging round.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions