Skip to content

Evict down to a low-water mark instead of once per export - #11

Draft
RhysSullivan wants to merge 1 commit into
mainfrom
fix/size-bound-hysteresis
Draft

RhysSullivan wants to merge 1 commit into
mainfrom
fix/size-bound-hysteresis

Conversation

@RhysSullivan

Copy link
Copy Markdown

Once the store passed its size or span bound, a retention pass evicted the oldest traces until it was back at exactly the bound. The next export crossed it again, and holdSizeBound runs after every export on workerd, so a store at its cap paid one eviction pass per export indefinitely.

A self-host user measured this on an idle Executor container: the bundled store reached 1 GiB in about six hours and then ran an eviction pass after nearly every export, part of about 13% of a core and about 2 MiB/s of writes at idle.

A store over a bound now evicts down to a low-water mark, 90% of the bound by default (MOTEL_OTEL_EVICTION_LOW_WATER_PERCENT), and keeps draining in later passes if a pass's time budget cuts it short. Exports after that find room until the store fills again. /api/ingest on workerd now also reports evictions (passes that evicted traces, and how many), so the cost is visible.

Measured on the workerd collector with a 4 MiB bound and 120 exports of 100 spans (about 30 MiB):

Low-water mark Eviction passes Traces evicted
100% (the old behaviour) 107 322
90% (default) 44 324
50% 12 323

At Executor's 1 GiB bound, each pass frees about 100 MiB.

Verified: typecheck; src/workerd.bounds.test.ts and src/workerd.integration.test.ts (7 pass). The test is in the PR above this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant