Repository navigation
Evict down to a low-water mark instead of once per export - #11
Draft
RhysSullivan wants to merge 1 commit into
Draft
RhysSullivan wants to merge 1 commit into
RhysSullivan wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once the store passed its size or span bound, a retention pass evicted the oldest traces until it was back at exactly the bound. The next export crossed it again, and
holdSizeBoundruns after every export on workerd, so a store at its cap paid one eviction pass per export indefinitely.A self-host user measured this on an idle Executor container: the bundled store reached 1 GiB in about six hours and then ran an eviction pass after nearly every export, part of about 13% of a core and about 2 MiB/s of writes at idle.
A store over a bound now evicts down to a low-water mark, 90% of the bound by default (
MOTEL_OTEL_EVICTION_LOW_WATER_PERCENT), and keeps draining in later passes if a pass's time budget cuts it short. Exports after that find room until the store fills again./api/ingeston workerd now also reportsevictions(passes that evicted traces, and how many), so the cost is visible.Measured on the workerd collector with a 4 MiB bound and 120 exports of 100 spans (about 30 MiB):
At Executor's 1 GiB bound, each pass frees about 100 MiB.
Verified: typecheck;
src/workerd.bounds.test.tsandsrc/workerd.integration.test.ts(7 pass). The test is in the PR above this one.