fix(streaming): keep the Go memory limit above the live heap so GC cannot spin - #935
Merged
Merged
Conversation
…nnot spin A user streaming one file saw 40-70% CPU for 18 hours with RSS flat at ~1 GB. The soft memory limit introduced in 49869f1 was derived as segment_cache.memory_mb plus a flat 256 MB, 512 MiB for their config, while the live heap was larger: the 256 MB article tier (on even with the disk cache disabled), up to 96 MB of read-ahead per reader across rclone's two chunk streams, and buffers for 341 TLS connections. With live above the limit the collector runs back to back and the GC CPU limiter pins at half the machine without freeing anything. Two layers fix it while keeping RAM bounded by the limit: - The automatic limit now budgets what is actually live: a 128 MB base, three read-ahead windows (the cap constant moves to config so the reader and the formula share it), and 256 KiB per enabled provider connection, on top of the memory tier and the PAR2 solver. - A pressure governor samples runtime/metrics every 2 s. When the GC CPU limiter engages or the live heap passes 90% of the limit it shrinks the memory tier in 25% steps to a 32 MB floor through a Source ceiling that survives file opens, logging a warning once; after 60 s of calm it restores the tier. Verified against a live instance with two rclone-style chunk readers: the old binary pinned below its live set ran at 201% CPU with GC scan dominating the profile; the fixed binary under the same config shrank the tier within 8 s and ran at 30%, the same as a healthy baseline, with RSS held under the limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A user streaming a single file hit sustained 40-70% CPU for 18+ hours with RSS flat at ~1 GB. Root cause: the Go soft memory limit added in 49869f1 was derived as
segment_cache.memory_mb+ flat 256 MB (512 MiB for their config), but the live heap was larger: the 256 MB in-memory article tier (on even when the disk cache is disabled), up to 96 MB read-ahead per reader across rclone's two chunk streams, and buffers for 341 TLS connections. With live heap above the limit the collector runs back to back and the GC CPU limiter pins at ~half the machine while freeing nothing.The fix keeps RAM bounded by the limit and stops the collector from spinning:
internal/config/manager.go): headroom is now a 128 MB base + 3 read-ahead windows (config.StreamReadAheadBytesCap, shared with the usenet reader) + 256 KiB per enabled provider connection, on top of the memory tier and PAR2 solver budget. The reporting user's limit moves from 512 MiB to ~760 MiB.internal/nzbfilesystem/segcache/pressure.go): a governor samplesruntime/metricsevery 2 s. When the GC CPU limiter engages or live heap exceeds 90% of the limit, it shrinks the article tier in 25% steps down to a 32 MB floor viaSource.SetMemoryCeiling(which survives the per-open capacity re-apply inStore()), logs one warning, and restores the tier after 60 s of calm.config.sample.yamlupdated.Verification
Live runs against the dev instance with real providers: one file, two rclone-style sequential 8 MiB range readers with periodic seeks, 90 s each, CPU sampled every 2 s plus a 30 s pprof profile.
The spiral reproduces only when live heap exceeds the limit (article buffers are noscan, so being near the limit is cheap). Streaming throughput was unchanged across runs.
Tests
Store()reopen, no-op above configured capacity./gc/memory/limit:bytesdoes not exist; it is/gc/gomemlimit:bytes), and a real 128 MB pin under a tight limit triggers a shrink.go test -race ./...passes; golangci-lint clean on touched packages.Follow-up (not in this PR)
nntppool ≥ v4.22 abort-drain closes the whole connection when a cancelled speculative read-ahead body has > 1 MiB left, failing pipelined neighbours and forcing TLS redials on every seek/reopen. Needs bench-gated measurement before changing. Also: per-provider
proxy_urlis never passed to nntppool.