Skip to content

fix(streaming): keep the Go memory limit above the live heap so GC cannot spin - #935

Merged
javi11 merged 1 commit into
mainfrom
fix/gc-soft-limit-pressure
Sep 7, 2026
Merged

javi11 merged 1 commit into
mainfrom
fix/gc-soft-limit-pressure

Conversation

@javi11

@javi11 javi11 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

A user streaming a single file hit sustained 40-70% CPU for 18+ hours with RSS flat at ~1 GB. Root cause: the Go soft memory limit added in 49869f1 was derived as segment_cache.memory_mb + flat 256 MB (512 MiB for their config), but the live heap was larger: the 256 MB in-memory article tier (on even when the disk cache is disabled), up to 96 MB read-ahead per reader across rclone's two chunk streams, and buffers for 341 TLS connections. With live heap above the limit the collector runs back to back and the GC CPU limiter pins at ~half the machine while freeing nothing.

The fix keeps RAM bounded by the limit and stops the collector from spinning:

  • Limit tracks the live set (internal/config/manager.go): headroom is now a 128 MB base + 3 read-ahead windows (config.StreamReadAheadBytesCap, shared with the usenet reader) + 256 KiB per enabled provider connection, on top of the memory tier and PAR2 solver budget. The reporting user's limit moves from 512 MiB to ~760 MiB.
  • Memory tier yields under pressure (internal/nzbfilesystem/segcache/pressure.go): a governor samples runtime/metrics every 2 s. When the GC CPU limiter engages or live heap exceeds 90% of the limit, it shrinks the article tier in 25% steps down to a 32 MB floor via Source.SetMemoryCeiling (which survives the per-open capacity re-apply in Store()), logs one warning, and restores the tier after 60 s of calm.
  • Docs and config.sample.yaml updated.

Verification

Live runs against the dev instance with real providers: one file, two rclone-style sequential 8 MiB range readers with periodic seeks, 90 s each, CPU sampled every 2 s plus a 30 s pprof profile.

Run Limit Avg CPU Max RSS Profile
baseline, auto 512 MiB 27.5% 538 MB network syscalls, no GC
baseline, pinned 384 MiB 29.5% 407 MB no GC
baseline, pinned below tier 200 MiB 201% 330 MB GC scan dominant; 256 MB article buffers live
fixed, same config 200 MiB 30.6% 266 MB no GC; governor shrank tier at 8 s
fixed, auto 684 MiB 30.2% 604 MB no GC

The spiral reproduces only when live heap exceeds the limit (article buffers are noscan, so being near the limit is cheap). Streaming throughput was unchanged across runs.

Tests

  • Limit formula tests including the user's exact 341-connection layout, backup counted, disabled skipped.
  • Governor policy: shrink on limiter engage / near-limit, floor, recovery after calm, middle-band resets, release when no limit, small-tier no-op.
  • Source ceiling applied immediately, survives Store() reopen, no-op above configured capacity.
  • Runtime tests: metric names resolve (caught a wrong name, /gc/memory/limit:bytes does not exist; it is /gc/gomemlimit:bytes), and a real 128 MB pin under a tight limit triggers a shrink.
  • go test -race ./... passes; golangci-lint clean on touched packages.

Follow-up (not in this PR)

nntppool ≥ v4.22 abort-drain closes the whole connection when a cancelled speculative read-ahead body has > 1 MiB left, failing pipelined neighbours and forcing TLS redials on every seek/reopen. Needs bench-gated measurement before changing. Also: per-provider proxy_url is never passed to nntppool.

…nnot spin

A user streaming one file saw 40-70% CPU for 18 hours with RSS flat at
~1 GB. The soft memory limit introduced in 49869f1 was derived as
segment_cache.memory_mb plus a flat 256 MB, 512 MiB for their config,
while the live heap was larger: the 256 MB article tier (on even with the
disk cache disabled), up to 96 MB of read-ahead per reader across
rclone's two chunk streams, and buffers for 341 TLS connections. With
live above the limit the collector runs back to back and the GC CPU
limiter pins at half the machine without freeing anything.

Two layers fix it while keeping RAM bounded by the limit:

- The automatic limit now budgets what is actually live: a 128 MB base,
  three read-ahead windows (the cap constant moves to config so the
  reader and the formula share it), and 256 KiB per enabled provider
  connection, on top of the memory tier and the PAR2 solver.
- A pressure governor samples runtime/metrics every 2 s. When the GC CPU
  limiter engages or the live heap passes 90% of the limit it shrinks the
  memory tier in 25% steps to a 32 MB floor through a Source ceiling that
  survives file opens, logging a warning once; after 60 s of calm it
  restores the tier.

Verified against a live instance with two rclone-style chunk readers:
the old binary pinned below its live set ran at 201% CPU with GC scan
dominating the profile; the fixed binary under the same config shrank
the tier within 8 s and ran at 30%, the same as a healthy baseline, with
RSS held under the limit.
@javi11
javi11 merged commit b921352 into main Sep 7, 2026
2 checks passed
@javi11
javi11 deleted the fix/gc-soft-limit-pressure branch September 7, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant