Summary
Under sustained client traffic the proxy starts emitting a continuous flood of identical panics (tens per second) originating in the foyer-memory dependency's in-memory LRU eviction. The process stays "active" (panics are caught at the tokio task boundary), but every client whose request triggers a cache insert into the affected shard has its session killed, producing widespread NNTP failures. The condition does not self-heal and persists until the service is restarted.
Panic
thread 'tokio-rt-worker' panicked at
foyer-memory-0.22.3/src/eviction/lru.rs:89:62:
called `Option::unwrap()` on a `None` value
ERROR nntp_proxy::runtime: Client session task failed:
JoinError::Panic(Id(...), "called `Option::unwrap()` on a `None` value", ...)
Observed: ~6,700 panics in a 2-minute window (~55/s), sustained for the life of the process.
Root cause
foyer-memory 0.22.3, src/eviction/lru.rs, Lru::may_overflow_high_priority_pool:
fn may_overflow_high_priority_pool(&mut self) {
while self.high_priority_weight > self.high_priority_weight_capacity {
strict_assert!(!self.high_priority_list.is_empty()); // compiled out in release
let record = self.high_priority_list.pop_front().unwrap(); // line 89 -> panics
...foyer tracks high_priority_weight (a running weight counter) separately from high_priority_list (the actual entries). The loop condition reports excess weight to evict while the list is already empty, so pop_front() returns None and .unwrap() panics. The counter and the list have drifted out of sync — a foyer-internal invariant violation. The strict_assert! guarding this is a no-op in release builds, so it falls through to the panicking unwrap() instead of a clean assertion.
This is an upstream foyer-memory bug, but it is reachable through this project's hybrid-cache configuration, which enables the high-priority pool with a custom weighter.
How this project triggers it
src/cache/hybrid.rs:
let mut builder = HybridCacheBuilder::new()
.with_name(HYBRID_CACHE_NAME)
.with_policy(HybridCachePolicy::WriteOnInsertion)
.memory(memory_capacity_usize)
.with_shards(config.shards)
.with_eviction_config(LruConfig {
high_priority_pool_ratio: 0.1, // enables the buggy code path
})
.with_weighter(|_key: &String, value: &DiskCachedArticle| value.payload_len().get())
...The in-place update path (re-inserting an existing key with a potentially different weight, hybrid.rs:387) is the most likely source of the weight/list desync, though I have not yet isolated a minimal reproducer.
Environment
- nntp-proxy: running from
main (foyer 0.22 -> foyer-memory 0.22.3 locked)
- rustc 1.95.0, release profile (
lto = "thin", codegen-units = 1, panic = "unwind")
panic = "unwind" is why the process survives each panic instead of aborting- Cache config in use:
article_cache_capacity = "12gb", disk capacity = "500gb", shards = 8, ttl 4h; process RSS ~21 GB at time of incident
- foyer-memory 0.22.3 is the latest published release — no upstream version bump available to pull in a fix
Impact
- Continuous panic spam in the journal (log volume + noise)
- Client NNTP sessions killed mid-request whenever they touch the poisoned shard
- Requires manual
systemctl restart to recover; recurs over time
Possible mitigations (for discussion)
- Disable the high-priority pool (
high_priority_pool_ratio: 0.0) to avoid may_overflow_high_priority_pool entirely — needs verification that ratio 0 fully bypasses the path rather than making every entry overflow.
- Switch the memory tier to the moka backend (already supported in
src/cache/mod.rs), sidestepping foyer's LRU.
- Report/track upstream in foyer and pin past a fixed release once available.
Summary
Under sustained client traffic the proxy starts emitting a continuous flood of identical panics (tens per second) originating in the
foyer-memorydependency's in-memory LRU eviction. The process stays "active" (panics are caught at the tokio task boundary), but every client whose request triggers a cache insert into the affected shard has its session killed, producing widespread NNTP failures. The condition does not self-heal and persists until the service is restarted.Panic
Observed: ~6,700 panics in a 2-minute window (~55/s), sustained for the life of the process.
Root cause
foyer-memory0.22.3,src/eviction/lru.rs,Lru::may_overflow_high_priority_pool:foyer tracks
high_priority_weight(a running weight counter) separately fromhigh_priority_list(the actual entries). The loop condition reports excess weight to evict while the list is already empty, sopop_front()returnsNoneand.unwrap()panics. The counter and the list have drifted out of sync — a foyer-internal invariant violation. Thestrict_assert!guarding this is a no-op in release builds, so it falls through to the panickingunwrap()instead of a clean assertion.This is an upstream
foyer-memorybug, but it is reachable through this project's hybrid-cache configuration, which enables the high-priority pool with a custom weighter.How this project triggers it
src/cache/hybrid.rs:The in-place update path (re-inserting an existing key with a potentially different weight,
hybrid.rs:387) is the most likely source of the weight/list desync, though I have not yet isolated a minimal reproducer.Environment
main(foyer0.22->foyer-memory0.22.3 locked)lto = "thin",codegen-units = 1,panic = "unwind")panic = "unwind"is why the process survives each panic instead of abortingarticle_cache_capacity = "12gb", diskcapacity = "500gb",shards = 8, ttl 4h; process RSS ~21 GB at time of incidentImpact
systemctl restartto recover; recurs over timePossible mitigations (for discussion)
high_priority_pool_ratio: 0.0) to avoidmay_overflow_high_priority_poolentirely — needs verification that ratio 0 fully bypasses the path rather than making every entry overflow.src/cache/mod.rs), sidestepping foyer's LRU.