Use a resizable receive ring with one inline descriptor

[thweetkomputer]

Replace Connection's deque with a receive-descriptor ring that stores its first descriptor inline. Depth-zero/one traffic performs no queue heap allocation. Overflow allocates 8 slots, then doubles when full; sustained low occupancy halves heap storage and eventually returns to the inline slot. Current shared-dataset cross-over results are recorded below; the historical fresh-fill initial-ring results remain separately labeled.

The measured throughput changes are mostly small: 10M scenario medians stay within ±0.36%. 1B SET/640 improves in all three pairs (+1.57%, +0.93%, +1.55%; median +1.55%), and 1B GET/640 has a median +0.41%. Tail latency is not uniformly better: 1B GET/320 p99.99 is higher in all three pairs (+2.84%, +2.19%, +9.95%); 10M GET/320 p99.99 has a +10.85% median change with mixed signs. These results do not support a blanket speedup or a no-regression claim.

The first 1B SET/320 pair is −11.83% QPS; the next two are +0.56% and +1.04%, and all are retained. The unchanged control itself moves from 903,807 QPS in the first round to 795,615 and 794,747 later, while the candidate is 796,890, 800,032 and 803,029. This exposes a substantial stage/time-dependent effect; the experiment does not isolate its cause or justify attributing the first-pair drop solely to the queue. Shared-data history and ordinary time effects remain confounded.

Current implementation

• Extract the implementation into InlineRingQueue<T, InlineCapacity> in bycorf/base/inline_ring_queue.h; Connection::received_buffers_ directly uses InlineRingQueue<ReceivedBuffer, 1>. The receive descriptor stays in connection.h; no separate receive-queue wrapper header or production alias is needed. Inline capacity is a compile-time positive power of two (up to 2^29), preserving masked indexing. Heap growth starts at max(8, 2 * InlineCapacity). The utility requires trivially copyable/destructible values with nonthrowing copy construction; default construction is not required. It is single-threaded and does not manage resources referenced by its values.
• Moving wrapped inline contents copies them in FIFO order and resets the destination head to zero. Heap moves transfer storage and preserve its head. Generic tests cover capacities 1/2/4/8/16/32, integer and over-aligned non-default-constructible values, move combinations including empty queues retaining heap allocations, aliased push arguments, wrapped resizing and shrinking back to inline with live values.
• Share inline storage and an owning raw heap pointer in a union. The queue releases heap storage in its destructor and on replacement, and transfers ownership on moves. capacity_ identifies the active member; transitions start the new member lifetime. Growth allocates/copies before releasing the old array; returning inline saves the pointer, copies live values, then deletes the array. Copying is statically required not to throw. Empty slots have no active descriptor; placement construction initializes only enqueued/copied elements. Both unused inline storage and unused heap slots avoid descriptor initialization. Explicit trivial-destructor invariants make pop, clear, overwrite and array destruction safe.
• Move inline values into the destination's own storage; transfer heap ownership for heap mode. A moved-from queue is empty and immediately reusable with inline capacity 1.
• Inline push/pop do not update the low-water counter. Heap capacity 8 contracts to inline after 8 consecutive low-water pops at depth ≤1; larger capacities halve after a full capacity's worth of low-water pops at ≤25% occupancy. A push above the relevant threshold resets the count. Shrinking is best-effort and nonthrowing; returning to inline allocates nothing.
• Keep growth and relocation in outlined helpers. Complete consumption saves the buffer ID, pops and returns the backend buffer without writing its discarded offset. Close returns all backend buffers before a single clear(), so close does not allocate or relocate through shrink.
• Element references must not survive push/pop or moving the queue. Payload bytes remain in backend storage; the queue contains IDs, sizes and partial-read offsets.

Evaluated and excluded: single-slot push/pop fast paths

A separate candidate added [[likely]] single-slot push/pop branches and outlined all remaining single-slot operations into PushSlow/PopSlow. GCC 13 x86-64 caller code shrank from 176 to 62 bytes for push and from 85 to 29 bytes for pop, with unchanged object layout. It was compared directly against this PR's 3cc3586 on the same retained pair: 32 client threads, GET/SET, 320/640 connections, 10M/1B keys, three paired rounds and 48 formal windows. The 1B dataset was reused; 10M was loaded once. No previous measurement rows were pooled.

Paired median QPS changes ranged from -0.50% to +0.23%, without a clear throughput gain. 10M GET/320 p99.99 worsened in all three pairs: 1.847→6.335 ms, 3.999→5.695 ms, 4.015→6.175 ms (median paired change +53.80%). 10M GET/640 p99.99 worsened in two of three pairs (median +47.20%). Both versions passed sanitizer/allocation checks; the candidate also passed five runtime checks. The experiment does not identify the microscopic cause or exclude storage-history and VM effects. This candidate was rejected and is not included in this PR; 3cc3586 retains its original push/pop implementation. Smaller caller code alone is not evidence of improved throughput or tail latency.

Layout and allocation checks

Native GCC 13.3 x86-64, using the retained server's actual linked mimalloc library:

Version	Queue object	Connection object	Connection allocation class	Extra slots at depth 1
Pre-inline ring (c53adc0)	24 B	176 B	192 B	96 B
Current inline union ring	32 B	184 B	192 B	0 B

Thus the measured depth-one allocation blocks total 288→192 bytes per connection, with one fewer allocation. This excludes backend receive buffers, allocator/page overhead and other connection state; it is not a measured RSS change. The original deque version's Connection was 232 bytes.

The first 64 bytes of Connection retain the merged receive-control layout. The queue still begins at offset 96; its inline descriptor and mutable counters fit in the first 32 bytes. The heap pointer shares the inline-storage union, and counters begin at queue offset 16. A 1,000-allocation mimalloc probe returned 64-byte-aligned 192-byte blocks. This alignment is allocator-dependent, not an alignof(Connection) guarantee. Validate actual addresses, generated accesses and paired throughput/tails before claiming a cache-line performance improvement. No Task, RemoteWork, storage-awaitable or Lavik layout edits are included.

Queue-depth evidence

A separate diagnostic build based on c53adc0 profiled the previous workloads on the retained Azure pair: 10M/1B keys, GET/SET, 320/640 total connections, 32 client threads. Across all 8 formal windows and 347,465,750 enqueues, every post-enqueue depth and every per-connection peak was 1; every pop emptied the queue. Capacity stayed 8 and no growth/shrink followed the initial allocation. Counts matched all 3,840 formal connection lifetimes and enqueue/dequeue/close conservation checks. The diagnostic counters were outside Connection and did not change its layout.

These are event-weighted observations of the existing non-pipelined workload, not production-wide or time-weighted occupancy estimates. High pipeline depth, large requests and bursts can still require the dynamic queue. Diagnostic-run QPS is not an optimization comparison.

Current correctness validation

The latest header-only organization change (7e1eebb) removes received_buffer_queue.h, restores ReceivedBuffer to connection.h, and directly instantiates the generic queue on Connection. Its fresh production build, backend smoke and five runtime checks pass; queue/Connection sizes and offsets are unchanged. The queue algorithm is unchanged. Performance tables below were acquired on 3cc3586 before this header cleanup; they are not a new benchmark of 7e1eebb.

• Native x86 GCC strict ASan/UBSan: randomized comparison against deque, wrapped growth/contraction, partial offsets, inline↔heap move combinations, source reuse/destruction, self-move, repeated 1→8→1 transitions and clear/reuse.
• Allocation audit: no queue heap allocation across 10,000 depth-one cycles; unused inline/grow/shrink slot bytes untouched; growth failure preserves queued values; shrink allocation failure remains nonthrowing; heap storage returns to inline without leaks.
• Current raw-pointer union layout is 32/184 B for queue/Connection, while the actual Connection allocation class stays 192 B. GCC 13.3 -O3 -march=x86-64-v2 probe sizes: push 176 B, pop 85 B, front 28 B, Grow 321 B, Shrink 415 B. Code size is not a throughput measurement.
• Actual x86 layout and allocator-class probe; complete production Lavik build and backend smoke; five runtime checks covering the queue, connect timer, connection storage lifetime, buffered reads/close regression and io_uring backpressure.
• Current performance has been measured for 3cc3586 using the shared-dataset protocol below. Historical results are retained separately, including unfavorable results.

Current benchmark — shared-dataset cross-over (3cc3586)

Both arms freshly rebuilt with Lavik 2ae16e9b86ce274a7705542d29c63e3c7a79fb2f: control Bycorf 86d686724ddd1c5617fb1b62f8131e05571e8423, candidate 3cc3586ab421f567ab1cae97a5c53841d5c8a271.

At the user's request, load each size only once in a separate baseline process, checkpoint and exit, then restart the two versions on that same persisted dataset in AB/BA/AB order. Both sizes checkpoint on every switch; all formal starts verify exact DBSIZE and five sampled value lengths. An initial attempt stopped before candidate execution because copying the ELF omitted its executable mode. Its four baseline windows are archived separately and excluded. Permissions were fixed without changing binary hashes; the existing 10M dataset and those prior SET updates were retained. SET changes and storage history persist between cells, so sequence only partially balances carryover. This is distinct from the historical fresh-fill protocol below.

All nine CI jobs passed for 3cc3586, including amd64/arm64, ordinary/GCC Debug, bypass OFF/ON and formatting.

Retained Korea Central pair; 16 server logical CPUs, 32 client threads, 320/640 connections, 10M/1B keys, GET/SET, 1,024-byte values, pipeline 1; 30-second 10M / 60-second 1B windows. Six 10M warmups, two one-time fills and 48 formal windows. No historical rows pooled. All formal windows passed configuration, duration, zero-error/miss and JSON/text checks; 806 selected server files and 224 cross-host JSON pairs verified. The original 494 source-file identities were also checked. Background controls restored and both VMs kept running.

Median of three paired percentage changes; positive QPS is better, positive latency is worse. The contrast covers the complete PR, not union/raw-pointer changes in isolation.

Scenario	QPS	p99	p99.9	p99.99
10M GET 320	-0.35%	+0.00%	-2.36%	+10.85%
10M GET 640	+0.11%	+0.63%	-5.79%	+3.98%
10M SET 320	+0.11%	+0.00%	+0.88%	+4.84%
10M SET 640	-0.11%	+0.66%	+0.47%	-2.27%
1B GET 320	-0.39%	+1.19%	+0.88%	+2.84%
1B GET 640	+0.41%	+0.00%	+0.00%	-2.34%
1B SET 320	+0.56%	+0.00%	-0.86%	-3.14%
1B SET 640	+1.55%	+0.00%	-0.47%	-8.30%

All current paired changes

Scenario	QPS R1/R2/R3	p99.9 R1/R2/R3	p99.99 R1/R2/R3
10M GET 320	-0.42% / +1.21% / -0.35%	-8.40% / -2.36% / +1.61%	-17.59% / +10.85% / +25.79%
10M GET 640	-0.09% / +0.11% / +0.22%	-26.50% / -5.79% / +4.79%	-14.23% / +3.98% / +13.83%
10M SET 320	+0.11% / +2.24% / -0.51%	+4.47% / -1.71% / +0.88%	+14.53% / +0.00% / +4.84%
10M SET 640	-1.03% / +0.17% / -0.11%	-0.45% / +0.47% / +1.84%	+0.79% / -2.78% / -2.27%
1B GET 320	+0.61% / -0.74% / -0.39%	-0.87% / +0.88% / +3.54%	+2.84% / +2.19% / +9.95%
1B GET 640	+0.73% / +0.41% / +0.25%	+0.00% / -1.80% / +2.33%	+4.03% / -7.64% / -2.34%
1B SET 320	-11.83% / +0.56% / +1.04%	-0.86% / +0.00% / -1.74%	-11.77% / -3.14% / -2.06%
1B SET 640	+1.57% / +0.93% / +1.55%	-0.47% / +0.95% / -3.21%	-8.30% / +1.26% / -9.69%

Server/client snapshots recorded zero steal, softnet drops/time-squeeze and interface error/drop increments. Aggregate execution ranges were 92.3–99.5% across 16 server logical CPUs and 51.7–62.6% across 32 client CPUs; snapshots include setup/exit overhead and do not exclude individual core limits or VM noise.

Three pairs do not establish statistical equivalence or a universal improvement. Cache/storage history, restarts and ordinary VM variation remain limitations.

Completed benchmark — initial ring revision

Same prior workload protocol on the retained Korea Central pair: L16aos_v4 server (16 logical CPUs, cores 0–15) and F32als_v6 client (32 threads on cores 0–31). Kernel TCP / SPDK storage; pipeline 1; 1,024-byte values; 320/640 total connections; 10M/1B keys; GET/SET; fresh fills. Three paired rounds in AB/BA/AB order, with 30-second 10M and 60-second 1B measurement windows. All 48 formal windows / 12 cells completed. Historical compact-layout or 16-thread results are not pooled.

Both arms were freshly rebuilt with Lavik 2ae16e9b86ce274a7705542d29c63e3c7a79fb2f. Control: merged Bycorf 86d686724ddd1c5617fb1b62f8131e05571e8423. Candidate production code: 6da7a471d13b8ab1069906ff46abd8e543c6ff57. The tables below describe only 6da7a471d13b8ab1069906ff46abd8e543c6ff57, which allocated its first 8-slot array lazily. These historical results predate direct-front, outlined slow-path, consume/close and inline-storage changes. The current 3cc3586 shared-dataset experiment is reported separately above; do not apply these historical QPS or latency changes to the current implementation.

Each entry below is the median of three paired percentage changes, (ring / deque - 1) × 100. Positive QPS is better; positive latency is worse. Percentiles are full-window measurements, not averages of per-second percentiles.

Keys / operation / connections	QPS	p99	p99.9	p99.99
10M GET 320	+0.11%	+1.18%	+0.00%	+11.38%
10M GET 640	+0.56%	+1.88%	+3.71%	+7.93%
10M SET 320	+1.73%	-1.16%	+0.00%	+4.47%
10M SET 640	+0.23%	+0.00%	+0.46%	+6.67%
1B GET 320	+0.20%	+0.00%	+0.00%	-2.07%
1B GET 640	+0.38%	-1.33%	-2.79%	-1.55%
1B SET 320	-0.16%	+0.00%	+0.86%	-1.53%
1B SET 640	+0.55%	-0.64%	+0.47%	+3.46%

• 10M SET/320 QPS improved by +0.50%, +1.86%, +1.73% (median +1.73%).
• 10M GET/640 p99.9 worsened in all three pairs: 2.607→2.703 ms, 2.591→2.687 ms, 2.591→2.847 ms (+3.68%, +3.71%, +9.88%). Its p99.99 changed 5.247→5.663 ms, 5.727→5.439 ms, 5.599→7.263 ms (+7.93%, −5.03%, +29.72%). The latter is variable, not uniformly worse in every pair.
• 1B GET/640 QPS improved in all three pairs (+0.13%..+0.62%) and p99.9 decreased in all three (−0.46%..−4.11%).
• 1B SET/320 QPS decreased in all three pairs (−0.11%..−0.75%); p99.9 increased by about 0.86%. 1B SET/640 p99.99 increased in all three pairs (+1.70%..+5.09%).

All paired changes, including unfavorable rounds

Scenario	QPS R1 / R2 / R3	p99.9 R1 / R2 / R3	p99.99 R1 / R2 / R3
10M GET 320	-0.36% / +0.31% / +0.11%	+3.45% / +0.00% / -0.84%	+11.38% / +11.95% / -1.25%
10M GET 640	-0.01% / +0.56% / +0.72%	+3.68% / +3.71% / +9.88%	+7.93% / -5.03% / +29.72%
10M SET 320	+0.50% / +1.86% / +1.73%	+2.68% / +0.00% / -2.59%	+4.47% / +7.66% / -0.52%
10M SET 640	+0.37% / -0.20% / +0.23%	-0.46% / +0.46% / +0.91%	-1.60% / +10.77% / +6.67%
1B GET 320	-0.13% / +0.20% / +0.92%	+1.76% / +0.00% / -1.71%	-3.18% / +0.00% / -2.07%
1B GET 640	+0.12% / +0.62% / +0.38%	-0.46% / -2.79% / -4.11%	+0.00% / -4.30% / -1.55%
1B SET 320	-0.11% / -0.75% / -0.16%	+0.86% / +0.86% / +0.86%	+6.00% / -1.53% / -9.71%
1B SET 640	+0.55% / -0.17% / +0.67%	+0.00% / +0.47% / +1.87%	+1.70% / +3.46% / +5.09%

Validation and limits

• All formal windows had zero connection errors and GET misses, correct 32-thread affinity/configuration, expected runtime, and matching JSON/text QPS/percentiles. Verified 872 archived server files and 264 matching server/client JSON pairs; audited 494 original source files and unchanged production ELF hashes.
• Server/client steal, softnet drops/time-squeeze, and interface error/drop deltas were zero in the sampled windows. Client aggregate CPU execution was 54.1–62.7% across the selected 32 CPUs; server 94.6–97.5% across 16 logical CPUs. These snapshots include setup/exit overhead; aggregate client utilization does not exclude individual hot-core limits. No CPU-saturation or noise-free claim is made.
• Background log collection and apt timers were controlled and restored afterward. Agent heartbeats/manifest polling remained active; agent, kernel, boot IDs and dynamic library identities were unchanged. Ordinary VM variability is not excluded. Both VMs remain running.
• Randomized FIFO comparison with std::deque, wrapped growth, delayed contraction with queued entries, partial offsets, moves, clear/reuse; Linux ASan/UBSan and local ARM64 UBSan passed.
• Fresh builds and backend smoke checks for both arms. Candidate passed 3 TCP/Unix accept checks, disconnect cancellation, and 7 CTest checks covering the queue, connect timers, connection lifetime, buffered reads, io_uring backpressure, large and fragmented replies with TLS.
• All 9 CI jobs passed for the preceding 0d1335c revision, including amd64/arm64, ordinary/GCC Debug, bypass OFF/ON, and formatting.

Three pairs describe this experiment, not statistical equivalence or a universal result. The 10M GET/640 tail signal needs investigation before treating this as a no-regression optimization. These standard workloads do not isolate worst-case resize latency or measure resize frequency, so the tail changes cannot be attributed specifically to resizing or cache-line effects.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f1f9e711-49c7-4422-8932-bfc0dd0177fa

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}