SSD vs GPU PCIE bandwidth balancing

[constantOut]

Thank you for an awesome OSS project!
I wonder if following makes sense performance wise:

1. pcie 5.0 SSD will dramatically improve performance?
2. IO load with 19MB should be closer to serial read than random 4k.
3. Threadripper based system with crazy number of SSDs (it has to store
4. Threadripper based system using DirectStorage to stream experts from 15 SSDs to 4 GPUs.

Threadripper has a nasty issue of number of CCDs affecting memory bandwidth available to CPUs, I have no idea whether this affects DMA and DirectStorage performance and whether one needs 4 or 8 CCDs.

[JustVugg]

Great questions, and mostly on the right track. The key thing to internalise first is where disk sits in the pipeline, because it changes every answer below.

The engine reads experts with a chunked pread of each expert's contiguous weight slice, immediately followed by posix_fadvise(..., DONTNEED) to keep RSS low, and it prefetches speculatively across layers (PREFETCH/SPEC/COUPLE/PILOT). But there's a learned residency cache: once it warms, the hot working set lives in RAM, and disk time drops toward zero. On the Graviton4 report in #634, 98.2% of experts became RAM-resident and expert disk time was 0.0 s of a 2800 s decode — the run was attention- and matmul-bound, not disk-bound. So:

1. PCIe 5.0 SSD — dramatic?
It dramatically improves the cold phase (first warm-up) and the miss stream when you're RAM-starved and constantly faulting experts from disk. Once your RAM holds the working set, the SSD stops being on the hot path and you get diminishing returns. Rule of thumb: buy SSD bandwidth to make warm-up fast and to survive misses; buy RAM to get out of the disk regime entirely.

2. 19 MB reads closer to serial than random 4k — yes, correct.
Each expert is read as a large contiguous slice (int4 gate/up/down), not 4k random. Multiple experts per layer are issued in parallel and prefetched ahead, so the workload is bound by sequential bandwidth + queue depth, not 4k IOPS. PCIe 5 sequential throughput and high QD both help the miss stream. (c/iobench.c lets you measure your drive under exactly this access pattern, buffered or O_DIRECT.)

3. Threadripper + many SSDs.
The pragmatic path today is a striped filesystem (RAID0 / btrfs raid0) rather than engine-level multi-device logic: the engine reads through the filesystem, so aggregate bandwidth is transparent. A user in #381 measured 12.6 GB/s on a btrfs RAID0 NVMe pair, and the engine just sees a fast file. So 15 SSDs in a stripe = one very fast namespace, no special support needed.

4. DirectStorage streaming disk -> 4 GPUs.
Honest status: this is not implemented. The current path is disk -> RAM (pread) -> compute on CPU, or RAM -> VRAM (cudaMemcpyHostToDevice) for GPU tiers. There is no GPUDirect Storage / cuFile / nvidia-fs path that DMAs disk straight into VRAM bypassing system RAM. It's a legitimate future direction, but today a 15-SSD -> 4-GPU rig works by: stripe the SSDs, let RAM/page-cache hold the hot experts, and let the GPUs compute the resident tiers. The disk->VRAM bypass would mainly help the cold/miss path, which — see point 1 — is not where a well-provisioned box spends its time.

On the CCD / memory-bandwidth worry — this is the one to take seriously.
Warm decode is memory-bandwidth-bound (resident experts streamed into SwiGLU matmuls) plus attention. Threadripper memory bandwidth scales with the number of populated CCDs, and a low-CCD part starves the cores regardless of how many DIMMs you install. For PCIe DMA specifically the CCD count matters less (that's the I/O die), but since you compute on the data right after loading it, the CCD-limited memory-bandwidth ceiling is what caps tok/s. So for this engine: prefer the higher-CCD parts (8 CCDs over 4). More CCDs buys you more of the thing that's actually the bottleneck once you're warm.

Net: past a fast-enough NVMe (or a modest stripe) and enough RAM to hold the working set, the highest-leverage spend is memory bandwidth (CCDs) and compute, not more disk. #634 is a good concrete data point for all of this.