feat: Add NUMA-aware RAM-disk streaming

[BColsey]

Summary

• add a Linux-only, Python standard-library coli ramdisk TUI and equivalent scriptable plan, prepare, status, benchmark, start, stop, and destroy commands
• discover NUMA/kernel capabilities, calculate exact staging and runtime reserves, mount THP-enabled tmpfs safely, and stage full models or profile-guided shard closures
• add descriptor-verified tmpfs direct mapping for complete experts while preserving the existing buffered, O_DIRECT, pipeline, and io_uring paths for SSD fallback
• support interleaved and per-node managed engines with durable KV/usage state, crash-safe usage merging, process/mount identity checks, benchmarking, and cleanup
• package and document the new control-plane module and add focused lifecycle, CLI, engine, integration, and packaging tests

Motivation

Large MoE models are frequently limited by repeated expert reads from SSD. This adds a managed tmpfs tier that can stage all weights, or only profile-selected shard closures, while letting eligible experts bypass slab allocation and LRU I/O through read-only mappings. Unstaged experts continue to use Colibri's existing SSD streaming path.

The canonical model is never modified. Managed state stays on durable storage, and the implementation does not change global swap settings or HugeTLB reservations.

Safety and compatibility

• sudo is limited to the exact mount and unmount operations
• mounts, PIDs, process groups, state paths, fingerprints, and safetensors headers are verified before destructive or signaling operations
• preparation rechecks memory reserve before every copy and uses atomic publication
• COLI_RAMMAP=1 and legacy COLI_MMAP=1 are mutually exclusive
• behavior is unchanged when the new RAM-disk variables are unset

Validation

• make check — portable build, full C suite, and 141 Python tests passed with zero compiler warnings; one privileged integration remains opt-in in the default suite
• isolated real-tmpfs prepare/status/destroy integration passed, including no swap growth and durable KV preservation
• git diff --check origin/dev...HEAD passed
• rebased onto current origin/dev; upstream storage-read, sampling, filesystem-detection, Windows, Nix, and documentation changes remain covered by their tests

Hardware follow-up

Full 744B-model token parity, multi-node placement/throughput, and full/partial physical SSD-read acceptance still require a suitably provisioned NUMA host and model fixture.

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

[BColsey]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

Thanks, ill correct that before i take it out of draft

[BColsey]

@JustVugg Rebase complete . should be good to go.

[BColsey]

Real-world validation and measured benefit

Tested at commit 5e592f0c2fe77e2ccf6d20db4a03c46608f5beea with a clean tracked worktree.

Host and model

• Linux 7.0.0-22-generic
• AMD Ryzen 9 3900X, 12 physical cores / 24 threads, AVX2
• 60 GiB RAM, one NUMA node
• Samsung 970 EVO Plus 2 TB, ntfs3, read-only model mount
• Real GLM-5.2 int4 snapshot: 144 safetensors shards, 383,739,826,712 bytes (about 358 GiB on disk)
• Test shard: out-00069.safetensors, 2,693,193,272 bytes

Exact I/O commands and run policy

make -C c iobench

MODEL=/run/media/ben/Storage/glm52_i4

# Physical SSD: O_DIRECT, no warm-up, three measured runs.
./c/iobench "$MODEL/out-00069.safetensors" 19 64 8 1

# Warm page cache: one warm-up, then three measured runs.
./c/iobench "$MODEL/out-00069.safetensors" 19 64 8 0

# tmpfs: copy the same shard, one warm-up, then three measured runs.
TMP=$(mktemp -d /dev/shm/colibri-pr377.XXXXXX)
cp --reflink=never "$MODEL/out-00069.safetensors" "$TMP/"
./c/iobench "$TMP/out-00069.safetensors" 19 64 8 0
rm "$TMP/out-00069.safetensors"
rmdir "$TMP"

Each measured run performs 64 deterministic random 19 MiB reads with eight threads (about 1.3 GB read per run).

path	measured runs (GB/s)	median	median effective latency/block
physical SSD, O_DIRECT	3.46, 3.50, 3.44	3.46 GB/s	5.8 ms
SSD file, warm page cache	6.94, 6.88, 7.41	6.94 GB/s	2.9 ms
tmpfs	7.32, 7.75, 7.37	7.37 GB/s	2.7 ms

On this host, tmpfs delivered 2.13× the median physical-SSD throughput and about 53% lower effective block latency. It was only about 1.06× faster than an already-warm page cache, which is an important caveat: this is an I/O-path measurement, not a claim of a 2.13× token-rate improvement. The practical benefit is making selected weights predictably memory-resident and eligible for direct mapping instead of depending on cache residency and repeated SSD reads.

The temporary 2.69 GB tmpfs copy was removed after the measurements.

Production planning against the real model

Commands:

python3 c/coli ramdisk plan \
  --model /run/media/ben/Storage/glm52_i4 \
  --json

python3 c/coli ramdisk plan \
  --model /run/media/ben/Storage/glm52_i4 \
  --mode partial \
  --capacity-gb 8 \
  --profile /run/media/ben/Storage/glm52_i4/.coli_usage \
  --json

Full mode identified all 144 shards and calculated 429,249,168,387 bytes of required staging/runtime/OS reserve against about 20.2 GB available. It correctly refused preparation with:

available memory would breach the global runtime/OS reserve

The 8 GiB partial plan selected three complete shard closures (8,058,234,192 bytes). From the real usage profile, the planner estimated 43,108,032,512 bytes of expert traffic avoided (5.35 predicted bytes avoided per staged byte), but only 2.81% profile coverage versus a 30.01% same-budget hot-expert PIN estimate. That is a useful negative result: on this particular 60 GiB machine, partial shard staging is not the best use of the RAM budget, and the planner exposes that instead of mounting anyway.

No reserve override, swapoff, model mutation, or full-model preparation was attempted. Swap was already saturated, so overriding the safety blocker would not have been a valid benchmark.

Regression validation

env PYTHONDONTWRITEBYTECODE=1 make -C c test-python
make -C c test-c

• Python: 208 passed, 12 skipped
• C test runner: passed; environment-dependent io_uring/AVX-512/direct-I/O cases reported their normal skips
• Conflict-marker and whitespace checks passed

The opt-in host mount lifecycle could not be rerun through this automation session because sudo requires interactive authentication and this kernel disallows the isolated user-namespace mount. The earlier real-tmpfs lifecycle result remains documented in the PR, but I am not presenting it as a fresh run at this merge commit.

Remaining hardware acceptance

A full engine-level SSD/slab/direct-map token-throughput A/B still needs a host with enough free memory for the planner's reserve (about 400 GiB for this snapshot in full mode, or at least 52.8 GB for this specific 8 GiB partial benchmark plan). That run should use coli ramdisk benchmark --json, which performs one warm-up plus three deterministic 32-token runs per variant and verifies identical greedy output and physical-read accounting.

[JustVugg]

Sorry for the wait on the workflow approval — that was on me, not you. GitHub holds runs from contributors whose first PR hasn't merged yet, and I hadn't noticed how many were queued. Approved now, and CI has spoken:

• CI (build + test suites): green on all jobs ✅
• check (make check on three platforms): Linux green, Windows and macOS fail ❌

Both failures are in the new tests/test_ramdisk.py, and both look like portability assumptions rather than anything wrong with the streaming work itself:

Windows — AttributeError: <module 'os'> does not have the attribute 'statvfs' in test_system_score_uses_concurrent_aggregate_rss_and_mount_shmem. os.statvfs is POSIX-only; on Windows it doesn't exist, so the test errors before it can assert anything. Either skip the test there or gate the call.

macOS — RamdiskError: private state path contains a symlink, in three tests. This one is interesting and worth a careful look: on macOS /var is a symlink to /private/var, so tempfile.mkdtemp() hands you a path that legitimately contains one. Your symlink guard is doing exactly what it should; it's the test that feeds it a path macOS resolves differently. Path.resolve() on the temp root before building the state path should settle it — but please check whether the guard itself needs to tolerate a resolved-equal symlink in production too, because a Mac user pointing state at anything under /var will hit the same refusal for real. That would be a genuine bug this test just found for you.

Nothing here touches the substance of the PR, which is large (+7,717) and which I still owe a proper read. Fix the two platform issues and I'll go through it. And since make check runs on all three platforms in CI, running it locally on Linux only won't catch this class — worth knowing for next time.

[JustVugg]

Follow-up: this now also conflicts with dev (colibri.c moved in #532 today), on top of the two CI failures from the run I approved earlier.

Recap of where it stands, so nothing is ambiguous:

• CI (build + tests): green ✅
• check on three platforms: Linux green, Windows and macOS fail ❌ — both in the new tests/test_ramdisk.py
  • Windows: os.statvfs doesn't exist there, so the test errors before asserting anything
  • macOS: three tests hit RamdiskError: private state path contains a symlink — /var is a symlink to /private/var on macOS, so tempfile.mkdtemp() legitimately hands you one
• conflicts with current dev

The macOS one is worth a second look beyond the test: your symlink guard is behaving correctly, but a real Mac user pointing state anywhere under /var will hit the same refusal in production, not just in the test. If that's true, this PR found a genuine bug in the guard and fixing it is worth more than fixing the test.

Nothing here touches the substance, which I still owe a proper read — it's +7,700 lines and I want to give it real attention rather than a skim. Rebase, fix the two platform issues, and I'll go through it. Since make check runs on all three platforms in CI, running it locally on Linux alone won't catch this class of problem — worth knowing for next time.

[BColsey]

@JustVugg Thanks for the detailed recap — I’ve addressed each item on the current pushed head (cdaf08c):

• Rebase/conflicts: the branch is rebased onto current dev (11cd1c) and GitHub now reports it mergeable. The engine work is against c/colibri.c and the extracted c/st.h / c/telemetry.h; there are no remaining c/glm.c changes.
• Windows / os.statvfs: benchmark sampling now treats statvfs as optional (getattr(os, "statvfs", None) in c/ramdisk.py, with guarded handling in c/ramdisk_support/benchmark.py). c/tests/test_ramdisk_benchmark_module.py verifies the no-statvfs path, and c/tests/test_ramdisk_platform.py imports/runs the control plane in a fresh simulated Windows process with POSIX-only os APIs removed.
• macOS /var alias: state-oriented tests now canonicalize host-provided temporary roots with Path.resolve() (c/tests/ramdisk_test_support.py and c/tests/test_ramdisk_state_lifecycle.py). I intentionally kept production symlink rejection strict: the RAM-disk lifecycle/TUI is Linux-only and unsupported hosts stop before Linux lifecycle operations, while rejecting symlinked durable-state paths remains a safety invariant. The explicit malicious-symlink rejection coverage remains in place.
• Cross-platform coverage: .github/workflows/check.yml runs make -C c check on Linux, native MSYS2/UCRT64 Windows, and macOS, plus locked Nix checks on Linux/macOS.
• Reviewability: c/ramdisk.py was reduced from 7,419 lines to a 1,150-line compatibility facade, with the implementation split into the 15-file c/ramdisk_support/ package and tests split by responsibility. I also added docs/ramdisk-tui.md as an operator/developer guide.

Validation at this head: local make -C c check passes (400 Python tests, 34 skipped, plus all native suites), git diff --check passes, and the opt-in real-tmpfs prepare → status → destroy test passed manually (1 test in 2.735s).

One GitHub-side item remains: the new CI and check workflow runs for cdaf08c are currently marked action_required, so they need maintainer approval before the updated cross-platform matrix can run.

[BColsey]

I refactored the tui and upgraded the aesthetic and ease of use. Right now this feature is only supported on linux. subsequent updates will include macos and windows.