feat(p2p): add direct RDMA reads into GPU KV cache

[GentleCold]

Cross-node KV hits currently pass through requester host staging before being copied into GPU KV slots. This draft adds opt-in RDMA READs from remote registered host memory directly into GPU KV slots after vLLM allocates destination blocks. Set PEGA_DIRECT_GPU_RDMA=1 on the participating processes. Direct-mode errors fail explicitly; no host-staging fallback is added.

The initial scope is dense attention storage group 0, including multi-layer and split K/V destinations. Page-first and recurrent/HMA direct layouts remain unsupported. Direct-loaded data is neither inserted into the requester host ReadCache nor advertised as a persistent owner. Ordinary host staging remains an independently selected reference/control.

Implementation

• Carry ordered direct plans through protobuf, PyO3, scheduler/TP queries and the existing query lease. Reuse the existing 600-second expiry and consume/release lifecycle; no additional TTL mechanism.
• Export DMA-BUF from the Python CUDA allocation owner with its device context bound. CUDA IPC importers cannot re-export this allocation on the tested B300 stack. Transfer real FDs through a registration-scoped Unix socket using SCM_RIGHTS and one-use capabilities; serialized descriptors contain allocation/view metadata, never a process-local FD integer.
• Validate the full allocation and view offsets, deduplicate shared allocations and register each configured NIC eagerly for Python IPC registrations. Native same-process callers can export their own allocations. MR entries and queued operations retain the allocation/FD owner; a GPU context guard spans normal asynchronous loads.
• Resolve destination block ranges, hold owner transfer locks before short-response validation, drain accepted batch receivers and verify completion bytes. Flush GPUDirect writes with cuFlushGPUDirectRDMAWrites(CURRENT_CTX, TO_OWNER) before reporting success. RC transport failure teardown remains a draft blocker below.
• Correct dense standardized [B,H,N,C] registration: group physical rows when physical blocks are smaller than logical blocks, as already done for MLA. On vLLM 0.29, 64-token physical rows back 128-token logical blocks. This shared connector defect caused incorrect cache-hit outputs on both raw master and the initial feature build.
• Add independent-process FD tests, real two-GPU gRPC byte verification, dense-layout regression cases, and a non-vacuous long same-process warm prompt in the standard vLLM E2E fixture.

Validation at da22d2d

Maiyi: one 8×B300 host, CUDA 13/RDMA, vLLM 0.29, torch 2.13+cu130, NIXL 1.4.1. Only GPUs 0–3 were used; production GPUs 4–7 were untouched. Local/remote SHA256 values match all 22 files changed by this repair.

• Linux source-only default gate: 308 passed, 14 deselected, with only pytest/numpy/requests. Mac: 307 passed, 1 Linux-only skip, 14 deselected.
• Release Rust gate: 344 passed, 8 ignored including doc ignores. Workspace/all-targets CUDA 13/RDMA Clippy with -D warnings passed. Only GPU 3 was exposed to this Rust gate; cases requiring additional GPUs can return early.
• Final server/native wheel build, Rust format, changed Python Ruff checks and local prek passed. CUDA release/Clippy hooks were run remotely instead of on the Mac.
• Real direct gRPC integration: passed, without a hardware skip. Independent Python owner, metaserver and two services; GPU 0→1 allocation setup with remote host memory→destination GPU reads, two split K/V layers sharing an allocation, 128-byte view offset, single/contiguous/fragmented destination blocks, and whole-tensor byte equality including untouched locations. This is same-host cross-process coverage, not two physical nodes.
• Standard vLLM correctness E2E: 5/5 passed in 867.02 s, covering cold, same-process warm, restart warm and partial-prefix cases; 50 blocks actually loaded. This uses a separate host-stage control and is not itself direct cross-node E2E.

Longer PD replay and output verification

Qwen3-0.6B, 2P2D/TP1, 8 GiB host pool per instance, concurrency 1, one output token, no explicit warmup or retries. Both variants independently replayed the full 19,610-record trace: 11,188 records exceed the 32K context limit, leaving the same 8,422 eligible requests, with zero preparation failures. The underlying tar SHA256 is d142c26674bada87414ab37f26587ed103a74a8aef68c250b563bd7127b2ecf3; lossless recompression for transfer changed only the gzip container.

Raw master also has the dense-layout defect. The control is master 1a20960 with only the common BHNC mapping repair in worker.py, using unchanged master native/server binaries. Feature is da22d2d, based on 2ab8fbf; the master snapshot additionally includes #451. Feat ran first, then the control, each from a cold service start.

Run	Successful requests	Measured replay	Mean TTFT	p50	p99	Throughput
Master + BHNC repair, host staging	8,422/8,422	1,581.45 s	187.46 ms	117.35 ms	578.64 ms	5.32550 req/s
Feature, direct GPU	8,422/8,422	1,582.42 s	187.53 ms	116.22 ms	583.59 ms	5.32223 req/s

Both had zero HTTP failures/retries and zero load failures. Startup, trace preparation, output checks and cleanup are excluded from measured replay; total wall time was 3,304.01 s for control and 3,308.81 s for feature. Workload manifests, normalized replay configurations, per-request index/token/block/completion/attempt shapes and all six synthetic output fingerprints match across variants.

Feature recorded 1,387 direct successes, 0 direct errors, 0 MR registration failures and 0 pinned-pool allocation failures during replay. Post-replay cache-hit output checks matched independent D0 recomputation 6/6 on each variant; feature added 7 direct successes in that separate window. The replay exercised cache churn: 1,411,413 feature and 1,447,213 control block evictions.

Control recorded 735 successful host RDMA fetches and 1 failed fetch. At 05:29:19, a 256 MiB NUMA0 fetch allocation failed (global pool usage 3.8/8.0 GiB); the prefetch returned 0/203 remote blocks. The connector retained 12 local hit blocks and the request still completed through ordinary computation. The pool-allocation and host-fetch error counters each increased by one. HTTP/load success therefore does not mean the control transport had no errors. Generic host-tier counters do not include direct loads, and direct-operation counts cannot be compared directly with host block counts.

Feature mean TTFT was 0.036% higher, p99 0.857% higher, and throughput 0.061% lower. This single pair shows no end-to-end speedup. In request-order quarters, relative mean-TTFT changes were -1.295%, +0.457%, +0.293%, +0.650%; input lengths vary by quarter. It remains one host, one model, TP1 and concurrency 1, not a repeated or two-node performance claim.

During the final approximately 15 minutes, both variants had constant GPU memory, bounded server FD counts, and less than 4 MiB RSS variation per server. Feature FD counts stayed fixed. There were no resource/metrics sampling failures. Fixed 8 GiB pools and RSS including shared mappings do not establish pinned-host memory savings or prove every failure-path lifetime.

HTTP replay success is separate from the six synthetic output checks; fingerprints for every trace response were not recorded. The additional 1032-token cold PD probe still differs from recomputation in both variants. The same fingerprint difference was previously reproduced with PegaFlow removed and pure NIXL enabled; the internal NIXL/vLLM cause remains unresolved. Both long runs logged unregister RPC connection refused during SIGTERM cleanup after validation, so teardown is not claimed warning-free.

Before ready for review / merge

• Add startup capability/version negotiation so an older server cannot silently ignore direct mode.
• Drain/reset RC session in-flight work on post/CQ errors before notification. Receiver draining alone does not fix backend early returns, and timeout draining can wait indefinitely on a wedged transport.
• Establish concurrent registration/unregister and fault-path allocation/MR/lock lifetime with injected transport failures.
• Complete real two-node, multi-TP and multi-NIC byte correctness and vLLM E2E, plus repeated latency and memory comparisons.

Long-run evidence: /home/linux2/kexi/pd_bench/direct_gpu_long19610_20260917{,_feat,_master-layout}/. Local raw copies, paired analysis and report: .local/pd_bench/artifacts/maiyi-direct-gpu-long-20260917/. Earlier build/integration/E2E and pure-NIXL isolation evidence remains in .local/pd_bench/artifacts/maiyi-direct-gpu-fix-20260917/ and the corresponding remote directories. All test processes exited; GPUs 0–3 were free after each run. Production GPUs 4–7 stayed at 273,102 MiB throughout resource samples, and PID 25188/kern-serve remained alive. No product source changed during this longer replay.

Tracking: AIINF-286. Keep this PR Draft and the issue In Progress until the remaining acceptance criteria are met.