rpc: cut the protocol cost of uploading weights to a remote server

[danielhanchen]

Loading a model that is split across two machines spends most of its time pushing the
remote half of the weights over RPC, and most of that time is protocol overhead rather
than wire time. On a pair of DGX Sparks with 111 Gb/s RoCE rails a 27B Q4_K_XL layer
split takes about fourteen minutes to become ready while both CPUs sit at 99 percent and
both GPUs read zero. This branch attacks the protocol side of that.

Where the time goes

Measured on a single node, with an ggml-rpc-server bound to the RoCE interface so the
whole transport is exercised (RDMA active, mtu=4096), pushing all 15.7 GiB of
Qwen3.8-27B-UD-Q4_K_XL to the remote backend. Numbers from the load profiler added here
(GGML_RPC_LOADPROF=1), for the span from the first SET_TENSOR to the last:

	base	this branch
upload span	41.5 s	23.5 s
hashing	12.6 s	0
host staging	4.3 s	0
wire	4.3 s	4.9 s
client side stalls between calls	20.4 s	18.6 s
whole run, wall	47.4 s	29.1 s

876 SET_TENSOR calls carry the 15.7 GiB; 369 of them are over 10 MiB and were the ones
being hashed. The client side stalls are four events, one of them 10.7 s, and are not per
tensor overhead: they are outside the RPC calls and are left for a separate change.

What changed

Everything is inside ggml/src/ggml-rpc/. GGML_RPC_LOAD_OPT=0 restores the previous
behaviour on both sides for A/B.

The hash pass is skipped when it cannot help. SET_TENSOR_HASH lets a server that
keeps a tensor cache answer "I already have this" instead of taking the upload. A server
started without -c has no cache, so the answer is always no and the FNV-1a pass over
every tensor above 10 MiB is pure cost, about 1.2 GiB/s of pure serial multiply. The
server now advertises whether it has a cache in the byte of the HELLO response that used
to be padding, and the client only hashes when that bit is set. The message keeps its
size and its version, an older server sends a zero byte and an older client ignores it,
so old and new interoperate in both directions.

No staging copy for the upload. The client used to allocate a zero filled buffer the
size of header plus tensor, copy the header and then the tensor into it, and send that.
For a 27B split that is several gigabytes of zero fill and several more of copy. The
header and the caller's payload are now written straight to the socket. The bytes on the
wire are byte for byte what they were.

The server streams the message instead of buffering it. RPC_CMD_SET_TENSOR is read
off the connection: header first, then the payload directly into the destination when the
backend buffer is host memory, or into one reused staging allocation that is never zero
filled when it is not (CUDA, or when a cache dir means the payload has to be hashed). It
used to resize() a fresh vector per message, which zero fills, and then copy out of it.

The RDMA transport stops waiting for every chunk. rdma_send posted one 256 KiB
chunk and polled it to completion before posting the next, so the link was idle for a
full round trip on every chunk and only one chunk was ever in flight. It now keeps up to
eight in flight against a ring of registered buffers and drains them at the message
boundary that flush() already marks. The receive ring is 24 deep, so the sender cannot
outrun it, and the number of send slots degrades down to one if a tight memlock limit
refuses the registration.

Receives are consumed byte by byte. rdma_recv used to copy a whole completed
receive into the caller's buffer and decrement by the full byte_len, so a peer whose
message framing differed would have the remainder of a frame dropped, or would overrun
the caller's buffer. A completion is now consumed across as many recv_data calls as it
takes. That is what makes the split write above safe against an older peer.

Interoperability

All four combinations were run on the 27B, each loading the whole model over RPC and
generating the same text:

client	server	upload span	wall
old	old	44.1 s	49.7 s
new	new	23.2 s	29.4 s
old	new	40.4 s	46.2 s
new	old	30.1 s	35.5 s
new	new, GGML_RPC_LOAD_OPT=0	41.5 s	47.4 s

New client against old server keeps the hash saving and the send side pipelining but not
the server side streaming receive, which is why its wire time is higher than new against
new. Old client against new server keeps the streaming receive only.

Non-RPC workloads

The whole diff is three files in ggml/src/ggml-rpc/. No file outside that directory is
touched, so a build without the RPC backend compiles identical code.

• -DGGML_RPC=OFF -DGGML_CUDA=ON configures and builds clean, all 81 targets.

• test-backend-ops -b CUDA0 on the new build: 2/2 backends passed, OK.

• Single GPU greedy, no --rpc, three prompts, 64 tokens, temperature 0, seed 1:
  md5 111e44cd31da973afbe5aa190b2fbed7 for base and for this branch. The raw output
  differs only in the tokens per second footer llama-cli prints, which is not the same
  from run to run on either build.

• Single GPU llama-batched-bench, no --rpc, 27B, -npp 512 -ntg 128 -npl 1,4,
  base / new / base in one window with the new build inside the bracket:

  arm	pp B=1	tg B=1	pp B=4	tg B=4
  base	823.12	11.67	826.69	38.27
  new	814.45	11.64	826.93	38.34
  base	823.25	11.67	827.77	38.20

Pair

Two DGX Sparks, 27B UD-Q4_K_XL split across both,
--device CUDA0,RPC0 -sm layer -c 16384 --parallel 32 --cache-ram 0 -t 6, 8538 MiB of
weights going to the peer, RDMA active on both ends. Load is the model loaded stamp in
the llama-server log; the arms alternate base, new, base, new in one window, both nodes
uncapped:

arm	load	prefill	decode
base	19.50 s	569.9 t/s	11.64 t/s
new	6.57 s	572.7 t/s	12.10 t/s
base	18.60 s	587.2 t/s	12.16 t/s
new	10.71 s	524.5 t/s	11.16 t/s

Load 19.1 s to 8.6 s on the mean of two arms each. Prefill and decode are unchanged, as
expected: nothing on the inference path changed.

Greedy output over the split is byte identical in all four arms,
md5 371c236256c5c537b2e72ed4add8571b (400 token prompt, 64 tokens, temperature 0).

The client side breakdown of a base load on the pair, from the profiler: of the 19.4 s,
6.85 s is hashing, 2.39 s is host staging and 3.06 s is wire, so 12.3 s of the load is
inside the RPC calls and a third of the whole load is a hash whose answer is thrown away.

Profiler

GGML_RPC_LOADPROF=1 prints, on both client and server, the number of SET_TENSOR
calls, the bytes, and the split between hashing, staging, wire and the gaps between
calls, plus a per command count and time on the client. It is off by default and costs
one relaxed atomic load per call when off.