Can one decoder block execute correctly across isolated local processes while every tensor transfer and failure boundary remains inspectable? This repository supplies an executable protocol and evidence path, not a new parallel algorithm or performance result.
The frozen block uses hidden size 16, eight query heads, four KV heads, head dimension two, and intermediate width 24:
X1 = X + Wo Attention(RMSNorm(X))
Y = X1 + Wdown (SiLU(Wgate RMSNorm(X1)) * Wup RMSNorm(X1))
Grouped-query causal attention assigns two query heads per KV head and applies rotary position transforms to Q and K. Seeded float32 inputs and weights are shared with a separate, standard-library float64 oracle. For R ranks, contiguous query heads, KV heads, and MLP columns divide evenly. Each rank returns attention and MLP down-projection partials; the coordinator sums them in rank or balanced pairwise order using float32. Both oracle error gates are 2e-5.
Every frame has a 36-byte network-order header:
magic | version | kind | flags | epoch | sequence | metadata bytes | payload bytes | CRC32
Canonical ASCII JSON metadata is bounded at 65,536 bytes; payloads at 16,777,216. CRC32 covers both. Per-direction sequences are monotonic, epochs reject stale traffic, and tensor metadata fixes identifier, rank, phase, shape, and little-endian float32 dtype. Phase fences reject duplicate or late partials.
A clean worker sends six frames (hello, two partials, two fences, result) and receives 13 (ack, two commands, input, seven weight shards, two reductions). With batch B, sequence T, hidden size H, and per-rank query/KV/intermediate widths q, k, i:
Eweights = 2 H q + 2 H k + 3 H i
payload bytes = 4 R (Eweights + 6 B T H)
The ledger accounts for headers and metadata separately.
The 192 clean cases cross ranks 1/2/4, sequences 1/2/4/7, batches 1/2, four seeds, and two sum orders. All completed; maximum absolute and relative L2 errors were 5.551680815e-08 and 4.640014905e-08, with byte-identical workers and exact accounting in 192/192.
The 96 four-rank fault cases cross four seeds, three phases, and rank exit, delay, checksum corruption, stale epoch, duplicate sequence, metadata mismatch, truncated frame, and cancellation race. Every case must emit no coordinator output and leave no worker after cleanup. All 96/96 met both conditions: 36 contract errors, 36 protocol errors, 24 timeouts. These are controlled failure-path tests, not incident or recovery estimates.
The package verifier checks schemas, hashes, source identity, matrix coverage, run/output identities, error gates, frame sequences, accounting, summary, and report. The independent verifier imports neither the package nor NumPy; it regenerates tensors, runs the float64 block, decodes every output, recomputes metrics and identities, and audits ledger and summaries. Tamper tests rehash altered artifacts to ensure semantic checks do not trust the manifest alone. The full gate then rebuilds all 288 cases and the SVG byte-for-byte.
Execution is one small full-sequence block on one CPU host over loopback with centralized reductions. It excludes accelerators, collectives, multi-host behavior, complete models, tokenization, KV caches, batching servers, routing, performance, cost, and real outage distributions. CRC32 detects corruption but is not authentication; hashes detect drift but do not prove authorship.
- Mohammad Shoeybi et al., Megatron-LM, grounds the column/row tensor-parallel decomposition.
- Hugo Touvron et al., LLaMA, grounds the block structure: RMSNorm, rotary attention, and gated MLP.
This implementation and its measurements are independent.