Tensor Parallelism implemented using NCCL for a hand-optimized GEMM in a PyTorch Transformer block with custom backend for causal FlashAttention in Triton. Validated and benchmarked against PyTorch single-GPU and TP standard implementations
Quick disclaimer, we don't truly save memory right now by running this because this is a pedagogical implementation of TP, so each rank still stores the full matrices and I call ncclAllGather after every columnwise GEMM, which of course in a production implementation would not be used.
Linux + CUDA + NCCL, Execution model: one process per GPU (torch.multiprocessing.spawn):
| Path | ms / forward |
|---|---|
TP ref (cuBLAS + torch.distributed + Triton attn) |
88.16 |
| TP custom (myGEMM + NCCL bind + Triton attn) | 89.94 |
*Grabbed a node off of Vast.ai
Let's begin by considering the ncclAllReduce ran on the 4xA100 node and assuming a standard ring model (which I believe nccl-tests does for busbw):
Where S = message size, p = #GPUs and B = bandwidth of slowest link in the ring.
In results/results_per_op.txt, we see that the allReduce operation took 25.616ms. Treating latency as negligible (we can because of the massive message size) and using p = 4 and
We can rearrange to get
(Verified with allGather at 12.901 ms results in 3.901 GB/s, approximates same busbw)
Pretty slow... shouldn't NVLink bandwidth be in the hundreds of GB/s?
We can get a hint when looking at the adjacency list in results/gpu_link_stats.txt from running nvidia-smi topo -m:
GPU0 GPU1 GPU2 GPU3
GPU0 X NV12 NODE NODE
GPU1 NV12 X NODE NODE
GPU2 NODE NODE X NV10
GPU3 NODE NODE NV10 X Of the 4 GPUs in my node, NVLink is only connected for GPUs 0<->1 and 2<->3. In ring reduce, the bandwidth of the slowest link is the limiting bandwidth of the whole op.
Since the slowest link is marked "NODE," we know that the path between the other GPUs goes through a PCIe CPU host bridge. Additionally, we know from the topo -p2p output that GPU Direct P2P is available between all the GPUs as well, including ones without NVLink.
By running main.py after export NCCL_DEBUG=INFO, we get this block of lines:
dbaaa2700fd2:1893:1893 [3] NCCL INFO Channel 02 : 3[3] -> 0[0] via SHM/direct/direct
dbaaa2700fd2:1891:1891 [1] NCCL INFO Channel 02 : 1[1] -> 2[2] via SHM/direct/direct
dbaaa2700fd2:1892:1892 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dbaaa2700fd2:1890:1890 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dbaaa2700fd2:1890:1890 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM/readBetween GPUs 2<->3 and 0<->1 which have NVLink, NCCL chooses to use P2P. However, for all other connections, it chooses to go through PCIe SHM (shared memory, I believe this is not P2P but instead goes through host), implying that SHM could be better than the non-NVLink P2P on this node.
I tried proving this by running profile_ops.py with export NCCL_P2P_LEVEL=SYS. I thought this might allow P2P use instead of host bridge regardless of whether NVLink is available. However, the process hangs after the graph is created and I'm not able to get around it. Debug output and nvidia-smi revealed 100% utilization but relatively low power draw at around 50W/400W, and the P2P/CUMEM desired paths (such as 1[1] -> 2[2] were indeed being used, but I was unable to find the source of the hang as of yet.
So instead, I created link_bandwidth.py which measures the P2P bandwidth between every GPU, and as we can see in results/P2P_bandwidth.txt, the peak bandwidth of the NVLink nodes is an exceptional 274 and 230 GB/s, while the other nodes have a dismal 2.7 GB/s. We can see that this figure, which is the peak of non-NVLink P2P bandwidth on this node, can't even compete with SHM. It is clear why NCCL chose the SHM/PCIe route (especially considering that the other one just hangs).
- Triton attn ~3x faster than fp32 SDPA (9.8 vs 30.1 ms, kind of a trivial result but also interesting when starting from fp32 SDPA as a baseline).
- NCCL bind matches
torch.distributed(all_gather 12.9 ms, all_reduce 25.616 ms). Seeresults/results_per_op.txt - myGEMM ~8-9 TFLOPS vs cuBLAS ~13-14.5 (fp32 peak 19.5) -> ~2 ms gap.
python main.py [-o out.txt] [-B 4] [-N 16384] [--reps 3]
python main.py --shapes -o <file.txt>
python profile_ops.py [-o ops.txt] [-B 2] [-N 16384] # per-op timing (GEMMs, attention, collectives)
python kernels/benchmark.py -o <file.txt>- Optimize GEMM and place the file in this workspace
- Triton FlashAttention implementation
- Pytorch Transformer block implementation
- Add GELU gated by a bool to gemm epilogue for linear layer 1 of MLP
- Hook up QKV projections and MLP block to my GEMM backend.
- Split tensors to n GPUs
- Use NCCL to coordinate the GEMM
- Benchmark against single-GPU and multi-GPU standard pytorch implementations
- Profile with NSYS to verify bottlenecks that I discussed earlier
- Use TP for the attention launch as well
- Implement pipeline parallelism for n/2 GPU layers (n = # gpus) while keeping the other n/2 GPUs for TP
- Eventually remove extra all gathers and fully isolate GPU-specific tensor slices so we ACTUALLY save
- Switch to bf16/fp16 across the repo to utilize tensor cores and compare with production level NCCL and CUDA
- Benchmark full suite
I used AI for bindings, planning, and structuring benchmarks (essentially for scaffolding).