antlobach · Aug 9, 2026 · Aug 9, 2026 · Aug 9, 2026 · Aug 9, 2026 · Aug 9, 2026
@@ -34,21 +34,27 @@ concurrency:
 
 jobs:
   test:
-    name: Clojure tests
+    name: Clojure tests (Java ${{ matrix.java }})
     runs-on: ubuntu-latest
     timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        java: ["21", "22", "23", "24", "25"]
+
     env:
       CLORCH_FORCE_CPU: "1"
+      JAVA_TOOL_OPTIONS: "--enable-native-access=ALL-UNNAMED"
 
     steps:
       - name: Check out source
         uses: actions/checkout@v4
 
-      - name: Set up Java 17
+      - name: Set up Java ${{ matrix.java }}
         uses: actions/setup-java@v4
         with:
           distribution: temurin
-          java-version: "17"
+          java-version: ${{ matrix.java }}
 
       - name: Set up Clojure CLI
         uses: DeLaGuardo/setup-clojure@13.4

@@ -174,6 +174,8 @@ This document tracks what's been ported from PyTorch to Clorch, and what still n
 | - `bce-loss`, `bce-with-logits-loss` | [x] |
 | **Similarity** | [x] |
 | - `cosine-similarity`, `pairwise-distance` | [x] |
+| **Fused Attention** | [x] |
+| - `scaled-dot-product-attention` | [x] |
 
 ---
 
@@ -203,17 +205,44 @@ This document tracks what's been ported from PyTorch to Clorch, and what still n
 | `IDataset` protocol, `dataset`, `defdataset` | [x] |
 | `dataloader` (Thread & Process workers) | [x] |
 | `tensor-dataset` | [x] |
+| `distributed-sampler`, `sample-indices`, `set-epoch!` | [x] |
+| `sampler-state`, `load-sampler-state!` | [x] |
 
 ---
 
 ### CUDA Support (`clorch.cuda`)
 
 | Feature | Status |
 |---------|--------|
-| `available?` | [x] |
+| `available?`, `device-count` | [x] |
+| `set-device!`, `current-device` | [x] |
+| `synchronize` | [x] |
+| `manual-seed`, `manual-seed-all` | [x] |
 
 ---
 
+### Automatic Mixed Precision (`clorch.amp`)
+
+| Feature | Status |
+|---------|--------|
+| `autocast`, `call-with-autocast` | [x] |
+| `grad-scaler`, `scale-loss`, `backward!`, `step!` | [x] |
+| `scaler-state`, `load-scaler-state!` | [x] |
+
+---
+
+### Distributed Training (`clorch.distributed`, `clorch.nn.parallel`)
+
+| Feature | Status |
+|---------|--------|
+| NCCL process-group lifecycle and rank metadata | [x] |
+| All-reduce, broadcast, reduce, gather, reduce-scatter, all-to-all | [x] |
+| Point-to-point send/receive and barriers | [x] |
+| Managed local worker launch, status, logs, cancellation, failure propagation | [x] |
+| `DistributedDataParallel`, bucketed gradient synchronization, `no-sync` | [x] |
+| Rank-zero model, optimizer, sampler, scaler, CPU RNG, and training checkpoints | [x] |
+---
+
 ## What's Not Yet Implemented
 
 ### Specialized Tensors & Ops
@@ -262,10 +291,12 @@ This document tracks what's been ported from PyTorch to Clorch, and what still n
 | Autograd | 5 | ~5 |
 | NN Layers | ~80 | ~20 |
 | Optimizers | 7 | ~10 |
-| Data | 5 | ~5 |
-| CUDA | 1 | ~10 |
-| **Total** | **~268** | **~60** |
+| Data | ~8 | ~4 |
+| CUDA | ~7 | ~8 |
+| AMP | ~6 | ~4 |
+| Distributed | ~20 | ~10 |
+| **Total** | **Pending version-pinned recount** | **Pending version-pinned recount** |
 
 ---
 
-*Last updated: 2026-03-13*
+*Last updated: 2026-08-08*
@@ -10,13 +10,14 @@
   <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-4c1.svg" alt="MIT license"></a>
 </p>
 
-Clorch is a Clojure deep-learning library backed by LibTorch. It provides PyTorch-style tensors, autograd, neural-network modules, optimizers, data loading, and explicit CPU/CUDA device placement through a REPL-friendly API.
+Clorch is a Clojure deep-learning library backed by LibTorch. It provides PyTorch-style tensors, autograd, neural-network modules, optimizers, data loading, explicit CPU/CUDA placement, and NCCL distributed training through a REPL-friendly API.
 
 ## Highlights
 
-- **Tensor and training APIs:** tensor operations, autograd, losses, optimizers, data loaders, state dictionaries, and native-memory scopes
+- **Tensor and training APIs:** tensor operations, autograd, losses, optimizers, data loaders, state dictionaries, AMP, and native-memory scopes
 - **Model building:** standard layers, custom `nn/defmodel` modules, architecture summaries, and checkpoint loading
-- **LLM components:** RMSNorm, RoPE, grouped-query attention, SwiGLU, causal masks, KV caches, and autoregressive generation
+- **Distributed CUDA:** NCCL collectives, managed rank processes, distributed sampling, synchronous DDP, gradient accumulation, and rank-zero checkpoints
+- **LLM components:** RMSNorm, RoPE, grouped-query attention, SwiGLU, fused scaled-dot-product attention, causal masks, KV caches, and autoregressive generation
 - **NanoChat-inspired example:** a compact, single-device Llama-style training and chat demo with checkpointing and optional KV caching, inspired by [Karpathy's NanoChat](https://github.com/karpathy/nanochat)
 - **CPU and CUDA:** native LibTorch execution with explicit tensor and model placement
 
@@ -37,6 +38,18 @@ Start a REPL from the project directory:
 clj
 ```
 
+### Supported runtime
+
+| Component | Supported version |
+|---|---|
+| Java | OpenJDK/Temurin 21 through 25; CI runs the full 21–25 matrix |
+| Clojure | Clojure 1.12.x |
+| Clojure CLI | A current 1.12.x CLI; CI uses `1.12.5.1664` |
+| Native PyTorch | JavaCPP PyTorch `2.10.0-1.5.13` |
+
+Use a JDK, not a JRE. Java 24 and 25 should be started with
+`--enable-native-access=ALL-UNNAMED` for JavaCPP native loading.
+
 ## Quick start
 
 ### Tensors and autograd
@@ -103,11 +116,14 @@ Non-trainable params: 0
 
 The constructor arguments configure the model, the binding vector registers modules or parameters, and the `forward` form defines execution. Registered fields participate in `nn/parameters`, `nn/to`, and state dictionaries. Read [Custom Models with `defmodel`](docs/nn.md#custom-models-with-defmodel) or the [minimal source example](examples/simple.clj).
 
-## CPU and CUDA
+## CPU, CUDA, and multi-GPU training
 
-Clorch selects the native backend when `clorch.torch` loads. It uses CUDA when GPU natives are available and NVIDIA hardware is detected; otherwise it loads the CPU backend.
+Clorch selects the native backend when `clorch.torch` loads. It uses CUDA when
+GPU natives are available and NVIDIA hardware is detected; otherwise it loads
+the CPU backend.
 
-Backend selection does not move tensors or models. Choose a device once, then place models and data on it:
+Backend selection does not move tensors or models. Choose a device once, then
+place models and data on it:
 
 ```clojure
 (require '[clorch.cuda :as cuda])
@@ -120,9 +136,127 @@ Backend selection does not move tensors or models. Choose a device once, then pl
 Environment overrides:
 
 - `CLORCH_FORCE_CPU=1` forces the CPU backend.
-- `CLORCH_FORCE_GPU=1` requests the CUDA backend. It still requires compatible natives, hardware, and an NVIDIA driver.
+- `CLORCH_FORCE_GPU=1` requests the CUDA backend. It still requires compatible
+  native libraries, hardware, and an NVIDIA driver.
+
+### Multi-GPU requirements
+
+Distributed training uses one worker JVM per GPU and NCCL for communication.
+The managed launcher covers multiple GPUs on one Linux host.
 
-Clorch currently supports single-device training. Distributed training, DDP, FSDP, collectives, and distributed checkpointing are not implemented.
+- Two or more NVIDIA GPUs visible to the same process
+- A working NVIDIA driver with CUDA 13 support
+- CUDA 13.1 user-space libraries, including cuBLAS
+- cuDNN 9 and NCCL 2; the validated stack uses cuDNN 9.19 and NCCL 2.29.2
+- Java 21 through 25 and Clojure 1.12.x
+- One distinct CUDA device per rank
+- Enough host RAM and CUDA VRAM for one model replica per rank
+- A writable checkpoint directory and an available local TCP port
+
+For an Ubuntu host configured with NVIDIA's CUDA package repository, the
+required runtime packages are:
+
+```bash
+sudo apt-get update
+sudo apt-get install cuda-libraries-13-1 libcudnn9-cuda-13 libnccl2
+```
+
+Confirm that the host exposes each GPU before starting Clojure:
+
+```bash
+nvidia-smi -L
+java -version
+clojure -Sdescribe
+```
+
+Set native-loading variables before the JVM starts. `JAVA_TOOL_OPTIONS` is
+inherited by launcher-created worker JVMs:
+
+```bash
+export CLORCH_FORCE_GPU=1
+export LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}"
+export JAVA_TOOL_OPTIONS="--enable-native-access=ALL-UNNAMED"
+
+CLOJURE_DISABLE_RLWRAP=1 clojure -M:dev
+```
+
+Check CUDA from the REPL:
+
+```clojure
+(require '[clorch.cuda :as cuda]
+         '[clorch.torch :as t])
+
+{:available (cuda/available?)
+ :devices (cuda/device-count)}
+;; => {:available true, :devices 2}
+```
+
+### Launch the distributed training example
+
+Clone this repository and start its nREPL so the `examples` path is available.
+The example below launches ranks for physical devices 0 and 1, waits for both
+workers, and returns status plus per-rank logs:
+
+```clojure
+(require '[distributed-training :as training])
+
+(def result
+  (training/run-local!
+   [0 1]
+   {:epochs 4
+    :sample-count 1024
+    :batch-size 32
+    :accumulation 2
+    :precision :bfloat16
+    :checkpoint-path "/checkpoints/clorch-ddp.pt"}))
+```
+
+Use `:float16` for autocast with dynamic gradient scaling, or `:bfloat16` for
+autocast without a scaler. `:accumulation 2` performs two micro-batches per
+optimizer step and suppresses DDP synchronization until the final
+micro-batch.
+
+The launcher sets `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, `LOCAL_WORLD_SIZE`,
+`MASTER_ADDR`, `MASTER_PORT`, and `CUDA_VISIBLE_DEVICES` for every child. Worker
+code should place its model and batches on `:cuda`; the launcher maps that
+logical device to the assigned physical GPU.
+
+For an application worker, launch a namespace-qualified function:
+
+```clojure
+(require '[clorch.distributed :as dist])
+
+(def job
+  (dist/launch!
+   {:nproc-per-node 2
+    :devices [0 1]
+    :main 'my.training/train-worker
+    :args {:epochs 10}
+    :timeout-ms 300000}))
+
+(dist/job-status job)
+(dist/await-job! job)
+(dist/job-logs job 0)
+```
+
+Every rank must execute collectives in the same order and train from a
+disjoint `distributed-sampler` partition. Create the model on `:cuda` before
+wrapping it with `distributed-data-parallel`, call `data/set-epoch!` each
+epoch, and use `ddp/optimizer-step!` instead of stepping the optimizer
+directly. Only rank zero writes checkpoints; all ranks participate in save and
+restore barriers.
+
+Run the GPU release check after configuring the host:
+
+```bash
+CLORCH_FORCE_GPU=1 clojure -Sthreads 1 -M -m clorch.release-check --mode gpu
+```
+
+On a host with at least two GPUs, the CUDA smoke includes a two-rank DDP check
+that verifies every model parameter remains synchronized. Read
+[Distributed CUDA Training](docs/distributed.md) for worker code, collectives,
+sampling, AMP, gradient accumulation, failure handling, and checkpoint
+restore.
 
 ## Native memory
 
@@ -151,6 +285,7 @@ Full documentation: **[antlobach.github.io/clorch](https://antlobach.github.io/c
 | [Neural networks](docs/nn.md) | Modules, custom models, layers, and summaries |
 | [Functional API](docs/functional.md) | Stateless layers, activations, pooling, and losses |
 | [Optimizers](docs/optimizers.md) | SGD, Adam, AdamW, RMSprop, and Adagrad |
+| [Distributed training](docs/distributed.md) | NCCL collectives, worker launch, DDP, AMP, sampling, and checkpoints |
 | [Memory management](docs/memory.md) | Native allocation scopes and pointer lifecycles |
 | [PyTorch parity](docs/pytorch-parity.md) | Measured coverage, current gaps, and roadmap |
 
@@ -164,6 +299,9 @@ Clorch follows PyTorch concepts, but it does not expose every PyTorch symbol. Ch
 |---|---|
 | `torch` | `clorch.torch` |
 | `torch.cuda` | `clorch.cuda` |
+| `torch.amp` | `clorch.amp` |
+| `torch.distributed` | `clorch.distributed` |
+| `DistributedDataParallel` | `clorch.nn.parallel` |
 | `torch.autograd` | `clorch.autograd` |
 | `torch.nn` | `clorch.nn` |
 | `torch.nn.functional` | `clorch.nn.functional` |
@@ -178,7 +316,7 @@ When generating Clorch code:
 - Keep long-lived models and optimizers outside `with-torch`; use one scope per allocating batch or generation step.
 - Return a JVM scalar or `nil` from `with-torch` unless a tensor must escape.
 - Reuse patterns from the [examples](examples/) instead of inventing wrapper APIs.
-- Check [PyTorch Parity](docs/pytorch-parity.md) before assuming support for distributed training, AMP, Flash Attention, or `torch.compile`.
+- Read the [Distributed CUDA Training](docs/distributed.md) guide before generating multi-process training code.
 
 ## Examples
 
@@ -187,10 +325,11 @@ When generating Clorch code:
 - [Synthetic training](examples/synthetic.clj): end-to-end model training
 - [Modern Llama](examples/modern_llama.clj): RoPE, GQA, SwiGLU, and KV caching
 - [Compact Llama chat](examples/nanochat.clj): small-corpus training, checkpointing, and token generation
+- [Distributed CUDA training](examples/distributed_training.clj): NCCL workers, DDP, AMP, accumulation, sampling, and checkpoints
 
 ## Project status
 
-The cross-language comparison suite currently passes 40 of 40 numerical scenarios. The repository's tracked feature catalog marks 268 of approximately 328 capabilities as implemented, or about 81.7%. That catalog does not enumerate the entire upstream PyTorch API.
+The cross-language comparison suite currently passes 40 of 40 numerical scenarios. The tracked feature catalog predates the PyTorch 2.10 distributed-training milestone and needs a version-pinned recount before Clorch publishes another breadth percentage.
 
 Read [PyTorch Parity](docs/pytorch-parity.md) for the capability table and roadmap to 100% of a version-pinned target.
 
@@ -208,6 +347,12 @@ Run the CPU release suite:
 clojure -Sthreads 1 -M -m clorch.release-check --mode cpu
 ```
 
+Run the GPU release suite on a Linux NVIDIA host:
+
+```bash
+clojure -Sthreads 1 -M -m clorch.release-check --mode gpu
+```
+
 Run all shipped examples:
 
 ```bash

@@ -1,8 +1,6 @@
 {:paths ["src" "test" "examples"]
- :deps {org.bytedeco/pytorch-platform {:mvn/version "2.1.2-1.5.10"}
-        org.bytedeco/pytorch-platform-gpu {:mvn/version "2.1.2-1.5.10"}
-        org.bytedeco/cuda-platform-redist {:mvn/version "12.3-8.9-1.5.10"}
-        org.bytedeco/openblas-platform {:mvn/version "0.3.26-1.5.10"}
+ :deps {org.bytedeco/pytorch-platform {:mvn/version "2.10.0-1.5.13"}
+        org.bytedeco/pytorch-platform-gpu {:mvn/version "2.10.0-1.5.13"}
         cnuernber/dtype-next {:mvn/version "11.007"}
         nrepl/nrepl {:mvn/version "1.1.1"}
         org.clojure/data.json {:mvn/version "2.5.0"}