Skip to content

Add CUDA distributed training runtime - #1

Merged
antlobach merged 7 commits into
mainfrom
feat/cuda-distributed-training
Aug 9, 2026
Merged

antlobach merged 7 commits into
mainfrom
feat/cuda-distributed-training

Conversation

@antlobach

@antlobach antlobach commented Aug 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • upgrade JavaCPP PyTorch/CUDA presets and expose CUDA device controls
  • add NCCL collectives, managed rank launch, DDP, distributed sampling, AMP, and rank-zero checkpoints
  • commit reduced gradient buckets only after NCCL completion so every model parameter stays synchronized
  • document Java, Clojure, CUDA 13.1, cuDNN, NCCL, multi-GPU setup, launch, training, and validation
  • test the CPU suite across Java 21, 22, 23, 24, and 25

Verification

  • Linux Java 25 CPU release suite: 163 tests, 722 assertions, 0 failures, 0 errors
  • dual RTX A5000 GPU release suite: 163 tests, 722 assertions, 0 failures, 0 errors
  • CUDA smoke: two-rank NCCL/DDP, all-parameter synchronization, AMP, fused attention, checkpoint restore, worker failure propagation, and process cleanup passed
  • deterministic two-rank Clorch/PyTorch comparison: 12 losses and trained parameters were float32-exact; maximum decimal error was 1.4984131269102363E-8
  • clj-kondo MCP: 0 errors and 0 warnings in changed Clojure files
  • MkDocs strict build passed

Use direct NCCL bindings because current JavaCPP ProcessGroup constructors cannot safely model intrusive pointers.
@antlobach
antlobach marked this pull request as draft August 9, 2026 00:25
@antlobach
antlobach marked this pull request as ready for review August 9, 2026 01:10
Gradient hooks could return before NCCL results reached every parameter, leaving later gradients rank-local. Commit reduced buckets after waiting for collective completion.
@antlobach
antlobach merged commit 07642ac into main Aug 9, 2026
5 checks passed
@antlobach
antlobach deleted the feat/cuda-distributed-training branch August 9, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant