Repository navigation
Add PyTorch seeding for multi-GPU and multi-node jobs - #2
Merged
Merged
Conversation
- Derive seed_number as the MD5 digest of the UTF-8 input modulo 2^32, as the Python package does, so both languages give the same seed number. - Default SeedHashGenerator$new() to [0, 2^31 - 1] like create_seedhash() and Python; the default range no longer fails in generate_seeds(). - Compute draws in doubles to avoid integer overflow to NA, return an integer vector, and support ranges wider than .Machine$integer.max. - Validate bounds and count up front (whole numbers, no NA, not -2^31). - Pin the RNG kind in generate_seeds() and restore the caller's state. - Make fields read-only active bindings documented with @field. - Add set_seed(), since set.seed(seed_number) fails above 2^31 - 1. - Import R6 and digest in NAMESPACE, run the examples in R CMD check, and require R >= 3.6.0 for set.seed(sample.kind =). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old script caught every error and only printed FAILED, so CI stayed green while two of its tests failed. Move the tests to testthat, install the package in CI and run them with stop_on_failure = TRUE. Add tests for the seed values shared with Python, pinned generate_seeds() output, RNG state handling, input validation and read-only fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hierarchy nodes are keyed by seed value, so a child seed equal to an existing seed overwrote that node's parent and could make a seed its own ancestor; add_experiment_result() then looped forever. Keep the record from a seed's first occurrence, guard the parent walk against cycles, and copy the caller's metrics and metadata dicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- typing.Literal needs Python 3.8, so importing seedhash failed on 3.7. - pyproject.toml's [project] table made setuptools ignore setup.py's extras_require, so the built wheel had no extras. Declare them in pyproject.toml and install the experiment extra in CI. - torch.use_deterministic_algorithms(warn_only=True) needs torch 1.11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Binary classification metrics no longer assume labels 0 and 1, and multi-class F1 is the mean of per-class F1 (both match scikit-learn). - Flatten y_true/y_pred and reject size mismatches instead of silently broadcasting (n,) against (n, 1). - Silhouette: singletons score 0 and identical points are excluded by index; results always have the same keys. - RL improvement_rate divides by |early mean| and its windows no longer overlap for 10-19 episodes. - Stratified and systematic sampling raise clear errors for ranges too narrow to sample; outputs for valid inputs are unchanged. - generate_seeds() uses a private random.Random (same output) instead of reseeding the global random module. - A broken torch/TensorFlow install no longer breaks import seedhash, TF uses enable_op_determinism(), and CUBLAS_WORKSPACE_CONFIG is kept if already set. - Drop the import-time warning when pandas is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State that Python and R share the seed number but not the seed lists from generate_seeds(), which come from each language's own RNG. Replace the porting guide's advice to use the first 7 hex digits and -1e9..1e9 defaults, which caused the mismatch with Python. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- R docs still described the old -1e9..1e9 default, the "Range is too large" error and R >= 3.5.0; describe the 0..2^31-1 default, the full integer range, the -2^31 limit, set_seed() and R >= 3.6.0. - Replace the porting guide's outdated R implementation and advice (7 hex digits, @description for fields) and mark its old fixes superseded. - Test instructions pointed at a nonexistent tests/ directory; run pytest on the root test_*.py files and the R testthat suite instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
set_seed() and seed_all() take per_rank and rank. By default every process uses seed_number; with per_rank=True each process uses rank_seed() = (seed_number + global rank) mod 2^32, so rank 0 and single-GPU runs keep seed_number. get_global_rank() reads the rank from torch.distributed or from RANK, OMPI_COMM_WORLD_RANK, PMI_RANK or SLURM_PROCID, never LOCAL_RANK, which repeats on every node. All visible GPUs are seeded, whatever the current device. Tests launch real jobs with torchrun on one node, on two simulated nodes and with mp.spawn (gloo on the CPU); a new CI job installs CPU PyTorch to run them. Add an example and document the three setups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It stacks on unmerged PR #1 and bundles that PR's reproducibility-breaking seed-derivation changes with new distributed code whose NCCL/SLURM/MPI paths are explicitly untested, so merge-order coordination and human verification are needed.
Review effort: Balanced
Findings: None
What changed in this PR
This PR adds PyTorch seeding support for multi-GPU and multi-node training to the seedhash package. The core addition is a get_global_rank() helper plus per_rank/rank arguments on set_seed()/seed_all() and a new rank_seed() method, so the same seeding call works identically for one GPU, several GPUs on one node, and GPUs across nodes (torchrun/SLURM/MPI). Because it is stacked on top of PR #1, the diff also carries #1's correctness/packaging fixes for the Python and R packages; the distributed-seeding feature itself is the final commit (32a11fc).
Changes:
- Add global-rank detection (process group →
RANK/OMPI_COMM_WORLD_RANK/PMI_RANK/SLURM_PROCID, neverLOCAL_RANK) and per-rank seeding(seed_number + rank) mod 2^32, seeding all visible GPUs viatorch.cuda.manual_seed_all. - Add a runnable
distributed_seeding.pyexample, unit + real multi-process launcher tests, and a dedicatedtest-python-torchCI job that installs CPU PyTorch. - Carry PR #1's fixes: cross-language seed parity, R package rewrite (read-only fields, testthat suite), ML-metric corrections, and extensive doc updates.
| File | Description |
|---|---|
| Python/seedhash/core.py | Adds get_global_rank(), rank_seed(), and per_rank/rank support in set_seed/seed_all; seeds all GPUs. |
| Python/seedhash/__init__.py | Exports get_global_rank. |
| Python/examples/distributed_seeding.py | New example showing identical seeding across single/multi-GPU/multi-node. |
| test_distributed_seeding.py | New unit tests for rank detection/seeds plus real torchrun/mp.spawn launcher tests. |
| Python/seedhash/experiment.py | PR #1: sampling validation, hierarchy/cycle fixes, metric corrections, dict copies. |
| Python/README.md | Documents multi-GPU/multi-node API and relocated test commands. |
| Python/pyproject.toml | Moves extras here, bumps torch≥1.11, requires Python≥3.8, adds dev/3.12. |
| Python/setup.py | Aligns classifiers/extras (3.8+, torch≥1.11). |
| Python/requirements.txt | Bumps torch to ≥1.11. |
| .github/workflows/tests.yml | Adds test-python-torch job; runs new suites; switches R job to testthat. |
| README.md | Adds multi-GPU feature bullet, example entry, updated test commands. |
| CONTRIBUTING.md | Updates test locations/commands to repo-root test_*.py and R/tests/testthat/. |
| paper.md | Corrects cross-language seed claim. |
| PYTHON_TO_R_GUIDE.md | Rewrites hash-to-seed guidance (digest mod 2^32); updates dates. |
| JOSS_PREPARATION_SUMMARY.md | Updates listed test commands. |
| JOSS_CHECKLIST.md | Updates R test path. |
| GITHUB_INSTALL.md | Updates R integer-range guidance/defaults. |
| test_correctness.py | PR #1: regression tests for seeds, imports, metrics. |
| R/R/seedhash.R | PR #1: seed parity, read-only fields, set_seed(), RNG-state restoration. |
| R/NAMESPACE | Adds importFrom entries for R6/digest. |
| R/DESCRIPTION | Requires R≥3.6.0; adds testthat config. |
| R/NEWS.md | Documents breaking changes/new features. |
| R/man/SeedHashGenerator.Rd | Regenerated docs incl. active bindings/set_seed(). |
| R/man/create_seedhash.Rd | Doc wording/examples update. |
| R/tests/testthat.R | New testthat entry point. |
| R/tests/testthat/test-seedhash.R | New testthat suite. |
| R/tests/test_seedhash.R | Removes old pass-through test script. |
| R/examples/example_usage.R | Formats seed_number with %.0f. |
| R/README.md | Updates R version and cross-language notes. |
| R/README_PACKAGE.md | Updates default range and integer-limit guidance. |
| R/README_CRAN.md | Updates test directory layout. |
| R/INSTALL.md | Updates R version and test commands. |
| R/INTEGER_RANGE_SOLUTION.md | Updates integer-range explanation/examples. |
| R/PYTHON_VS_R.md | Updates cross-language seed explanation. |
| R/SUBMIT_NOW.md | Updates R dependency version. |
Files not reviewed (2)
- R/man/SeedHashGenerator.Rd: Generated file
- R/man/create_seedhash.Rd: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1. Merge that first. Until then, this PR also shows #1's commits; its own change is the last commit, 32a11fc.
Seeds PyTorch the same way on one GPU, on several GPUs on one node, and on GPUs across several nodes.
Changes
set_seed()andseed_all()take two new arguments,per_rankandrank.seed_number. DistributedDataParallel copies rank 0's weights to every rank, so model initialization stays identical either way.per_rank=True, each process usesrank_seed() = (seed_number + global_rank) mod 2^32, so ranks draw different augmentations and dropout. Rank 0, and any single-GPU run, keepsseed_number.get_global_rank()reads the rank fromtorch.distributedonce it is initialized. Otherwise it readsRANK,OMPI_COMM_WORLD_RANK,PMI_RANKorSLURM_PROCID. It never usesLOCAL_RANK, which repeats on every node and would give processes on different nodes the same seed.mp.spawnsets none of those variables, so the rank can be passed explicitly.torch.cuda.manual_seed_all, so the result doesn't depend on which device is current.Python/examples/distributed_seeding.py, plus a multi-GPU / multi-node section in the Python README.Testing
Unit tests run everywhere, with no PyTorch needed.
Launcher tests start real jobs on the CPU with the gloo backend:
mp.spawnwith an explicit rankEach global rank gets the same seed whether the job runs as 4 processes on one node or as 2 × 2 across nodes.
A new CI job,
test-python-torch, installs CPU PyTorch to run the launcher tests.Not tested: real GPUs with NCCL, and real SLURM or MPI launches. The SLURM and MPI rank variables are covered by unit tests only.
🤖 Generated with Claude Code