Hydra Release 2021.11

[knagrecha]

Current Release:

• Standard model-parallel sharding supported
• Pilot-run style partitioning
• Sharded-LRTF Scheduling
• Standard linear execution patterns for forward/backward passes
• Arbitrarily deep models can be trained on one GPU
• Near-linear speedups for end-to-end runtimes in single-node multi-GPU setting

Next Release TODOs (Target Completion: 2022.06):

• Tensor Parallel Support
• Recurrent/Autoregressive Network Support
• Cluster setting scheduler (prioritize turnaround time vs makespan)
• Disk spilling
• Multi-node scaling
• More examples
• Additional documentation
• More robust partitioner

Long-term TODOs

• (Potential) FlexFlow integration?
• Pip package
• AMD GPU support
• Unit tests
• Website
• TensorFlow support
• Data Parallel Support
• Model Selection APIs

[knagrecha]

Also, really need to get support for PyTorch 1.10 and the corresponding torchtext. The codebase itself should be fine, even have some possible optimization opportunities (see saved_intermediates branch) but the examples just break entirely when I try to upgrade PyTorch.