Skip to content

Latest commit

 

History

History
65 lines (44 loc) · 2 KB

README.md

File metadata and controls

65 lines (44 loc) · 2 KB

🏗️ Loom-Train

A Simple & Efficient Training Framework for Long-Context LLMs

Optimized for scalability, memory efficiency, and seamless integration — built to unlock the full potential of long-context large language models.

📅 Update Log

  • 📅 2025-10-07 — 🚀 Initial Release: Loom-Train is now live!
    ✅ Native support for 🤗 Hugging Face Trainer
    ✅ Optimized attention with 🌀 Ring-Flash-Attention
    ✅ Lightweight, plug-and-play design for long-sequence training (128K+ tokens)

✨ Key Features

  • 🔧 Plug-and-Play: Drop-in replacement for HF Trainer — no major code changes needed.
  • 🚀 Memory-Efficient: Leverages Ring-Flash-Attention to reduce GPU memory footprint by up to 50%.
  • 📈 Scalable: Seamlessly scales to 100K+ context lengths without sacrificing speed.
  • ⚡ Fast Setup: Minimal dependencies, easy installation via pip install loom-train.

💻 Environment & Installation

To install theloomtrain package from the gitee repository, run:

git clone https://github.com/LCM-Lab/LOOM-Train.git
conda create -n loom_train python=3.10 -y
conda activate loom_train
cd LOOM-Train/loomtrain
pip install -e .

To install flash attention, run the command below to obtain the required flah-attn version:

loomtrain-required-flash-attn

Download the suitable version of flash_attn from https://github.com/Dao-AILab/flash-attention/releases

pip install <path_to_flash_attn_whl_file>
pip install ring_flash_attn

🛠️ Getting Started

Coming soon ...

🤝 Contributing

We welcome contributions! Whether it’s bug fixes, new features, or documentation improvements — feel free to open an issue or PR.
Let’s build the future of long-context training, together. 💪

📬 Contact

Questions? Suggestions? Reach out at: iiiigray19@gmail.com and zctang2000@gmail.com