Long-context Pre-Training of Large Language Models

This folder contains the codes and models for our research papers on LongContext Post-Training.

• 🎉 Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model, accepted as ICLR 2025 conference paper !

• 🎉 NExtLong: Toward Effective Long-Context Training without Long Documents, ranked 1st among LLMs under 10B on the LongBench v2 leaderboard (2025/01/23) and accepted as ICML 2025 conference paper !

• 🎉 LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs, accepted as AAAI 2026 conference paper !

• 👀 EntropyLong: Effective Long-Context Training via Predictive Uncertainty, under review .

Long-context Instruct-Tuning of Large Language Models

• 🎉 LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions, accepted as NeurIPS 2025 conference paper !

Long-context Evaluation of Large Language Models

• 🔥 LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark!