Skip to content

planning.md

Latest commit

 

History

History
54 lines (38 loc) · 3.98 KB

planning.md

File metadata and controls

54 lines (38 loc) · 3.98 KB

OpenWorldLib Planning

Methods Being Integrated

We are continuously expanding the World Model method library in our framework.
The representative methods currently being integrated are listed below. We warmly welcome more developers to submit PRs and help us improve the ecosystem together!

Method Paper Code Repository Main Direction / Task
Infinite-World paper code Long Navigation Video Generation
MatrixGame-3 paper code Long Navigation Video Generation
Solaris paper code Navigation Video Generation (multiplayer Minecraft world modeling)
Cambrian-S paper code Spatial Reasoning
Ctrl-World paper code VLA / Robot Manipulation (controllable generative world model)
FantasyWorld paper code 3D Scene Generation (unified video and 3D prediction)
MemFlow paper code Long video generation

If you have methods you’d like to integrate (e.g., new world models, evaluation benchmarks, or inference pipelines), feel free to open an Issue or submit a PR directly.

Todo List

Status Legend: ✅ Completed; 🚧 In Progress; 🔄 Ongoing; 📝 Not Started.

  • ✅ Integrate multiple world model tasks into a single unified framework.
  • 🔄 Continuously add cutting-edge and valuable world model-related works to OpenWorldLib.
  • 🚧 Support more works related to long-video generation.
  • 🚧 Support more works related to autonomous driving.
  • 🚧 Optimize the contribution workflow to make it easier for other contributors to submit PRs, and continuously refactor the OpenWorldLib codebase.
  • 🔄 Provide comprehensive documentation and detailed explanations for the interactions within each pipeline.
  • 📝 Support inference data parallelism across all pipelines.
  • 🔄 Streamline the codebase and simplify the outputs of the pipelines.

Code Optimization

  • The current repository may still contain some redundant code, which we will further streamline;
  • The interactions between different modules are not yet seamless enough. We will work towards enabling more flexible and natural integration between components;
  • The performance of some existing methods still requires further improvement, specifically:
    • The rendering effectiveness of vggt should improve;

Roadmap

  • Continuous Benchmark Enhancement
    We will further improve the current benchmark suite by expanding task coverage, refining evaluation dimensions, and strengthening data quality control. We also plan to increase dataset scale to improve robustness and discriminative power of evaluation results.

  • Optimization for Multi-Stream Generation
    Since interaction signals are typically variable-length inputs, the current framework is not yet optimal for high-concurrency generation. We will improve support for multi-stream generation through better scheduling, batch alignment, and throughput optimization.

  • Training Framework Development
    OpenWorldLib primarily aims to provide a standardized and extensible framework for world models and their inference pipelines, with a strong emphasis on comprehensiveness. In the next phase, we will introduce world model training frameworks built on several mainstream foundation models.

  • Datasets and Tooling Release
    Along with the training framework, we will release corresponding datasets, preprocessing pipelines, and starter examples to help users with reproduction, fine-tuning, and further development.