2025.05 · Release of First Work: DeepMath-Creative Benchmark
2025.05 · Paper "DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models" Published on arXiv
2026.03 · Next Frontier: Exploring LLMs' Capacity to Originate Mathematical Research Problems
2026.03 · Historic Release: 665 Expert-Verified Research-Level Problems in Differential Geometry, Conceived by an LLM Agent — The DeepMath-Generator Series
DeepMath is more than just another AI project—it's a bold exploration into the unknown territories of machine creativity. Initiated by the DeepMath team at the School of Mathematical Sciences, Tongji University, our mission is audacious: to train a large language model that doesn't just solve math problems, but thinks, creates, and discovers at the level of a PhD mathematician.
We are not merely chasing higher scores on existing benchmarks. We are asking the questions that matter:
- Can machines truly create mathematics, or merely mimic it?
- Could an AI become a genuine collaborator in pushing the frontiers of human knowledge?
Through rigorous evaluation and groundbreaking generation of novel research problems, DeepMath stands at the intersection of artificial intelligence and pure mathematics, challenging the very definition of creativity itself.
DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models
The AI world is obsessed with reasoning. Benchmarks abound for testing how well models solve known problems. But what about the ability that defines true mathematicians—the ability to create, to construct, to innovate? This critical dimension of mathematical intelligence has remained in the shadows, unmeasured and unexplored.
Until now.
As the inaugural work of the DeepMath initiative, we dared to measure the unmeasurable. We introduced DeepMath-Creative—the first benchmark specifically designed to probe the mathematical creativity of large language models. Featuring carefully crafted construction problems across algebra, geometry, and analysis, this dataset pushes models beyond pattern recognition into the realm of genuine structural insight.
Our findings reveal a profound gap between reasoning and creativity:
- Under generous grading (focusing on core ideas, forgiving minor errors), the top-performing model—OpenAI's O3 Mini—managed only 70% accuracy on undergraduate-level creative problems.
- As problem complexity increased, performance collapsed dramatically.
- For truly open-ended challenges, models offered little more than silence.
The verdict? Today's AI excels at recombining memorized patterns but struggles profoundly with genuine creative understanding. DeepMath-Creative doesn't just expose this limitation—it provides the roadmap for overcoming it.
The DeepMath-Creative dataset is fully open-sourced and available in the DeepMath/DeepMath-Creative/datasets/ folder of this project.
[Can LLM generate interesting mathematical research problems?]
If creativity can be measured, can it also be generated? This question drove our second, even more ambitious exploration.
We built DeepMath-Generator—an advanced LLM-powered agent system with a singular mission: to conceive mathematical problems that have never existed before. Not textbook exercises. Not variations of known theorems. But genuine, research-level questions that could occupy mathematicians for years.
665 original research-level problems in differential geometry.
Every single one generated entirely by our LLM agent. But we didn't stop there. Each problem underwent rigorous verification by domain experts, who confirmed what we had hoped but hardly dared to believe:
- Many problems were completely unknown to specialists in the field
- They possessed authentic research value—not mere mathematical curiosities
- They opened new avenues for exploration, suggesting directions no human had yet considered
This is not pattern recognition. This is not recombination. This is the emergence of machine creativity—an AI system functioning not as a calculator, but as a mathematical discoverer.
The complete collection—665 expert-verified, LLM-generated research-level problems in differential geometry—is now publicly available in the DeepMath/DeepMath-generator/problems/ folder.
This is more than a dataset. It's a glimpse into the future of mathematical discovery.
Welcome all friends interested in mathematics and machine learning to join the DeepMath community!
Whether you are a mathematics researcher, a machine learning engineer, or an enthusiast passionate about the intersection of AI and science, you can find like-minded partners here to jointly explore the infinite possibilities of mathematics and artificial intelligence.
- Contribute Code: Participate in dataset construction, model training, and evaluation system development
- Pose Questions: Share your mathematical problems to enrich our evaluation set
- Collaborate on Research: Jointly explore cutting-edge issues of large models in the realm of mathematical creativity
For any questions, suggestions, or collaboration inquiries, please feel free to contact us via email: [xychen100@tongji.edu.cn]
DeepMath stands on the shoulders of every researcher, contributor, and supporter who believes that mathematics and artificial intelligence, together, can achieve something extraordinary. Thank you for being part of this journey.