A curated list of benchmarks, evaluations, and testing frameworks for AI agents and frontier models. This resource tracks the capabilities of AI agents, providing insights into progress toward artificial super intelligence (ASI).
Last Updated: May 9, 2025
This project is maintained by Parni and Ian. Follow Supernal Intelligence for more updates.
Website: supernalintelligence.com
Join our Discord: Supernal Intelligence Discord
For more complete data and the latest information, please visit our website: supernalintelligence.com
If you see an error or want to contribute, please email i@supernal.ai or submit a PR.
- What are AI Agent Benchmarks?
- Benchmark Selection Methodology
- General Reasoning Benchmarks
- Agent-Specific Benchmarks
- Domain-Specific Benchmarks
- Multi-Modal Benchmarks
- Specialized Capability Benchmarks
- Safety and Alignment Benchmarks
- Benchmark Aggregators
- Mobile and Edge Devices Benchmarks
- Resources
- Limitations of Current Benchmarks
- Commercial vs. Open-Source Performance
- Contribution
- License
AI agent benchmarks are standardized evaluation frameworks designed to assess the capabilities, performance, and limitations of artificial intelligence systems. These benchmarks serve several critical functions:
- Progress Tracking: Measuring AI advancement over time across various domains
- Capability Assessment: Evaluating specific abilities like reasoning, tool use, and knowledge
- Comparative Analysis: Benchmarking different models and approaches against each other
- Risk Monitoring: Identifying areas where AI systems may exceed human capabilities
- Forecasting: Providing data to predict future AI development trajectories
In the context of AGI research, these benchmarks help researchers, policymakers, and the public understand the rate and direction of AI progress, potentially identifying risks and opportunities as systems become more capable.
The benchmarks included in this list are selected based on the following criteria:
- Relevance: Benchmarks must specifically test AI agent or frontier model capabilities
- Rigor: Must have a clearly defined evaluation methodology and metrics
- Reproducibility: Results should be independently verifiable
- Impact: The benchmark should be recognized or adopted by the research community
- Recency: Focus on benchmarks that represent current state-of-the-art challenges
- Diversity: We prioritize including benchmarks that cover different capabilities and domains
We regularly review and update this list as new benchmarks emerge and existing ones evolve. Our goal is to provide a comprehensive yet focused resource that tracks meaningful progress in AI capabilities.
Benchmarks that evaluate broad reasoning capabilities across multiple domains.
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| MMLU | 57-domain knowledge across academic subjects | GPT-4.1 (90.2%), Gemini 2.5 Pro (89.0%) | 89% |
| MMLU-Pro | Expert-level knowledge assessment | GPT-4.1 (80.5%), Claude 3.7 Sonnet (78.2%) | Expert |
| MMLU-X | Extended domains with cultural and interdisciplinary topics | GPT-4.5 (88.9%), Claude 3.8 (87.6%) | 92% |
| BIG-bench | 200+ tasks: logic, abstraction, social reasoning | GPT-4 (90+ tasks over 80%) | Varied |
| BigBench Hard | Most challenging BIG-bench tasks | GPT-4 (78.5%), Claude 3.7 Sonnet (76.3%) | Expert |
| ARC | Elementary-level science reasoning | GPT-4 (96.0%), Claude 3.5 Sonnet (93.0%) | 95% |
| HELLASWAG | Common-sense text completion | Claude 3.7 Sonnet (91.7%) | 95.3% |
| AGIEval | Standardized human exams (SAT, LSAT, math competitions) | GPT-4 (95% on SAT Math) | Varies |
| ARC-AGI-1 | Grid-based visual reasoning with minimal training | OpenAI o3 (87% high compute) | 73-77% |
| ARC-AGI-2 | Skill acquisition efficiency | o3 (4.0%), o1-pro (1.3%) | ~80% |
| Analogies Benchmark | Analogical pattern recognition across domains | Gemini Ultra 2 (83.7%), Claude 3.7 Sonnet (81.2%) | 95% |
| AdvGLUE | Adversarial language understanding | GPT-4.1 (73.1%), Claude 3.7 Sonnet (72.6%) | 89% |
| TruthfulQA | Truthfulness and resistance to falsehoods | Claude 3.5 Sonnet (85.0%), GPT-4.1 (85.0%) | N/A |
| Cultural Context Benchmark | Cross-cultural understanding | Claude 3.8 Universal (85.7%), Polyglot-AI 2 (84.2%) | 98% |
| RAFT | Instruction-following assessment | Claude 3.5 Sonnet (90.2%), GPT-4o (88.7%) | N/A |
Benchmarks focused specifically on evaluating AI agent capabilities.
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| GAIA | General AI assistants across difficulty levels | Manus (SOTA at all levels), Trase Agent (SOTA) | N/A |
| AgentEval 360 | Comprehensive agent capabilities | Gemini 3.0 Agent (76.2%), Claude Agent X (75.8%) | N/A |
| AgentBench | Diverse agent environments (8 domains) | Various research agents | N/A |
| AGENTS | Automated evaluation of natural thinking | Various agent systems | N/A |
| Agent S2 | GUI interaction across operating systems | Agent S2 (OSWorld SOTA) | N/A |
| AgentSims | Simulation-based agent evaluation | UI-TARS (~60%), Voyager (~58%), ReAct (~55%) | 90% |
| MLE-bench | Agent self-improvement and autonomy | Top GPT-4-based agents (~60%) | N/A |
| TP3T | Cross-platform performance benchmark | Agent S2 (52.81% Windows, 16.51% Android improvement) | N/A |
Benchmarks that evaluate capabilities in specific domains or fields.
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| GSM8K | Grade school math with multi-step reasoning | Claude 3.7 Sonnet w/thinking (97.2%), Claude 3 Opus (97.2%) | 97% |
| MATH | Advanced math competition problems | Gemini 2.5 Pro (75.0%), Claude 3.7 Sonnet (73.0%) | 90% |
| AIME | Advanced high school math competition | Grok 3 Beta (93.3%), o3-mini (87.3%) | 90% |
| MGSM | Grade school math in multiple languages | Claude 3.5 Sonnet (91.6%), Llama 3.1 405b (91.6%) | 97% |
| FrontierMath | Extremely difficult math problems | Grok 3 Beta (~35%), o3 (~33%) | N/A |
| OlympiadBench | Olympiad-level math and physics | GPT-4V (17.97%) | Expert |
| OlympicArena | 7 Olympic competition disciplines | GPT-4o (39.97%) | Expert |
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| GPQA | Graduate-level scientific reasoning | Claude 3.7 Sonnet w/thinking (84.8%), Grok 3 (84.6%) | 89% |
| SuperGPQA | 285 graduate-level disciplines | Various frontier models | Expert |
| HLE | Safety-critical, long-horizon reasoning | DeepSeek R1, GPT-4, Claude 3.7 (all ~67%) | N/A |
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| HumanEval | Python code generation | Claude 3.5 Sonnet (92.0%), Claude 3 Opus (89.8%) | 92% |
| HumanEval++ | Advanced programming problems | Claude 3.7 Opus (83.5%), GPT-4o (82.9%) | 94% |
| SWE-bench Verified | Real-world software engineering | OpenAI o3 (72.0%), Claude 3.7 Sonnet w/scaffold (70.3%) | 100% |
| LiveCodeBench | Real-time coding challenges | Grok 3 (79.4%), DeepSeek R1 (76.2%) | N/A |
| MBPP | Basic Python programming | Claude 3.7 Sonnet (90.0%), GPT-4.1 (89.0%) | 95% |
| Codeforces | Competitive programming | o3-mini high (2130 ELO) | 2500+ |
| SWE-Lancer | Freelance-style software tasks | Claude 3.5 Sonnet ($208K), OpenAI o1 ($195K) | N/A |
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| WebArena | Web navigation and form interaction | Various Research Agents (~44%) | 92% |
| BrowseComp | Complex web browsing and search | Deep Research Agent (51.5%), GPT-4o (1.9%) | 88% |
| WebVoyager | Cross-site navigation and task chaining | Research agents (52.0%) | 91% |
| MiniWoB++ | Mini web tasks with HTML/CSS/JS | Research agents (60.0%) | 95% |
| WorkArena | Office task automation | Claude 3.7 Sonnet (65.0%), Various agents (57.0%) | 90% |
| OSWorld | Desktop and OS task automation | Agent S2 (SOTA) | 87% |
| Android World | Android mobile tasks | Agent S2 (16.51% TP3T improvement) | N/A |
| Windows Agent Arena | Windows-specific tasks | Agent S2 (52.81% TP3T improvement), Navi (19.5%) | 74.5% |
| MobileAgentBench | Android app interaction | AppAgent (highest SR), MobileAgent (second) | 100% |
| ScreenSpot-Pro | Professional GUI grounding | OS-Atlas-7B (18.9%) | 100% |
| UI-TARS Benchmark | GUI interaction efficiency | UI-TARS-72B (highest), UI-TARS-7B (93.6%) | N/A |
| CRAB | Cross-environment action benchmark | GPT-4o (38.01%) | 100% |
| Mind2Web | 2,350 tasks across 137 websites | Various agent models | N/A |
| Real Website Benchmark | 15 real-world website tasks | WebVoyager (59.1% success) | N/A |
| ChatGPT Operator | Web tasks including sign-ins | ChatGPT Operator (50%) | 100% |
Benchmarks that evaluate capabilities across multiple modalities (text, vision, etc.).
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| MMMU | College-level multimodal tasks | Gemini 2 Pro (62.4%), Claude 3 Opus (60.5%) | 90% |
| DocVQA | Visual question answering on documents | Molmo-72B (96.3) | N/A |
| MTEB | Text embedding benchmark (58 tasks) | E5-Mistral-7B (64.1%), text-embedding-3-large (64.0%) | N/A |
Benchmarks that focus on specific AI capabilities.
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| TAU-bench | Multi-tool decision making | Claude 3.7 Sonnet (81.2% retail, 58.4% airline) | 93% |
| τ-bench | Multi-tool decision making | Galileo Agent Models (85%) | 93% |
| ToolACE | Tool selection consistency | Claude 3.7 Sonnet (82.0%) | 90% |
| xLAM | Real-world tool usage | Claude 3.7 Sonnet (84.0%) | 91% |
| BFCL | Function calling accuracy | Claude 3.5 Sonnet (90.2%) | 95% |
| PlanBench | Multi-step planning evaluation | Claude 3.7 Sonnet (65%), OpenAI o1 (62%) | 85% |
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| SkillBench | Procedural knowledge learning | LLaMA 2 (~65%), GPT-4 (~65%) | N/A |
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| Chain-of-Thought Bench | Step-by-step reasoning evaluation | Various reasoning models | N/A |
| CoT-HUB | Chain-of-thought reasoning analysis | Claude 3.7 w/thinking (88.9%), Grok 3 Pro (86.2%) | 97% |
Benchmarks focused on evaluating safety, robustness, and alignment of AI systems.
| Benchmark | Focus | Top Performers | Human Performance |
|---|---|---|---|
| HELMa | Factual consistency and hallucination | Claude 3.5 (91.0%) | 98% |
| Responsible AI Benchmark | Ethics, fairness, bias assessment | Anthropic Guardian Model (95.1%), Claude 3.7 Opus (92.4%) | N/A |
| RobustBench | Adversarial attack resistance | Claude 3.7 Fortress (80.6%), GPT-4 Secure (79.3%) | N/A |
| Prompt Injection Benchmark | Resistance to prompt injection | Various models | N/A |
| MLCommons AI Safety Benchmark | Safety risks of chat-tuned LLMs | Various models | N/A |
| TrustML Leaderboard | Model trustworthiness and safety | Various models | N/A |
| CyberEval | Cybersecurity capabilities | Various security models | N/A |
| ST-WebAgentBench | Safety and trustworthiness in web agents | Various web agents | N/A |
| Superintelligence Alignment Tests | Power-seeking behavior and deception | Various frontier models | N/A |
Platforms that compile results from multiple benchmarks.
| Aggregator | Focus | Notable Models |
|---|---|---|
| HELM | Multi-metric evaluation across dimensions | Gemini Ultra (92.1%) |
| HELM Lite | Streamlined version of HELM | Various frontier models |
| AI Benchmarking Dashboard | GPQA, PlanBench, MMLU, ARC aggregation | Claude 3.5 (~82% avg), GPT-4 (~80% avg) |
| Eleuther AI LM Evaluation Harness | Unified framework for model evaluation | Various language models |
| OpenLLM Leaderboard | Open-source model comparison | Llama-3-70B (83.2%), Mistral Large (82.7%) |
| Artificial Analysis | 30+ model performance comparison | GPT-4o, Claude 3.5 |
| Nebuly's Leaderboards | Comprehensive leaderboard collection | Varied |
| Vellum AI LLM Leaderboard | Multi-dimensional LLM assessment | Claude 3.5, GPT-4, Gemini |
| Klu.ai LLM Leaderboard | In-depth performance metrics | Varied |
| Deep Ranking AI | Commercial/OSS model comparison | Leading commercial/OSS models |
| Pareto Frontier Evaluations | Cost vs. performance optimization | Varies by cost efficiency |
Benchmarks specialized for resource-constrained environments.
| Benchmark | Focus | Top Performers | Hardware Type |
|---|---|---|---|
| AI-Benchmark | Mobile hardware AI capabilities | Various mobile devices | Mobile Processors |
| Mobile AI Benchmark | Hardware AI performance | Various mobile processors | Mobile Hardware |
| MobileBERT | Compact language models | MobileBERT (GLUE +0.2 vs BERT-base) | Mobile Devices |
| GLUE (Mobile) | Mobile device language understanding | MobileBERT (+0.2 vs BERT-base) | Mobile NLP |
| TinyML Benchmark | Microcontroller performance | Various microcontroller models | Microcontrollers |
- Supernal Intelligence Discord - Join our community to discuss AI benchmarks, share resources, and connect with others
- X/Twitter: @supernalasi - Follow for updates and news about AI agent benchmarks and advancements
- Website: supernalintelligence.com - Official website with more resources and information
- Stanford CRFM - Center for Research on Foundation Models
- Princeton HAL Team - Holistic Agent Leaderboard developers
- Epoch AI - Tracks frontier model progress across domains
- MLCommons - Industry-standard benchmarking organization
- Alignment Research Center - Superintelligence alignment testing
- Galileo Team - Tool use benchmark development
- Benchmarking Foundation Models: Progress and Open Questions
- Levels of AGI: Operationalizing Progress on the Path to AGI
- The Dawn of LLM Agents: Perspectives on Evaluation
- Evaluating Human-AI Collaboration for Task Completion
- Awesome GUI Agents - Curated list of GUI agents
- Awesome AI Agent Leaderboards - Comprehensive list of leaderboards for AI agents
While benchmarks provide valuable insights into AI progress, they have important limitations:
- Narrow Scope: Many benchmarks test specific, isolated capabilities rather than integrated performance in open-ended scenarios
- Overfitting: Models can be specialized to perform well on specific benchmarks without generalizing to real-world tasks
- Static Nature: Benchmarks quickly become outdated as models improve, requiring continuous development of harder challenges
- Lack of Adaptability: Few benchmarks measure how AI systems learn and adapt in dynamic environments
- Measurement Challenges: Critical capabilities like creativity, common sense, and social intelligence remain difficult to quantify
- Human Baseline Issues: "Expert" human performance can be inconsistently defined across benchmarks
- Limited Diversity: Most benchmarks are primarily in English and centered on Western cultural contexts
Researchers should consider these limitations when interpreting benchmark results and making claims about AI progress.
The performance gap between commercial and open-source models continues to evolve:
| Benchmark Type | Commercial Lead | Notable OSS Challengers | Gap Trend |
|---|---|---|---|
| General Reasoning | Substantial (10-20%) | Llama-3-70B, Mistral Large | Narrowing |
| Mathematical Reasoning | Moderate (5-15%) | Gemma 2, DeepSeek MoE | Narrowing rapidly |
| Code Generation | Small (3-8%) | WizardCoder, CodeLlama | Minimal gap |
| Agent Tasks | Large (20-40%) | AutoGPT, BabyAGI | Widening |
| Multimodal | Very large (30-50%) | LLaVA, CLIP | Steady gap |
| Tool Use | Large (25-40%) | Toolformer-based models | Narrowing slowly |
The gap appears to be smallest in straightforward language tasks and largest in complex reasoning, agent capabilities, and multimodal understanding. Recent trends suggest open-source models are catching up in some domains while commercial models continue to advance in others.
Contributions welcome! Please read the contribution guidelines first or email i@supernal.ai if you see an error or want to contribute.
This awesome list is maintained by Parni and Ian, and is released under the MIT Open Source License.