Candidate bench
Repo: https://github.com/lechmazur/elimination_game
Owner: lechmazur (GitHub)
Source-of-truth data URL: Structured game logs and model summaries in the repo (/logs, /model_summary_results directories); leaderboard uses Microsoft TrueSkill ratings
What it measures
Multiple LLMs play rounds of a social deduction / elimination game. Each round, players negotiate, form coalitions, and vote to eliminate opponents. The benchmark evaluates social reasoning under pressure: cooperative reliability, coalition engineering, strategic deception, deception resistance, negotiation, persuasion, reputation management, theory of mind, and long-horizon planning.
Scores are TrueSkill ratings (μ±σ) computed across randomized passes through game logs. 61 models currently ranked, including all major frontier labs.
Why it fits
- Personal/indie benchmark by a single GitHub user (lechmazur, same author as several other strong community benches)
- Completely novel question type: how well does a model play social games with other LLMs? — no academic equivalent
- 302 GitHub stars — strong community signal
- 61 models on the leaderboard, actively maintained
- Structured data exists in the repo (game logs + model summary JSONs)
Notes for adapter
The ingestable data is likely in /model_summary_results (per-model JSON summaries). The score to display is the TrueSkill μ (higher-better). Check whether a flat leaderboard CSV/JSON exists at the repo root before parsing individual log files. May need direction: "higher-better" with scoreLabel: "TrueSkill rating".
Candidate bench
Repo: https://github.com/lechmazur/elimination_game
Owner: lechmazur (GitHub)
Source-of-truth data URL: Structured game logs and model summaries in the repo (
/logs,/model_summary_resultsdirectories); leaderboard uses Microsoft TrueSkill ratingsWhat it measures
Multiple LLMs play rounds of a social deduction / elimination game. Each round, players negotiate, form coalitions, and vote to eliminate opponents. The benchmark evaluates social reasoning under pressure: cooperative reliability, coalition engineering, strategic deception, deception resistance, negotiation, persuasion, reputation management, theory of mind, and long-horizon planning.
Scores are TrueSkill ratings (μ±σ) computed across randomized passes through game logs. 61 models currently ranked, including all major frontier labs.
Why it fits
Notes for adapter
The ingestable data is likely in
/model_summary_results(per-model JSON summaries). The score to display is the TrueSkill μ (higher-better). Check whether a flat leaderboard CSV/JSON exists at the repo root before parsing individual log files. May needdirection: "higher-better"withscoreLabel: "TrueSkill rating".