Meta-Chess is a benchmark that evaluates cognitive flexibility in large language models — the ability to generalize novel rules rather than rely on memorized patterns. It uses a modified chess game with a progressively complex rule hierarchy to distinguish genuine adaptive reasoning from surface-level pattern matching.
The benchmark consists of four levels of increasing complexity. The full rules for each level are defined in the prompt inside each respective file.
| Level | File | Description |
|---|---|---|
| 0 | RegularChess_openAI.py |
Standard chess — establishes the memorisation baseline |
| 1 | Ground-Chess_openAI.py& Ground-ChessKnights_openAI.py |
Fixed rules based on standard chess: only one chess piece can move, while all the other rules in standard chess apply. |
| 2 | Meta-Chess_openAI.py&Meta-ChessKnights_openAI.py |
Rules change dynamically during gameplay |
| 3 | Meta-Chess2_openAI.py |
Rules governing how rules themselves change |
Models that rely on memorization degrade as levels increase. Models capable of true rule generalization maintain performance.
- Python 3.x
- OpenAI API key
Install dependencies:
pip install openaiAdd your OpenAI API key directly in each file:
openai.api_key = "your-api-key-here"If using a different LLM provider, replace the API call and key lines in each file accordingly.
Evaluated on OpenAI frontier models via the OpenAI API.