Candidate: LLM Divergent Thinking Benchmark

[Dan-Cleary]

Candidate bench

Repo: https://github.com/lechmazur/divergent
Owner: lechmazur (GitHub / @lechmazur on X)
Source-of-truth data URL: https://raw.githubusercontent.com/lechmazur/divergent/main/graded_results.csv

What it measures

Models are given 50 random seed words and asked to generate 25 new words starting with a specified letter that are maximally distinct from each other AND from all 50 seed words. Four independent LLM judges score each generated word on its divergence from all other words in the set and rule compliance (must be real English words, no proper nouns, no hyphens). The final score is the average minimum-divergence grade across all generated words — higher means the model produced more genuinely original, uncorrelated words.

This is an operationalization of the classic open-ended divergent-thinking creativity test from cognitive psychology, applied to LLMs.

Why it fits

• Personal/indie benchmark by a single GitHub user (lechmazur, same creator as the already-tracked LLM Sycophancy Benchmark and LLM Elimination Game issues)
• Novel, opinionated question: how creatively can a model think outside the box? — no academic equivalent that runs across frontier models
• Machine-readable data: graded_results.csv in the repo root with columns model, grade, error; confirmed accessible via raw GitHub URL
• Actively maintained, last updated 2026
• Direction: higher-better (confirmed in README: "Higher scores indicate better performance")

Notes for adapter

The data file is a flat CSV: model, grade (primary score), error (±margin). Rank by grade descending. The adapter should throw if fewer than 5 rows parse. scoreLabel could be "Divergence grade", scoreExplainer: "Average originality score for the 25 words the model generated — higher means the words were more genuinely distinct from each other and from the seed set."

[Dan-Cleary]

Verified the data before building, and it's not usable yet. graded_results.csv is the only machine-readable file and it's stale: last committed 2024-12-28 ("first commit"), never updated since. 19 models, all 2024-era (o1-preview, gpt-4o, gemini-2.0-flash-exp). No current frontier models, and the live leaderboard in the README is images. Good bench, but nothing fresh to ingest. Holding until the CSV is refreshed.

[Dan-Cleary]

@lechmazur, I'd love to add your Divergent Thinking benchmark to BenchDirectory (https://bench.directory), a directory of indie AI benchmarks that links every score back to its creator. One blocker: graded_results.csv looks like it's from Dec 2024 and has no current frontier models. If you refresh it (or point me at a current machine-readable export), I'll wire up the adapter and credit you. Thanks for building these.