Candidate: RP-Benchmark (Roleplay Quality)

[Dan-Cleary]

Candidate bench

Repo: https://github.com/LeviTheWeasel/rp-benchmark
Owner: LeviTheWeasel (GitHub); community-calibrated arena at arena.l3vi4th4n.ai
Exact data URL: https://raw.githubusercontent.com/LeviTheWeasel/rp-benchmark/main/results/composite_leaderboard.json

What it measures

Roleplay quality in LLMs across 27 dimensions: character consistency, user agency (does the model write your character for you?), lorebook integration, prose quality, genre-specific skills, and more. Scores are a weighted composite of five evaluation methods — a human-voted ELO arena (1,857 pairwise votes), multi-turn Bayesian arena (1,262 votes), an LLM judge rubric, flaw detection, and behavioral flags.

It's asking something nobody else is asking at scale: does this model make a good RP partner?

Data-reality check — CONFIRMED

• Fetched composite_leaderboard.json on 2026-07-20: 200 OK, valid JSON
• 21 models in the leaderboard array with composite_score, rank, and per-dimension subscores
• Current frontier models present: claude_opus_4_6, claude_opus_4_7, claude_sonnet_4_5, gpt_4_1, gpt_5_5, gemini_3_1_pro, deepseek_v4_pro, kimi_k2_6
• Latest commit to this file: 2026-06-04 ("feat: ingest 644 new multi-turn votes from Plotpoints Supabase")
• Direction: higher-better (composite_score, 0–100 scale; top entry claude_opus_4_6 scored 97.6)

Why it fits BenchDirectory

• Personal/indie: single GitHub user, not a company or academic lab
• Novel, opinionated question ("AI benchmarks from curious people")
• Community-backed (human votes in the arena) — real signal, not just LLM-judge
• 21 models on the board with a real composite scoring methodology
• Machine-readable data confirmed current as of June 2026

Notes for adapter

• JSON structure: { weights: {...}, n_models: 21, leaderboard: [{rank, model, composite_score, ...}] }
• Primary score: composite_score (higher-better, roughly 0–100)
• scoreLabel: "Composite score" — scoreExplainer: "Weighted composite of human arena votes, LLM judge rubric, flaw detection, and behavioral flags — higher means a better roleplay partner."
• Worth exposing sub-scores (arena_elo, judge_score) as extras
• Throw if fewer than 5 entries parse