Frontier vs Open — a blind taste test for LLMs

The question every AI forum keeps asking: "Has anyone actually tried Kimi / GLM / DeepSeek / Qwen, and how do they compare to GPT / Claude?" Benchmarks don't answer it — they're gamed, contaminated, and meaningless to a small-business owner, a part-time dev, or a vibe-coder.

This is the answer those people actually want: real tasks, answers shown blind, you pick the one you'd ship, then the reveal shows who wrote it — and what it cost. Spoiler the format is built to surface: the $0.30 Chinese model often wins the blind test against the $5 frontier one.

What's here

• index.html — the blind side-by-side viewer. Open it in any browser. Ships with clearly-labeled demo data so it's clickable immediately; load a real results.json to populate it with live outputs. Features: blind answers, one-click voting, reveal with US/CN origin + price + latency + cost, a running "your verdict" tally (incl. cost-vs-all-Claude), and plain-English model fingerprints.
• tasks.json — the curated task suite. Real-world categories (cold email, bug fix, messy-data→JSON, regex, landing page) plus myth-buster tasks designed to test the "Chinese models are just distilled" claim (multi-step reasoning, strict instruction-following, knowledge honesty). Each task has a look_for so a non-expert can judge.
• run_comparison.py — the harness. Sends every task to every model through OpenRouter (the one API where closed frontier and open Chinese models are callable identically and priced comparably), and records outputs + token counts + computed cost into results.json.

Run it for real

export OPENROUTER_API_KEY=sk-or-...      # https://openrouter.ai/keys
pip install requests                      # (stdlib urllib also works; see script)
python run_comparison.py                  # all tasks x all models
# then open index.html and load the generated results.json via the file picker

Subsets: python run_comparison.py --tasks cold_email bug_fix --models deepseek/deepseek-v4-pro anthropic/claude-opus-4.8

Extend it

• Add a task: append to tasks.json (prompt, category, look_for, myth_busting). Re-run the harness.
• Add/swap a model: edit meta.models in tasks.json (use the exact OpenRouter id; refresh price_in/price_out from the catalog). Fingerprints live in index.html's demo meta and are reused for labels.
• Prices are USD per 1M tokens, approximate from the OpenRouter catalog — re-pull periodically; they move.

Why this is credible (and not just another leaderboard)

• You judge, blind. The tool never declares a winner — it shows raw outputs side-by-side and lets you decide. The value is the curated tasks + the difference annotations + the cost overlay, not a handed-down verdict. That sidesteps judge bias and is exactly the format that travels on forums.
• Real money, one arena. OpenRouter prices make the cost comparison concrete and un-gameable.
• Myth-buster tasks put the "they're just distilled" claim to a test anyone can read.

Honest limitations

• Demo outputs shipped in index.html are illustrative placeholders, not real model responses — run the harness for real data.
• Single-shot, single-temperature; real eval should sample multiple times and across temperatures.
• Task suite is small by design (a sharp v0); breadth and crowd-voting are the obvious next steps.
• Prices drift; re-pull before publishing any numbers.