feat(benchmarks): completeness judge — LOC wins can't hide under-delivery

[ousamabenyounes]

Follow-up to #126 (the benchmark-fairness thread with @ColinEberhardt) and a companion to #169/#170.

These two PRs share one goal: make the benchmark better at every iteration so the project's claims are earned, not asserted.

• fix(benchmarks): portable plugin-dir resolution for agentic arms #170 made the benchmark reproducible by anyone — it removed hardcoded personal plugin-cache paths so the ponytail/caveman arms run off the maintainer's machine, not just on it.
• This PR makes the benchmark more relevant — it closes the "less code might just mean less feature" hole, which is the most credible attack left on the headline number after the Benchmark issues - baseline scores are ~7 times better #126 rebuild.

Problem

The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on git diff alone. The only quality gate is weak:

• score_vibe → "it compiles"
• score_fixture (tmpl-fe) → "a new file exists"
• score_open → correct=1 always

So an arm can win the LOC metric by shipping a stub — fewer lines because it did less, not because it is less bloated. That is exactly the "you wrote less because you did less" objection, and the bench currently can't see it.

Fix

complete.py — a second LLM judge on the same auditable footing as the over-engineering judge in judge.py: fixed model (claude-sonnet-4-6), temperature 0, published rubric. It rates how fully each submission implements its task, 0 (stub) .. 3 (fully implements). Read alongside the LOC table: a low-LOC arm whose completeness also drops is doing less, not less-bloated — and now the bench shows it.

• judge_call gains a system= parameter so the HTTP/key/source plumbing is reused, not duplicated — one rubric is the only delta between the two passes.
• Output: mean completeness per arm + a list of under-delivered cells (completeness <= 1).
• README: new "Completeness judge" section + updated can/cannot-show limitations.

Test verification (RED → GREEN)

The judge model is validated by --selftest (must rank a complete reference strictly above a stub). The gate logic is validated with no API key by --selftest-offline:

GREEN — with the gate intact:

offline gate -- well-ordered (expect ok):
ok cache: complete(3) > stub(0)
offline gate -- stub out-scores complete (expect XX):
XX cache: did not rank complete above stub

completeness gate selftest (offline): valid

RED — gate neutered to a no-op (regression that stops catching under-delivery):

completeness gate selftest (offline): BROKEN

(exit 1)

So the check fails the moment the gate stops catching a stub that out-scores a real implementation.

🤖 Generated with Claude Code