[Platform] Judge panel + scoreboard for hackathon PRs

[mariagorskikh]

Why

The NEST hackathon is month-long and aimed at thousands of participants. Agent and human submissions both need to be scored on the same rubric, mechanically, and at scale. This PR ships the judge panel and the scoreboard wiring on the platform/judge-panel track.

What's in the box

Rubric (scripts/judge/rubric.md, version: 1)

Six dimensions, each scored 1-5 with concrete 1/3/5 anchors:

• correctness — does the code do what the PR claims; any obvious bugs.
• test_rigor — coverage, property-based vs example-based, adversarial cases.
• api_fit — drop-in compatibility, registry wiring, idiomatic to NEST.
• docs_quality — PR body clarity, docstrings, runnable examples.
• novelty — what's interesting vs. textbook restatement.
• persona_fidelity — does it actually reflect the claimed engineer persona.

Versioned. Re-runs record the rubric version they used, so historical scores stay interpretable when the rubric evolves.

Judge module (scripts/judge/judge_pr.py)

async def judge_pr(pr_number: int, *, n_judges: int = 3,
                   model: str = "claude-opus-4-7") -> JudgeResult

• Fetches the PR diff, body, author, head SHA, and check-run summary via the GitHub REST API (stdlib urllib; optional GITHUB_TOKEN for rate-limit headroom).
• Runs N judges in parallel via anthropic.AsyncAnthropic + asyncio.gather. Each judge is self-contained: it sees rubric + PR body + diff, no conversational context, no chained calls.
• Prompt caching on the rubric: the rubric is sent as a system block with cache_control: {"type": "ephemeral"}. For 3 judges in the same 5-min window, the rubric tokens are billed once.
• Determinism: temperature=0.0 on every judge. The output records model and rubric_version.
• The anthropic SDK is imported lazily so the module imports cleanly in CI environments without it.
• ANTHROPIC_API_KEY is read from env at call time; missing key raises a clear error before any I/O. No .env file needed or supported.

Aggregation

• Median per dimension uses statistics.median_low (deterministic tie-break to the low value).
• Missing/errored judges are skipped per-dimension; the verdict is still recorded with its error field. If every judge errors, medians become NaN and the consensus explicitly says so.
• Consensus is a deterministic 3-sentence narrative stitched from the numeric medians plus a snippet from the strongest judge's rationale — no second LLM call.

Diff truncation

Per-file diffs over 5000 lines are clipped with a marker (... [truncated N lines ...] ...) and the output sets diff_truncated: true. The truncator walks diff --git boundaries, so one giant file doesn't drown the others.

CLI (scripts/judge/run_all.py)

# Live: 10 PRs x 3 judges = 30 Anthropic calls
ANTHROPIC_API_KEY=... uv run python -m scripts.judge.run_all \
    --output docs/hackathon/scores.json

# Offline / CI smoke (mock judges keyed off head_sha)
uv run python -m scripts.judge.run_all --mock \
    --prs-cache scripts/judge/fixtures/hackathon-prs-2026-05-26.json \
    --output docs/hackathon/scores.json

• Idempotent: re-running skips any PR whose head_sha matches the prior scoreboard. Pass --force to override, or change the rubric version to invalidate everything.
• Offline mode: when ANTHROPIC_API_KEY is unset, falls back to a deterministic MockJudgeClient whose scores are derived from sha256(head_sha | judge_id | dimension). Same input → same scores, every time. The scoreboard's top-level mock: true flag and per-row model: "mock:..." make this unmistakable.
• --prs-cache lets the CLI run without GitHub access — useful for sandboxed CI and offline reproducibility. The committed fixture (scripts/judge/fixtures/hackathon-prs-2026-05-26.json) snapshots the 10 hackathon PRs as of this PR's creation.

Scoreboard schema (docs/hackathon/scores.json)

{
  "version": 1,
  "generated_at": "2026-05-26T20:04:17+00:00",
  "mock": true,
  "submissions": [
    {
      "pr": 2, "handle": "harvard-phd", "layer": "trust",
      "title": "...", "author": "mariagorskikh",
      "head_sha": "4e0f4d...", "head_ref": "hackathon/harvard-phd-eigentrust",
      "model": "mock:claude-opus-4-7", "rubric_version": 1,
      "diff_truncated": false,
      "scores": {"correctness": 3.0, "test_rigor": 2.0, ...},
      "median": 21.0,
      "consensus": "PR #2 from the harvard-phd persona scored ...",
      "judges": [{"judge_id": 0, "scores": {...}, "rationale": "...", "error": null}, ...]
    }
  ]
}

Persona handle is preferred from the [Hackathon] <handle>: title prefix (explicit) and falls back to the branch heuristic. Layer is inferred from title/body against the canonical 12 layers.

Bootstrap scoreboard

The committed docs/hackathon/scores.json was generated from mock judges because this sandbox has no ANTHROPIC_API_KEY. The schema is fully exercised; the numeric scores are deterministic stand-ins. To replace with real judgments:

export ANTHROPIC_API_KEY=...
uv run python -m scripts.judge.run_all --force --output docs/hackathon/scores.json

mock: true in the top-level metadata and model: "mock:claude-opus-4-7" per submission flag this clearly. A live re-run will flip both.

Cost model

• Live: 10 PRs × 3 judges = 30 calls.
• Each call: rubric (~1.8k tokens, cached after first call in the 5-min window) + PR body (variable) + diff (capped at ~5k lines/file). Realistic per-call input: ~5k-20k tokens.
• Output capped at max_tokens: 2048 per judge.
• With claude-opus-4-7 at current pricing, a full live run is on the order of $3-5. Prompt caching cuts that roughly in half on the rubric portion.
• Idempotency means re-runs only re-score PRs whose HEAD SHA changed, so steady-state cost during the hackathon is one call per push, not per CI run.

Tests (scripts/judge/tests/test_judge.py)

25 tests covering:

• Score aggregation: odd/even counts, ties (low tie-break), empty, single, all-judges-errored.
• Missing-judge handling: per-dimension skip, still appears in judges with error.
• parse_verdict fault tolerance: plain JSON, fenced markdown, garbage, out-of-range, missing dimension, non-int score, refusals.
• Diff truncation: short diffs untouched, oversized files clipped, multi-file diffs where only the big one is cut.
• Persona inference: title-wins for 3-token cases like stanford-ml-phd, branch fallback otherwise.
• JSON schema round-trip on JudgeResult.to_dict().
• MockJudgeClient determinism.
• End-to-end judge_pr() with a fake JudgeClient (no SDK required).
• "One judge errors, two succeed" → result still computes.

Live API tests are behind @pytest.mark.live and skipped by default (addopts = ["-m", "not live"]). Run them with pytest -m live and a real key.

Local CI parity

uv sync
uv run ruff check .            # 0 errors
uv run ruff format --check .   # 100 files clean
uv run pyright                 # 0 errors (strict mode)
uv run pytest -v               # 289 passed, 1 deselected (live)

All five exit 0.

How to re-run

Live judges against current open PRs:

export ANTHROPIC_API_KEY=sk-ant-...
uv run python -m scripts.judge.run_all --output docs/hackathon/scores.json

Subset:

uv run python -m scripts.judge.run_all --pr 7 --pr 11 \
    --output docs/hackathon/scores.json

Force a full re-score (e.g. after bumping the rubric version):

uv run python -m scripts.judge.run_all --force --output docs/hackathon/scores.json

Offline schema smoke:

uv run python -m scripts.judge.run_all --mock \
    --prs-cache scripts/judge/fixtures/hackathon-prs-2026-05-26.json \
    --output /tmp/scores.json

Scope discipline

• No participant PR was modified.
• No other platform track was touched.
• No new runtime dependencies (anthropic is already an optional dep in nest-shell; the judge imports it lazily and uses stdlib for everything else).

https://claude.ai/code/session_01C5j2D4MgCkPgsjSCqBVpWW

---

Generated by Claude Code

[sourcery-ai]

Sorry @mariagorskikh, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[mariagorskikh]

Follow-up commit 696b28b extends the judge panel with an OpenAI provider so live judges can run with either an Anthropic or OpenAI key.

What's new

• OpenAIProvider in scripts/judge/judge_pr.py — wraps openai.AsyncOpenAI against chat.completions with temperature=0.0, response_format={"type": "json_object"}, and default model gpt-5.5 (OpenAI's current best reasoning model on chat.completions as of May 2026). The SDK is imported lazily; no prompt-caching gymnastics, since OpenAI's caching is implicit.
• AnthropicProvider is now a public alias for the existing AnthropicJudgeClient.
• make_provider(name, *, model=None, api_key=None) factory returns the right client; unknown names raise a clear ValueError.
• run_all.py gets --provider {anthropic,openai} (default anthropic) and --model <name> (defaults to the provider's recommended model). If the selected provider's API key env var is unset, the mock judge fallback kicks in just like before, and the error message names the right env var.
• pyproject.toml adds an optional-dependencies group: [project.optional-dependencies] judge = ["anthropic>=0.30", "openai>=1.0"]. Install with uv sync --extra judge.
• New scripts/judge/README.md documents providers, env vars, and CLI usage.

Invariants preserved

• Same rubric, same six dimensions, same JSON output schema, same median-low aggregation, same scores.json shape.
• Default-provider behavior is bit-for-bit identical to the previous Anthropic-only flow.

Tests

10 new tests for the provider factory and the OpenAI provider (mocked SDK; no live API calls). All 299 tests pass; ruff/format/pyright clean.

---

Generated by Claude Code

[mariagorskikh]

Superseded by #17 (now merged to main at 1771cdb). Closing — the content of this PR (including the live judge scoreboard, OpenAI provider, schema reconciliation, and gpt-5.5 temperature fix) is part of that integration merge.

---

Generated by Claude Code