Skip to content

Add GPT-5.5 n=3 results and refresh leaderboard - #31

Merged
Shchoholiev merged 2 commits into
mainfrom
codex/gpt55-n3-leaderboard
Aug 3, 2026
Merged

Add GPT-5.5 n=3 results and refresh leaderboard#31
Shchoholiev merged 2 commits into
mainfrom
codex/gpt55-n3-leaderboard

Conversation

@Shchoholiev

@Shchoholiev Shchoholiev commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add three independent GPT-5.5 full-corpus runs with all 96 saved predictions, reports, metadata, and per-sample statuses
  • add an aggregate summary with arithmetic means, sample standard deviations, usage, costs, result paths, and report hashes
  • replace the GPT-5.5 leaderboard single-run row with the three-run aggregate
  • extend the generic replicate checker, release consistency checks, exporter tests, and documentation

GPT-5.5 n=3 result

Metric Mean ± sample SD
Exact-record recall 96.6% ± 1.5 pp
Complete documents 6.3 ± 0.6 / 32
Field micro-F1 99.3% ± 0.3 pp
Structural exact recall 89.5% ± 5.1 pp
Scale-control exact recall 99.4% ± 0.1 pp
API-equivalent full-run cost $43.19 ± $3.34

The three runs used the same 32 released OCR transcripts, field contracts, prompt, GPT-5.5 model, xhigh effort, Codex CLI 0.146.0, four workers per run, 272K context cap, and repository-denied ephemeral isolation as the GPT-5.6 replicate rows. All 96 document jobs completed on their first attempt. Run 1 was selected as the reference before evaluation; the aggregate does not select the best run.

Leaderboard change

The GPT-5.5 row now reports n=3 · arithmetic mean and uses:

  • $43.19 full-run cost
  • 96.6% exact recall
  • 6/32 complete documents in the table; the info popover retains the exact 6.3/32 ± 0.6 mean and sample SD
  • 89.5% structural recall
  • 99.4% scale-control recall
  • 99.3% field F1

The paper's earlier single-run GPT-5.5 result remains unchanged; this PR updates the current leaderboard and repository summary table only.

Verification

  • python -m pytest -q — 203 passed
  • python benchmarks/check_replicate_summary.py — verified 4 summaries and their full run packages
  • fresh leaderboard export plus JSON assertion for the GPT-5.5 n=3 row
  • git diff --cached --check
  • staged artifact audit: 96 predictions, 3 reports, 3 metadata files, 3 status tables, 0 transient logs

@Shchoholiev
Shchoholiev merged commit 629ed97 into main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant