[oss] Add read-only jevbench validate-data subcommand

[fstandhartinger]

What changed

• jevbench/validate_data.py (new): read-only checks over task JSONL files. It loads each line with Task.from_dict from jevbench/tasks.py, the same path load_jsonl uses, so every task's validate() runs. A bad line is reported with file:line and loading moves on to the next line, so you see every problem in one run instead of only the first. On top of validate() it checks:
  • ids are unique across all the files given
  • question.type is an allowed kind (QUESTION_TYPES: noul/choice/score)
  • expected has the type its kind needs: an int for score (a bool is rejected), "no"/"yes" for noul, a label string for choice. None (unmeasured) is allowed.
  • group consistency, the same check as tests/test_protocol.py::test_public_suite_pairs: all items that share a group must share str(expected)
  • It prints a per-file summary, one PROBLEM line per issue and an OK/FAIL total. It exits 1 on any problem.
• jevbench/cli.py: registers the validate-data PATH... subcommand. It is 12 added lines and nothing existing changed.
• tests/test_validate_data.py (new): 6 tests that use tiny invented fixtures (fake-* ids, placeholder text) written to a temp dir.
• CONTRIBUTING.md: a short "Checking task files" section.

The diff is 4 files, +224 lines. No changes to datasets/, results/, scoring, constants or published numbers.

Verification

Before (on main, python -m pytest -q):

250 passed, 1 xfailed, 5 subtests passed in 3.06s

After:

256 passed, 1 xfailed, 5 subtests passed in 2.82s

python -m jevbench.cli validate-data datasets/public/*.jsonl:

datasets/public/easy.jsonl: 48 tasks (choice=36, noul=12); 0 groups
datasets/public/hard.jsonl: 111 tasks (choice=67, noul=38, score=6); 111 groups
datasets/public/original.jsonl: 72 tasks (choice=36, noul=24, score=12); 36 groups
OK: 3 files, 231 tasks, 0 problems
exit=0

It found no problems in the public data, and the data was not touched.

Open doubts / conservative choices

• "Task kind" is read as question.type. I did not also check family against jevbench.FAMILIES + HARD_FAMILIES. The easy tier uses fact and tool_selection, which are in neither tuple, so that check would flag published data. If those families should be registered, that is a maintainer decision.
• Group check is limited to what test_protocol checks: members must agree on expected. It does not require pairs of size 2, because every group in hard.jsonl has exactly one item. Groups are pooled across all files given; the public files share no group names.
• In practice a bad kind is already rejected by Task.validate() when the line loads. The separate kind check is there as a backstop.
• Nothing is left for Sandy: no credentials, network or GPUs were needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W43cN22Vpksq7669pf5f5K

---

Generated by Claude Code

[fstandhartinger]

Thanks — Harold has logged this for review. Contributor text and code are treated as untrusted data; no external code was run on the benchmark host.

• Harold