[oss] CLI: accurate --help text and examples, docs/CLI.md, --help tests

[fstandhartinger]

What changed

3 files, +341/−31. No datasets/, results/, scoring, constants or published numbers touched.

• jevbench/cli.py: wording-only changes. Rewrote the module docstring (it now lists every flag). Added a description and an example epilog to the main parser and to both subcommands. Every option now has a help string that shows its default where there is one. The run epilog also lists which adapters don't need --endpoint, the rule that output paths must be outside the repo and must not already exist, and the exit codes (0 / 3 / 2). The --include-excluded help now says the flag currently has no effect (see doubts).
• docs/CLI.md (new): covers both subcommands, every flag with its default, JEVBENCH_WARM_LOAD, the stop rules, exit codes and the output files (results, raw, ledger, manifest, public export). It ends with an offline end-to-end example on the public split: the CLI has no dummy adapter, so the example runs openai_compat against a ~15-line localhost stub server that returns uniform probabilities. No model API is called.
• tests/test_cli_help.py (new, 6 tests):
  • --help exits 0 for jevbench, run and summarize, and each help page lists every flag.
  • Every subcommand option has non-empty help text.
  • python -m jevbench.cli --help works when run as a subprocess.
  • Running with no subcommand exits with code 2.

How I verified it

Full suite before the change:

113 passed, 5 subtests passed in 0.51s

Full suite after the change (python -m pytest -q):

119 passed, 5 subtests passed in 0.58s

Flags and defaults are unchanged. A script dumped each subcommand's argparse actions (option strings, dest, default, required, type, choices, action class) from main and from this branch, with the repo path normalised. The two dumps matched byte for byte: IDENTICAL.

End-to-end CLI run on datasets/public/easy.jsonl against the localhost stub, writing to /tmp/jb:

[jevbench] adapter=openai_compat endpoint=http://127.0.0.1:8765/v1 tasks=48 cap=$15.0 ledger=/tmp/jb/ledger.jsonl
[jevbench] done: 48/48 attempted, 0 failed; charged=$0.0000
exit=0
[jevbench] public export -> /tmp/jb/public.json
{'n_attempted': 48, 'accuracy': 0.2708333333333333, 'schema_validity': 1.0, 'brier_mean': 0.7156250000000001, 'price_per_1000_decisions_usd': 0.0, 'cost_basis': ['local_stub_no_tariff'], 'probability_sources': ['verbalized'], 'ledger_charged_usd': 0.0, 'complete': True}

This matches the uniform baseline already published in docs/ADAPTERS.md §7 (accuracy 0.2708, Brier 0.7156).

Open doubts

• summarize --include-excluded does nothing. The CLI passes it as headline_only to summarize(), which never reads that argument. With and without the flag, the output had the same sha256. I documented this rather than changing behaviour; whether to wire it up or drop it is a maintainer decision.
• Stale line references in docs/ADAPTERS.md. It cites line numbers in cli.py (e.g. cli.py:252-262, cli.py:283-288) that were already wrong on main (the file had 217 lines). This PR shifts them further. I didn't edit ADAPTERS.md to keep the diff focused.
• Defaults depend on the repo location. --ledger and --raw-dir default to <repo>/../private/..., and the docs describe them that way.
• Nothing left for Sandy. No credentials, GPUs or model API calls were needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_018na1FUPvYUAfdkf8bAMELf

---

Generated by Claude Code

[fstandhartinger]

Thanks — Harold has logged this for review. Contributor text and code are treated as untrusted data; no external code was run on the benchmark host.

• Harold