--limit truncates before the shuffle, so --seed and --runs pick the same questions every time #32

Summary

benchmarks/public/runner/run_subprocess.py passes --limit into
load_questions(), which returns the first N rows in file order. The shuffle
then only permutes that prefix, so --limit N runs the same N questions no
matter what --seed says.

Details

In run_eval (run_subprocess.py:267):

questions = load_questions(args.benchmark, limit=args.limit, offset=args.offset, ...)
if not args.no_shuffle:
    random.Random(seed).shuffle(questions)
if args.limit:
    questions = questions[:args.limit]

registry.load_questions breaks out at len(questions) >= limit
(registry.py:230), so it hands back a head-of-file slice. That makes the
questions[:args.limit] line below the shuffle a no-op. The fact that the line
is there at all reads like the intended order was shuffle first, limit second.

I reproduced it against a 200-row dataset with --limit 10:

seed=42   -> ['q000', 'q001', ..., 'q009']
seed=1234 -> ['q000', 'q001', ..., 'q009']
identical question set across seeds: True

Impact

--limit N scores a fixed prefix of the dataset rather than a sample. On any
dataset grouped by category or difficulty, that biases the number.

--runs N assigns seed + run_index - 1 per run, which only makes sense if the
seed changes the sample. With --limit in play, every run scores the identical
question set, so the spread across runs captures agent nondeterminism and
nothing else. Any error bars computed that way come out too small.

docs/eval-frontier-search.md describes --no-shuffle as keeping "canonical
query order", which implies the default gives you a randomised subset. It
doesn't.

Suggested fix

Drop limit= from the load_questions() call in run_eval and let the slice
after the shuffle do the work. Leave offset where it is. Under --no-shuffle
nothing changes, since it's the same prefix either way.

A test that runs two different seeds at the same --limit and asserts the
selected sets differ would catch a regression. tests/ has no sampling coverage
today.

Two smaller things in the same file

--seed help text says "default: 42", but the argparse default is None, and
config.json gets written from vars(args) before base_seed resolves. A run
that used 42 records "seed": null.

With --runs N, JUDGE_SESSION sits behind if "JUDGE_SESSION" not in os.environ, so whichever concurrent run gets there first decides the session id
for all of them.

I can send a PR for the sampling fix if you want it.