pass_at_k implements the unbiased estimator and the runner reports pass@5 and
pass@10 when enough samples exist, but samples_per_problem: 1 in every shipped
config, so only pass@1 is ever produced.
The blocker is cost, not correctness. Generation dominates eval wall-clock, and
samples_per_problem: 5 multiplies it by five — on a T4 that turns a ~1.2h eval
into ~6h, which does not fit alongside training in one Kaggle session.
Options worth measuring:
- Batch the
num_return_sequences > 1 generation instead of looping — currently
infer.make_generator already passes num_return_sequences, so this may
mostly work already and just needs verifying under memory pressure
- Run eval as its own kernel session, separate from training
- vLLM for eval only, if it can be made to load a 4-bit adapter
pass@1 with greedy decoding is a defensible headline number, but pass@k is where
sampling-based approaches show their value, and right now the project cannot
measure that at all.
pass_at_kimplements the unbiased estimator and the runner reports pass@5 andpass@10 when enough samples exist, but
samples_per_problem: 1in every shippedconfig, so only pass@1 is ever produced.
The blocker is cost, not correctness. Generation dominates eval wall-clock, and
samples_per_problem: 5multiplies it by five — on a T4 that turns a ~1.2h evalinto ~6h, which does not fit alongside training in one Kaggle session.
Options worth measuring:
num_return_sequences > 1generation instead of looping — currentlyinfer.make_generatoralready passesnum_return_sequences, so this maymostly work already and just needs verifying under memory pressure
pass@1 with greedy decoding is a defensible headline number, but pass@k is where
sampling-based approaches show their value, and right now the project cannot
measure that at all.