Problem
When KATA_SN60_PROJECT_SAMPLE_SIZE narrows the benchmark, resolve_sn60_project_keys() in kata/validator_system/project_selection.py passes secrets.token_hex(16) as sample_nonce on every call.
That makes each evaluation draw a different project subset for the same king/candidate inputs. Re-running evaluate_submission, rerun-stale, or manual re-evaluations becomes a lottery and enables re-roll gaming.
Expected
Sampling should be deterministic for stable inputs (KATA_SN60_PROJECT_SAMPLE_SECRET, king hash, candidate hash, submission id) so outcomes are reproducible and auditable.
Proposed fix
Use a stable nonce (empty string) in resolve_sn60_project_keys() and add a regression test that two consecutive calls with identical inputs return identical project_keys.
A fix branch exists at RealDiligent:fix/critical-issue-deterministic-sn60-sampling for maintainer cherry-pick/review.
Problem
When
KATA_SN60_PROJECT_SAMPLE_SIZEnarrows the benchmark,resolve_sn60_project_keys()inkata/validator_system/project_selection.pypassessecrets.token_hex(16)assample_nonceon every call.That makes each evaluation draw a different project subset for the same king/candidate inputs. Re-running
evaluate_submission,rerun-stale, or manual re-evaluations becomes a lottery and enables re-roll gaming.Expected
Sampling should be deterministic for stable inputs (
KATA_SN60_PROJECT_SAMPLE_SECRET, king hash, candidate hash, submission id) so outcomes are reproducible and auditable.Proposed fix
Use a stable nonce (empty string) in
resolve_sn60_project_keys()and add a regression test that two consecutive calls with identical inputs return identicalproject_keys.A fix branch exists at
RealDiligent:fix/critical-issue-deterministic-sn60-samplingfor maintainer cherry-pick/review.