Observation
Found while verifying the top-p pre-filter in #300, whose parity tests compare selected token
indices between two implementations.
At 32K+ vocabularies with near-uniform logit distributions, distinct logits can round to the same
float probability after softmax. Array.Sort is an unstable IntroSort, so the relative order of
equal-probability entries is unspecified — and therefore which token top-p selects among a tie is
undefined.
This is pre-existing and unchanged by #300; that PR's implementation has exactly the same
behaviour as the one it replaces. It surfaced only because writing a differential test against
32K-vocab distributions forced the question of what "the same result" means under ties.
Why it is worth recording
Two consequences, neither catastrophic but both worth knowing:
- Seeded runs are not fully reproducible across environments under ties. The sampler is seeded,
so a user reasonably expects identical output for identical seed + prompt + model. Sort
implementation details can differ across runtime versions or array sizes (IntroSort switches
strategy by size), so the guarantee is weaker than it appears.
- Index-level parity assertions are testing sort internals, not sampler semantics. Any future
test comparing selected indices under near-uniform distributions is measuring the wrong thing and
may become flaky for reasons unrelated to the sampler.
Probability-level behaviour is unaffected: the set of tokens above the top-p mass is correct
either way, and any tie member is an equally valid sample.
Options
- Do nothing, document it. Cheapest. Record in
docs/SAMPLING.md that tie order is unspecified
and that determinism guarantees hold per-environment, not across them.
- Break ties deterministically by token id (a stable secondary key in the comparison). Makes
seeded output reproducible across environments; costs a marginally more expensive comparer on a
path that already sorts the vocabulary.
- Sort stably. More invasive and probably not worth it for this alone.
I lean towards the second if cross-environment reproducibility is a property worth advertising, and
the first if it is not — but it should be a decision rather than an accident.
Acceptance criteria
References
Observation
Found while verifying the top-p pre-filter in #300, whose parity tests compare selected token
indices between two implementations.
At 32K+ vocabularies with near-uniform logit distributions, distinct logits can round to the same
floatprobability after softmax.Array.Sortis an unstable IntroSort, so the relative order ofequal-probability entries is unspecified — and therefore which token top-p selects among a tie is
undefined.
This is pre-existing and unchanged by #300; that PR's implementation has exactly the same
behaviour as the one it replaces. It surfaced only because writing a differential test against
32K-vocab distributions forced the question of what "the same result" means under ties.
Why it is worth recording
Two consequences, neither catastrophic but both worth knowing:
so a user reasonably expects identical output for identical seed + prompt + model. Sort
implementation details can differ across runtime versions or array sizes (IntroSort switches
strategy by size), so the guarantee is weaker than it appears.
test comparing selected indices under near-uniform distributions is measuring the wrong thing and
may become flaky for reasons unrelated to the sampler.
Probability-level behaviour is unaffected: the set of tokens above the top-p mass is correct
either way, and any tie member is an equally valid sample.
Options
docs/SAMPLING.mdthat tie order is unspecifiedand that determinism guarantees hold per-environment, not across them.
seeded output reproducible across environments; costs a marginally more expensive comparer on a
path that already sorts the vocabulary.
I lean towards the second if cross-environment reproducibility is a property worth advertising, and
the first if it is not — but it should be a decision rather than an accident.
Acceptance criteria
docs/SAMPLING.md.logits that round to the same probability) and asserts a stable selection.
References
not mis-read
src/DotLLM.Engine/Samplers/TopPSampler.cs