Skip to content

engine(samplers): top-p tie-breaking is undefined at large vocab (float ties + unstable sort) #423

Description

@jamesburton

Observation

Found while verifying the top-p pre-filter in #300, whose parity tests compare selected token
indices between two implementations.

At 32K+ vocabularies with near-uniform logit distributions, distinct logits can round to the same
float probability after softmax
. Array.Sort is an unstable IntroSort, so the relative order of
equal-probability entries is unspecified — and therefore which token top-p selects among a tie is
undefined.

This is pre-existing and unchanged by #300; that PR's implementation has exactly the same
behaviour as the one it replaces. It surfaced only because writing a differential test against
32K-vocab distributions forced the question of what "the same result" means under ties.

Why it is worth recording

Two consequences, neither catastrophic but both worth knowing:

  1. Seeded runs are not fully reproducible across environments under ties. The sampler is seeded,
    so a user reasonably expects identical output for identical seed + prompt + model. Sort
    implementation details can differ across runtime versions or array sizes (IntroSort switches
    strategy by size), so the guarantee is weaker than it appears.
  2. Index-level parity assertions are testing sort internals, not sampler semantics. Any future
    test comparing selected indices under near-uniform distributions is measuring the wrong thing and
    may become flaky for reasons unrelated to the sampler.

Probability-level behaviour is unaffected: the set of tokens above the top-p mass is correct
either way, and any tie member is an equally valid sample.

Options

  • Do nothing, document it. Cheapest. Record in docs/SAMPLING.md that tie order is unspecified
    and that determinism guarantees hold per-environment, not across them.
  • Break ties deterministically by token id (a stable secondary key in the comparison). Makes
    seeded output reproducible across environments; costs a marginally more expensive comparer on a
    path that already sorts the vocabulary.
  • Sort stably. More invasive and probably not worth it for this alone.

I lean towards the second if cross-environment reproducibility is a property worth advertising, and
the first if it is not — but it should be a decision rather than an accident.

Acceptance criteria

  • The chosen behaviour is stated in docs/SAMPLING.md.
  • If deterministic tie-breaking is adopted, a test constructs an explicit tie (two tokens with
    logits that round to the same probability) and asserts a stable selection.
  • No throughput regression on the sampling path.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions