Skip to content

fix(estimators): make RAINPROOF score orientation configurable - #465

Merged
ArtemVazh merged 2 commits into
IINemo:mainfrom
ArtemVazh:fix/rainproof-uncertainty-score-type
Sep 1, 2026
Merged

ArtemVazh merged 2 commits into
IINemo:mainfrom
ArtemVazh:fix/rainproof-uncertainty-score-type

Conversation

@ArtemVazh

Copy link
Copy Markdown
Collaborator

Summary

  • keep uncertainty as the default orientation, where higher scores mean greater uncertainty
  • add score_type="rainproof" to return the original positive RAINPROOF divergence/distance orientation
  • distinguish estimator names by score orientation to avoid mixing cached results
  • correct the Rényi divergence normalization against the uniform distribution

Usage

RenyiNeg(score_type="uncertainty")  # default: higher = more uncertain
RenyiNeg(score_type="rainproof")    # higher = farther from uniform

FisherRao(score_type="uncertainty")
FisherRao(score_type="rainproof")

This supersedes #459 while preserving Yedson54 as the author of the original orientation-fix commit.

Validation

  • Black and flake8 pass for the changed files
  • numerically checked uniform and peaked distributions for both orientations and Rényi alpha=0.5 / alpha=1

Yedson54 and others added 2 commits August 4, 2026 14:41
…core convention

RenyiNeg and FisherRao compute a divergence/distance to the uniform
token distribution. The raw quantity increases when token distributions
are sharper, so it behaves as a certainty score.

Most LM-Polygraph uncertainty estimators use the opposite convention,
where higher scores indicate higher uncertainty. Negate the returned
scores and clarify the convention in the estimator docstrings.

BREAKING CHANGE: RenyiNeg and FisherRao scores change sign. Existing
results using these estimators should be reinterpreted accordingly or
recomputed.
@ArtemVazh
ArtemVazh merged commit 0d420e5 into IINemo:main Sep 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants