Skip to content

Conversation

@moibra05
Copy link
Contributor

Because

  • Upon first visiting the new results page it is unclear the results of the experiment and whether there were any significant changes

This commit

  • Adds an arrow indicator next to the metric names to indicate how that experiment affected that particular metric
  • Fixed an issue where metrics would incorrectly be placed in the "Notable Changes" section

Fixes #14460

@moibra05
Copy link
Contributor Author

image image

Copy link
Contributor

@mikewilli mikewilli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing this, it really drives home that the answer to "was the overall significance of this metric positive or negative?" is quite a bit more nuanced than it sounds. I wonder if it'd be better to simply annotate them with statuses instead. Something like:

  • Notable metric change
  • Metric missing results
  • (no special styling/icon) No error or notable change

What do you think?

@jaredlockhart
Copy link
Collaborator

Oh yeah I think I suggested four possible states

  1. All neutral
  2. Neutral or positive
  3. Neutral or negative
  4. Mixed (neutral, positive, negative)

But you're right @mikewilli there's also a case for

  1. Missing data

I think that should cover all the states.

Yeah theoretically we could fold any positive/negative together into a 'notable results' I just think it would be great to see all the directionality in one zoomed out view before you zoom in, which we don't have right now. If we find it's confusing or misleading we can always change it later.

@moibra05
Copy link
Contributor Author

Oh yeah I think I suggested four possible states

  1. All neutral

  2. Neutral or positive

  3. Neutral or negative

  4. Mixed (neutral, positive, negative)

But you're right @mikewilli there's also a case for

  1. Missing data

I think that should cover all the states.

Yeah theoretically we could fold any positive/negative together into a 'notable results' I just think it would be great to see all the directionality in one zoomed out view before you zoom in, which we don't have right now. If we find it's confusing or misleading we can always change it later.

Yep, I planned to address better ui for metrics that are missing data in a different pr

@jaredlockhart
Copy link
Collaborator

Oh the double arrows means mixed! That's clever, but actually I'd use up/down together since we're using up and down already for each of positive and negative.

Copy link
Collaborator

@jaredlockhart jaredlockhart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep this is all looking good! Try playing with set and see if it helps, otherwise this should be good to land 🎉

from experimenter.outcomes import Outcomes


class MetricSignificance(StrEnum):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect 🙏


significances = {metric_significance, overall_change}

if significances == {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 🙏

@moibra05 moibra05 enabled auto-merge February 3, 2026 15:31
@moibra05 moibra05 added this pull request to the merge queue Feb 3, 2026
Merged via the queue into main with commit 6f0d5da Feb 3, 2026
17 of 20 checks passed
@moibra05 moibra05 deleted the 14460 branch February 3, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add metric result indicators next to subtitles

5 participants