Skip to content

Improve Benchmarking UI and dataset alignment flow - #118

Draft
nicooxxx wants to merge 3 commits into
mainfrom
fix/benchmark-ui-minor-changes
Draft

Improve Benchmarking UI and dataset alignment flow#118
nicooxxx wants to merge 3 commits into
mainfrom
fix/benchmark-ui-minor-changes

Conversation

@nicooxxx

Copy link
Copy Markdown
Collaborator

Overview

This is a first iteration of the Benchmarking UI improvements.
The current changes focus mainly on the Dataset and Alignment flow for Ranking and Classification.

V1 – implemented in this draft

Simplified Ranking and Classification dataset setup
Clearer distinction between Ground Truth and Benchmark data
Improved alignment wizard labels and step descriptions
Added help text/tooltips for less obvious fields
Simplified technical wording throughout the dataset flow
Improved consistency of the Benchmarking UI

V2 – next iteration

The next step would focus on the Evaluate flow, especially:
Reusing dataset and column selections from the alignment step
Reducing duplicate configuration
Simplifying Ranking configuration (Top K, Metrics at K)
Simplifying Classification evaluation
Improving the transition from evaluation to results

Feedback

This is still a work in progress. Feedback on the general flow and terminology would be helpful before I continue with the Evaluate section.

Screenshots

Dataset Managment

01_dataset_Ranking

Ranking Upload GT

02_dataset_Ranking_GT

Ranking Align GT

03_dataset_Ranking_GT_Align

Ranking Upload BM

04_dataset_Ranking_BM

Classification Upload

06_dataset_Clasification

Classification Align

07_dataset_Clasification_BM

Evaluate Ranking

09_evaluate_Ranking

Classification Ranking

11_evaluate_Classification

@suung suung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicooxxx FWIW

I would generally think like this

Benchmark Dataset is the exact same as Ground truth. I always found this confusing

So fo rme it is more like "Data you want to test" and either "Benchmark dataset" or "Ground thruth dataset"

Or both.

@suung

suung commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

@nicooxxx FWIW

I would generally think like this

Benchmark Dataset is the exact same as Ground truth. I always found this confusing

So fo rme it is more like "Data you want to test" and either "Benchmark dataset" or "Ground thruth dataset"

Or both.

@nicooxxx FWIW

I would generally think like this

Benchmark Dataset is the exact same as Ground truth. I always found this confusing

So fo rme it is more like "Data you want to test" and either "Benchmark dataset" or "Ground thruth dataset"

Or both.

Hey @nicooxxx I came across this again during #127 .. I think we should fix it, do you want to do it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants