I would like to be able to use real-valued labels, and compute DCG and NDCG metrics that respect the differences in labels that at less than 1. So for example:
- A document with relevance label = 1 would have Burges gain of
2**1 - 1 = 1.0
- But a document with relevance label = 1.9 would have Burges gain of
2**1.9 - 1 = 2.73
However, it seems like Ranx's Qrels object will cast my relevance labels to integers:
from ranx import Qrels
qrels_dict = {"q_1": {"d_11": 1.9}}
qrels = Qrels(qrels_dict)
print(qrels)
>>> {q_1: {d_11: 1}}I believe this happens in the Qrels constructor here. A consequence of this is that I can't properly use real-valued labels. The relevance values will be truncated, and a lot of information will be lost.
Questions for the maintainers:
- Am I missing something? Is there a way to achieve what I want with Ranx?
- If not, what is required to make Ranx support this? Is this a big lift or a small one?
I would like to be able to use real-valued labels, and compute DCG and NDCG metrics that respect the differences in labels that at less than 1. So for example:
2**1 - 1 = 1.02**1.9 - 1 = 2.73However, it seems like Ranx's
Qrelsobject will cast my relevance labels to integers:I believe this happens in the Qrels constructor here. A consequence of this is that I can't properly use real-valued labels. The relevance values will be truncated, and a lot of information will be lost.
Questions for the maintainers: