Skip to content

Fix inverted S/C relevance position mapping in prepare_trec_eval_files.py - #26

Open
iamjaygao wants to merge 1 commit into
amazon-science:mainfrom
iamjaygao:patch-1
Open

Fix inverted S/C relevance position mapping in prepare_trec_eval_files.py#26
iamjaygao wants to merge 1 commit into
amazon-science:mainfrom
iamjaygao:patch-1

Conversation

@iamjaygao

Copy link
Copy Markdown

prepare_trec_eval_files.py maps S to position 2 and C to position 3.
With -m 'ndcg.1=0,2=0.01,3=0.1,4=1' that gives S a gain of 0.01 and C
a gain of 0.1 — reversed. train.py uses S=0.1, C=0.01.

Correct mapping per @franbvalero in #9: E=4, S=3, C=2, I=1.

Also reported in #18. The trailing-newline change is from the web editor.

@dst111dst

dst111dst commented Sep 4, 2026 via email

Copy link
Copy Markdown

@iamjaygao

Copy link
Copy Markdown
Author

Added a minimal reproduction — no dataset, no dependencies:
https://github.com/iamjaygao/esci-gain-mapping-repro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants