[AdnanElAssadi]

SketchyI2IRetrieval has several rows under the same id. Each sketch is stored 6 times (the Sketchy augmentations: the same drawing shifted and scaled), and each photo twice (two crops), all with the same id: 452,886 query rows for 75,481 ids, and 25,000 corpus rows for 12,500 ids.

Results are keyed by id, so only one copy of each ends up counting, but every model still encodes all the copies, and the same photo id can appear twice in a ranking.

This adds SketchyI2IRetrieval.v2, which keeps the first row of each id in queries and corpus, and marks the old task as superseded. The descriptive stats are generated for v2.

openai/clip-vit-base-patch32, full test set:

	v1	v2
Query rows / corpus rows	452,886 / 25,000	75,481 / 12,500
hit_rate_at_1 (main score)	0.383	0.405
ndcg_at_10	0.373	0.385
Evaluation time	12.6 min	4.9 min

The score changes a little because in v1 the copy that ends up counting depends on how duplicate ids are merged.

[Samoed]

(the Sketchy augmentations: the same drawing shifted and scaled)

But shoudn't we modify ids instead to use augmented images?

[AdnanElAssadi]

(the Sketchy augmentations: the same drawing shifted and scaled)

But shoudn't we modify ids instead to use augmented images?

we can. I dont think scores would change much though but probably 6x compute