Skip to content

fix: Qwen3-VL embedders, reranker and UEmbed size video frames like qwen-vl-utils - #5766

Merged
Samoed merged 2 commits into
embeddings-benchmark:mainfrom
AdnanElAssadi:fix/qwen-vl-video-frame-size
Oct 25, 2026
Merged

Samoed merged 2 commits into
embeddings-benchmark:mainfrom
AdnanElAssadi:fix/qwen-vl-video-frame-size

Conversation

@AdnanElAssadi

@AdnanElAssadi AdnanElAssadi commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

The official Qwen3-VL-Embedding and Qwen3-VL-Reranker scripts resize video frames with qwen-vl-utils: each frame is fitted to the patch grid, then the clip to a total_pixels budget at 128–768 tokens per frame. mteb lets the transformers video processor resize instead. It spends one budget on the whole clip, so frames of short clips keep near-native size and small frames are not upscaled. Its cap_pixels_per_frame=True option is closer, but still matches the official size on only 11–31 of the 60 videos below.

This moves UEmbed's tensor resize from #5651 into a shared resize_video and adds the per-frame step it skipped. The Qwen3-VL wrappers go through sentence-transformers, so the resize runs in a small Qwen3VLVideoProcessor subclass before the usual processing, with do_resize=False, the same as UEmbed does.

Patch grid equal to the official one, 60 real videos from 10 MVEB tasks:

Model main This PR
Qwen3-VL-Embedding / DNA-VL-STEER 23 / 60 60 / 60
Qwen3-VL-Reranker (its own budget) 24 / 60 60 / 60
UEmbed 38 / 60 60 / 60

Embedding cosine to the official qwen-vl-utils path, 12 videos:

Model main This PR
Qwen3-VL-Embedding-2B min 0.943, mean 0.982 min 0.998, mean 0.999
UEmbed-2B min 0.967, mean 0.988 min 0.997, mean 0.998

Scores on 32-item slices:

Model Task main This PR
Qwen3-VL-Embedding-2B MSVDT2VRetrieval 0.956 0.957
Qwen3-VL-Embedding-2B DiDeMoV2TRetrieval 0.907 0.910
DNA-VL-STEER-2B DiDeMoV2TRetrieval 0.909 0.919
UEmbed-2B VATEXT2VRetrieval 0.957 0.961
Qwen3-VL-Reranker-2B XModBenchT2VReranking 0.531 0.625

@AdnanElAssadi
AdnanElAssadi force-pushed the fix/qwen-vl-video-frame-size branch from ea11dab to 252eaa7 Compare October 24, 2026 05:30
Comment thread mteb/models/model_implementations/qwen3_vl_embedding_models.py
@AdnanElAssadi
AdnanElAssadi force-pushed the fix/qwen-vl-video-frame-size branch from 252eaa7 to 5e9d48a Compare October 24, 2026 23:12
@Samoed
Samoed merged commit 5610d84 into embeddings-benchmark:main Oct 25, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants