Skip to content

fix: retry corrupt trailing video frames in resample_video - #5795

Merged
Samoed merged 1 commit into
embeddings-benchmark:mainfrom
AdnanElAssadi:fix/vcdb-broken-video
Oct 27, 2026
Merged

Samoed merged 1 commit into
embeddings-benchmark:mainfrom
AdnanElAssadi:fix/vcdb-broken-video

Conversation

@AdnanElAssadi

@AdnanElAssadi AdnanElAssadi commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

VCDBCoreRetrieval and VCDBCoreAudioVideoRetrieval crash during evaluation with RuntimeError: decode_av_frame ... Invalid data found when processing input. One video, beckham_70_yard_goal/ab12206048207553294916664d0ebf1cd9e81ef6.flv, has a corrupt last frame: frames 0–1149 decode, frame 1150 doesn't.

resample_video already retries with fewer trailing frames, but only when the error says "no more frames". This also retries on "Invalid data found", so the video is read without its last frame and the dataset stays as it is.

main This PR
VCDBCoreRetrieval, microsoft/xclip-base-patch32 crash 0.452
VCDBCoreAudioVideoRetrieval, facebook/pe-av-small crash 0.445

@Samoed Samoed left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dug into why this video "worked" originally. The file really is corrupt, so the dataset isn't the problem.

  • beckham_70_yard_goal/ab12206048207553294916664d0ebf1cd9e81ef6.flv reports 1151 frames. Frames 0–1149 decode fine, but the last one fails (Could not push packet to decoder: Invalid data found when processing input). PyAV hits the same packet after 1150 frames, so it's the file's tail rather than a torchcodec quirk. I checked this at the pinned dataset revision 61bcc06a….
  • The stats generation and cache hashing go through hash_video (mteb/_content_hashes.py), which uses num_frames - 1 and so never touches the last frame. That is probably why the stats and the original contribution looked fine. I couldn't check what the original run did, so this is an inference.
  • Evaluation reads the frames in resample_video (mteb/models/modality_collators.py). It already retries by dropping trailing frames, but only when the error contains "no more frames". This file raises "Invalid data found…", so the error propagates. Whether a model hits it depends on whether it samples the last frame index.

Because the file is only bad at the very end, I'd rather not drop the video and change the benchmark data. Instead, could we extend the retry in resample_video to also catch "Invalid data found"?

@AdnanElAssadi AdnanElAssadi mentioned this pull request Oct 27, 2026
7 of 22 tasks
@AdnanElAssadi AdnanElAssadi changed the title fix: drop the one undecodable video from the VCDB tasks fix: retry corrupt trailing video frames in resample_video Oct 27, 2026
@Samoed
Samoed merged commit 61750f8 into embeddings-benchmark:main Oct 27, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants