Making Video-MME-v2 an official benchmark on the hub

[NielsRogge]

Hi,

Niels here from the open-source team at Hugging Face. Congrats on the release of Video-MME-v2!

We recently introduced evaluation results on Hugging Face, where HF datasets can be turned into native leaderboards. Find the current supported ones here.

Would you be interested in making https://huggingface.co/datasets/MME-Benchmarks/Video-MME-v2 an official benchmark on the hub? It would be the reference benchmark for video understanding.

Let me know if this interests you!

Kind regards,

Niels

[dongyh20]

Hi, thank you for your kind message! We're excited about this proposal and would love to make Video-MME-v2 an official benchmark on the hub. Please let us know what we can do to support this. We'd be very happy to collaborate and help.

[NielsRogge]

Awesome! Feel free to check the guide here: https://huggingface.co/docs/hub/eval-results#registering-a-benchmark. Basically you will need to add an eval.yaml at the root of the HF dataset repository, similar to the one of OlmOCRBench. Next, we will need to register Video-MME-v2 as evaluation framework to the list here: https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/eval.ts.

Let me know when you have done these steps!

[dongyh20]

Hi, I have added the eval.yaml to the HF dataset repository. And I have also created an MR: huggingface/huggingface.js#2097 to add the evaluation framework of Video-MME-v2 to the list in huggingface.js. Please tell me if there is anything that I can do.

[dongyh20]

Hi, we have finished the required steps, could you please add the Video-MME-v2 as one of the supported evaluation results on the hub?

[NielsRogge]

Hi, https://huggingface.co/datasets/MME-Benchmarks/Video-MME-v2 is now an official benchmark on the hub!

Feel free to start submitting pull requests to the models on your leaderboard. This can be done programmatically using the huggingface_hub library/hf CLI with coding agents/manually. They should add an .eval_results/video_mme_v2.yaml file at the root of each model repository.

See e.g. here for an example: https://huggingface.co/datalab-to/chandra-ocr-2/blob/main/.eval_results/olmocrbench.yaml

[dongyh20]

Thanks!

[NielsRogge]

Hi @dongyh20 can we keep this issue open for the scores?

[NielsRogge]

Hi @dongyh20 I opened 2 sample PRs:

• Kimi-K2.5: https://huggingface.co/moonshotai/Kimi-K2.5/discussions/116.
• Qwen3.5:-397B-A17B https://huggingface.co/Qwen/Qwen3.5-397B-A17B/discussions/78

This ensures that these models show up at https://huggingface.co/datasets/MME-Benchmarks/Video-MME-v2.

Could you open pull requests for the remaining models?

Also, you can add multiple task IDs (leaderboards) to the eval.yaml, e.g. overall_no_subtitles, overall_subtitles (instead of the current task ID video-mme-v2). The first one mentioned will show up as default leaderboard on the HF dataset. For now I've picked overall with subtitles.

Let me know if you need any help!

[dongyh20]

OK, thanks! We will follow this to add other models on the leaderboard