Evaluation results

[yingift]

I found your benchmark interesting and implemented evaluation on the DT (Diagram and Table) with you code.
However, my results is significantly lower than the officially reported numbers. Here is my setting:
Model: Qwen2.5-vl-7B, resolution: 4k*4k, max_new_tokens=512.
The prompt follows your official code
The final results are:
Perception task: 53.03% (ours) 77.97% official leaderboard
Reasoning task: 46.40%(ours) 59.80% official leaderboard
Can the authors offer the evaluation details? This matters a lot.

Additionally, if I test with COT like adding 'Respond with the reason why you selected the choice', the results marginally improve.
I wonder if you use the COT in the evaluation? Can you specify these?

Hope you reply soon.

[yfzhang114]

This seems very unusual, as we didn’t use any chain-of-thought prompting and directly tested Qwen2.5 VL using LMMs-Eval.

[yingift]

@yfzhang114 Thanks for your quick reply. Can you offer me the detailed parameter when you tested the qwen2.5-vl? Especially the resolution. I will double-check my code.

[yfzhang114]

we use the default parameter in lmms-eval and test it on 8*A100. For remote sensing tasks, we use 1605632//2 max pixels

    min_pixels: int = 256 * 28 * 28,
    max_pixels: int = 1605632,
    max_num_frames: int = 32,

[Laqcce-cao]

I can not understand what means 1605632//2 max pixels. Can you explain what this parameter means?

[yfzhang114]

The default max pixel is 1605632, the image size of remote sensing is too large so we use 802816

[Laqcce-cao]

Thanks for your patient and timely response. My result is very different with yours.
My result with cot is only comparable to you.
Can you share your evaluation code with me?
I will appreciate it for your help.

[yfzhang114]

https://github.com/EvolvingLMMs-Lab/lmms-eval

[yingift]

@yfzhang114 As you suggested, we evaluate the dataset again with the setting of max_pixels1605632//2,max_num_frames: int = 32,min_pixels: int = 256 * 28 * 28. But we still cannot reproduce the results officially reported. Even worse?

[yfzhang114]

do you use lmms-eval?

[yingift]

Results fromthis code
Experiments with lmms-eval is running.
These two code are very similar.

[yfzhang114]

Thank you for the information! Based on what you've shared, it seems like the code from the MME-RealWorld GitHub repository is more tailored for the VLMEval kit, but might be slightly outdated due to our adjustments in datasets or prompts. As a result, you recommend following the lmmseval instead.

[Laqcce-cao]

Thank you for your suggestion. We are using lmms-eval for testing. We evaluated “yifanzhang114/MME-RealWorld-lite-Lmms-eval”, but all the results were E. Have you encountered this problem before?

This is my configs:

accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=/data/haoyan/MME-RealWorld/model/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=True \
    --tasks mmerealworld \
    --batch_size 1 \
    --output_path ./logs/

this is my result:

[Laqcce-cao]

Thank you for your help, I have successfully run this evaluation script.

[Laqcce-cao]

I have test the DT task, but the result is still lower than yours. And my max_pixels: int = 1605632. my result as follow:

Can you give me your other setting(eg., random seed, numpy seed ，torch manual seed )？