实验结果复现

[Qzy568]

我想复现在qwen2-VL 上的实验结果，但是结果和表里不同，请问有什么参数设置和版本的要求吗

[yfzhang114]

Perhaps you could provide your reproduction details, such as which evaluation toolkit you chose, lmmseval or vlmeval kit, the code and your reproduction results.

[Qzy568]

我想要复现您的实验结果，请问能否提供Qwen2-VL的代码

[yfzhang114]

We did not save the relevant code. This implementation is very easy and can be directly evaluated using lmmseval or vlmevalkit. The settings we use are also the default settings of Qwen/Qwen2-VL-7B-Instruct.

[Qzy568]

However, the maximum supported pixel value of qwen2-vl-7b model is 1280 * 28 * 28, which is smaller than the image size of the dataset. How to deal with it？

[yfzhang114]

Thanks for reaching out! Generally, most MLLM models, including Qwen2-VL, can't handle high-resolution images as per our benchmark, which is why their performance might not match expectations. The process_vision_info function in Qwen should automatically adjust the image to fit within its capabilities. Let me know if you need further assistance!

[Qzy568]

Thank you for your answer. Now I want to use gpt-4o-mini to reproduce your results, but the image processed by gpt-4o-mini needs to be less than 20MB, how did you handle it?

[yfzhang114]

#7 (comment)

[Qzy568]

对于您的数据集，超出了很多模型的最大图片限制，请问您是都按照这个压缩函数压缩到7MB以下吗

[yfzhang114]

压缩到最大能承受的上传限制~

[Qzy568]

I used the following code to compress your remote sensing image, but it could not be compressed, what is the reason?

from PIL import Image
import io
def compress_image_to_memory(image_path, max_size_bytes=7 * 1024 * 1024, initial_quality=85, min_quality=10, max_width=4096, max_height=4096):
    """
    Compress an image to less than 5MB and keep it in memory without saving to disk.
    
    :param image_path: Path to the original image
    :param max_size_bytes: Target maximum size in bytes (default 5MB)
    :param initial_quality: Starting compression quality (1-100)
    :param min_quality: Minimum allowable compression quality (1-100)
    :param max_width: Maximum width of the image
    :param max_height: Maximum height of the image
    :return: Compressed image byte data
    """
    try:
        with Image.open(image_path) as img:
            # Resize the image to limit maximum width and height
            img.thumbnail((max_width, max_height), Image.Resampling.LANCZOS)
            
            # Gradually reduce the image quality until the image is less than 5MB
            quality = initial_quality
            compressed_image_bytes = io.BytesIO()
            img.save(compressed_image_bytes, format=img.format, quality=quality, optimize=True)
            
            while len(compressed_image_bytes.getvalue()) > max_size_bytes and quality > min_quality:
                # Reduce quality and recompress
                quality -= 5
                compressed_image_bytes = io.BytesIO()
                img.save(compressed_image_bytes, format=img.format, quality=quality, optimize=True)
                compressed_size = len(compressed_image_bytes.getvalue())
                print(f"Compressed Image Size: {compressed_size / (1024 * 1024):.2f} MB")  # Output compressed size in MB
                
            if len(compressed_image_bytes.getvalue()) > max_size_bytes:
                print("Warning: Unable to compress image to under 5MB even at the lowest quality.")
                pass
            else:

                print(f"Compressed image size: {len(compressed_image_bytes.getvalue()) / (1024 * 1024):.2f} MB")
                pass
                
            return compressed_image_bytes.getvalue()
    except Exception as e:
        print(f"Error compressing image: {e}")
        return None


image_path = 'MME-RealWorld/remote_sensing/03949_Toronto.png'
compress_image_to_memory(image_path)

Output:

Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Compressed Image Size: 17.46 MB
Warning: Unable to compress image to under 5MB even at the lowest quality.