[Solved] FID calculation

[Bili-Sakura]

I would is it a mistake of FID calculation in all baseline works along with yours. To my undertanding, your generated image is awesome, so the FID should be a small number (take a reference in general computer vision domain where they achieve FID 1.5 - 10 in SOTA on ImageNet-256 dataset).

I suppose there is a mistake when using FrechetInceptionDistance. If we use image as uint8, we should set normalize=False otherwise, if we use image as [0,1] then set normalize=True.

from torchmetrics.image.fid import FrechetInceptionDistance
# Use normalize=False since we'll provide uint8 images in [0, 255] range
fid = FrechetInceptionDistance(normalize=False).to(device)

See fid.py from torchmetrics.

[Chen-Yang-Liu]

I would is it a mistake of FID calculation in all baseline works along with yours. To my undertanding, your generated image is awesome, so the FID should be a small number (take a reference in general computer vision domain where they achieve FID 1.5 - 10 in SOTA on ImageNet-256 dataset).

I suppose there is a mistake when using FrechetInceptionDistance. If we use image as uint8, we should set normalize=False otherwise, if we use image as [0,1] then set normalize=True.

from torchmetrics.image.fid import FrechetInceptionDistance

Use normalize=False since we'll provide uint8 images in [0, 255] range

fid = FrechetInceptionDistance(normalize=False).to(device)
See fid.py from torchmetrics.

Thank you for your interest in our work. I would like to clarify that in our work (as well as in the baselines we follow), we did not use torchmetrics.image.FrechetInceptionDistance to compute FID. Instead, we strictly follow the evaluation protocol of the prior work Txt2Img-MHN (IEEE TIP 2023, https://github.com/YonghaoXu/Txt2Img-MHN), which computes FID using the official implementation from pytorch-fid:
https://github.com/mseitzer/pytorch-fid

Therefore, the potential normalization issue related to normalize=True/False in torchmetrics does not apply in our case.

More importantly, it is worth noting that FID values are highly dependent on the Inception network used for feature extraction. Directly using an ImageNet-pretrained Inception model—while common in general computer vision—can be inappropriate for remote sensing imagery due to the significant domain gap. Following Txt2Img-MHN, we train an Inception network on remote sensing data and use this domain-specific model consistently across all methods for FID evaluation. This setting naturally leads to FID values that are not directly comparable to those reported on ImageNet-256 benchmarks (e.g., FID 1.5–10).

[Bili-Sakura]

A great thanks for clarification :)

By the way, can you share your pre-trained Inception model to measure FID or provide more detailed on how you train it (pertained dataset, training cost). I am currently heavily working with remote sensing visual generative models.