FLIPD results differ from paper using MNISTDiffusion — possible reasons?

[KOjuny]

Hello,

I trained a diffusion model using the code from the following repository:
MNISTDiffusion
Using this trained model, I implemented a method to calculate FLIPD, as shown in the attached image.

However, my results differ from those reported in the paper.

I am wondering if this discrepancy could be due to differences in the model architecture or if I might have misunderstood the method.

I am sharing my experimental results along with the GitHub repo for reference.
My repo

FMNIST result :

MNIST result :

Thank you for your help!

[HamidrezaKmK]

The architecture can play a large role! The results are highly dependent on how well the chosen architecture can approximate the score function, and FLIPD is particularly sensitive to that fit. In practice we’ve seen large shifts when switching from a U-Net to a simple MLP -- even though, in principle, a perfectly fitted model should recover the same intrinsic LID.

Because of this sensitivity, it’s safer to use FLIPD for relative comparisons rather than relying on its absolute values. Pick a timestep $t \in (0, 1)$, rank your images by their estimated LID, and you’ll find they order themselves by a meaningful notion of relative complexity. Also, try using MLP architectures, for some unknown reason, score networks based on MLPs better capture the true score function, even though their image quality is subpar.

In general, your plots look sensible: for the same $t$, the estimated LID for Fashion-MNIST is consistently higher than for MNIST, as expected. Also, you are observing mild "knee" structures here.

For more information on the effect of architecture, please check Appendix E, and in particular E.5 of https://arxiv.org/pdf/2406.03537

[HamidrezaKmK]

One more thing: I think you are not computing the $\ell_2$ term properly (a mistake we did for our other repo and you pointed it out). You would need to square the term. This repo does not have that bug and performs that "squaring".

Can you please double-check this and re-send your images?

[KOjuny]

Thanks for response!

I fix my FLIPD computation code, and the result is almost same :

The part I find most confusing is when $t \rightarrow 0$, since the results differ significantly from those reported in the paper.

Thank you.

[HamidrezaKmK]

Thanks for the response!

Can you please sort 20 different images according to your estimate? Just to be sure if there's an issue?
The architecture used in the paper is much different from yours so one must not expect it to give the exact same value.