Question about README_FMNIST.md

[KOjuny]

Thanks for your great work on this project.

I downloaded the model checkpoints using python scripts/download_resources.py, added the following to my .env file:
DIFFUSION_FMNIST_MLP_CHECKPOINT_N_EPOCH=540+2 # add the epoch-num of the checkpoint you want to use with 2 (TODO: fix this) DIFFUSION_FMNIST_MLP_CHECKPOINT=/workspace/FLIPD_RF/flipd/outputs/downloads/dgm-geometry/checkpoints/diffusions/fmnist/epoch=540-step=253729.ckpt
Then I ran:
python scripts/train.py dataset=fmnist +experiment=train_diffusion_greyscale +checkpoint=diffusion_fmnist +callbacks@all_callbacks.lid=flipd_curve all_callbacks.lid.frequency=1 all_callbacks.lid.lid_estimation_args.method=hutchinson_gaussian

But I got the following error:

I would be grateful if you could point out what I may have done incorrectly.
Thanks!

[HamidrezaKmK]

I will get back to you soon on this and double check the checkpoints we have uploaded to make sure they are correct. But in short: You'd have to set the environment variable DIFFUSION_FMNIST_N_EPOCH to the number of your epoch in the checkpoint + 2! This is a .env (dotenv) environment variable.

[KOjuny]

I do this, but i got same error...

thanks for answer

[HamidrezaKmK]

So sorry for the late response, I was away for a while.

Please pull the README again. I have also attached my log files after following the instruction below. This is an example checkpoint of FMNIST trained on a UNet architecture (the architecture is an important factor for how the curve looks like):

https://drive.google.com/file/d/1GhugGZfb4r-f5PUWscUq-OzoIagCv-eJ/view?usp=drive_link

You can reproduce this checkpoint on your machine by running the following:

python scripts/train.py dataset=fmnist +experiment=train_diffusion_greyscale

If your environment is set up properly, these artifacts will pop up in your output/mlruns/... directory.

Once this is done, we now need to get the FLIPD curve. To do so, first go to your .env file and add the following:

DIFFUSION_FMNIST_CHECKPOINT_N_EPOCH=(xxx+2) # add the epoch-num of the checkpoint you want to use with 2 (TODO: fix this)
DIFFUSION_FMNIST_CHECKPOINT=path/to/logs/epoch=xxx-step=yyyyy.ckpt

Next, simply run one of the scripts in the README. I would recommend the following version which is slower but is not stochastic:

python scripts/train.py dataset=fmnist +experiment=train_diffusion_greyscale +checkpoint=diffusion_fmnist +callbacks@all_callbacks.lid=flipd_curve all_callbacks.lid.frequency=1 all_callbacks.lid.lid_estimation_args.method=deterministic

Running this will log different artifacts. What we are interested in are the FlipdEstimatorCurve:

https://drive.google.com/file/d/1h_DO6A8cw7iEWTrjM69kFJxW8myNkkIk/view?usp=drive_link

with the following curve which is similar to Figure 10-d:

For the rest of the figures, a similar approach can be taken. If anything, feel free to reach out directly to me for faster response time: hamid@layer6.ai. I am planning to refactor the codebase and add some features, any form of feedback would be greatly appreciated :)