How to replace the stable-diffusion-v1-5 model with other SD model

[guijuzhejiang]

I tried to use another model instead of the stable-diffusion-v1-5 model by modifying pretrained_model_path in animation.yaml to the other model path. The run reported an error, and I modified the config in unet to change UNetMidBlock2DCrossAttn to UNetMidBlock3DCrossAttn, guessing that it is because you have added a time series dimension to your trained model. Although the modified model can be loaded successfully and animations are generated, the effect is still the same as the sd1.5 model, unchanged. So how to load the other models correctly?

[thepowerfuldeez]

Hi! I have also verified that pretrained_unet_path loads new model, but I don't see related changes from the checkpoint either. Seems like temporal attentions which are loaded later into the modified unet reduce the effect from SD Unet

[FurkanGozukara]

Hi! I have also verified that pretrained_unet_path loads new model, but I don't see related changes from the checkpoint either. Seems like temporal attentions which are loaded later into the modified unet reduce the effect from SD Unet

i was gonna ask this

so it is not supporting custom models?

[AugmentedRealityCat]

This is NOT using a checkpoint. This uses diffusers - basically everthing else in the stable-diffusion-v1-5 folder on Huggingface besides the ckpt and safetensors.

Try this: remove all the ckpt and safetensors files from your magic-animate\pretrained_models\stable-diffusion-v1-5 folder, (there should be 2 of each, for a total of 4 normally) start the magic-animate app and you will see it's still working perfectly.

If we want to replace the 1.5 model, we should try using another model in diffusers format, or use a script to translate a ckpt into a diffusers formatted model - which is a folder structure with a whole bunch of different files, but not a single ckpt or safetensors.

I haven't tried so there is no guarantee this is going to work, but if you are adventurous, the treasure might well be in that direction !

Here are a bunch of those models:
https://huggingface.co/models?other=stable-diffusion-diffusers

And here is a ckpt2diffusers script
https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

[guijuzhejiang]

This is NOT using a checkpoint. This uses diffusers - basically everthing else in the stable-diffusion-v1-5 folder on Huggingface besides the ckpt and safetensors.

Try this: remove all the ckpt and safetensors files from your magic-animate\pretrained_models\stable-diffusion-v1-5 folder, (there should be 2 of each, for a total of 4 normally) start the magic-animate app and you will see it's still working perfectly.

If we want to replace the 1.5 model, we should try using another model in diffusers format, or use a script to translate a ckpt into a diffusers formatted model - which is a folder structure with a whole bunch of different files, but not a single ckpt or safetensors.

I haven't tried so there is no guarantee this is going to work, but if you are adventurous, the treasure might well be in that direction !

Here are a bunch of those models: https://huggingface.co/models?other=stable-diffusion-diffusers

And here is a ckpt2diffusers script https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

Thanks for the reminder, I'm sorry I didn't make it clear, I used the model in the diffusers format (which includes the directory of feature_extractor, scheduler, text_encoder, tokenizer, unet, vae, etc.), but after directly replacing it, launching the program reported an error, so I also modified the config file, even though I successfully launched the program and generated an animation, it was the same as that generated by the 1. 5 model, so I don't think that the new model I used is valid!

[FurkanGozukara]

i tested and custom model still generates same thing

very weird

[liuwenran]

Check the weights of temporal_attention/temporal_attention.ckpt. This ckpt contains unet weights, so whatever your custom model is loaded, unet weights will be reloaded when you load this ckpt.

[FurkanGozukara]

Check the weights of temporal_attention/temporal_attention.ckpt. This ckpt contains unet weights, so whatever your custom model is loaded, unet weights will be reloaded when you load this ckpt.

when we delete motion module from config it doesnt work

so basically there is no way to use custom checkpoint?

[niqodea]

Seconding this, would be nice to try different checkpoints. Though from what I can understand it is not a given that different checkpoints will be compatible with other components of the model (temporal attention, appearance attention).

[liuwenran]

Check the weights of temporal_attention/temporal_attention.ckpt. This ckpt contains unet weights, so whatever your custom model is loaded, unet weights will be reloaded when you load this ckpt.

when we delete motion module from config it doesnt work

so basically there is no way to use custom checkpoint?

You can try to load the weights of temporal_attention first, and then load the custom ckpt. Not sure about the result. it seems that the author didn't fix the original unet weight while training motion module.

[alethea-deepak]

Same question. Anyone able to solve this so far?

[johndpope]

does this solve things? #111