Problem in generating infrared modality by using data-engine

[fallenskies00]

Firstly, thanks for your contribution!
When i used data-engine to generate infrared picture from RGB picture, i got an unsatisfactory results that didn't seem like an infrared picture. I followed the method from '#25 (comment)' but it didn't work , therefore, i hope you could help me.
Thank you.

[Info]: Loading config from tools/infrared/stylebooth_tb_pro.yaml
[Info]: System take tools/infrared/stylebooth_tb_pro.yaml as yaml, because we find yaml in this file
[Info]: ENV is not set and will use default ENV as {'SEED': 2023, 'USE_PL': False, 'BACKEND': 'nccl', 'SYNC_BN': False, 'CUDNN_DETERMINISTIC': True, 'CUDNN_BENCHMARK': False}; If want to change this value, please set them in your config.
[Info]: Parse cfg file as
{
"NAME": "EDIT",
"IS_DEFAULT": false,
"DEFAULT_PARAS": {
"PARAS": {
"RESOLUTIONS": [
[
512,
512
],
[
1024,
1024
]
]
},
"INPUT": {
"IMAGE": null,
"PROMPT": "",
"NEGATIVE_PROMPT": "",
"TARGET_SIZE_AS_TUPLE": [
1024,
1024
],
"PROMPT_PREFIX": "",
"SAMPLE": "ddim",
"SAMPLE_STEPS": 50,
"GUIDE_SCALE": {
"text": 7.5,
"image": 1.5
},
"GUIDE_RESCALE": 0.5,
"DISCRETIZATION": "trailing"
},
"OUTPUT": {
"LATENT": null,
"IMAGES": null,
"SEED": null
},
"MODULES_PARAS": {
"FIRST_STAGE_MODEL": {
"FUNCTION": [
{
"NAME": "encode",
"DTYPE": "float16",
"INPUT": [
"IMAGE"
]
},
{
"NAME": "decode",
"DTYPE": "float16",
"INPUT": [
"LATENT"
]
}
],
"PARAS": {
"SCALE_FACTOR": 0.18215,
"SIZE_FACTOR": 8
}
},
"DIFFUSION_MODEL": {
"FUNCTION": [
{
"NAME": "forward",
"DTYPE": "float16",
"INPUT": [
"SAMPLE_STEPS",
"SAMPLE",
"GUIDE_SCALE",
"GUIDE_RESCALE",
"DISCRETIZATION"
]
}
]
},
"COND_STAGE_MODEL": {
"FUNCTION": [
{
"NAME": "encode_text",
"DTYPE": "float16",
"INPUT": [
"PROMPT",
"NEGATIVE_PROMPT"
]
}
]
}
}
},
"MODEL": {
"PRETRAINED_MODEL": "weights/stylebooth/stylebooth-tb-5000-0.bin",
"SCHEDULE": {
"PARAMETERIZATION": "eps",
"TIMESTEPS": 1000,
"ZERO_TERMINAL_SNR": false,
"SCHEDULE_ARGS": {
"NAME": "scaled_linear",
"BETA_MIN": 0.00085,
"BETA_MAX": 0.012
}
},
"DIFFUSION_MODEL": {
"NAME": "DiffusionUNet",
"PRETRAINED_PATH": null,
"IN_CHANNELS": 8,
"OUT_CHANNELS": 4,
"MODEL_CHANNELS": 320,
"NUM_HEADS": 8,
"NUM_RES_BLOCKS": 2,
"ATTENTION_RESOLUTIONS": [
4,
2,
1
],
"CHANNEL_MULT": [
1,
2,
4,
4
],
"CONV_RESAMPLE": true,
"DIMS": 2,
"USE_CHECKPOINT": false,
"USE_SCALE_SHIFT_NORM": false,
"RESBLOCK_UPDOWN": false,
"USE_SPATIAL_TRANSFORMER": true,
"TRANSFORMER_DEPTH": 1,
"CONTEXT_DIM": 768,
"DISABLE_MIDDLE_SELF_ATTN": false,
"USE_LINEAR_IN_TRANSFORMER": false,
"IGNORE_KEYS": []
},
"FIRST_STAGE_MODEL": {
"NAME": "AutoencoderKL",
"EMBED_DIM": 4,
"IGNORE_KEYS": [],
"BATCH_SIZE": 4,
"ENCODER": {
"NAME": "Encoder",
"CH": 128,
"OUT_CH": 3,
"NUM_RES_BLOCKS": 2,
"IN_CHANNELS": 3,
"ATTN_RESOLUTIONS": [],
"CH_MULT": [
1,
2,
4,
4
],
"Z_CHANNELS": 4,
"DOUBLE_Z": true,
"DROPOUT": 0.0,
"RESAMP_WITH_CONV": true
},
"DECODER": {
"NAME": "Decoder",
"CH": 128,
"OUT_CH": 3,
"NUM_RES_BLOCKS": 2,
"IN_CHANNELS": 3,
"ATTN_RESOLUTIONS": [],
"CH_MULT": [
1,
2,
4,
4
],
"Z_CHANNELS": 4,
"DROPOUT": 0.0,
"RESAMP_WITH_CONV": true,
"GIVE_PRE_END": false,
"TANH_OUT": false
}
},
"TOKENIZER": {
"NAME": "ClipTokenizer",
"PRETRAINED_PATH": "weights/clip-vit-large-patch14/",
"LENGTH": 77,
"CLEAN": true
},
"COND_STAGE_MODEL": {
"NAME": "FrozenCLIPEmbedder",
"FREEZE": true,
"USE_GRAD": false,
"LAYER": "last",
"PRETRAINED_MODEL": "weights/clip-vit-large-patch14/"
}
},
"ENV": {
"SEED": 2023,
"USE_PL": false,
"BACKEND": "nccl",
"SYNC_BN": false,
"CUDNN_DETERMINISTIC": true,
"CUDNN_BENCHMARK": false
}
}
已处理到此: ['./figs/origin_image.jpg']
Processing Infrared Images: 0%| | 0/1 [00:00<?, ?it/s]{'image': None, 'prompt': 'Convert the image to an infrared image', 'negative_prompt': '', 'target_size_as_tuple': [512, 512], 'prompt_prefix': '', 'sample': 'ddim', 'sample_steps': 50, 'guide_scale': {'text': 7.5, 'image': 1.5}, 'guide_rescale': 0.5, 'discretization': 'trailing'}
cuda
scepter [INFO] 2025-09-30 17:15:31,325 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading diffusion_model model
scepter [INFO] 2025-09-30 17:15:40,131 [File: unet_module.py Function: init_from_ckpt at line 479] Restored from /media/user/3EEC56E1EC56934F/svn_codes/MINIMA/data_engine/weights/stylebooth/diffusion_model.pth with 0 missing and 0 unexpected keys
scepter [INFO] 2025-09-30 17:15:41,461 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading cond_stage_model model
scepter [INFO] 2025-09-30 17:15:42,030 [File: diffusion_inference.py Function: init_from_ckpt at line 215] Restored from /media/user/3EEC56E1EC56934F/svn_codes/MINIMA/data_engine/weights/stylebooth/cond_stage_model.pth with 0 missing and 1 unexpected keys
scepter [INFO] 2025-09-30 17:15:42,030 [File: diffusion_inference.py Function: init_from_ckpt at line 221]
Unexpected Keys:
['transformer.text_model.embeddings.position_ids']
scepter [INFO] 2025-09-30 17:15:42,183 [File: tuner_inference.py Function: register_tuner at line 48] Loading tuner model
tunner_model_folder weights/stylebooth/step-210000/
scepter [INFO] 2025-09-30 17:15:47,446 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading diffusion_model model
scepter [INFO] 2025-09-30 17:15:47,446 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading cond_stage_model model
scepter [INFO] 2025-09-30 17:15:47,446 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading cond_stage_model model
scepter [INFO] 2025-09-30 17:15:48,252 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading cond_stage_model model
scepter [INFO] 2025-09-30 17:15:48,280 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading first_stage_model model
scepter [INFO] 2025-09-30 17:15:48,281 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
scepter [INFO] 2025-09-30 17:15:49,107 [File: ae_kl.py Function: init_from_ckpt at line 142] Restored from /media/user/3EEC56E1EC56934F/svn_codes/MINIMA/data_engine/weights/stylebooth/first_stage_model.pth with 0 missing and 0 unexpected keys
scepter [INFO] 2025-09-30 17:15:50,679 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading first_stage_model model
scepter [INFO] 2025-09-30 17:15:50,679 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading diffusion_model model
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.18it/s]
scepter [INFO] 2025-09-30 17:16:00,343 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading diffusion_model model█████████████████| 50/50 [00:09<00:00, 5.12it/s]
scepter [INFO] 2025-09-30 17:16:00,343 [File: diffusion_inference.py Function: dynamic_load at line 268] Loading first_stage_model model
scepter [INFO] 2025-09-30 17:16:00,388 [File: diffusion_inference.py Function: dynamic_unload at line 293] Unloading first_stage_model model
scepter [INFO] 2025-09-30 17:16:00,388 [File: tuner_inference.py Function: unregister_tuner at line 35] Unloading tuner model
Processing Infrared Images: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:30<00:00, 30.79s/it]

[LSXI7]

The LoRA weights are not loaded correctly. Try deleting the corresponding weight folders and cache folders, then re-download them — this may help resolve the it.