Using text as conditioning in image generation models has some shortcomings. Image descriptions may be long, ambiguous, and loosely structured. Scene graphs could be a more structured alternative for representing image contents. In this project I investigated the performance of ControlNet conditioned on scene graphs with the current benchmark, sg2im on Visual Genome Dataset. The report can be found here. An overview of the project can be seen below:
The folder src contains the script for preprocessing the data for ControlNet and the scripts used for inference. The pretrained models can be optained from sg2im and HuggingFace. I used ControlNet v1.1 and Stable Diffusion v1.5. The preprocessing of the data is identical to sg2im. You can some of the examples in the folder images.