InstanceConsistency

Improving Instance-Level Semantic Consistency in
Grounded Text-to-Image Diffusion via
Inference-Time Attention Routing and Self-Guidance

---

Overview

InstanceConsistency is a training-free inference-time framework built on top of GLIGEN-XL. It improves instance-level semantic consistency under multi-object prompts by explicitly regulating semantic interactions during diffusion sampling.

The framework introduces:

• Attention Routing
  • Cross-attention masking (token ↔ region)
  • Self-attention gating (region ↔ region)
• Appearance-Level Self-Guidance
  • Late-stage refinement using feature–attention alignment

No retraining or finetuning is required.

---

Framework

---

Repository Structure

InstanceConsistency/
├── code/
│   ├── attention/
│   │   ├── alignment.py
│   │   ├── attention.py
│   │   ├── masks.py
│   │   ├── processors.py
│   │   ├── recorder.py
│   │   └── registry.py
│   ├── pipeline.py
│   ├── self_guidance_recorder.py
│   ├── presets.json
│   └── infer.py
│
├── src/
│   └── Ours_pipeline.jpg
│
├── test/
├── environment.yml
├── environment.lock.yml
├── requirements.txt
└── README.md

---

Environment Setup

Conda (Recommended)

conda env create -f environment.lock.yml
conda activate igligen-xl

Manual Installation

conda create -n igligen-xl python=3.10 -y
conda activate igligen-xl
pip install -r requirements.txt

Verified Versions

• torch: 2.7.0 + CUDA 12.6
• diffusers: 0.33.1
• transformers: 4.52.3
• accelerate: 1.7.0
• safetensors: 0.5.3
• huggingface_hub: 0.33.0

---

Base Model Weights (GLIGEN-XL)

We do not provide pretrained model weights.

Please follow the official GLIGEN-XL repository to obtain the base checkpoints:

https://github.com/jiuntian/igligen-xl

After downloading, place the weights locally, for example:

./autodl-tmp/gligenxl_weight/

---

Prompt Preset Format

All prompts are stored in a single JSON file:

code/presets.json

Example:

[
  {
    "prompt": "a blue bowl and a red cup",
    "phrases": ["a blue bowl", "a red cup"],
    "bboxes": [
      [0.1, 0.42, 0.31, 0.56],
      [0.8, 0.37, 0.93, 0.61]
    ]
  },
  {
    "prompt": "a brown bench and a green bird",
    "phrases": ["a brown bench", "a green bird"],
    "bboxes": [
      [0.1, 0.39, 0.53, 0.59],
      [0.76, 0.38, 0.93, 0.6]
    ]
  }
]

---

Inference

Basic Usage

python code/infer.py \
  --model_path ./autodl-tmp/gligenxl_weight \
  --preset_path code/presets.json \
  --output_dir ./result \
  --seed 8031 \
  --num_steps 50 \
  --guidance_scale 7.5 \
  --enable_self 1 \
  --enable_cross 0

Generated images will be saved to:

./result/

---

Evaluation Resources

T2I-CompBench++

https://github.com/Karine-Huang/T2I-CompBench

ImageReward

https://github.com/THUDM/ImageReward

pip install image-reward

---

Notes

• Attention maps can be recorded via AttnRecordConfig
• CPU offloading is recommended for long inference runs
• Appearance-level self-guidance is applied only at late denoising stages