SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion

Paul Engstler, Iro Laina, Christian Rupprecht, Andrea Vedaldi
Visual Geometry Group, University of Oxford

SynCity 3000 produces globally coherent 3D worlds while enabling fine-grained layout control. It is the follow-up work to SynCity, addressing its key limitations. By generating the entire scene all at once, SynCity 3000 produces scenes that are more visually and semantically coherent than those produced by SynCity or related methods. Prompting is no longer constrained to a rigid grid but allows for precise placement of objects within the scene.

📦 Installation

Prerequisites

• System: The code was tested on Ubuntu 22.04. We expect it to run on other Linux-based distributions too.
• Hardware: An NVIDIA GPU with at least 48GB of memory is required. We have used A40 and A6000 GPUs.
• Software:
  • The CUDA Toolkit is needed to compile certain submodules. We have tested CUDA versions 11.8 and 12.4.
  • Conda is used to create the environment to run the code. This environment uses Python version 3.10.

Installation Steps

1. Clone the repository:

git clone --recurse-submodules https://github.com/paulengstler/syncity-3k.git
cd syncity-3k

2. Install the dependencies:

source ./setup.sh --new-env --basic --xformers --diffoctreerast --spconv --mipgaussian --kaolin --nvdiffrast

Make sure to have set the environment variable CUDA_HOME, which should point to your CUDA Toolkit installation. If you run into issues while running this setup script, please refer to the README in the TRELLIS repository, which provides additional guidance.

3. Create a HuggingFace account if you do not already have one. Then, agree to the terms & conditions of black-forest-labs/FLUX.1-dev. Run huggingface-cli login and follow the instructions.

⚙️ Usage

Make sure to have activated the syncity-3k Conda environment.

The process to generate a world is split into two straightforward steps.

Step 1: Creating the Scene Template

The scene template is an image of the world that we will be turning into 3D. We generate it using a JSON file that provides the text prompts for the 2D image diffusion model to produce the template image. In this repository, we share example prompts in the folder prompts. We also provide a World Designer that makes it easy to create custom JSON files.

With a prompt JSON file of your choice, here prompts/ruins.json, run the following to produce a scene template:

python make_scene_template.py prompts/ruins.json --output_dir outputs/ruins

Step 2: Turn the Scene Template into 3D

To produce a 3D world from the scene template, we use our fine-tuned version of TRELLIS, which will be automatically pulled from the Hugging Face repository paulengstler/syncity-3k. Continuing with the example from step 1:

python convert_to_3d.py outputs/ruins/template.png --output_dir outputs/ruins --prompt_json prompts/ruins.json

Find the final generated 3D world at outputs/ruins/scene.ply and outputs/ruins/scene_color_adjusted.ply. For most scenes, the latter will look better as we adjust the color statistics to compensate for a loss in color vibrancy in the 3D conversion process.

Custom Scene Template

The second step does not strictly require scene templates generated by the first step. Instead, you may generate them in any way you like as long as they match the dimmetric framing. In the paper, we have demonstrated generating templates using ChatGPT Images. We recommend running step 1 once to generate a scene template that can be used as a reference. Please be aware that the checkpoints for step 2 were trained to support boundary structures that are at most 60px tall.

Prompting Guide

As described in the paper, scene layout prompts can be generated with an LLM (see section A of the supplementary materials). However, using the World Designer, you can also easily craft them yourself.

Compared to the original SynCity paper, our proposed method imposes virtually no restrictions on the scenes you can generate. Here, we share a few tips to help you obtain great results:

• Use short and simple scene and layout constraint prompts.
• Layout constraints are powerful tools to steer scene template generation, but come with some caveats:
  1. They act as nudges for the diffusion model but do not guarantee that the structure will appear.
  2. They slow down the generative process, so use them sparingly.
  3. Larger constraints generally work better than smaller ones.
• To generate convincing indoor scenes, we recommend creating an approximate floor plan with layout constraints.
• Avoid surfaces that reflect. These tend to confuse the 3D conversion model.
• Specify uniform, bright light in your theme prompt (e.g., "midday sunlight").
• Add "8k, photorealistic natural textures, isometric perspective" to your theme prompt.
• Try using multiple different random seeds when generating the scene template (--seed N).

Training

Please note that the inference scripts above automatically pull our pre-trained checkpoints. This section is only relevant if you want to retrain the models from the TRELLIS checkpoints.

First, consult the README in the data_engine directory to generate a synthetic dataset that mirrors what we have used to fine-tune the TRELLIS sparse structure and structured latent models. For our pre-trained checkpoints, we used a dataset with 320k samples.

To train the models, run:

# Sparse Structure
python train.py --config configs/ft_ss_flow_img_context.json --output_dir outputs/models/ft_ss_flow_img_context --data_dir /FIXME/dataset

# Structured Latent
python train.py --config configs/ft_slat_flow_img_context.json --output_dir outputs/models/ft_slat_flow_img_context --data_dir /FIXME/dataset

💐 Acknowledgements

We thank the authors of TRELLIS, Flux, and 3D Gaussian Splatting as well as the contributors to the Blender project. Without these projects, SynCity 3000 would not have been possible.

The authors of this work are supported by ERC 101001212-UNION and Meta Research.

📜 Citation

If you find this work helpful, please consider citing our paper:

@inproceedings{
    engstler2026syncity3k,
    title={SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion},
    author={Paul Engstler and Iro Laina and Christian Rupprecht and Andrea Vedaldi},
    booktitle={European Conference on Computer Vision},
    year={2026}
}