Implementation of "PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications"
Salimans et al., ICLR 2017 | Paper | Blog | Official Repo
PixelCNN++ is a generative model that learns the exact probability distribution of images pixel by pixel using autoregressive modeling. This repository contains a from-scratch PyTorch implementation of the paper, trained on a toy subset of CIFAR-10.
| # | Modification | Description |
|---|---|---|
| 1 | Discretized Logistic Mixture | Replaces 256-way softmax with a mixture of logistics |
| 2 | Whole Pixel Conditioning | Models RGB jointly instead of sub-pixel by sub-pixel |
| 3 | Downsampling | Stride-2 convolutions instead of dilated convolutions |
| 4 | Short-cut Connections | U-Net style skip connections between encoder-decoder |
| 5 | Dropout Regularization | Dropout rate 0.5 to prevent overfitting |
├── dataset.py ← CIFAR-10 data loading (toy subset)
├── Model.py ← PixelCNN++ architecture (U-Net encoder-decoder)
├── losses.py ← Discretized logistic mixture loss
├── train.py ← Training loop with evaluation and image generation
├── plot_results.py ← Loss curve plotting
├── loss_curve.png ← Training results plot
├── generated.png ← Sample generated image
├── requirements.txt ← Dependencies
└── README.md
# Clone the repository
git clone https://github.com/<Kumari124>/<Image-generation>.git
cd <Image-generation>
# Install dependencies
pip install -r requirements.txtrequirements.txt:
torch
torchvision
matplotlib
numpy
Place CIFAR-10 data folder (containing cifar-10-batches-py/) in the root directory.
The dataset is loaded automatically from the current directory (.).
python train.pyThis will:
- Train for 20 epochs on 10,000 CIFAR-10 images
- Evaluate on 2,000 test images every epoch
python plot_results.py- Dataset: CIFAR-10 (10,000 train / 2,000 test)
- Epochs: 20
- Filters: 64
- Best Test Loss: 2.889 bits/dim (Epoch 18)
The model demonstrates a consistent decrease in loss, indicating effective learning of pixel dependencies.
Although both training and test loss decrease initially, slight overfitting is observed after epoch 18, where the test loss begins to increase.
This behavior is expected due to:
- limited dataset size (10,000 training samples)
- high model capacity
- shared distribution between training and test data
Despite the increase, the test loss remains relatively low. However, this does not necessarily indicate strong generalization, but rather reflects the model fitting the dataset distribution closely.
| Metric | Paper (Full Training) | Our Implementation (Toy) |
|---|---|---|
| Framework | TensorFlow | PyTorch |
| Training images | 50,000 | 10,000 |
| Filters | 192 | 64 |
| Epochs | ~5,000 | 20 |
| Best Test bits/dim | 2.92 | 2.889 |
Although our model achieves a numerically similar bits/dim value (2.889 vs 2.92), this is not a direct comparison. The paper uses significantly larger datasets, higher model capacity, and much longer training.
The lower value in our case is likely due to training on a smaller dataset, which makes the distribution easier to fit, rather than indicating a better model.
Input (32×32 RGB)
↓ MaskedConv2d Type-A (autoregressive entry)
Block1 [32×32] ─────────────────────────────→ Block6 [32×32]
↓ stride-2 conv ↑ transposed conv
Block2 [16×16] ──────────────────────→ Block5 [16×16]
↓ stride-2 conv ↑ transposed conv
Block3 [8×8] ──────────→ Block4 [8×8]
(skip conn)
Each block = 5 × GatedResNet layers
Output: 10 × nr_mix channels per pixel
The quality of generative models like PixelCNN is highly dependent on:
- dataset size
- model capacity
- training duration
Even with a reduced dataset, the model successfully learns meaningful pixel dependencies and demonstrates generative capability.
- Salimans et al., PixelCNN++, ICLR 2017 — arxiv
- van den Oord et al., Pixel Recurrent Neural Networks, ICML 2016
- Official TF Implementation — openai/pixel-cnn
- Blog post — Medium
Kajal Kumari | GNR 638 Assignment 3