GhostVision is an advanced Deep Learning framework designed to translate single-channel thermal imagery into highly realistic, 3-channel RGB visible-light spectrum images. By bridging the visual divide between heat signatures and human intuition, this robust Conditional Generative Adversarial Network (cGAN) produces high-fidelity outputs suited for a variety of real-world operational environments.
- How It Works
- Use Cases
- Architecture Overview
- Composite Loss Function
- Repository Structure
- Installation
- Usage Guide
- License
Thermal cameras capture Long-Wave Infrared (LWIR) radiation emitted by objects, producing a greyscale representation maps of heat distribution rather than reflected light. While excellent for detecting objects in pure darkness or through obscurants (like fog or smoke), thermal images lack the semantic richness—such as texture and color—necessary for easy human interpretation or downstream computer vision tasks designed for standard RGB imagery.
GhostVision solves this by framing the Thermal-to-RGB translation as an Image-to-Image translation problem.
- Input: A tensor representing an normalized 2D thermal heat signature.
- Translation: The U-Net Generator extracts structural features (edges of a person, vehicle shape) and learned color priors to hallucinate a realistic visible-light spectrum texture over the thermal structure.
- Refinement: The generative process is constantly critiqued by an adversarial Patch Discriminator, forcing the generator to produce imperceptibly realistic RGB details while maintaining strict structural alignment enforced by custom perceptual and LAB color space losses.
The ability to translate thermal data into RGB imagery unlocks immediate value across multiple domains:
Thermal sensors excel at identifying pedestrians and wildlife on pitch-black country roads or in heavy fog. GhostVision can convert this thermal data into vivid RGB feeds, allowing existing RGB-trained object detection algorithms (like YOLO or Faster R-CNN) to function seamlessly in zero-visibility conditions without requiring complete retraining on thermal datasets.
In chaotic environments like building fires, smoke, or dense forests at night, search teams rely on thermal imaging. Translating those thermal feeds into RGB images allows rescue operators to contextualize environments much faster, recognizing clothing colors, terrain textures, and structural layouts.
Perimeter defense systems often combine thermal and RGB cameras. GhostVision can synthesize daylight-equivalent feeds from nighttime thermal captures, reducing cognitive load on human security operators who may struggle to interpret anomalous greyscale heat blobs over long shifts.
Thermography is used to monitor inflammation or blood flow. Translating thermal maps onto expected visible-light skin textures can help clinicians better visualize local abnormalities relative to the patient's physical anatomy.
GhostVision employs a Pix2Pix-inspired adversarial architecture tailored for high-fidelity cross-modal image synthesis.
An Encoder-Decoder architecture featuring robust skip connections (U-Net). It progressively downsamples the single-channel thermal input into a deep latent representation and decodes it back into a 3-channel RGB image.
- Encoder (Downsampling): Series of
Conv2d->BatchNorm2d->LeakyReLUlayers that extract high-level feature maps from the raw heat signature. - Decoder (Upsampling): Series of
ConvTranspose2d->BatchNorm2d->ReLUlayers. Crucially, the encoder's feature maps are concatenated with the decoder's feature maps at each respective level, preventing the loss of fine spatial details (like sharp edges) during the translation process. - Output: A
Tanhactivation bounding the synthesized image between[-1, 1], mapped to RGB normalized space.
A Markovian discriminator (PatchGAN) that classifies
- Why PatchGAN?: Standard discriminators often result in blurry images because they focus on global outlines. By evaluating structural patches independently, the PatchDiscriminator enforces high-frequency, crisp textures.
- It takes both the thermal source image and the RGB image (either real or generated) as paired inputs, ensuring the synthesized texture logically corresponds to the original heat map layout.
Training a GAN to invent textures from heat signatures is notoriously unstable. GhostVision stabilizes training and ensures physical realism using a meticulous, multi-objective loss function:
- Adversarial (GAN) Loss (MSE): The standard Minimax objective. It forces the generator to synthesize textures realistic enough to fool the discriminator.
- L1 Loss: Controls the overall structure. It computes the absolute pixel-wise difference between the generated image and the ground truth RGB image, ensuring spatial alignment (low-frequency correctness).
- LAB Color Space Loss:
- RGB distance alone is a poor measure of human color perception. GhostVision converts predictions into the CIE-LAB color space and computes L1 loss there.
- This explicitly penalizes color shifts and luminance inconsistencies independently, leading to far more natural coloration (e.g., green trees, grey roads).
- VGG Perceptual Loss:
- Pushes the generated image through a pre-trained ImageNet VGG-16 network.
- Computes L1 distance between the high-level feature maps of the real and generated images. This ensures semantic consistency (a generated car "feels" structurally like a car to CNN feature extractors, rather than just matching raw pixel values).
GhostVision/
├── data/
│ ├── raw/ # Put your raw paired thermal/RGB dataset here
│ └── processed/ # Processed 256x256 Train/Val/Test splits
├── experiments/
│ ├── checkpoints/ # Generator/Discriminator weights saved here
│ └── results/ # Inference and generated output validations
├── models/
│ ├── discriminator.py # PatchGAN Discriminator definition
│ ├── generator.py # UNet Generator definition
│ └── losses.py # Contains LAB space and VGG Perceptual logic
├── src/
│ ├── dataset.py # PyTorch Dataloader implementation for paired data
│ ├── train.py # Main training loop and optimization scheduler
│ ├── eval.py # Evaluation and benchmark script
│ └── utils.py # Core helpers (seed locking, device mapping)
├── prepare_dataset.py # Automated dataset resizing and splitting script
├── test.py # Dataloader sanity check with matplotlib visualizations
├── requirements.txt # Package dependencies
└── README.md # Documentation central
- Clone the repository:
git clone https://github.com/yourusername/GhostVision.git cd GhostVision - Setup your environment:
We recommend using a virtual environment (e.g.,
condaorvenv).python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
- Install exact dependencies:
pip install -r requirements.txt
Place your raw paired imaging datasets into data/raw/thermal and data/raw/rgb.
Ensure that matching thermal and RGB frames share identical filenames (e.g., frame_001.jpg).
Run the automated data preparation script. This normalizes image sizes to 256x256 and automatically splits them into training (70%), validation (15%), and testing (15%) directories inside data/processed/.
python prepare_dataset.pyOptional: To visually verify that your thermal/RGB pairs are correctly aligned before training:
python test.pyStart the automated training loop. The script automatically detects CUDA arrays and defaults to CPU if unavailable. Progress is tracked via tqdm, and model checkpoints (generator_epoch_X.pth) are periodically saved to experiments/checkpoints/.
python src/train.pyTuning Hyperparameters:
Open src/train.py to modify system configurations. Core tunable parameters include:
BATCH_SIZE = 8(Scale up to 16/32 depending on VRAM)LR_G / LR_D(Learning rates for Generator/Discriminator. Discriminator is kept weaker purposefully).- Loss Weights: Control the influence of specific losses:
LAMBDA_L1,LAMBDA_LAB,LAMBDA_VGG.
Once a satisfactory checkpoint is produced, you can run the evaluation script over the test dataset split to visualize the translations and compute quantitative metrics (PSNR, SSIM).
python src/eval.pyGenerated frames will be deposited in experiments/results/.
This project is licensed under the MIT License - see the LICENSE file for complete details. You are free to modify, distribute, and utilize this software in private or commercial settings.