SSL_SaN: Self-Supervised Learning with Sound and Negatives

Official PyTorch implementation for Learning from Silence and Noise for Visual Sound Source Localization (BMVC 2025).

SSL_SaN: Self-Supervised Learning with Sound and Negatives for Audio-Visual Sound Source Localization
Project Page | Dataset | Paper | PDF | Poster

Table of Contents 📑

• Overview
  • Key Features
• Training
• IS3+ Dataset Available!
• Evaluation
• Supported Models
• Datasets
• Project Structure
• Installation
• Citation
• License
• Contact
• Acknowledgments

---

Overview 📋

SSL_SaN is a self-supervised learning framework for audio-visual sound source localization that leverages negative samples (silence, noise, and off-screen audio) to improve robustness and localization accuracy. This repository provides the official implementation along with comprehensive evaluation tools for benchmarking against state-of-the-art methods.

Key Features ✨

• 🔇 Robust Negative Sample Learning: Systematic use of silence, noise, and off-screen audio for improved model robustness
• 🗂️ Multi-Dataset Evaluation: Comprehensive benchmarking across VGG-SS, IS3, IS3+, S4, and Flickr datasets
• 🔄 Cross-Modal Analysis: Tools for analyzing modality gap and cross-modal retrieval performance
• 📊 Comprehensive Metrics: CIoU, AUC, pIA, F-LOC, and modality gap measurements
• 🏆 Baseline Comparisons: Evaluation framework includes EZVSL, SLAVC, FNAC, LVS, SSL-TIE, and SSL-Alignment

---

Training 🚀

For details on training SSL_SaN, including its negative sample augmentation and all training options, please refer to the train/README.md. This file provides a complete overview of the training pipeline, key features, and instructions for running and configuring training.

---

IS3+ Dataset Available! 🎉⭐

The IS3+ dataset is now available for download!

IS3+ is our extended version of the IS3 dataset, providing additional annotations and test samples for comprehensive audio-visual sound source localization evaluation.

📥 Download: IS3+ Dataset on Zenodo

Click the [Dataset] button at the top of this README or download directly from Zenodo.

---

Evaluation 🧪

For a complete and detailed guide to evaluating SSL_SaN and all supported models—including quick start commands, dataset organization, evaluation scripts, metrics, and visualization—please refer to evaluate/README.md.

---

Supported Models 🤖

This repository implements and evaluates:

• SSL_SaN (Ours) ⭐
• SSL-TIE 🔗
  Authors: Liu et. al.
• SSL-Alignment (Senocak et al.) 🔗
  Authors: Senocak, Ryu and Kim et. al.
• LVS: Localizing Visual Sounds the Hard Way 🔗
  Authors: Chen et. al.
• SLAVC (Morgado et al.) 🔗
  Authors: Mo & Morgado
• EZVSL (Morgado et al.) 🔗
  Authors: Mo & Morgado
• FNAC 🔗
  Authors: [Sun & Zhang et. al.]

Datasets 📁

The framework supports evaluation on multiple benchmarks:

• 🎬 VGGSoundSources (VGG-SS) - Sound source localization dataset
• 🎵 IS3 - Audio-visual localization dataset
• 🎵 IS3+ - Extended IS3 dataset (Ours) ⭐
• 🎥 S4 (AVSBench) - Audio-visual segmentation benchmark
  • Bounding box annotations (s4_box)
  • Segmentation masks (s4_seg)
• 📸 Flickr SoundNet

Project Structure 🗂️

.
.
├── train/                            # Training code for SSL_SaN ⭐
│   ├── main_sam.py                   # Main training script
│   ├── opts.py                       # Training arguments and hyperparameters
│   ├── models/                       # Training model implementation
│   │   └── model.py                  # SSL_SaN training model
│   ├── networks/                     # Network architectures for training
│   ├── datasets/                     # Training data loaders
│   │   └── dataloader_sam.py         # Dataset with silence and noise augmentation
│   ├── utils/                        # Training utilities
│   ├── metadata/                     # Training metadata files
│   └── scripts/                      # Training scripts
│       └── SSL_SaN.sh                # Training script with SSL_SaN specific arguments
│
└── evaluate/                         # Evaluation code and resources ⭐
    ├── eval.py                       # Main evaluation script
    ├── infer.py                      # Inference and visualization
    ├── cma_exp.py                    # Cross-modal alignment experiments
    ├── modality_gap.py               # Modality gap analysis
    ├── recall_metrics.py             # Cross-modal retrieval metrics
    │
    ├── datasets/                     # Dataset loading and preprocessing
    │   ├── datasets.py               # General dataset implementations for EZ-VSL, FNAC and SLAVC
    │   ├── datasets_lvs.py           # LVS and SSL-Align specific datasets
    │   └── datasets_ssltie.py        # SSL-TIE and SSL-SaN specific datasets
    │
    ├── models/                       # Model architectures
    │   ├── model.py                  # EZVSL, SLAVC, FNAC models
    │   ├── model_lvs.py              # LVS and SSL-Align model
    │   └── model_ssltie.py           # SSL-TIE and SSL-SaN model
    │
    ├── networks/                     # Network components
    │   ├── audio_convnet.py          # Audio encoder networks
    │   ├── image_convnet.py          # Visual encoder networks
    │   └── resnet.py                 # ResNet backbones
    │
    ├── utils_dir/                    # Utility functions
    │   ├── utils.py                  # General utilities
    │   ├── eval_utils.py             # Evaluation metrics
    │   └── opts_ssltie.py            # SSL-TIE and SSL-SaN arguments
    │
    ├── metadata/                     # Dataset annotations and metadata
    ├── checkpoints/                  # Model checkpoints directory
    │
    └── scripts/                      # Evaluation scripts
        ├── eval_models.sh            # Sound localization evaluation
        ├── cross_modal_alignment.sh  # Cross-modal retrieval evaluation
        ├── modality_gap.sh           # Modality gap analysis
        ├── recall_metrics.sh         # Recall metrics computation
        └── infer_models_*.sh         # Inference scripts for different datasets

Installation 🛠️

Requirements 📦

• Python 3.7+
• PyTorch 1.7+
• CUDA (for GPU support)
• torchvision
• OpenCV
• scikit-learn
• numpy
• tqdm
• matplotlib
• PIL
• easydict

Setup ⚙️

# Clone the repository
git clone https://github.com/xavijuanola/SSL_SaN.git
cd SSL_SaN

# Install dependencies
pip install torch torchvision opencv-python scikit-learn numpy tqdm matplotlib pillow easydict

---

Citation 📝

If you find this work useful, please consider citing:

@misc{juanola2025learningsilencenoisevisual,
      title={Learning from Silence and Noise for Visual Sound Source Localization}, 
      author={Xavier Juanola and Giovana Morais and Magdalena Fuentes and Gloria Haro},
      year={2025},
      eprint={2508.21761},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.21761}, 
}

License 📄

This project is released under the MIT License.

Contact 📧

For questions or issues, please open an issue on GitHub or contact [xavier.juanola@upf.edu].

Acknowledgments 🙏

This repository builds upon and evaluates methods from multiple research works in audio-visual learning. Specifically, our evaluation code is adapted and continued from SSL-Alignment, while our training code is based on SSL-TIE. We thank the authors of EZVSL, SLAVC, FNAC, LVS, SSL-TIE, and SSL-Alignment for their contributions to the field and for making their code publicly available.