RGBT-GroundBench

Visual Grounding Beyond RGB in Complex Real-World Scenarios

ECCV 2026

Tianyi Zhao^1,*    Jiawen Xi^1,*    Linhui Xiao²    Junnan Li¹    Xue Yang³    Maoxun Yuan^1,†    Xingxing Wei^1,†

¹Beihang University    ²Pengcheng Laboratory    ³Shanghai Jiao Tong University

^*Equal contribution    ^†Corresponding author

Introduction

Visual grounding aims to localize an object described by natural language. Existing benchmarks, however, are predominantly built on RGB images collected under clean conditions. They offer limited coverage of the low illumination, adverse weather, occlusion, and small or distant objects that perception systems encounter in the real world.

RGBT-GroundBench extends visual grounding beyond RGB. It is the first large-scale benchmark for RGB-Thermal grounding in complex real-world scenarios, pairing natural-language expressions and target boxes with aligned RGB-TIR imagery and fine-grained condition labels. The benchmark further provides a unified protocol for fair comparison across RGB, TIR, and RGB+TIR inputs.

Benchmark characteristics, annotation distribution, and representative model performance on RGBT-GroundBench.

21,535	38,760	13	11
RGB-TIR pairs	Grounding instances	Scene categories	Evaluated models

What RGBT-GroundBench Provides

• A challenging RGB-TIR benchmark. More than 40K images from diverse real-world scenes, with a substantially higher proportion of low-light and small-object cases than classical visual grounding datasets.
• Fine-grained condition annotations. Each instance is characterized at three levels: scene type; illumination and weather; and object size and occlusion.
• A unified evaluation protocol. The same data splits and metrics support controlled comparison of RGB-only, TIR-only, and RGB+TIR grounding.
• A broad empirical study. Evaluation of 11 representative models exposes four underexplored challenges: sensitivity to scene complexity, severe low-light degradation, the robustness benefit of LoRA adaptation, and limitations of current multimodal LLMs.
• A reproducible reference baseline. RGBT-VGNet provides a simple, competitive design for language-guided and reliability-aware RGB-TIR fusion.

Dataset

RGBT-GroundBench is built from FLIR, M3FD, and MFAD. Each retained sample contains paired RGB-TIR images, a referring expression, a target bounding box, and condition labels. Its targets are frequently small, occluded, and off-center, while its scenes span daylight, nighttime, weak illumination, and adverse weather.

Overview of RGBT-GroundBench. Click the figure to view it at full resolution.

Statistics

Sub-dataset	Instances	Train	Val	Test	TestA	TestB	TestC	Source
RefFLIR	9,712	7,000	608	2,104	837	640	968	FLIR
RefM3FD	7,548	3,604	168	3,776	1,232	1,094	1,848	M3FD
RefMFAD	21,500	16,000	1,256	4,244	789	2,452	2,544	MFAD
Total	38,760	26,604	2,032	10,124	2,858	4,186	5,360	All

TestA, TestB, and TestC are condition-specific subsets of the test set. See the paper for their precise definitions and the original dataset citations.

Download

Resource	Link
All resources	RGBT-Ground-Dataset on Hugging Face
RefFLIR	Annotations · Images
RefM3FD	Annotations · Images
RefMFAD	Annotations · Images

Reference Baseline: RGBT-VGNet

Motivated by the benchmark findings, we introduce RGBT-VGNet, a reference baseline that adapts an RGB-pretrained vision-language model to cross-spectral grounding and fuses the two modalities according to their reliability.

• Asymmetric Modality Adaptation (AMA) assigns higher-rank LoRA adaptation to the TIR branch, reflecting the larger domain gap from RGB pretraining.
• Language-Aware Visual Synergy (LAVS) lets the referring expression guide cross-modal interaction and retain target-relevant evidence.
• Tri-Prior Fusion (TPF) combines RGB and TIR features using modality, illumination, and language-aware reliability cues.

Architecture of RGBT-VGNet.

Getting Started

Installation

git clone https://github.com/crazyxiaoxi/RGBT-GroundBench.git
cd RGBT-GroundBench

conda env create -f environment_full.yml
conda activate rgbtvg

For a lighter pip-based setup:

conda create -n rgbtvg python=3.9
conda activate rgbtvg
pip install -r requirements.txt

Data Preparation

Download the images and annotations from the links above, then set the corresponding dataset paths in the training and evaluation scripts. By default, the code expects visual grounding data under:

../dataset_and_pretrain_model/datasets/VG

Training

RGBT-VGNet uses IAFv3 RGB-T fusion and the multi-stage HiLoRA schedule by default. Train it sequentially on all three benchmark datasets:

bash script_train/RGBT_VGNet/train_all.sh

To train on one dataset only:

bash script_train/RGBT_VGNet/ref_m3fd/train.sh

Each dataset script initializes from the released MMVG pretraining checkpoint, then runs warm-up and HiLoRA stages 1–3. Image size, batch size, GPU selection, learning schedule, and checkpoint path can be overridden through the environment variables defined at the top of each script; the released model configuration remains fixed to RGB-T input and IAFv3 fusion.

Evaluation

bash script_eval/RGBT_VGNet/evaluate_all.sh

This entry point runs inference and reports grounding accuracy across the supported datasets and evaluation settings.

Visualization

bash script_visualize/all/run_all_rgbt_vgnet_tests.sh

Repository Structure

RGBT-GroundBench/
├── data_pre/              # Data preprocessing scripts
├── datasets/              # Dataset definitions and loaders
├── models/                # Model architectures
├── script_train/          # Training scripts
├── script_eval/           # Evaluation scripts
├── script_visualize/      # Visualization tools
├── train_val/             # Training and validation utilities
├── utils/                 # Helper functions
├── environment_full.yml   # Reproducible Conda environment
├── requirements.txt       # Pip dependencies
└── README.md

Citation

If this benchmark or codebase is useful in your research, please cite our paper:

@inproceedings{zhao2026rgbtgroundbench,
  title     = {RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios},
  author    = {Zhao, Tianyi and Xi, Jiawen and Xiao, Linhui and Li, Junnan and Yang, Xue and Yuan, Maoxun and Wei, Xingxing},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

License

This repository is released for academic research. The source datasets remain subject to their respective licenses; please consult the original dataset pages before redistribution or commercial use.

Contact

For questions, please open a GitHub issue or contact the authors using the email addresses provided in the paper.