HoloGeo

Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning

English | 中文

🌐 Project Page · 📄 Paper · 🗂️ Dataset · 🤗 Model · 💻 Code

---

🔍 Overview

Vision-language models have substantially advanced image geo-localization, but their reasoning can still be affected by landmark bias: a model may treat a salient building, statue, text sign, or cultural symbol as decisive evidence while overlooking complementary cues from the surrounding environment, architecture, vegetation, terrain, and infrastructure.

When an image contains a replicated landmark, visually similar architecture, or an ambiguous cultural cue, this reliance on a single signal can yield a plausible but incorrect and overconfident location prediction.

HoloGeo systematically studies this problem and offers a complete solution from diagnosis to mitigation:

• We use Bias Intensity (BI) and Bias Harmfulness (BH) to quantify how landmarks affect model decisions;
• We introduce LandmarkBias-3K, a benchmark for evaluating misleading-landmark reasoning;
• We construct BF-30K, a training dataset with structured multi-evidence reasoning chains;
• We propose HoloGeo, an evidence-driven framework that combines supervised fine-tuning with GRPO reinforcement learning.

Landmark bias examples: existing models can be misled by visually salient landmarks while overlooking environmental evidence that supports the true location.

---

✨ Key Contributions

1. Systematic Definition and Quantitative Diagnosis of Landmark Bias

We define landmark bias from the perspective of geographic reasoning and introduce two complementary metrics:

• Bias Intensity (BI): measures the increase in a model's preference for the landmark-induced location when the landmark is present;
• Bias Harmfulness (BH): measures whether a landmark shifts the model away from the ground-truth location toward an incorrect prediction.

BI describes the strength of landmark influence, while BH captures whether that influence genuinely harms localization. Together, they distinguish reasonable landmark use from harmful over-reliance.

2. LandmarkBias-3K and BF-30K

We introduce two data resources with distinct purposes:

Dataset	Scale	Purpose	Key Characteristics
LandmarkBias-3K	3K	Diagnosis and evaluation	Challenging images with salient but insufficient, ambiguous, or misleading landmarks
BF-30K	30K	Supervised fine-tuning and reinforcement learning	Region-level visual evidence and structured <Analyze>-<Think>-<Answer> reasoning chains

LandmarkBias-3K asks whether a model is affected by harmful landmark bias, whereas BF-30K trains models to conduct more comprehensive multi-evidence geographic reasoning.

3. Evidence-Driven HoloGeo Framework

Instead of mapping one salient landmark directly to a location, HoloGeo:

1. Identifies geographically informative regions in an image;
2. Analyzes visual evidence from different regions independently;
3. Aggregates cues from architecture, vegetation, terrain, climate, text, and cultural symbols;
4. Resolves ambiguities and conflicts among evidence;
5. Produces country- and city-level predictions.

The model first acquires structured reasoning ability through supervised fine-tuning, then undergoes GRPO optimization with multi-dimensional rewards to reduce reliance on single visual cues.

---

📏 Landmark Bias Evaluation

For an original image $x$, we extract a landmark region $r$ and construct three inputs:

• Original image: retains all visual information;
• Landmark-only: retains only the landmark region;
• Landmark-removed: removes the landmark while retaining the remaining scene context.

By comparing predictions across these three inputs, BI measures the anchoring strength of a landmark, while BH measures whether this anchoring weakens a model's preference for the ground-truth location.

LandmarkBias-3K construction: landmark extraction, BI/BH computation, difficult-sample selection, and category statistics.

LandmarkBias-3K covers six classes of landmarks that may induce landmark bias:

• Architecture
• Language Signs
• Cultural Cues
• Vegetation
• Landform
• Climate

---

🧭 Method Overview

HoloGeo consists of two primary stages.

Stage I: BF-30K Construction

BF-30K is built through the following process:

1. Data filtering
   We select images with meaningful geographic information from MP-16 and GLDv2, covering landmark-bias, ordinary-landmark, and non-landmark scenarios.

2. Region extraction and semantic filtering
   GroundingDINO extracts candidate visual regions, which Qwen2.5-VL-72B then filters for geographic discriminativeness.

3. Multi-model reasoning annotation
   Qwen2.5-VL-72B and ChatGPT-o3 generate structured multi-evidence reasoning chains:

   <Analyze>
   Describe visual facts and geographic cues from different regions.
   </Analyze>
   
   <Think>
   Aggregate evidence, establish geographic associations, resolve ambiguities, and progressively narrow the location.
   </Think>
   
   <Answer>
   Output the country and city.
   </Answer>
   

4. Multi-dimensional quality validation
   InternVL3-78B conducts cross-model visual-semantic consistency checks, together with manual spot checks that filter hallucinations, factual errors, and logical contradictions.

Stage II: Two-Stage HoloGeo Training

• SFT warm-up: LoRA supervised fine-tuning of Qwen2.5-VL-7B-Instruct on 1.5K BF-30K samples;
• GRPO post-training: multi-dimensional reward optimization on 28.5K samples.

BF-30K construction and the two-stage HoloGeo training pipeline.

---

🎯 Multi-Dimensional Rewards

During GRPO, HoloGeo uses the following rewards:

Reward	Objective
$R_{\mathrm{format}}$	Enforces the <Analyze>-<Think>-<Answer> output structure
$R_{\mathrm{geo}}$	Optimizes country- and city-level geo-localization accuracy
$R_{\mathrm{box}}$	Encourages coverage of multiple geographically informative regions rather than a single salient landmark
$R_{\mathrm{CLR}}$	Evaluates visual factuality, reasoning consistency, and answer clarity

The total reward is defined as:

$$ R = R_{\mathrm{format}} +\lambda_1 R_{\mathrm{geo}} +\lambda_2 R_{\mathrm{box}} +\lambda_3 R_{\mathrm{CLR}}. $$

Here, $R_{\mathrm{box}}$ explicitly encourages the use of multiple regional visual cues, while $R_{\mathrm{CLR}}$ constrains the reasoning chain to remain consistent with the image content and final answer.

---

📊 Results

We evaluate geo-localization by the geographic distance between predicted and ground-truth coordinates:

• City: within 25 km;
• Region: within 200 km;
• Country: within 750 km.

Standard Geo-localization Benchmarks

Benchmark	City ↑	Region ↑	Country ↑
IM2GPS	47.3	60.3	76.8
IM2GPS3K	38.5	53.7	70.8
YFCC4K	18.9	31.7	51.5

HoloGeo achieves strong city- and region-level performance on IM2GPS and the best results across all three granularities in the table on YFCC4K, which contains more complex real-world scenes.

LandmarkBias-3K

Model	City ↑	Region ↑	Country ↑
Qwen2.5-VL-7B	16.83	28.67	44.57
GLOBE	23.27	44.27	63.90
GeoAgent	23.57	45.27	67.07
HoloGeo-SFT	20.03	30.33	64.05
HoloGeo (SFT + GRPO)	27.27	47.07	68.20
Gain over the base model	+10.44	+18.40	+23.63

On LandmarkBias-3K, HoloGeo substantially improves over the Qwen2.5-VL-7B base model and surpasses the compared domain-specific methods. These results indicate that multi-evidence reasoning and reinforcement-learning rewards improve robustness in misleading-landmark scenarios.

---

🔬 Analysis

More Balanced Visual Attention

Gradient-based saliency analysis shows that the base model and existing geo-localization models often concentrate on a small number of salient landmarks while neglecting surrounding architecture, environmental context, and scene details.

In contrast, HoloGeo distributes attention across both landmark and non-landmark regions, allowing it to jointly use diverse visual evidence.

Comparison of saliency distributions across geo-localization models.

Case Study

Existing models may commit too early to an incorrect location based on superficial similarity in a single building or statue. HoloGeo instead combines evidence from architectural style, public space, vegetation layout, and cultural context to reduce landmark-induced erroneous anchoring.

Reasoning comparison between HoloGeo and baseline models on a representative case.

---

📁 Repository Structure

HoloGeo/
├── Data_Examples/
│   ├── Fig_BF30K/                       # BF-30K image examples
│   ├── Fig_LandmarkBias-3K/             # LandmarkBias-3K image examples
│   ├── Examples_BF30K.jsonl             # BF-30K example annotations
│   └── Examples_LandmarkBias-3K.jsonl   # LandmarkBias-3K example annotations
├── Train_code/
│   └── examples/
│       ├── train/grpo/hologeo/           # SFT and GRPO training scripts
│       └── infer/                        # Inference scripts and examples
├── Eval_Code/
│   ├── evaluate_hologeo.py               # Evaluation pipeline for merged HoloGeo checkpoints
│   └── run_eval_hologeo.sh               # Example evaluation launch script
├── BIBH/
│   ├── infer.py                          # BI/BH computation script
│   └── run_compute_bibh.sh               # Example BI/BH launch script
├── assets/                               # README image assets
├── requirements.txt
├── README.md
└── README_zh.md

Data_Examples/ currently provides a small number of demonstration samples. Eval_Code/ provides the merged-checkpoint evaluation pipeline, while BIBH/ provides scripts for computing BI and BH. The full BF-30K, LandmarkBias-3K, model weights, and processed metadata are available through the links above.

---

⚙️ Setup

conda create -n hologeo python=3.11 -y
conda activate hologeo

pip install -r requirements.txt

Key dependencies include:

• PyTorch
• Transformers
• DeepSpeed
• vLLM
• FlashAttention
• ModelScope Swift

Before training, install PyTorch, FlashAttention, and vLLM versions compatible with your local CUDA and GPU environment.

---

🚀 Training

Training scripts are located in:

Train_code/examples/train/grpo/hologeo/

Before running a script, configure the following variables:

base_model=/path/to/Qwen2.5-VL-7B-Instruct
dataset=/path/to/BF-30K
output_dir=/path/to/output

Stage I: LoRA Supervised Fine-Tuning

bash Train_code/examples/train/grpo/hologeo/train_sft.sh

Stage II: GRPO Reinforcement Learning

bash Train_code/examples/train/grpo/hologeo/train_all_rewards.sh

The complete training setting jointly uses $R_{\mathrm{format}}$, $R_{\mathrm{geo}}$, $R_{\mathrm{box}}$, and $R_{\mathrm{CLR}}$.

---

🔮 Inference and Evaluation

For batched inference and geo-localization evaluation with a merged HoloGeo checkpoint, use the HoloGeo entry point in Eval_Code/:

cd Eval_Code
# Edit MODEL_DIR, DATASET_FILE, and OUTPUT_FILE in run_eval_hologeo.sh first.
bash run_eval_hologeo.sh

run_eval_hologeo.sh calls evaluate_hologeo.py and supports configurable prompt types, devices, generation settings, and geographic distance thresholds.

Model outputs follow this structure:

<Analyze>
Analyze architecture, text, vegetation, terrain, and cultural cues from multiple regions.
</Analyze>

<Think>
Aggregate evidence, resolve conflicts, and progressively narrow the country and city.
</Think>

<Answer>
{"country": "China", "city": "Suzhou"}
</Answer>

---

📦 Data Notes

BF-30K Example Format

Each sample contains:

• An image URL or image reference;
• Country, city, and coordinate information;
• Bounding boxes for geographically informative regions;
• Region-level visual analyses;
• Multi-evidence aggregated reasoning;
• Final country and city answers.

LandmarkBias-3K Example Format

Each sample contains:

• The original image;
• A landmark region;
• Landmark-only and landmark-removed inputs;
• Ground-truth location;
• Landmark-induced location;
• BI and BH metrics;
• Landmark category.

---

⚠️ Data Use, Limitations, and Responsible Use

• This repository references source images through URLs or metadata and does not redistribute upstream image binaries. Image use remains subject to the licenses, copyrights, and terms of service of the original sources.
• BF-30K reasoning chains are generated with assistance from vision-language models and undergo cross-model validation and manual spot checks, but may still contain omissions, biases, hallucinations, or reasoning errors.
• BI and BH depend on model probabilities and landmark-region extraction; they should not be interpreted as absolute, model-independent measures of bias.
• The data primarily originates from MP-16 and GLDv2. Remote regions, rare landforms, non-English contexts, and data-sparse areas may be underrepresented.
• Image geo-localization can raise privacy, surveillance, and spatial-security risks. This project must not be used for unlawful tracking, identity localization, privacy infringement, or other harmful purposes.
• Reasoning annotations are intended for model research and training and should not be treated as independently verified sources of geographic fact.

---

📝 Citation

If you find HoloGeo useful, please cite it as follows. Accepted at ACM Multimedia 2026. Citation details will be updated after publication.

@inproceedings{zhou2026hologeo,
  title     = {HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning},
  author    = {Zhou, Pengcheng and Liu, Xuanyu and Yin, Yanchen and Li, Bobo and Wu, Shengqiong and Lee, Mong-Li and Hsu, Wynne},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026}
}

---

⚖️ License

The code in this repository is licensed under the Apache License 2.0. Data annotations and model weights are governed by the terms specified in their respective release repositories.

Upstream images remain subject to their original copyrights and license terms. Unless explicitly authorized for a resource, this repository's license does not automatically cover third-party image content.

---

🙏 Acknowledgements

This project uses or builds upon the MP-16 and Google Landmarks Dataset v2 data resources, as well as GroundingDINO, Qwen2.5-VL, InternVL3, GeoCLIP, DeepSpeed, vLLM, and ModelScope Swift. We thank the corresponding researchers and open-source communities for their contributions.