[test]: Run a baseline for tasks

[Dinghye]

Run a baseline for the target tasks within the ESA Phi Lab embed2heights challenge.

Tasks

• Identify the required datasets and evaluation metrics for each task.
• Implement a simple baseline approach for each task(Based on current repo)
• Run the baseline and record the results.
• Summarize findings, challenges, and areas needing further work.

This issue tracks the baseline implementation and result reporting for all four tasks. Please document your process for reproducibility and discussion.

[Dinghye]

Emb2Heights Baseline Evaluation Report

Date: 2026-04-12 (metrics updated 2026-04-19)
Task: Predict sub-pixel land cover (Building, Vegetation, Water) and surface height (nDSM) from GFM embeddings.
Dataset: 2024 patches, 256x256 @ 10m resolution (France), 4-band labels [building%, vegetation%, water%, height_m].

---

1. Baseline Overview

Six baselines were evaluated, each using the same training pipeline but with embeddings from a different Geospatial Foundation Model (GFM) and sensor combination. The only differences are the input embedding and the model architecture matched to its tensor shape.

1.1 Embedding Characteristics

Baseline	GFM	Sensor	Channels	Spatial Size	Tensor Type
AlphaEarth	AlphaEarth	\	64	256 x 256	Pixel-aligned
Tessera	Tessera	\	128	256 x 256	Pixel-aligned
TerraMind S2	TerraMind	Sentinel-2	768	16 x 16	ViT tokens
TerraMind S1	TerraMind	Sentinel-1	768	16 x 16	ViT tokens
THOR S2	THOR	Sentinel-2	768	16 x 16	ViT tokens
THOR S1	THOR	Sentinel-1	768	16 x 16	ViT tokens

1.2 Model Architecture

The model is selected based on the embedding spatial structure:

• Pixel-aligned embeddings (AlphaEarth, Tessera): LightUNet — a lightweight encoder-decoder with skip connections. Input and output are at the same 256x256 resolution. Data is loaded via PixelEmbeddingDataset.

• ViT token embeddings (TerraMind, THOR): EfficientDecoder256 — a bottleneck + 4-stage transposed-convolution decoder with residual blocks that upsamples 16x16 tokens to 256x256 output. Data is loaded via LatentTokenDataset (scale_factor=16).

1.3 Training Configuration (shared across all baselines)

Parameter	Value
Optimizer	AdamW (lr=2e-4, weight_decay=1e-4)
LR Scheduler	ReduceLROnPlateau (factor=0.5, patience=2)
Gradient Clipping	max_norm=1.0
Loss Function	ImprovedCompositeLoss (4-term weighted sum)
Epochs	30
Batch Size	32
Patch Size	256
Train/Val Split	80% / 20% (random, seed=42)

1.4 Loss Function

ImprovedCompositeLoss combines four terms with weights [1.0, 0.5, 0.5, 2.0]:

1. MAE (weight 1.0): Pixel-level regression with foreground/background split.
2. SSIM Loss (weight 0.5): Enforces structural similarity and sharp boundaries on land-cover channels.
3. Gradient Loss (weight 0.5): Penalizes blurred edges in predictions.
4. Tversky Loss (weight 2.0): Asymmetric segmentation loss (alpha=0.3, beta=0.7) that penalizes false negatives heavily, designed to capture sparse building footprints. Includes a structure-boosted height component where building-pixel height errors are penalized 2x more.

---

2. Evaluation Metrics

The leaderboard score is a weighted mean of five metrics. The formulas below were reverse-engineered via an all-zero submission probe (see ../../emb2heights-backbone/logs/METRIC_PROBE_REPORT.md, 2026-04-17) because the challenge does not document them.

Metric	Weight	Measures
IoU_buildings	25%	Positive-class Jaccard for buildings
IoU_trees	15%	Positive-class Jaccard for vegetation
IoU_water	15%	Positive-class Jaccard for water
RMSE_building_height	25%	Height error (m) on GT-building pixels
RMSE_vegetation_height	20%	Height error (m) on GT-vegetation pixels

IoU — per-image positive-class Jaccard with the "both empty → 1.0, one empty → 0.0" convention (equivalent to sklearn.metrics.jaccard_score(zero_division=1.0)). Prediction binarized at threshold 0.5; label binarized at threshold 0 (any non-zero fraction counts as positive).

RMSE — per-image macro average: for each image that has any GT-positive pixel of the class, compute RMSE over those pixels, then average across images.

Composite score (with our assumed normalization constants):

score = 0.25 · IoU_build
      + 0.15 · IoU_veg
      + 0.15 · IoU_water
      + 0.25 · max(0, 1 − RMSE_h_build / 3.0)
      + 0.20 · max(0, 1 − RMSE_h_veg  / 5.0)

The server's true X_bld and X_veg normalization constants are not disclosed; the probe only bounds them by X_bld < 4 m and X_veg < 10.9 m. We use X_bld = 3.0, X_veg = 5.0 as working assumptions within those bounds. Absolute scores scale with these constants; the ranking between baselines is largely stable.

---

3. Results

3.1 Validation Set (20%, 405 samples, seed=42)

Evaluated on the held-out validation split — the same split used during training, reproduced deterministically via each run's saved split.json (where available) or splits/val.json.

#	Baseline	IoU_bld	IoU_tree	IoU_wat	RMSE_bH (m)	RMSE_vH (m)	Score
1	AlphaEarth	0.2238	0.7197	0.3822	2.17	3.83	0.3374
2	TerraMind S2	0.0744	0.5587	0.1891	2.90	6.38	0.1392
3	TerraMind S1	0.0768	0.5384	0.0706	2.92	6.93	0.1174
4	THOR S2	0.0346	0.5104	0.0967	2.92	6.99	0.1065
5	THOR S1	0.0505	0.5298	0.0934	3.05	7.04	0.1061
6	Tessera	0.0123	0.4622	0.1062	3.13	10.16	0.0883

Recomputed via ../tools/recompute_backbone_metrics.py. JSON: ../runs/backbone_metrics_summary.json.

Note on clamping: with X_bld = 3.0, THOR S1 (RMSE_bH = 3.05) and Tessera (RMSE_bH = 3.13) both exceed the normalization bound, so their building-height contribution is clamped to 0. This is plausible given the tight bound — the server likely treats these predictions as "effectively useless" for that metric.

---

4. Analysis

4.1 AlphaEarth leads by a wide margin

AlphaEarth is ~2.4x the score of the next best baseline and wins on every metric. Its pixel-aligned 64-channel embedding at native 256x256 resolution preserves fine spatial detail that the LightUNet decoder exploits directly via skip connections — no upsampling means no information loss. Buildings in particular benefit: IoU_bld = 0.22 vs ≤ 0.08 for every other baseline.

4.2 ViT token embeddings pay a heavy spatial-detail cost

TerraMind and THOR compress the scene into 768-channel tokens on a 16x16 grid (16x spatial downsampling). Reconstructing 256x256 predictions from this coarse representation fundamentally caps boundary sharpness and small-object detection. The effect is most visible on sparse classes: building IoU collapses to 0.03–0.08, and water IoU to 0.07–0.19.

4.3 S1 vs S2 within the same GFM

• TerraMind: S2 (0.1392) > S1 (0.1174). S2's optical bands give better IoU on every land-cover class; S1's advantage on building height RMSE (2.92 vs 2.90) is negligible.
• THOR: S2 (0.1065) ≈ S1 (0.1061). S1 has better building and vegetation IoU; S2 has slightly better water IoU and vegetation RMSE. They're effectively tied.

The story is more nuanced than "SAR helps height, optical helps land cover" — once the IoU scores collapse to the low-positive-class regime, sensor differences get noisy. The complementarity is real but small at this level of absolute performance.

4.4 Tessera is the worst baseline

Tessera's 128-channel pixel-aligned embedding is beaten by AlphaEarth's 64-channel embedding on every metric by large margins (IoU_tree 0.46 vs 0.72, RMSE_vH 10.2 m vs 3.8 m). The pre-training objective evidently does not encode the information needed for this task.

4.5 Height prediction remains the bottleneck for non-AlphaEarth baselines

AlphaEarth's RMSE (2.17 m building, 3.83 m vegetation) is the only case that stays inside the X = 3/5 normalization window and thus contributes meaningfully to the composite score. Every other baseline's vegetation-height term is heavily penalized, and THOR S1 / Tessera are fully clamped to 0 on the building-height term.

4.6 Vegetation height is harder than building height — universally

Across all six baselines, RMSE_vegetation_height > RMSE_building_height. Building heights are approximately piecewise-constant within a footprint, while vegetation heights are diffuse and vary continuously — a strictly harder regression target.

---

5. Possible Improvements

1. Pin down the true X_bld / X_veg — the current 3/5 assumption clamps two baselines to zero on building height. Running a second probe (e.g. --height-value 5 as suggested in ../../emb2heights-backbone/logs/METRIC_PROBE_REPORT.md) would reveal the real constants and make absolute score comparisons meaningful.
2. Ensemble multiple embeddings — fuse AlphaEarth + TerraMind S2 (or all six) as multi-source input. S1/S2 fusion is still worth trying; even small complementarity helps on sparse classes.
3. Deeper / token-aware decoders — for ViT-token embeddings, the 16→256 upsample is the bottleneck. Attention-based decoders, cross-scale fusion, or learned token→pixel mappings could recover some of the IoU loss.
4. Height-specific loss tuning — give RMSE-relevant terms more weight, or split the decoder into separate land-cover and height heads.
5. Data augmentation — spatial flips and 90° rotations are the only valid augmentations on pre-computed embeddings (they must be applied identically to embedding and label). Spectral / color augmentations are not applicable — embedding channels are abstract latent features, not spectral bands.
6. Post-processing — CRF or morphological operations to sharpen building boundaries, especially for ViT-token baselines where the decoder struggles with small objects.

[Dinghye]

Visualization on the test set.

Region 1

Region 2

Region 3

Region 4