Motivation
We currently train separate single-model baselines for each GFM embedding (AlphaEarth, Tessera, TerraMind, THOR). The next step is to fuse embeddings from multiple models to exploit complementary information. However, naively concatenating features of very different nature (pixel-aligned vs. ViT tokens, 64–128 channels vs. 768 channels, 256×256 vs. 16×16 spatial resolution) is unlikely to be optimal and may even hurt performance due to modality laziness.
This issue proposes a set of controlled experiments to identify effective fusion strategies.
This is a huge evaluation project. We can create a few sub-issues to record these. You can pick up a few that you are interested in. And you don't have to follow every solution I write here if you have a better idea.
Hypotheses
H1: Compress high-resolution embeddings before fusion
AlphaEarth (64ch, 256×256) and Tessera (128ch, 256×256) carry spatially dense, channel-compact information, while THOR and TerraMind tokens (768ch, 16×16) are spatially sparse but channel-rich. When fusing them:
- The pixel-aligned embeddings should first pass through dedicated compression layers (e.g., a small CNN or strided convolutions) to reduce spatial resolution and increase channel depth, producing representations that are structurally compatible with the ViT tokens.
- Only after this independent compression step should the features be concatenated or cross-attended with THOR/TerraMind embeddings.
- Rationale: without compression, the pixel-aligned features dominate spatially but are shallow in semantic abstraction, leading to an imbalanced fusion where the ViT branch is effectively ignored.
H2: Task-specific modality selection matters
Different downstream tasks benefit from different input modalities:
- Water segmentation: SAR (S1) is particularly valuable — water appears as specular dark regions in SAR regardless of cloud cover or optical conditions. Fusing THOR-S1 and TerraMind-S1 as the primary signal for the water head, supplemented by optical embeddings, should significantly boost water mIoU.
- Building segmentation & height: Optical (S2) embeddings capture fine structural edges and shadows critical for building detection. AlphaEarth's high spatial fidelity should anchor the building/height heads.
- Vegetation: Multi-spectral indices embedded in S2 models (NDVI, Red Edge) make optical embeddings dominant, but SAR-derived canopy structure from S1 can help with height estimation.
This suggests a task-aware fusion architecture where modality routing or attention weights are conditioned on the prediction head, rather than a single shared fusion trunk.
H3: Feature dimension alignment to prevent modality laziness
Prior experiments have shown that when fused features have large dimension disparities, the model tends to become lazy toward the lower-dimensional modality — effectively learning to ignore it. For example:
- Concatenating a 12-dim feature with a 128-dim feature causes the optimizer to rely disproportionately on the 128-dim branch.
- This was empirically confirmed in our earlier trials.
To mitigate this:
- Project all modality features to a shared dimensionality (e.g., 256-dim) via learned linear layers or 1×1 convolutions before fusion.
- Alternatively, use gating mechanisms or modality dropout during training to force the model to leverage all branches.
Proposed Experiments
| # |
Experiment |
What it tests |
| 1 |
Naive concatenation (all embeddings → single decoder) |
Baseline for fusion |
| 2 |
Compress AlphaEarth/Tessera to 16×16 then concat with THOR/TerraMind |
H1 — compression before fusion |
| 3 |
Shared-dim projection (all → 256-dim) then concat |
H3 — dimension alignment |
| 4 |
Task-conditioned fusion: separate attention weights per head |
H2 — task-specific routing |
| 5 |
Modality dropout (randomly zero out one modality during training) |
H3 — robustness to laziness |
| 6 |
S1-anchored water head vs. S2-anchored building/height head |
H2 — SAR for water |
| 7 |
Best combination of the above |
Combined |
Evaluation Plan
- Compare against the current best single-model baseline (AlphaEarth, score 0.741).
- Report all 5 competition metrics (mIoU for buildings/vegetation/water, RMSE for building/vegetation height) plus composite score.
Notes
- Tessera currently underperforms AlphaEarth on vegetation despite having 2× channels — worth investigating whether this is a pre-training limitation or a decoder capacity issue before committing to Tessera-heavy fusion.
Motivation
We currently train separate single-model baselines for each GFM embedding (AlphaEarth, Tessera, TerraMind, THOR). The next step is to fuse embeddings from multiple models to exploit complementary information. However, naively concatenating features of very different nature (pixel-aligned vs. ViT tokens, 64–128 channels vs. 768 channels, 256×256 vs. 16×16 spatial resolution) is unlikely to be optimal and may even hurt performance due to modality laziness.
This issue proposes a set of controlled experiments to identify effective fusion strategies.
Hypotheses
H1: Compress high-resolution embeddings before fusion
AlphaEarth (64ch, 256×256) and Tessera (128ch, 256×256) carry spatially dense, channel-compact information, while THOR and TerraMind tokens (768ch, 16×16) are spatially sparse but channel-rich. When fusing them:
H2: Task-specific modality selection matters
Different downstream tasks benefit from different input modalities:
This suggests a task-aware fusion architecture where modality routing or attention weights are conditioned on the prediction head, rather than a single shared fusion trunk.
H3: Feature dimension alignment to prevent modality laziness
Prior experiments have shown that when fused features have large dimension disparities, the model tends to become lazy toward the lower-dimensional modality — effectively learning to ignore it. For example:
To mitigate this:
Proposed Experiments
Evaluation Plan
Notes