Point-Supervised Semantic Segmentation for Remote Sensing

Point-Supervised Semantic Segmentation for Remote Sensing Imagery via Partial Focal Cross-Entropy Loss Haris bin Shakeel · Deep Learning Lab (DLL), TUKL — NUST SEECS · 2026 Paper submitted to arXiv (cs.CV / eess.IV) — link will be added on acceptance of the listing.

Overview

Dense pixel masks are the most expensive artifact in remote-sensing ML. This repo trains urban land-cover segmentation from as few as 5 labeled pixels per class per image using a Partial Focal Cross-Entropy (PFCE) loss that (1) restricts gradients to labeled points only and (2) applies focal weighting (1 − p_t)^γ to concentrate learning on hard, minority classes.

On LoveDA (NeurIPS 2021) with DeepLabV3+ (ResNet-50), the 5-point focal configuration reaches 44.35% mIoU — 97.5% of the fully supervised baseline (45.48%) from a ~3–7× cheaper annotation budget.

Key results

Configuration	Points/class	γ	mIoU (%)	vs. full supervision
Full supervision (baseline)	all	—	45.48	—
Point-supervised (k=1)	1	2.0	36.93	−8.55 pp
Point-supervised (k=5, no focal)	5	0.0	41.30	−4.18 pp
Point-supervised (k=5, focal)	5	2.0	44.35	−1.13 pp
Point-supervised (k=10)	10	2.0	43.70	−1.78 pp

SR-preprocessing ablations (downstream segmentation)

Controlled ablation of super-resolution preprocessing before segmentation, comparing L1-based reconstruction against GAN-based and interpolation baselines:

SR preprocessor	Type	Downstream mIoU
RRDBNet (L1)	L1 reconstruction	0.3804 (best)
Real-ESRGAN	GAN-based	see notebook
SatESRGAN	GAN-based (satellite)	see notebook
Bicubic	interpolation	see notebook

Finding: L1-based SR preserves radiometric fidelity and transfers better to downstream segmentation than GAN-based SR, whose perceptual losses distort the very statistics the segmenter depends on. Full per-method numbers are in point_supervised_segmentation.ipynb and the submitted paper.

Literature gap this closes

No prior work combines point supervision with urban–rural domain adaptation on LoveDA. This project identifies the gap, and the submitted paper is the first to address it.

Qualitative results

Left: input RGB · Center: dense ground truth · Right: prediction from 5 labeled points per class.

⬜ Background · 🟥 Building · 🟨 Road · 🟦 Water · 🟫 Barren · 🟩 Forest · 🟧 Agriculture

Method — Partial Focal Cross-Entropy

class PartialFocalCELoss(nn.Module):
    """CE loss only at labeled (point-annotated) pixels, with focal weighting
    on hard, minority-class examples. gamma=0 → standard masked CE."""
    def __init__(self, gamma=2.0, ignore_index=255):
        super().__init__()
        self.gamma = gamma
        self.ce = nn.CrossEntropyLoss(reduction='none', ignore_index=ignore_index)

    def forward(self, logits, target, mask_labeled):
        ce_loss = self.ce(logits, target)          # per-pixel CE
        pt = torch.exp(-ce_loss)                   # estimated softmax prob
        focal = ((1 - pt) ** self.gamma) * ce_loss
        return (focal * mask_labeled).sum() / (mask_labeled.sum() + 1e-7)

$$\mathcal{L}_\text{PFCE} = \frac{\sum_{p,q} s[p,q] \cdot (1 - p_t[p,q])^\gamma \cdot L[p,q]}{\sum_{p,q} s[p,q] + \varepsilon}$$

On-the-fly point sampling — at each step, k pixels/class are sampled uniformly from the dense mask, so every epoch sees a different label subset (annotation augmentation):

def simulate_point_labels(mask, num_points_per_class=5):
    h, w = mask.shape
    mask_labeled = np.zeros((h, w), dtype=np.float32)
    gt_labels = np.full((h, w), 255, dtype=np.int64)   # 255 = ignore
    for c in np.unique(mask):
        if c == 255: continue
        coords = np.argwhere(mask == c)
        n = min(len(coords), num_points_per_class)
        for r, col in coords[np.random.choice(len(coords), n, replace=False)]:
            mask_labeled[r, col] = 1.0
            gt_labels[r, col] = c
    return mask_labeled, gt_labels

Architecture & dataset

• Backbone: ResNet-50 (ImageNet) · Decoder: DeepLabV3+ with ASPP (dilations 6/12/18) · 7 classes at full resolution · segmentation-models-pytorch
• LoveDA: 2,522 train / 1,669 val images, 1024×1024 @ 0.3 m/px, urban + rural domains

Setup & data

git clone https://github.com/Haris-bin-shakeel/point-supervised-rs-segmentation.git
cd point-supervised-rs-segmentation
python -m venv venv && source venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install segmentation-models-pytorch albumentations opencv-python scikit-learn tqdm matplotlib

wget https://zenodo.org/records/5706578/files/Train.zip
wget https://zenodo.org/records/5706578/files/Val.zip
unzip Train.zip -d data/ && unzip Val.zip -d data/

Usage

All training, ablation, and evaluation runs live in point_supervised_segmentation.ipynb (single-notebook research repo). Final metrics: results_final.json / results_checkpoint.json.

Tech stack

Python · PyTorch · segmentation-models-pytorch · LoveDA · Albumentations · scikit-learn

Citation

@article{shakeel2026pointsupervised,
  title  = {Point-Supervised Semantic Segmentation for Remote Sensing Imagery
            via Partial Focal Cross-Entropy Loss},
  author = {Shakeel, Haris bin},
  year   = {2026},
  note   = {arXiv preprint (submitted, cs.CV/eess.IV)}
}

License

MIT