Upload Validation: Prevent DSM, Corrupt, and Invalid Image File Uploads

[JesJehle]

Problem

Users can upload unsupported file types that pass initial validation but fail during processing. Recent analysis shows 3 additional failure modes beyond ODM issues:

1. COG Generation Failure (10% of failures) - DSM files uploaded as RGB orthomosaics
2. Ortho Conversion Failure (10% of failures) - Corrupt or invalid image files
3. Treecover Edge Cases (10% of failures) - Images too small or with invalid data

These failures waste processing time and provide unclear error messages to users.

Current Failure Modes

1. DSM Files Uploaded as RGB Orthomosaics (Dataset 6198)

File: Sime Darby XME1 DSM.tif

Problem:

• User uploaded Digital Surface Model (DSM - elevation data)
• System expects RGB orthomosaic (3-band color image)
• Ortho processing succeeds (file is valid GeoTIFF)
• COG generation fails when trying to extract RGB bands

Error:

Command 'gdal_translate ... -b 1 -b 2 -b 3 ...' returned non-zero exit status 1

Root Cause: File only has 1 band (elevation), not 3 bands (RGB)

Detection: Check band count during upload

with rasterio.open(file) as src:
    if src.count == 1:
        raise ValueError("File has only 1 band. Upload RGB orthomosaics (3 bands), not DSM/DTM files.")

---

2. Corrupt or Invalid Image Files (Dataset 6182)

File: WhatsApp Image 2025-10-15 at 08.47.18.tif

Problem:

• File uploaded via WhatsApp (likely compressed/corrupted)
• Passes upload but fails during standardization
• Error message: "convert processing failed: Conversion failed"

Root Cause: File corruption or unsupported compression format

Detection: Validate file integrity during upload

from PIL import Image
try:
    with Image.open(file) as img:
        img.verify()  # Check file integrity
        img.load()    # Attempt to read pixel data
except Exception as e:
    raise ValueError(f"File appears to be corrupt or invalid: {str(e)}")

---

3. Treecover Segmentation Edge Cases (Datasets 3182, 6036)

Empty Batch Error (Dataset 3182)

Problem:

• Image too small: 1024×1024 px at 0.1m resolution
• TCD pipeline skips all batches
• Error: "Empty batch, skipping" → No output tiles

Root Cause: Image below minimum size for treecover detection

Detection: Check image dimensions before treecover processing

MIN_SIZE_FOR_TREECOVER = 2048  # pixels
if width < MIN_SIZE_FOR_TREECOVER or height < MIN_SIZE_FOR_TREECOVER:
    raise ValueError(f"Image too small for treecover segmentation: {width}×{height}. Minimum: {MIN_SIZE_FOR_TREECOVER}×{MIN_SIZE_FOR_TREECOVER}")

Chunk and Warp Failed (Dataset 6036)

Problem:

• Large untiled GeoTIFF causes GDAL reprojection failure
• Error: "Chunk and warp failed"

Status: Should be fixed by local file caching mechanism (processor-pipeline rules)

Verification Needed: Check if fix is deployed in production

---

Proposed Solution

Upload-Time Validation

File: api/src/upload/geotiff_processor.py

Add comprehensive GeoTIFF validation during upload:

async def validate_geotiff_for_processing(file_path: Path) -> dict[str, str | None]:
    """
    Validate uploaded GeoTIFF is suitable for processing pipeline.
    Returns: {"error": None} if valid, {"error": "message"} if invalid
    """
    try:
        with rasterio.open(file_path) as src:
            # Check band count (need RGB)
            if src.count == 1:
                return {
                    "error": "File has only 1 band (likely DSM/DTM). "
                            "Please upload RGB orthomosaics with 3 or 4 bands. "
                            "DSM files are not supported."
                }
            
            if src.count not in [3, 4]:
                return {
                    "error": f"File has {src.count} bands. "
                            "Supported: 3 bands (RGB) or 4 bands (RGBA)."
                }
            
            # Check dimensions (minimum for processing)
            if src.width < 512 or src.height < 512:
                return {
                    "error": f"Image too small: {src.width}×{src.height} pixels. "
                            "Minimum: 512×512 pixels."
                }
            
            # Check for CRS (required for processing)
            if src.crs is None:
                return {
                    "error": "File lacks coordinate reference system (CRS). "
                            "Please upload georeferenced GeoTIFFs only."
                }
            
            # Try reading sample pixels (detect corruption)
            try:
                _ = src.read(1, window=Window(0, 0, 100, 100))
            except Exception as e:
                return {
                    "error": f"Failed to read pixel data. File may be corrupt: {str(e)}"
                }
            
            return {"error": None}
            
    except Exception as e:
        return {"error": f"Invalid GeoTIFF file: {str(e)}"}


# In process_geotiff_upload():
async def process_geotiff_upload(...):
    # Validate file before creating dataset
    validation = await validate_geotiff_for_processing(temp_file_path)
    
    if validation["error"]:
        raise ValueError(validation["error"])
    
    # ... continue with upload

Processing-Time Checks

Files:

• processor/src/process_cog.py (check bands before COG generation)
• processor/src/process_treecover_segmentation.py (check dimensions before segmentation)

Add defensive checks before expensive operations:

# Before COG generation
if band_count == 1:
    raise ValueError("Cannot create RGB COG from single-band file. Skipping COG generation.")

# Before treecover segmentation  
if width < 2048 or height < 2048:
    logger.warning(f"Image too small for treecover: {width}×{height}. Skipping segmentation.")
    # Mark as completed without error (not a failure)

---

Impact

Current Failure Statistics (Past 30 Days)

• ODM Exit Code 1: 6 datasets (60%) - Addressed by ODM: Improve error message when files lack GPS EXIF data #223, Frontend: Validate GPS EXIF data before accepting raw drone image uploads #224
• ODM Exit Code 134: 2 datasets (20%) - Addressed by ODM Processing Failure: Exit Code 134 During Image Undistortion (Dataset 6266) #226
• COG GDAL Error: 1 dataset (10%) - This issue
• Ortho Conversion: 1 dataset (10%) - This issue
• Treecover Edge Cases: 2 datasets (historic) - This issue

Expected Benefits

• Fail fast: Reject invalid files at upload (not after 20 minutes processing)
• Clear errors: Explain what's wrong and how to fix it
• Save resources: Avoid wasted processing time on invalid files
• Better UX: Users understand requirements before uploading

---

Acceptance Criteria

Upload Validation

• Check GeoTIFF band count (must be 3 or 4)
• Check minimum dimensions (512×512 px)
• Verify CRS exists
• Test pixel data readability (detect corruption)
• Clear error messages for each failure type

Processing Validation

• COG generation checks bands before processing
• Treecover checks dimensions before segmentation
• Graceful degradation (skip optional tasks if requirements not met)

Testing

• Test upload of DSM file (should reject)
• Test upload of corrupt file (should reject)
• Test upload of 1-band file (should reject)
• Test upload of small image (should warn or reject)
• Test upload of valid RGB GeoTIFF (should accept)

---

Priority

Medium - Affects 20-30% of failures, but less frequent than ODM issues

Recommend implementing after #223 and #224 (which address 60% of failures).

---

Related Issues

• ODM: Improve error message when files lack GPS EXIF data #223 - ODM error messages (missing GPS EXIF)
• Frontend: Validate GPS EXIF data before accepting raw drone image uploads #224 - Frontend validation (GPS EXIF check)
• ODM Processing Failure: Exit Code 134 During Image Undistortion (Dataset 6266) #226 - ODM crash (exit code 134)

This issue complements those by covering non-ODM failure modes.

[linear]

DT-144