Debris Estimation

Project Summary

A modular machine learning pipeline for estimating post-disaster debris volume using staged classification and regression models on geospatial and structural data.

Provenance

This repository is now maintained as an independent private repository. Keep the original source repository link here permanently for provenance:

Original repository: https://github.com/jskelly2021/debris-estimate

The current working repository is: https://github.com/rayford295/debris-estimate

Current Research Progress

This repository has moved from a notebook-style debris-estimation prototype to a reproducible research pipeline for testing how high-resolution remote-sensing representations can improve post-disaster debris-volume estimation. The current study focuses on AlphaEarth 10 m embeddings, H3 polygon aggregation, spatially grouped validation, and competitive model-family baselines.

Status Snapshot

Area	Current status
Repository identity	Independent private repository, with the original upstream link preserved above for provenance.
Core pipeline	Packaged Python workflow under src/, with reusable presets, staged-model training, sweep utilities, and checked output writers.
Data integration	H8 and H9 debris tables are paired with AlphaEarth 2023 10 m features sampled at centroids and aggregated over H3 polygons.
Feature design	Centroid sampling, polygon zonal mean/std, train-split-safe PCA, and regularized XGBoost variants have been tested.
Validation design	Random repeated splits, H7 parent-group holdout, and stricter H6 parent-group holdout are available for AlphaEarth experiments.
Model families	Staged cascade, single XGBoost, Tweedie XGBoost, hurdle XGBoost, and naive group-mean baselines are compared under the same spatial protocol.
Paper artifacts	Manuscript draft, paper-ready figures, checked Markdown result tables, and external-review response notes are stored in the repository.
Imagery change analysis	Pre/mid/post-storm Sentinel-2 mosaics (9/19, 10/4, 10/14) plus Maxar/NOAA high-res chips are downloaded, and per-hexagon dNDVI change features correlate with debris volumes (positives-only Spearman ≈ −0.25); see docs/imagery_change_analysis.md.

Key Findings So Far

Centroid AlphaEarth features are technically feasible but are not a reliable drop-in improvement. H8 centroid sampling worsened all three targets, while H9 centroid sampling produced only target-specific gains. This motivated the shift from one-pixel centroid extraction to H3 polygon-level zonal summaries.

Polygon-level AlphaEarth features show the strongest signal at H9. Under H6 grouped spatial validation, raw zonal mean/std improves C&D debris RMSE by 5.22 percent and MAE by 7.74 percent, and improves vegetative debris RMSE by 5.73 percent and MAE by 8.52 percent. The combined H9 target improves more modestly with a lower-dimensional PCA variant.

H8 is useful but more conditional. Raw zonal mean/std improves combined and vegetative debris under H6 grouped validation, but H8 C&D does not retain a positive AlphaEarth gain under the stricter H6 holdout.

The staged cascade is no longer framed as the universal best model. Across the checked model-family comparison, H8 often favors a hurdle XGBoost model and H9 often favors Tweedie XGBoost. The staged cascade remains useful as a structured diagnostic baseline, but the final paper should treat model family as an empirical choice.

The current study is event-informed rather than strict pre-landfall forecasting. Predictors include static exposure, AlphaEarth 2023 surface context, and event-observed hazard fields; debris labels are post-event grid-level outcomes. This timing distinction matters for the manuscript claims and for the next forecast-mode experiment.

Research Artifacts

Artifact	Location
AlphaEarth manuscript draft	manuscript/alphaearth_debris_paper/alphaearth_debris_manuscript.docx
AlphaEarth full experiment summary	docs/alphaearth_full_results.md
Model-family baseline summary	docs/model_family_baseline_results.md
Paper figure inventory and captions	docs/alphaearth_paper_figures.md
External research review	docs/research_review_2026-07-14.md
Review response and implementation plan	docs/research_review_response_2026-07-14.md
Per-seed significance and narrowed claims	docs/alphaearth_statistical_robustness.md
Feature importance and AlphaEarth replaceability	docs/alphaearth_feature_importance.md
Best AlphaEarth validation table	docs/tables/alphaearth_best_validation_summary.md
Best model-family table	docs/tables/model_family_best_summary.md
Machine-readable model-family results	data/alphaearth/model_family/model_family_best_summary.csv
Pre/post-storm imagery change analysis	docs/imagery_change_analysis.md
Per-hexagon Sentinel-2 change features	data/imagery/h9_s2_change_features.csv
Post-storm street-view (Mapillary) survey	data/imagery/mapillary_poststorm_survey.csv

Next Research Steps

1. Repeat AlphaEarth representation sweeps inside the strongest model family for each resolution and target, rather than only inside the staged cascade.
2. Build a storm-long table for Helene and Milton so the study can test event-to-event transfer instead of only grid-level pooled prediction.
3. Add uncertainty estimates through quantile models or split-conformal calibration, then report empirical coverage and interval width.
4. Define a forecast-mode feature subset that separates pre-event/static predictors from event-observed predictors and post-event labels.
5. Convert the strongest result tables and figures into the manuscript with calibrated claims about AlphaEarth, spatial validation, and model-family dependence.

Setup

Create a Python virtual environment.

python3 -m venv .venv
source .venv/bin/activate

Install dependencies

pip install -e .

Install optional dependencies for the AlphaEarth 10 m embedding probe:

pip install -e ".[alphaearth]"

Project Structure

The reusable pipeline lives under src/; executable experiment workflows live under scripts/.

src/
  config_presets/        # baseline plus dataset/target-specific config factories
  debris_estimate/
    data/                 # loading, preprocessing, splitting, and clipping
    evaluation/           # metrics and diagnostic plots
    model/                # staged model training and prediction
    sweep/                # sweep summaries, leaderboards, and plots
    config.py             # pipeline configuration dataclasses
    outputs.py            # run and experiment artifact writers
    run.py                # reusable single-run workflow
scripts/                  # sweep, smoke-test, and analysis entry points
data/                     # input CSV datasets
docs/                     # project notes and supporting documentation

Data

The repository includes grid-level modeling data under data/. Raw Excel workbooks are preserved under data/raw/, and CSV copies are stored at the top level because the packaged pipeline loads CSV files. See data/README.md for the dataset inventory, target definitions, quality checks, and raw-to-CSV mapping.

Presets and Scripts

Presets cover four datasets (gh8_v3, gh9_v3, h8_v3, and h9_v6) and three targets (both, cd, and vg). Each preset module exposes build_run_config(), which returns a fresh RunConfig with its dataset, target, clipping values, threshold, and model parameters.

from pathlib import Path

from config_presets.gh8_v3 import both
from debris_estimate.run import run_model

config = both.build_run_config()
run_model(config, Path("outputs/runs") / config.run_name)

config_presets also exports ALL, BOTH, CD, and VG collections for running groups of presets. RunConfig.prediction_mode defaults to end_to_end. Set it to oracle only for stage-level diagnostics.

Sweeps use dotted configuration paths and evaluate the Cartesian product of their values:

from config_presets.gh8_v3 import both
from debris_estimate.config import ExperimentConfig
from scripts.run_sweep import run_sweep

run_sweep(ExperimentConfig(
    experiment_name="gh8_v3_both_thresholds",
    output_dir="outputs/threshold_sweeps",
    base_run_config=both.build_run_config(),
    swept_fields={"model.threshold": [500, 850, 1000]},
))

Run scripts from the repository root after installing the package:

Script	Purpose
run_sweep.py	Example configurable threshold sweep and reusable sweep functions.
run_full_clip_sweep.py	Sweep feature and target clipping across all presets.
run_full_threshold_sweep.py	Sweep preset-specific tier thresholds across all presets.
run_full_threshold_clip_sweep.py	Sweep thresholds and clipping values together across all presets.
run_skew_analysis.py	Write numeric skew summaries and histograms for each dataset.
run_smoke_test.py	Single-run smoke workflow using the baseline preset.
probe_alphaearth_source.py	Probe AlphaEarth 10 m Source Cooperative COGs at H3 grid centroids.
extract_alphaearth_zonal.py	Extract H3 polygon-level AlphaEarth 10 m mean/std features from Source Cooperative COGs.
build_alphaearth_dataset.py	Join AlphaEarth feature tables to debris data and write matched baseline tables.
run_alphaearth_ablation.py	Run matched baseline vs AlphaEarth feature ablations, including optional train-split-safe PCA and XGBoost regularization for AlphaEarth variants.
run_alphaearth_robustness.py	Run repeated-split AlphaEarth robustness and PCA-sweep summaries without writing full per-run plots.
run_alphaearth_spatial_validation.py	Run H3 parent-group spatial validation summaries for AlphaEarth variants.
run_model_family_baselines.py	Compare staged, single XGBoost, Tweedie, hurdle, and naive group-mean baselines under the same spatial protocol.
make_alphaearth_paper_figures.py	Generate paper-ready AlphaEarth tables and SVG/PDF/PNG figures from checked CSV outputs.
make_model_family_figures.py	Generate checked model-family baseline tables and paper-ready figures from full baseline runs.
make_readme_progress_figure.py	Generate the README multi-stage research roadmap figure as SVG, PDF, and PNG.

Data Processing Flow

flowchart TD
    A[Raw CSV path] --> B[load_dataset]
    B --> C[Raw DataFrame df]

    C --> D[preprocess_features]
    D --> E[Remove leakage columns]
    E --> F[Create hazard features]
    F --> G[Log-transform selected columns]
    G --> H[Convert distance columns to binary threshold features]
    H --> I[Ordinal encode selected columns]
    I --> J[One-hot encode categorical columns]
    J --> K[Processed feature matrix X]

    C --> L[Select target y = target_col]

    K --> M[split_data]
    L --> M

    M --> N[X_train]
    M --> O[X_test]
    M --> P[y_train]
    M --> Q[y_test]

    N --> R[Fit feature clipping caps on X_train only]
    R --> S[Clip X_train]
    R --> T[Clip X_test using train caps]
    O --> T

    P --> U[Fit positive-target cutoff on y_train only]
    U --> V[Drop training rows above cutoff]

    S --> W[Training features]
    V --> X[Cut training target]
    T --> Y[Test features]
    Q --> Z[Unclipped test target for evaluation]

Loading

1. The dataset is loaded from CSV into a raw DataFrame.
2. Feature preprocessing is applied to create the model input matrix X.
3. The preset's configured target column is selected separately as y.
4. The processed features and raw target are split into train/test sets using shared indices.
5. Feature clipping caps are fit on X_train only, then applied to both X_train and X_test.
6. A target cutoff is fit on positive y_train values only, then rows above that cutoff are excluded from the training split.
7. The model trains on clipped training features and cutoff-filtered training targets.
8. Evaluation compares predictions against the original, unclipped y_test.

Model Training and Prediction Flow

The staged model is trained as three connected parts:

flowchart TD
    A[Training split: X_train, y_train] --> B[Zero-vs-positive classifier]

    A --> C[Filter actual positive rows]
    C --> D[X_train_pos, y_train_pos]

    D --> E[Create tier labels using threshold]
    E --> F[Tier classifier: low vs high]

    D --> G[Low-tier rows only]
    D --> H[High-tier rows only]

    G --> I[Low regressor]
    H --> J[High regressor]

    K[Test / prediction rows] --> L[Zero-vs-positive prediction]
    L -->|Predicted zero| M[Final prediction = 0]
    L -->|Predicted positive| N[Tier prediction]
    N -->|Predicted low| O[Low regressor prediction]
    N -->|Predicted high| P[High regressor prediction]

    O --> Q[Invert log target with expm1]
    P --> Q
    Q --> R[Clip negative predictions to 0]
    R --> S[Final debris volume prediction]

Loading

What Each Model Is Trained On

Model	Input rows	Target used	SMOTE?	Output
Zero-vs-positive classifier	All training rows	1 if y_train > 0, else 0	Yes	Predicts whether a row has debris
Tier classifier	Actual positive training rows only	0 if y_train <= threshold, else 1	Yes	Predicts low-tier vs high-tier debris
Low regressor	Actual positive low-tier rows only	log1p(y_train)	No	Predicts debris volume for low-tier positives
High regressor	Actual positive high-tier rows only	log1p(y_train)	No	Predicts debris volume for high-tier positives

Prediction Behavior

During end-to-end prediction, the model routes each row through the stages:

1. The zero-vs-positive classifier decides whether the row should receive a debris prediction.
2. Rows predicted as zero receive a final prediction of 0.
3. Rows predicted as positive are passed to the tier classifier.
4. The tier classifier routes each positive row to either the low regressor or high regressor.
5. Regressor predictions are converted back from log space using expm1.
6. Negative predictions are clipped to 0.

Prediction Modes

RunConfig.prediction_mode controls test-time routing:

Mode	Use case	Routing behavior
end_to_end	Main operational evaluation	Uses the model's own zero-vs-positive and tier predictions to route test rows.
oracle	Stage-level diagnostics only	Uses observed y_true to route rows into positive, low, and high regression paths.

Use end_to_end for paper-facing operational metrics. Use oracle only when diagnosing how the regressors behave under ideal routing.

Prediction Results

predict_staged_model returns a PredictionResults object containing predictions from each stage of the pipeline.

Rows that do not reach a stage are represented as NaN. For example, rows predicted as zero debris do not receive tier or regressor predictions.

Field	Description
zero_pos_pred, zero_pos_prob	Zero-vs-positive classifier outputs
tier_pred, tier_prob	Low/high tier classifier outputs
low_pred	Low-tier regressor predictions
high_pred	High-tier regressor predictions
reg_pred	Combined low/high regressor predictions
final_pred	Final end-to-end debris volume prediction

Helper methods are provided to simplify evaluation by automatically aligning predictions with the correct subset of ground-truth values.

y_tier_true, y_tier_pred, y_tier_prob = preds.tier_pairs(y_true)
y_low_true, y_low_pred = preds.low_pairs(y_true)
y_high_true, y_high_pred = preds.high_pairs(y_true)
y_reg_true, y_reg_pred = preds.reg_pairs(y_true)

final_pred should be used to evaluate overall system performance, while the pair helper methods are intended for stage-level evaluation and diagnostics.

Outputs

Single runs write to the directory passed to run_model. Sweeps write to:

<output_dir>/<experiment_name>/
  experiment.json
  runs/
    <run_id>/
      config.json
      metrics.json
      predictions.csv
      plots/
  analysis/
    summary.csv
    leaderboard.csv
    plots/

Run Outputs

File	Description
config.json	Stores the complete RunConfig used to generate the run, including preprocessing, splitting, clipping, and model parameters.
metrics.json	Stores the complete EvaluationResults object, including system-level, classifier, and regressor metrics.
predictions.csv	Stores one row per sample containing the ground-truth target, target column name, grid ID, final prediction, and stage-level model outputs.
plots/	Run-level evaluation and diagnostic visualizations.

Sweep Analysis Outputs

File	Description
experiment.json	The base run configuration, swept fields, and ranking settings.
summary.csv	Flattened configuration values and metrics for each successful run.
leaderboard.csv	Successful runs ranked by the configured primary metric and mode.
plots/	Metric comparisons for each swept field.

Sweep run IDs include the index and swept values, for example 000_threshold_500_fclip_0p9. A failed or invalid run is logged and skipped; analysis includes only runs that produced both config.json and metrics.json.