From e680a1794df90766fb40dd03161edb9955560faf Mon Sep 17 00:00:00 2001 From: Phil Brodrick Date: Sat, 28 Mar 2026 06:09:17 -0700 Subject: [PATCH] docs rewrite --- README.md | 223 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 144 insertions(+), 79 deletions(-) diff --git a/README.md b/README.md index 9cf8407..a907654 100644 --- a/README.md +++ b/README.md @@ -1,152 +1,217 @@ -# SpectralUtils +# SpectralUtil -This is a package for basic manipulation of imaging spectroscopy data. It is designed to accommodate data from a variety of instruments, abstracting out the specifics of the file delivery. Currently, the package supports data from the following instruments / product levels: +SpectralUtil provides command-line and Python tools for common imaging spectroscopy workflows, including quicklooks, plotting, mosaicking, Earthdata download helpers, and format conversion. + +Supported products include: - [AVIRIS-3 L1B Radiance](https://doi.org/10.3334/ORNLDAAC/2356) - [AVIRIS-3 L2A Reflectance](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2357) - AVIRIS-NG L2A Reflectance - [EMIT L1B Radiance](https://lpdaac.usgs.gov/products/emitl1bradv001/) - [EMIT L2A Reflectance](https://lpdaac.usgs.gov/products/emitl2arflv001/) -- Any data in ENVI format +- ENVI format inputs ## Installation -To install the package, we recommend using [pixi](https://pixi.sh): +Recommended with [pixi](https://pixi.sh): ```bash pixi install ``` -Or you can install via pip, though depending on the system you may experience gdal-related issues: +Or with pip: ```bash pip install spectral_util ``` -## CLI Interface +## Running the CLI + +If installed into your current environment: -The package provides a unified CLI interface with the following commands: +```bash +spectral_util --help +``` -### Quicklooks (RGB, NDVI, NBR) +If using the local pixi environment: ```bash -# Standard RGB -spectral_util quicklooks rgb input_file.tif output_rgb.tif +pixi run spectral_util --help +``` -# RGB with custom wavelengths -spectral_util quicklooks rgb input_file.tif output_rgb.tif --red_wl 660 --green_wl 560 --blue_wl 460 +The top-level command groups are: -# NDVI calculation -spectral_util quicklooks ndvi input_file.tif output_ndvi.tif +- download +- mosaic +- quicklooks +- plot +- reformat -# NBR calculation -spectral_util quicklooks nbr input_file.tif output_nbr.tif -``` +## CLI Examples by Section -### Mosaic Operations +### download +#### download get-fid ```bash -# Build observation mosaic -spectral_util mosaic build-obs-nc output_file.nc input_file_list.txt --x_resolution 10 --y_resolution -10 +# Download files for one specific FID and product short-name +spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000 -# Apply GLT to files -spectral_util mosaic apply-glt glt_file.txt raw_files.txt output_file.tif +# Download only the RDN granule component (used in examples below) +spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000 --subfile RDN --version 1 -# Stack GLTs -spectral_util mosaic stack-glts glt_files.txt obs_file_lists.txt output_glt_file.txt output_file_list.txt +# Overwrite previously-downloaded files +spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000 --subfile RDN --version 1 --overwrite ``` -### Data Download (AV3 and EMIT) +Assume the downloaded file used in subsequent examples is: +```text +./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc +``` + +### quicklooks + +#### quicklooks rgb ```bash -# Download AV3 data -spectral_util av3-download /path/to/output --temporal 2024-10-04T16:00:00 2024-10-04T17:00:00 --bounding_box -103.74460188 32.22680624 -103.74481188 32.22700624 +# Basic RGB quicklook +spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb.tif -# Download EMIT data -spectral_util emit-download /path/to/output --temporal 2024-10-04T16:00:00 2024-10-04T17:00:00 --bounding_box -103.74460188 32.22680624 -103.74481188 32.22700624 -``` +# RGB with custom wavelengths and percentile stretch +spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb.tif --red_wl 660 --green_wl 560 --blue_wl 460 --stretch 2 98 -### Reformat an EMIT file -``` -spectral_util reformat nc-to-envi EMIT_L2A_RFL_001_20250525T090616_2514506_004.nc test_emit_rfl --ortho +# RGB with explicit per-channel scaling (disable stretch) +spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb_scaled.tif --stretch -1 -1 --scale 0 0.2 0 0.2 0 0.2 + +# Orthorectified RGB output when GLT is available +spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb_ortho.tif --ortho ``` -## Python API +#### quicklooks ndvi +```bash +# Default NDVI +spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi.tif -For programmatic use, the package can be imported directly: +# NDVI with custom wavelengths and band widths +spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi_custom.tif --red_wl 665 --nir_wl 842 --red_width 20 --nir_width 20 -```python -from spectral_util.common import quicklooks -from spectral_util.mosaic import mosaic -from spectral_util.ea_assist import earthaccess_helpers_AV3, earthaccess_helpers_EMIT +# Orthorectified NDVI +spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi_ortho.tif --ortho ``` -## Utilities +#### quicklooks nbr +```bash +# Default NBR +spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr.tif -A series of utility scripts are provided to help with common tasks, such as: +# NBR with custom NIR/SWIR settings +spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr_custom.tif --nir_wl 860 --swir_wl 2200 --nir_width 20 --swir_width 40 -### Standard RGB, with stretching -```bash -spectral_util quicklooks rgb EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_rgb.tif +# Orthorectified NBR +spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr_ortho.tif --ortho ``` -### Standard RGB, custom wavelengths +### plot + +#### plot plot-basic-overview ```bash -spectral_util quicklooks rgb EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_rgb.tif --red_wl 2360 --green_wl 1800 --blue_wl 1000 +# Show RGB + selected spectra interactively +spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc + +# Save basic overview plot and choose random sampling +spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file spectra_overview.png --n_points 8 --method random + +# K-means representative spectra +spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file spectra_kmeans.png --n_points 10 --method kmeans ``` -### Spectral Indices +#### plot plot-pcs ```bash -spectral_util quicklooks nbr EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_nbr.tif -spectral_util quicklooks ndvi EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_ndvi.tif +# Plot first 20 principal components +spectral_util plot plot-pcs ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file pcs.png + +# Plot a custom PC range with controlled sampling +spectral_util plot plot-pcs ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file pcs_10_29.png --first_pc 10 --last_pc 29 --n_points 20000 --seed 42 ``` -## Command Reference +#### plot plot-mnf +```bash +# Plot first 20 MNF components +spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf.png -### Quicklooks Commands -- `rgb`: Create RGB composite -- `ndvi`: Calculate NDVI (Normalized Difference Vegetation Index) -- `nbr`: Calculate NBR (Normalized Burn Ratio) +# Custom MNF count and covariance sampling options +spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf_30.png --n_mnf 30 --n_points 20000 --seed 42 -### Mosaic Commands -- `build-obs-nc`: Build observation mosaic from input files -- `apply-glt`: Apply Geolocation Transformation to files -- `stack-glts`: Stack multiple GLTs into a single output +# Estimate noise along rows instead of columns +spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf_diff_rows.png --diff_dim 0 +``` -### Download Commands -- `get-fid`: Download a specific FID for a given product via CMR. -- `av3-download`: Download AV3 data from Earthdata - currently niche usage, to be updated -- `emit-download`: Download EMIT data from Earthdata - currently niche usage, to be updated +### mosaic -## Reformat Commands -- `nc-to-envi`: Convert a netcdf data file to an envi binary file, with options +#### mosaic build-obs-nc +```bash +# Build GLT mosaic from a text file of OBS inputs +spectral_util mosaic build-obs-nc mosaic_glt.tif obs_files.txt --x_resolution 30 --y_resolution -30 --output_epsg 4326 + +# Build with explicit extent and criteria selection +spectral_util mosaic build-obs-nc mosaic_glt_utm.tif obs_files.txt \ + --x_resolution 60 --y_resolution -60 --output_epsg 32611 \ + --target_extent_ul_lr 3800000 420000 3700000 520000 \ + --criteria_band 0 --criteria_mode min --n_cores 8 + +# Exclude selected files and set max nearest-neighbor distance +spectral_util mosaic build-obs-nc mosaic_filtered.tif obs_files.txt \ + --ignore_file_list ignore_files.txt --x_resolution 30 --y_resolution -30 --max_distance 45 +``` -## Options +#### mosaic apply-glt +```bash +# Apply GLT to a single raw file +spectral_util mosaic apply-glt mosaic_glt.tif ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ortho.tif -Most commands have custom options - call the given utility with --help to see all options. +# Apply GLT to a list of raw files and subset bands +spectral_util mosaic apply-glt mosaic_glt.tif raw_files.txt output_rgb.tif --bands 34 --bands 22 --bands 10 -## Examples +# Write ENVI output with custom nodata handling +spectral_util mosaic apply-glt mosaic_glt.tif raw_files.txt output_envi --output_format envi --nodata_value -9999 --glt_nodata_value 0 +``` -### RGB with custom wavelengths and stretching +#### mosaic stack-glts ```bash -spectral_util quicklooks rgb input.nc output.tif --red_wl 660 --green_wl 560 --blue_wl 460 --stretch 2 98 +# Merge several GLTs and their file lists into one stack +spectral_util mosaic stack-glts glt_files.txt obs_file_lists.txt stacked_glt.tif stacked_file_list.txt ``` -### Download with specific parameters +### reformat + +#### reformat nc-to-envi ```bash -spectral_util emit-download /data/output --temporal 2024-01-01T00:00:00 2024-12-31T23:59:59 --bounding_box -120 30 -110 40 +# Convert NetCDF to ENVI +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi + +# Convert NetCDF to orthorectified ENVI output +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi_ortho --ortho + +# Overwrite existing output +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi --overwrite ``` -## Help +## Python API -For detailed help on any command: -```bash -spectral_util --help +For programmatic use: + +```python +from spectral_util.common import quicklooks +from spectral_util.mosaic import mosaic +from spectral_util.ea_assist import earthaccess_helpers_AV3, earthaccess_helpers_EMIT ``` -For example: +## Help + +Use help at any level: +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi ```bash +spectral_util --help +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi_ortho --ortho spectral_util quicklooks rgb --help -spectral_util mosaic build-obs-nc --help -spectral_util av3-download --help -``` \ No newline at end of file +spectral_util plot --help +spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi --overwrite \ No newline at end of file