Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 144 additions & 79 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,152 +1,217 @@
# SpectralUtils
# SpectralUtil

This is a package for basic manipulation of imaging spectroscopy data. It is designed to accommodate data from a variety of instruments, abstracting out the specifics of the file delivery. Currently, the package supports data from the following instruments / product levels:
SpectralUtil provides command-line and Python tools for common imaging spectroscopy workflows, including quicklooks, plotting, mosaicking, Earthdata download helpers, and format conversion.

Supported products include:

- [AVIRIS-3 L1B Radiance](https://doi.org/10.3334/ORNLDAAC/2356)
- [AVIRIS-3 L2A Reflectance](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2357)
- AVIRIS-NG L2A Reflectance
- [EMIT L1B Radiance](https://lpdaac.usgs.gov/products/emitl1bradv001/)
- [EMIT L2A Reflectance](https://lpdaac.usgs.gov/products/emitl2arflv001/)
- Any data in ENVI format
- ENVI format inputs

## Installation

To install the package, we recommend using [pixi](https://pixi.sh):
Recommended with [pixi](https://pixi.sh):

```bash
pixi install
```

Or you can install via pip, though depending on the system you may experience gdal-related issues:
Or with pip:

```bash
pip install spectral_util
```

## CLI Interface
## Running the CLI

If installed into your current environment:

The package provides a unified CLI interface with the following commands:
```bash
spectral_util --help
```

### Quicklooks (RGB, NDVI, NBR)
If using the local pixi environment:

```bash
# Standard RGB
spectral_util quicklooks rgb input_file.tif output_rgb.tif
pixi run spectral_util --help
```

# RGB with custom wavelengths
spectral_util quicklooks rgb input_file.tif output_rgb.tif --red_wl 660 --green_wl 560 --blue_wl 460
The top-level command groups are:

# NDVI calculation
spectral_util quicklooks ndvi input_file.tif output_ndvi.tif
- download
- mosaic
- quicklooks
- plot
- reformat

# NBR calculation
spectral_util quicklooks nbr input_file.tif output_nbr.tif
```
## CLI Examples by Section

### Mosaic Operations
### download

#### download get-fid
```bash
# Build observation mosaic
spectral_util mosaic build-obs-nc output_file.nc input_file_list.txt --x_resolution 10 --y_resolution -10
# Download files for one specific FID and product short-name
spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000

# Apply GLT to files
spectral_util mosaic apply-glt glt_file.txt raw_files.txt output_file.tif
# Download only the RDN granule component (used in examples below)
spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000 --subfile RDN --version 1

# Stack GLTs
spectral_util mosaic stack-glts glt_files.txt obs_file_lists.txt output_glt_file.txt output_file_list.txt
# Overwrite previously-downloaded files
spectral_util download get-fid ./downloads AV3_L1B_RDN_2356 AV320250809t182459_000 --subfile RDN --version 1 --overwrite
```

### Data Download (AV3 and EMIT)
Assume the downloaded file used in subsequent examples is:

```text
./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc
```

### quicklooks

#### quicklooks rgb
```bash
# Download AV3 data
spectral_util av3-download /path/to/output --temporal 2024-10-04T16:00:00 2024-10-04T17:00:00 --bounding_box -103.74460188 32.22680624 -103.74481188 32.22700624
# Basic RGB quicklook
spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb.tif

# Download EMIT data
spectral_util emit-download /path/to/output --temporal 2024-10-04T16:00:00 2024-10-04T17:00:00 --bounding_box -103.74460188 32.22680624 -103.74481188 32.22700624
```
# RGB with custom wavelengths and percentile stretch
spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb.tif --red_wl 660 --green_wl 560 --blue_wl 460 --stretch 2 98

### Reformat an EMIT file
```
spectral_util reformat nc-to-envi EMIT_L2A_RFL_001_20250525T090616_2514506_004.nc test_emit_rfl --ortho
# RGB with explicit per-channel scaling (disable stretch)
spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb_scaled.tif --stretch -1 -1 --scale 0 0.2 0 0.2 0 0.2

# Orthorectified RGB output when GLT is available
spectral_util quicklooks rgb ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_rgb_ortho.tif --ortho
```

## Python API
#### quicklooks ndvi
```bash
# Default NDVI
spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi.tif

For programmatic use, the package can be imported directly:
# NDVI with custom wavelengths and band widths
spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi_custom.tif --red_wl 665 --nir_wl 842 --red_width 20 --nir_width 20

```python
from spectral_util.common import quicklooks
from spectral_util.mosaic import mosaic
from spectral_util.ea_assist import earthaccess_helpers_AV3, earthaccess_helpers_EMIT
# Orthorectified NDVI
spectral_util quicklooks ndvi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ndvi_ortho.tif --ortho
```

## Utilities
#### quicklooks nbr
```bash
# Default NBR
spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr.tif

A series of utility scripts are provided to help with common tasks, such as:
# NBR with custom NIR/SWIR settings
spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr_custom.tif --nir_wl 860 --swir_wl 2200 --nir_width 20 --swir_width 40

### Standard RGB, with stretching
```bash
spectral_util quicklooks rgb EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_rgb.tif
# Orthorectified NBR
spectral_util quicklooks nbr ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_nbr_ortho.tif --ortho
```

### Standard RGB, custom wavelengths
### plot

#### plot plot-basic-overview
```bash
spectral_util quicklooks rgb EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_rgb.tif --red_wl 2360 --green_wl 1800 --blue_wl 1000
# Show RGB + selected spectra interactively
spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc

# Save basic overview plot and choose random sampling
spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file spectra_overview.png --n_points 8 --method random

# K-means representative spectra
spectral_util plot plot-basic-overview ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file spectra_kmeans.png --n_points 10 --method kmeans
```

### Spectral Indices
#### plot plot-pcs
```bash
spectral_util quicklooks nbr EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_nbr.tif
spectral_util quicklooks ndvi EMIT_L1B_RAD_001_20240715T195403_2419712_015.nc emit_ndvi.tif
# Plot first 20 principal components
spectral_util plot plot-pcs ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file pcs.png

# Plot a custom PC range with controlled sampling
spectral_util plot plot-pcs ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file pcs_10_29.png --first_pc 10 --last_pc 29 --n_points 20000 --seed 42
```

## Command Reference
#### plot plot-mnf
```bash
# Plot first 20 MNF components
spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf.png

### Quicklooks Commands
- `rgb`: Create RGB composite
- `ndvi`: Calculate NDVI (Normalized Difference Vegetation Index)
- `nbr`: Calculate NBR (Normalized Burn Ratio)
# Custom MNF count and covariance sampling options
spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf_30.png --n_mnf 30 --n_points 20000 --seed 42

### Mosaic Commands
- `build-obs-nc`: Build observation mosaic from input files
- `apply-glt`: Apply Geolocation Transformation to files
- `stack-glts`: Stack multiple GLTs into a single output
# Estimate noise along rows instead of columns
spectral_util plot plot-mnf ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc --output_file mnf_diff_rows.png --diff_dim 0
```

### Download Commands
- `get-fid`: Download a specific FID for a given product via CMR.
- `av3-download`: Download AV3 data from Earthdata - currently niche usage, to be updated
- `emit-download`: Download EMIT data from Earthdata - currently niche usage, to be updated
### mosaic

## Reformat Commands
- `nc-to-envi`: Convert a netcdf data file to an envi binary file, with options
#### mosaic build-obs-nc
```bash
# Build GLT mosaic from a text file of OBS inputs
spectral_util mosaic build-obs-nc mosaic_glt.tif obs_files.txt --x_resolution 30 --y_resolution -30 --output_epsg 4326

# Build with explicit extent and criteria selection
spectral_util mosaic build-obs-nc mosaic_glt_utm.tif obs_files.txt \
--x_resolution 60 --y_resolution -60 --output_epsg 32611 \
--target_extent_ul_lr 3800000 420000 3700000 520000 \
--criteria_band 0 --criteria_mode min --n_cores 8

# Exclude selected files and set max nearest-neighbor distance
spectral_util mosaic build-obs-nc mosaic_filtered.tif obs_files.txt \
--ignore_file_list ignore_files.txt --x_resolution 30 --y_resolution -30 --max_distance 45
```

## Options
#### mosaic apply-glt
```bash
# Apply GLT to a single raw file
spectral_util mosaic apply-glt mosaic_glt.tif ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_ortho.tif

Most commands have custom options - call the given utility with --help to see all options.
# Apply GLT to a list of raw files and subset bands
spectral_util mosaic apply-glt mosaic_glt.tif raw_files.txt output_rgb.tif --bands 34 --bands 22 --bands 10

## Examples
# Write ENVI output with custom nodata handling
spectral_util mosaic apply-glt mosaic_glt.tif raw_files.txt output_envi --output_format envi --nodata_value -9999 --glt_nodata_value 0
```

### RGB with custom wavelengths and stretching
#### mosaic stack-glts
```bash
spectral_util quicklooks rgb input.nc output.tif --red_wl 660 --green_wl 560 --blue_wl 460 --stretch 2 98
# Merge several GLTs and their file lists into one stack
spectral_util mosaic stack-glts glt_files.txt obs_file_lists.txt stacked_glt.tif stacked_file_list.txt
```

### Download with specific parameters
### reformat

#### reformat nc-to-envi
```bash
spectral_util emit-download /data/output --temporal 2024-01-01T00:00:00 2024-12-31T23:59:59 --bounding_box -120 30 -110 40
# Convert NetCDF to ENVI
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi

# Convert NetCDF to orthorectified ENVI output
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi_ortho --ortho

# Overwrite existing output
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi --overwrite
```

## Help
## Python API

For detailed help on any command:
```bash
spectral_util <command> --help
For programmatic use:

```python
from spectral_util.common import quicklooks
from spectral_util.mosaic import mosaic
from spectral_util.ea_assist import earthaccess_helpers_AV3, earthaccess_helpers_EMIT
```

For example:
## Help

Use help at any level:
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi
```bash
spectral_util --help
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi_ortho --ortho
spectral_util quicklooks rgb --help
spectral_util mosaic build-obs-nc --help
spectral_util av3-download --help
```
spectral_util plot --help
spectral_util reformat nc-to-envi ./downloads/AV320250809t182459_000_L1B_RDN_4842d6a3_RDN.nc output_envi --overwrite
Loading