Updates to EMIT Mineral Notebooks

README.md

@@ -1,5 +1,11 @@
 # EMIT-Data-Resources  
 
+<img alt="Python resources and guides for working with EMIT data." src="img/earthdata_logo.png" width="50%" align="center" />
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)
+[![Cite this repo](https://img.shields.io/badge/Cite-EMIT--Data--Resources-blue)](https://github.com/nasa/EMIT-Data-Resources/blob/main/CITATION.cff)
+
 Welcome to the EMIT-Data-Resources repository. This repository provides guides, short how-tos, and tutorials to help users access and work with data from the [Earth Surface Mineral Dust Source Investigation (EMIT) mission](https://lpdaac.usgs.gov/data/get-started-data/collection-overview/missions/emit-overview/). In the interest of open science this repository has been made public but is still under active development. All notebooks and scripts should be functional, however, changes or additions may be made. Make sure to consult the [CHANGE_LOG.md](CHANGE_LOG.md) for the most recent changes to the repository. Contributions from all parties are welcome.  
 
 ---
@@ -8,7 +14,7 @@ Welcome to the EMIT-Data-Resources repository. This repository provides guides,
 
 The [EMIT](https://earth.jpl.nasa.gov/emit/) Project delivers space-based measurements of surface mineralogy of the Earth’s arid dust source regions. These measurements are used to initialize the compositional makeup of dust sources in Earth System Models (ESMs). The dust cycle, which describe the generation, lofting, transport, and deposition of mineral dust, plays an important role in ESMs. Dust composition is presently the largest uncertainty factor in quantifying the magnitude of aerosol direct radiative forcing. By understanding the composition of mineral dust sources, EMIT aims to constrain the sign and magnitude of dust-related radiative forcing at regional and global scales. During its one-year mission on the International Space Station (ISS), EMIT will make measurements over the sunlit Earth’s dust source regions that fall within ±52° latitude. EMIT will schedule up to five visits (three on average) of each arid target region and only acquisitions not dominated by cloud cover will be downlinked. EMIT-based maps of the relative abundance of source minerals will advance the understanding of the current and future impacts of mineral dust in the Earth system.  
 
-EMIT Data Products are distributed by the [LP DAAC](https://lpdaac.usgs.gov/). Learn more about EMIT data products from [EMIT Product Pages](https://lpdaac.usgs.gov/product_search/?query=emit&status=Operational&view=cards&sort=title) and search for and download EMIT data products using [NASA EarthData Search](https://search.earthdata.nasa.gov/search?q=%22EMIT%22)  
+EMIT Data Products are distributed by the [LP DAAC](https://www.earthdata.nasa.gov/centers/lp-daac). Learn more about EMIT data products from [EMIT Product Pages](https://www.earthdata.nasa.gov/data/catalog?keyword=%22EMIT%22) and search for and download EMIT data products using [NASA EarthData Search](https://search.earthdata.nasa.gov/search?q=%22EMIT%22)  
 
 ---
 
@@ -27,7 +33,7 @@ Below are the resources available for EMIT Data.
 |[Exploring EMIT L2A Reflectance](python/tutorials/Exploring_EMIT_L2A_Reflectance.ipynb)|Jupyter Notebook|Explore EMIT L2A Reflectance data using interactive plots|
 |[Visualizing Methane Plume Timeseries](python/tutorials/Visualizing_Methane_Plume_Timeseries.ipynb)|Jupyter Notebook|Find EMIT L2B CH4 Plume Data and build a timeseries of CH4 plume complexes|
 |[Generating_Methane_Spectral_Fingerprint](python/tutorials/Generating_Methane_Spectral_Fingerprint.ipynb)|Jupyter Notebook|Extract Radiance Spectra and build an in-plume/out-of-plume ratio to compare with CH4 absorption coefficient|
-|[Finding_EMIT_L2B_Data](python/tutorials/Finding_EMIT_L2B_Data.ipynb)|Jupyter Notebook|Use the `earthaccess` Python library to find EMIT L2B Mineral Identification Band Depth and Uncertainty data|
+|[Finding_EMIT_L2B_Mineral_Data](python/tutorials/Finding_EMIT_L2B_Mineral_Data.ipynb)|Jupyter Notebook|Use the `earthaccess` Python library to find EMIT L2B Mineral Identification Band Depth and Uncertainty data|
 |[Working with EMIT L2B Mineralogy](python/tutorials/Working_with_EMIT_L2B_Mineralogy.ipynb)|Jupyter Notebook|Work with the EMIT L2B Mineral Identification Band Depth and Uncertainty Data and aggregate individual spectral library constituents into the EMIT-10 minerals and estimate abundance| 
 |[How to find and access EMIT data](python/how-tos/How_to_find_and_access_EMIT_data.ipynb)|Jupyter Notebook|Use the `earthaccess` Python library to find and download or stream EMIT data|
 |[How to Convert to ENVI Format](python/how-tos/How_to_Convert_to_ENVI.ipynb)|Jupyter Notebook|Convert from downloaded netCDF4 (.nc) format to .envi format|
@@ -38,25 +44,58 @@ Below are the resources available for EMIT Data.
 |[How to use Direct S3 Access with EMIT](python/how-tos/How_to_Direct_S3_Access.ipynb)|Jupyter Notebook|Use S3 from inside AWS us-west2 to access EMIT Data|
 |[How to find EMIT Data using NASA's CMR API](python/how-tos/How_to_find_EMIT_data_using_CMR_API.ipynb)|Jupyter Notebook|Use NASA's CMR API to programmatically find EMIT Data|
 
----
+## Citation
+This repository is the product of a collaborative effort from the NASA Land Processes Distributed Active Archive Center (DAAC) and NASA Jet Propulsion Laboratory. If you use this resource or an associated data product for your research, we appreciate a citation.
+
+[![Cite this repo](https://img.shields.io/badge/Cite-EMIT--Data--Resources-blue)](https://github.com/nasa/EMIT-Data-Resources/blob/main/CITATION.cff)
+
+For each data product used, you can retrieve a citation in your desired format from the product pages below.
+
+|Product|Product Page|Collection Shortname|
+|:---|:---|:---|
+|EMIT L1B At-Sensor Calibrated Radiance and Geolocation Data|https://doi.org/10.5067/EMIT/EMITL1BRAD.001|EMITL1BRAD|
+|EMIT L2A Estimated Surface Reflectance and Uncertainty and Masks|https://doi.org/10.5067/EMIT/EMITL2ARFL.001|EMITL2ARFL|
+|EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty|https://doi.org/10.5067/EMIT/EMITL2BMIN.001|EMITL2BMIN|
+|EMIT L2B Estimated Methane Enhancement Data|https://doi.org/10.5067/EMIT/EMITL2BCH4ENH.001|EMITL2BCH4ENH|
+|EMIT L2B Estimated Methane Plume Complexes|https://doi.org/10.5067/EMIT/EMITL2BCH4PLM.001|EMITL2BCH4PLM|
+
+
+## Contributors
+
+If you would like to contribute, please view our [contributing guide](CONTRIBUTING.md).
 
-## Helpful Links  
+[![Contributors](https://contrib.rocks/image?repo=nasa/emit-data-resources)](https://github.com/nasa/emit-data-resources/graphs/contributors)
 
+## Related Resources
+
+LP DAAC also develops and maintains additional resources, available at <https://github.com/nasa/LPDAAC-Data-Resources>. This landing repository provides general guides and tutorials, as well as links to mission-specific repositories that offer detailed guides, tutorials, how-tos, and scripts to help users find, access, and work with each mission’s data products.
+
+| GitHub Repository | Summary | Contents Description |
+|----|-----|----|
+|[AppEEARS Data Resources](https://github.com/nasa/AppEEARS-Data-Resources) |How to use the Application for Extracting and Exploring Analysis Ready Samples (AppEEARS) |Tutorials, AppEEARS API, Direct S3 Access |
+|[AppEEARS QGIS Plugin](https://github.com/nasa/AppEEARS-QGIS-Plugin)|Browse and load cloud-optimized geotiff output files from AppEEARS directly into QGIS|QGIS Plugin|
+|[ASTER Data Resources](https://github.com/nasa/ASTER-Data-Resources)| How to find, access, and work with ASTER data |Tutorials|
+|[ECOSTRESS Data Resources](https://github.com/nasa/ECOSTRESS-Data-Resources)|How to find, access, and work with ECOSTRESS data (The ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station)|Tutorials, Scripts, Direct S3 Access|
+|[EMIT Data Resources](https://github.com/nasa/EMIT-Data-Resources) |How to find, access, and work with EMIT data (Earth Surface Mineral Dust Source Investigation)|Tutorials, Scripts, Direct S3 Access |
+|[GEDI Data Resources](https://github.com/nasa/GEDI-Data-Resources) |How to find, access, and work with GEDI data (Global Ecosystem Dynamics Investigation)|Tutorials |
+|[HLS Data Resources](https://github.com/nasa/HLS-Data-Resources)|How to find, access, and work with HLS data (Harmonized Landsat Sentinel-2)|Tutorials, Scripts, Direct S3 Access|
+|[LPDAAC Data Resources](https://github.com/nasa/LPDAAC-Data-Resources)|How to find, access, and work with LPDAAC data |Tutorials, Scripts, Direct S3 Access|
+|[MODIS-VIIRS Data Resources](https://github.com/nasa/MODIS-VIIRS-Data-Resources)|How to find, access, and work with MODIS and VIIRS data|Tutorials|
+|[VITALS](https://github.com/nasa/VITALS)|How to find and work with EMIT and ECOSTRESS data together |Tutorials|
+
+### Other Helpful Links   
+
++ [NASA Earthdata Website](https://www.earthdata.nasa.gov/)
++ [LP DAAC Website](https://www.earthdata.nasa.gov/centers/lp-daac)
 + [JPL EMIT Website](https://earth.jpl.nasa.gov/emit/)  
 + [EMIT Tutorial Webinar Series Recordings](https://www.youtube.com/playlist?list=PLO2yB4LGNlWrC5NdxeHMxyAxdwQhSypXe)
 + [LP DAAC EMIT Product Pages](https://lpdaac.usgs.gov/product_search/?query=emit&status=Operational&view=cards&sort=title) - Learn more about available EMIT products  
 + [VISIONS Open Data Portal](https://earth.jpl.nasa.gov/emit/data/data-portal/coverage-and-forecasts/) - Learn about current and forecasted EMIT coverage  
-
 + [EMIT on Earth Data Search](https://search.earthdata.nasa.gov/search?q=%22EMIT%22) - Download EMIT Data from NASA
-
 + [EMIT Github Repository](https://github.com/emit-sds) - Main EMIT Repository  
-
 + [EMIT Utilities Github Repository](https://github.com/emit-sds/emit-utils) - General convenience utilities for working with EMIT data
-
 + [L2A Reflectance User Guide](https://lpdaac.usgs.gov/documents/1569/EMITL2ARFL_User_Guide_v1.pdf)  
-
 + [L2A Algorithm Theoretical Basis Document](https://lpdaac.usgs.gov/documents/1571/EMITL2A_ATBD_v1.pdf)  
-
 + [EMIT on Slack]( https://forms.gle/XefLVG6e6A7ezwpY9) - Join the EMIT slack community!
 
 ---
@@ -67,6 +106,6 @@ Email: <LPDAAC@usgs.gov>
 Voice: +1-866-573-3222  
 Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹  
 Website: <https://lpdaac.usgs.gov/>  
-Date last modified: 08-01-2024  
+Date last modified: 06-28-2024  
 
 ¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I.  

data/min_results_s3_urls.txt

@@ -0,0 +1,3 @@
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230427T173309_2311711_010/EMIT_L2B_MIN_001_20230427T173309_2311711_010.nc
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230804T191650_2321613_007/EMIT_L2B_MIN_001_20230804T191650_2321613_007.nc
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230808T173953_2322012_011/EMIT_L2B_MIN_001_20230808T173953_2322012_011.nc

data/min_uncert_s3_urls.txt

@@ -0,0 +1,3 @@
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230427T173309_2311711_010/EMIT_L2B_MINUNCERT_001_20230427T173309_2311711_010.nc
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230804T191650_2321613_007/EMIT_L2B_MINUNCERT_001_20230804T191650_2321613_007.nc
+s3://lp-prod-protected/EMITL2BMIN.001/EMIT_L2B_MIN_001_20230808T173953_2322012_011/EMIT_L2B_MINUNCERT_001_20230808T173953_2322012_011.nc

python/modules/emit_tools.py

@@ -20,12 +20,11 @@
 import math
 from skimage import io
 import pandas as pd
-import geopandas as gpd
 import xarray as xr
 import rasterio as rio
 import rioxarray as rxr
 import s3fs
-from rioxarray.merge import merge_arrays
+from rioxarray.merge import merge_datasets
 from fsspec.implementations.http import HTTPFile
 
 
@@ -49,6 +48,8 @@ def emit_xarray(filepath, ortho=False, qmask=None, unpacked_bmask=None):
         granule_id = filepath.info()["name"].split("/", -1)[-1].split(".", -1)[0]
     elif type(filepath) == HTTPFile:
         granule_id = filepath.path.split("/", -1)[-1].split(".", -1)[0]
+    elif hasattr(filepath, 'path'):
+        granule_id = filepath.path.split("/", -1)[-1].split(".", -1)[0]
     else:
         granule_id = os.path.splitext(os.path.basename(filepath))[0]
 
@@ -624,80 +625,76 @@ def is_adjacent(scene: str, same_orbit: list):
     return all(b - a == 1 for a, b in zip(scene_nums[:-1], scene_nums[1:]))
 
 
-def merge_emit(datasets: dict, gdf: gpd.GeoDataFrame):
+def merge_emit(datasets, bounds=None, nodata=-9999):
     """
-    A function to merge xarray datasets formatted using emit_xarray. This could probably be improved,
-    lots of shuffling data around to keep in xarray and get it to merge properly. Note: GDF may only work with a
-    single geometry.
+    This function merges adjacent emit scenes. If pixels aren't align it matches the first
+    in the list.
+
+    Parameters
+    datasets : dict[str, xr.Dataset] | list[xr.Dataset]
+        Orthorectified EMIT datasets (from emit_xarray(..., ortho=True)).
+    bounds : tuple (minx, miny, maxx, maxy) | GeoDataFrame | None
+        Output extent. A GeoDataFrame uses .total_bounds. None = combined extent.
+    nodata : numeric
+        Fill value. Defaults to EMIT's -9999.
     """
-    nested_data_arrays = {}
-    # loop over datasets
-    for dataset in datasets:
-        # create dictionary of arrays for each dataset
-
-        # create dictionary of 1D variables, which should be consistent across datasets
-        one_d_arrays = {}
-
-        # Dictionary of variables to merge
-        data_arrays = {}
-        # Loop over variables in dataset including elevation
-        for var in list(datasets[dataset].data_vars) + ["elev"]:
-            # Get 1D for this variable and add to dictionary
-            if not one_d_arrays:
-                # These should be an array describing the others (wavelengths, mask_bands, etc.)
-                one_dim = [
-                    item
-                    for item in list(datasets[dataset].coords)
-                    if item not in ["latitude", "longitude", "spatial_ref"]
-                    and len(datasets[dataset][item].dims) == 1
-                ]
-                # print(one_dim)
-                for od in one_dim:
-                    one_d_arrays[od] = datasets[dataset].coords[od].data
-
-                # Update format for merging - This could probably be improved
-            da = datasets[dataset][var].reset_coords("elev", drop=False)
-            da = da.rename({"latitude": "y", "longitude": "x"})
-            if len(da.dims) == 3:
-                if any(item in list(da.coords) for item in one_dim):
-                    da = da.drop_vars(one_dim)
-                da = da.drop_vars("elev")
-                da = da.to_array(name=var).squeeze("variable", drop=True)
-                da = da.transpose(da.dims[-1], da.dims[0], da.dims[1])
-                # print(da.dims)
-            if var == "elev":
-                da = da.to_array(name=var).squeeze("variable", drop=True)
-            data_arrays[var] = da
-            nested_data_arrays[dataset] = data_arrays
-
-            # Transpose the nested arrays dict. This is horrible to read, but works to pair up variables (ie mask) from the different granules
-    transposed_dict = {
-        inner_key: {
-            outer_key: inner_dict[inner_key]
-            for outer_key, inner_dict in nested_data_arrays.items()
-        }
-        for inner_key in nested_data_arrays[next(iter(nested_data_arrays))]
+    if isinstance(datasets, dict):
+        datasets = list(datasets.values())
+    if bounds is not None and hasattr(bounds, "total_bounds"):
+        bounds = tuple(bounds.total_bounds)
+
+    def _prep(ds):
+        ds = ds.copy()
+        if "elev" in ds.coords and "elev" not in ds.data_vars:
+            ds = ds.reset_coords("elev")
+        ds = ds.rename({"latitude": "y", "longitude": "x"})
+        # reorder dims - rioxarray.merge expects 3D arrays as (band, y, x)
+        for v in ds.data_vars:
+            if ds[v].ndim == 3:
+                band_dim = next(d for d in ds[v].dims if d not in ("y", "x"))
+                ds[v] = ds[v].transpose(band_dim, "y", "x")
+        return ds
+
+    prepped = [_prep(ds) for ds in datasets]
+    src = datasets[0]
+
+    shared_coords = {
+        n: src.coords[n]
+        for n in src.coords
+        if src.coords[n].ndim == 1
+        and n not in ("y", "x", "latitude", "longitude")
     }
 
-    # remove some unused data
-    del nested_data_arrays, data_arrays, da
-
-    # Merge the arrays using rioxarray.merge_arrays()
-    merged = {}
-    for _var in transposed_dict:
-        merged[_var] = merge_arrays(
-            list(transposed_dict[_var].values()),
-            bounds=gdf.unary_union.bounds,
-            nodata=-9999,
-        )
-
-    # Create a new xarray dataset from the merged arrays
-    # Create Merged Dataset
-    merged_ds = xr.Dataset(data_vars=merged, coords=one_d_arrays)
-    # Rename x and y to longitude and latitude
-    merged_ds = merged_ds.rename({"y": "latitude", "x": "longitude"})
-    del transposed_dict, merged
-    return merged_ds
+    merged = merge_datasets(prepped, bounds=bounds, nodata=nodata)
+    merged = merged.rename({"y": "latitude", "x": "longitude"})
+    merged = merged.assign_coords(shared_coords)
+
+    merged.attrs = dict(src.attrs)
+    for v in merged.data_vars:
+        if v in src.data_vars:
+            merged[v].attrs = dict(src[v].attrs)
+        elif v in src.coords:
+            merged[v].attrs = dict(src.coords[v].attrs)
+    for c in merged.coords:
+        if c in src.coords:
+            merged.coords[c].attrs = dict(src.coords[c].attrs)
+
+    if "elev" in merged.data_vars:
+        merged = merged.set_coords("elev")
+
+    # Restore (y, x, band) dim order
+    for v in merged.data_vars:
+        if merged[v].ndim == 3:
+            band_dim = next(
+                d for d in merged[v].dims if d not in ("latitude", "longitude")
+            )
+            merged[v] = merged[v].transpose("latitude", "longitude", band_dim)
+
+    # Set _FillValue
+    for v in merged.data_vars:
+        merged[v].encoding["_FillValue"] = nodata
+
+    return merged
 
 
 def ortho_browse(url, glt, spatial_ref, geotransform, white_background=True):