Automated COG generation for OMI Ozone dataset

[abarciauskas-bgse]

This set of steps should be generalizable to other datasets, but we will start with OMI Ozone.

We will deploy a workflow (AWS Step Function) per dataset. Each dataset workflow will be triggered to run on a schedule to discover new files, generate (and publish metadata for) COGs.

We will write infrastructure as code (CDK) to:

1. Deploy the workflow (step function state machine)
2. Trigger the workflow to run on a schedule (cloudwatch event rules)
3. Deploy as step 1 in the workflow the generate file URLs step
4. Deploy as step 2 in the workflow the generate (and publish STAC metadata) COG step

Tasks breakdown

Infra as code for workflow

• 1. Deploy a skeleton workflow
• 2. Deploy trigger to schedule workflow
• 3. Add steps in workflow to workflow deployment
• Update workflow to use processing script for OMDOAO3 version 003

There will need to be a way to trigger a parallel workflow for all files discovered in step 1 of the workflow. It looks like, for AWS Step Functions, there is a Map state type that can be used for this.

Steps in workflow

1. Docker which generates list of file URLs

• Write scripts which queries CMR for a given collection short name, version and temporal range and outputs list of file URLs
• Check it works for this dataset (OMDOAO3 version 003)

2. Docker which creates COG from file URL

• Check for any existing scripts which might be relevant in this repository and https://github.com/orgs/stactools-packages/repositories
• In-progress: Create script which takes file URL as input and generates COG
• Validate the output with a dataset expert
• In-progress: Write to S3

3. Docker which creates COG also publishes to STAC

• Check for any existing scripts which might be relevant in this repository and https://github.com/orgs/stactools-packages/repositories
• Write script which takes COG file or URL and writes to STAC API

Infrastructure details for workflow

Technologies proposed: CDK + AWS StepFunctions (Lambda, Fargate) + AWS CloudWatch

We are using the AWS SF Map State

Alternatives: Prefect (Cloud Agnostic), Batch (necessary?)

[abarciauskas-bgse]

Update January 11:

• I updated includes a workflow for OMI NO2 dataset which works to map inputs discovered to COG generation using a schedule and a trigger, to support ongoing discovery, COG generation (and eventually STAC generation)
• @jvntf is working on an update to the script for the OMDOAO3 version 003

Next steps cc @jvntf :

• integrate new code for OMDOAO3 version 003 into workflow and cdk deploy. Rename this workflow for the given collection
• add code to write to S3
• Understand STAC metadata specification
• Design STAC metadata Add ADR for CO2 stac api entry covid-api#147
• STAC metadata module
  • add stac generation code. Open question: should this be a part of the COG generation docker or separate step? It's a good separation of concerns to include it as a second step but implies that the lambda may have to read information from the COG over HTTP as opposed to the local file system.
  • AFAIK the STAC API is not ready for inputs yet, so this step should stub a request to a STAC API endpoint to post the STAC metadata
  • add stac generation step to workflow

[abarciauskas-bgse]

@jvntf in case you are new to the STAC spec, @anayeaye and @slesaad have already been designing STAC metadata and may have some other ideas to contribute to getting onboarded to designing STAC metadata for a new dataset but here is how I'm thinking to get started:

• Poke around https://stacspec.org/, specifically the link to https://github.com/radiantearth/stac-spec
• Look at some of the examples in "STAC in action"
• https://github.com/orgs/stactools-packages/repositories?type=all has examples of generating STAC
• ADR designing the STAC for OMI Ozone data

[abarciauskas-bgse]

@jvntf as discussed next steps are to:

• Pass along the granule_ids in the cmr-query handler.py#handler output to downstream tasks
• Add a new function which queries cmr with the granule_id and creates a STAC item

TODO: Where and how to handle collection STAC metadata generation

[abarciauskas-bgse]

Closing since we are moving forward with other datasets pipelines