Feature/sqs by sliu008 · Pull Request #360 · podaac/hydrocron

sliu008 · 2026-06-15T19:18:40Z

This pull request migrates the CNM (Collection Notification Message) and granule loading workflow from using direct Lambda invocations via SNS to a decoupled, queue-based architecture using Amazon SQS. This improves reliability, scalability, and error handling for message processing between components. The changes update both the application logic and Terraform infrastructure to support this new design.

Key changes include:

Infrastructure: Migration to SQS-based Messaging

Introduced new SQS queues and dead letter queues for both CNM and granule load messages, including event source mappings to trigger the respective Lambda functions from SQS instead of SNS or direct Lambda invocation. (terraform/hydrocron-sqs.tf)
Updated the SNS topic subscription to deliver CNM messages to the SQS queue (hydrocron_cnm_queue) instead of directly invoking the Lambda. (terraform/hydrocron-sns.tf)
Updated environment variables and IAM roles/policies to support SQS access, including permissions for sending messages and consuming from SQS. (terraform/hydrocron-lambda.tf, terraform/hydrocron-iam.tf) [1] [2] [3] [4] [5]

Application Logic: Decoupling and SQS Integration

Refactored the CNM handler to send granule load messages to SQS instead of invoking the granule load Lambda directly, and updated the granule handler to process messages from SQS. (hydrocron/db/load_data.py) [1] [2] [3]
Updated S3 file download calls to include the RequestPayer: requester argument for compatibility with requester-pays buckets. (hydrocron/db/io/swot_shp.py) [1] [2]

Dependency Updates

Added aiohttp as a new dependency in the Python project configuration. (pyproject.toml)

Copilot

Pull request overview

This PR migrates the CNM (Collection Notification Message) → granule loading flow from direct Lambda invocation to an SQS-based, decoupled architecture, and updates both Terraform and application code to support queue-triggered processing and DLQs.

Changes:

Added CNM and granule SQS queues + DLQs and wired Lambda event source mappings (SQS → Lambdas).
Updated SNS CNM response delivery to target SQS instead of invoking the CNM Lambda directly.
Refactored cnm_handler to enqueue granule-load work items to SQS; updated granule handler to accept SQS event shapes; added requester-pays args to S3 downloads; added aiohttp dependency.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`terraform/hydrocron-sqs.tf`	Introduces CNM + granule SQS queues, DLQs, queue policy for SNS→SQS, and event source mappings.
`terraform/hydrocron-sns.tf`	Switches CNM SNS subscription protocol from `lambda` to `sqs` (raw delivery enabled).
`terraform/hydrocron-lambda.tf`	Updates CNM lambda env var to `GRANULE_QUEUE_URL`; removes direct invoke permission from CNM to granule lambda.
`terraform/hydrocron-iam.tf`	Adds SQS send policy for CNM lambda; adds SQS execution managed policy for granule loader role; removes SNS→Lambda permission.
`hydrocron/db/load_data.py`	Updates handlers for SQS event shapes; CNM handler now sends granule-load messages to SQS.
`hydrocron/db/io/swot_shp.py`	Adds `RequestPayer=requester` to S3 `download_file` calls.
`pyproject.toml`	Adds `aiohttp` dependency.
`CHANGELOG.md`	Documents the SQS migration and requester-pays change (wording needs cleanup).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sliu008 · 2026-06-15T21:34:34Z

+    if 'Records' in event:
+        record = json.loads(event['Records'][0]['body'])
+        body = record['body']
+    else:
+        body = event['body']


added tests

sliu008 · 2026-06-15T21:34:29Z

@@ -186,19 +192,23 @@ def cnm_handler(event, _):
                else:
                    raise MissingTable(f"Error: Cannot load granule: {granule_uri}")

-                event2 = ('{"body": {"granule_path": "' + granule_uri
-                          + '","table_name": "' + table_name
-                          + '","track_table": "' + track_table
-                          + '","checksum": "' + checksum
-                          + '","revisionDate": "' + revision_date
-                          + '","load_benchmarking_data": "' + load_benchmarking_data + '"}}')
-
-                logging.info("Invoking granule load lambda with event json %s", str(event2))
-
-                lambda_client.invoke(
-                    FunctionName=os.environ['GRANULE_LAMBDA_FUNCTION_NAME'],
-                    InvocationType='Event',
-                    Payload=event2)
+                msg_body = json.dumps({
+                    "body": {
+                        "granule_path": granule_uri,
+                        "table_name": table_name,
+                        "track_table": track_table,
+                        "checksum": checksum,
+                        "revisionDate": revision_date,
+                        "load_benchmarking_data": load_benchmarking_data
+                    }
+                })
+
+                logging.info("Sending granule load message to SQS: %s", msg_body)
+
+                sqs_client.send_message(
+                    QueueUrl=queue_url,
+                    MessageBody=msg_body
+                )


added tests

sliu008 · 2026-06-15T21:34:25Z

        if filepath.startswith('s3'):
            bucket_name, key = filepath.replace("s3://", "").split("/", 1)
            logging.info("Opening granule %s from bucket %s", key, bucket_name)

-            s3_resource.Bucket(bucket_name).download_file(key, lambda_temp_file)
+            s3_resource.Bucket(bucket_name).download_file(key, lambda_temp_file, ExtraArgs={"RequestPayer": "requester"})



added tests

sliu008 · 2026-06-15T21:22:18Z

 moto = "^5.0.9"
 vcrpy = "^8.0.0"
 python-cmr = "^0.13.0"
+aiohttp = "3.13.5"



aiohttp latest breaks another library so im pinning it to the newest one possible

tloubrieu-jpl · 2026-06-22T12:12:37Z

Hi @sliu008 , @jamesfwood , do we have an issue to track that activity ?
Thanks

sliu008 added 15 commits May 13, 2026 15:17

update aws provider to 5.0

95fb90c

update changelog

00d3482

removed lifecycle for iam policy

f89ad0f

update aws provider to 6.0

dde303c

fix attachment of policys

5b54a4d

poetry update

748a639

update so test runs later

644fce3

update so sonar cloud after tests

3449023

ignore snyk for now as aiohttp conflicts with vcrpy

f10999d

update required provider syntax

d411871

add an sqs so we trigger lambda from sns to sqs to lambda

2ea6384

update how load data from sqs now instead of sns

8e316df

add a second queue that triggers the load granule lambda

5efdc49

add requester pay

a764389

poetry update

c0b86d4

sliu008 requested a review from jamesfwood June 15, 2026 19:18

tloubrieu-jpl added this to podaac Jun 15, 2026

tloubrieu-jpl moved this to needs:triage in podaac Jun 15, 2026

update changelog

50f0f50

sliu008 mentioned this pull request Jun 15, 2026

Permission Issue with Requestor Pays in ops vs uat vs sit #358

Open

sliu008 temporarily deployed to SIT June 15, 2026 19:53 — with GitHub Actions Inactive

jamesfwood approved these changes Jun 15, 2026

View reviewed changes

jamesfwood requested a review from Copilot June 15, 2026 21:05

Copilot started reviewing on behalf of jamesfwood June 15, 2026 21:05 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

update changelog and add test for sqs and requester pay

97db994

sliu008 temporarily deployed to SIT June 15, 2026 21:56 — with GitHub Actions Inactive

Merge branch 'develop' into feature/sqs

6d09175

merge develop

4fc1090

sliu008 temporarily deployed to SIT June 23, 2026 17:14 — with GitHub Actions Inactive

merge in aws provider update

9351958

sliu008 deployed to SIT June 23, 2026 17:54 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/sqs#360

Feature/sqs#360
sliu008 wants to merge 20 commits into
developfrom
feature/sqs

sliu008 commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

sliu008 Jun 15, 2026

Uh oh!

sliu008 Jun 15, 2026

Uh oh!

sliu008 Jun 15, 2026

Uh oh!

sliu008 Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

tloubrieu-jpl commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sliu008 commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

sliu008 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

sliu008 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

sliu008 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

sliu008 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tloubrieu-jpl commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants