Are you sure this is not a new requirement or bug?
Yes
💡 Description
Background
Currently in PDS Nucleus, each Airflow DAG is associated with a Product Completion Checker Lambda.
The Product Completion Checker Lambda queries the Nucleus database to identify a batch of products for which all expected files have been received. Once a batch is identified, the Lambda performs the preparation required to launch the downstream Airflow workflow. This includes activities such as:
- Selecting completed products from the Nucleus database
- Preparing the product batch
- Generating harvest configuration
- Generating manifest files
- Writing the generated configuration and artifacts to S3
- Invoking the appropriate Airflow DAG
The Airflow DAG then reads the generated configuration and executes the batch.
Conceptually, the current flow is:
Nucleus Database → Product Completion Checker Lambda → Generate Batch/Configuration → Invoke DAG → DAG Execution
Current Assumption
The current architecture assumes that the Product Completion Checker Lambda can be customized for the requirements of a particular DAG.
For example, a DAG developer may want to process only completed products whose files belong to a particular S3 prefix. Under the current approach, this could be implemented by creating a customized version of the Product Completion Checker Lambda with a DAG-specific SQL query or additional filtering logic.
This provides flexibility, but it also means that part of the DAG's workflow logic exists outside Airflow.
Problem
This creates an operational dependency between DAG development and Nucleus infrastructure deployment.
A DAG developer who needs custom product-selection or batch-preparation behavior may need to:
- Modify or create a Product Completion Checker Lambda.
- Obtain access to deploy the Lambda and its associated AWS resources, or
- Work with the PDS/Nucleus team to have the customized Lambda deployed.
While this model can work, it increases coupling between application-level DAG development and the underlying Nucleus infrastructure.
Ideally, a DAG developer should be able to develop, deploy, configure, and maintain a custom workflow primarily through Airflow DAGs, the Airflow UI, and DAG-related configuration constructs, without requiring a separate Lambda deployment for each variation in workflow behavior.
This becomes increasingly important as Nucleus supports more DAGs and more diverse processing requirements. Otherwise, each new DAG or customization can potentially introduce another Lambda variant that must be deployed and maintained as infrastructure.
Proposed Direction
Investigate an architecture that separates generic product-completion detection and DAG triggering from DAG-specific product-selection and batch-processing logic.
The goal would be to keep the Nucleus-managed component generic while allowing DAG developers to express workflow-specific behavior through Airflow.
A possible target model is:
Nucleus Database → Generic Completion/Trigger Mechanism → Airflow DAG → DAG-Specific Selection and Processing
Under this model, the Nucleus infrastructure would provide the common mechanisms required to identify or signal available work, while DAG-specific requirements—such as:
- Product filtering
- S3 prefix selection
- Batch composition
- Source-specific rules
- Harvest configuration
- Manifest generation
- Destination-specific behavior
could, where appropriate, be implemented or configured within the Airflow workflow.
Design Goal
The objective is not necessarily to eliminate the Product Completion Checker Lambda.
Instead, the objective is to minimize DAG-specific implementation inside infrastructure-managed Lambda functions and establish a clean boundary between:
Nucleus-managed infrastructure
and
DAG-developer-managed workflow logic
A successful approach should allow most new DAGs and DAG-specific processing requirements to be introduced without requiring the PDS/Nucleus infrastructure team to deploy a new Lambda implementation.
Key Question
How can Nucleus provide a generic product-completion and DAG-triggering mechanism while allowing DAG developers to define product selection, batching, and workflow-specific behavior using Airflow constructs rather than custom infrastructure?
This should be investigated with consideration for scalability, database query efficiency, concurrency, failure recovery, configuration management, and backward compatibility with existing Nucleus DAGs.
Are you sure this is not a new requirement or bug?
Yes
💡 Description
Background
Currently in PDS Nucleus, each Airflow DAG is associated with a Product Completion Checker Lambda.
The Product Completion Checker Lambda queries the Nucleus database to identify a batch of products for which all expected files have been received. Once a batch is identified, the Lambda performs the preparation required to launch the downstream Airflow workflow. This includes activities such as:
The Airflow DAG then reads the generated configuration and executes the batch.
Conceptually, the current flow is:
Nucleus Database → Product Completion Checker Lambda → Generate Batch/Configuration → Invoke DAG → DAG Execution
Current Assumption
The current architecture assumes that the Product Completion Checker Lambda can be customized for the requirements of a particular DAG.
For example, a DAG developer may want to process only completed products whose files belong to a particular S3 prefix. Under the current approach, this could be implemented by creating a customized version of the Product Completion Checker Lambda with a DAG-specific SQL query or additional filtering logic.
This provides flexibility, but it also means that part of the DAG's workflow logic exists outside Airflow.
Problem
This creates an operational dependency between DAG development and Nucleus infrastructure deployment.
A DAG developer who needs custom product-selection or batch-preparation behavior may need to:
While this model can work, it increases coupling between application-level DAG development and the underlying Nucleus infrastructure.
Ideally, a DAG developer should be able to develop, deploy, configure, and maintain a custom workflow primarily through Airflow DAGs, the Airflow UI, and DAG-related configuration constructs, without requiring a separate Lambda deployment for each variation in workflow behavior.
This becomes increasingly important as Nucleus supports more DAGs and more diverse processing requirements. Otherwise, each new DAG or customization can potentially introduce another Lambda variant that must be deployed and maintained as infrastructure.
Proposed Direction
Investigate an architecture that separates generic product-completion detection and DAG triggering from DAG-specific product-selection and batch-processing logic.
The goal would be to keep the Nucleus-managed component generic while allowing DAG developers to express workflow-specific behavior through Airflow.
A possible target model is:
Nucleus Database → Generic Completion/Trigger Mechanism → Airflow DAG → DAG-Specific Selection and Processing
Under this model, the Nucleus infrastructure would provide the common mechanisms required to identify or signal available work, while DAG-specific requirements—such as:
could, where appropriate, be implemented or configured within the Airflow workflow.
Design Goal
The objective is not necessarily to eliminate the Product Completion Checker Lambda.
Instead, the objective is to minimize DAG-specific implementation inside infrastructure-managed Lambda functions and establish a clean boundary between:
Nucleus-managed infrastructure
and
DAG-developer-managed workflow logic
A successful approach should allow most new DAGs and DAG-specific processing requirements to be introduced without requiring the PDS/Nucleus infrastructure team to deploy a new Lambda implementation.
Key Question
How can Nucleus provide a generic product-completion and DAG-triggering mechanism while allowing DAG developers to define product selection, batching, and workflow-specific behavior using Airflow constructs rather than custom infrastructure?
This should be investigated with consideration for scalability, database query efficiency, concurrency, failure recovery, configuration management, and backward compatibility with existing Nucleus DAGs.