Are you sure this is not a new requirement or bug?
Yes
Task Type
Sub-task
💡 Description
Description:
The existing pipeline has some possible bottlenecks that may prevent it from handling terabytes to petabytes-scale PDS data ingestion efficiently in future. This task will address the core scalability and correctness issues.
Planned changes:
Per-node databases — migrate from a single shared pds_nucleus database to dedicated databases per PDS node within the same Aurora cluster, isolating query load and failure domains at no additional cost.
Archive tables — move COMPLETE products to product_archive and product_data_file_mapping_archive after dispatch, keeping the active tables bounded to only in-flight products so the completion checker query performance does not degrade over time.
Batch DB writes — replace per-file execute_statement calls with a single batch_execute_statement per XML label, reducing RDS API round-trips from N+1 to 2 regardless of how many files a product references.
Batch size 50 → 500 — increase the completion checker batch size and add a drain loop so each Lambda invocation processes up to 500 products at a time and continues until the queue is empty, clearing 10× more products per minute.
DISPATCHING status — introduce an intermediate DISPATCHING state so products are stamped before the DAG is triggered and reset to INCOMPLETE on failure, preventing duplicate DAG runs when two Lambda invocations overlap or when the trigger fails mid-execution.
Are you sure this is not a new requirement or bug?
Yes
Task Type
Sub-task
💡 Description
Description:
The existing pipeline has some possible bottlenecks that may prevent it from handling terabytes to petabytes-scale PDS data ingestion efficiently in future. This task will address the core scalability and correctness issues.
Planned changes:
Per-node databases — migrate from a single shared pds_nucleus database to dedicated databases per PDS node within the same Aurora cluster, isolating query load and failure domains at no additional cost.
Archive tables — move COMPLETE products to product_archive and product_data_file_mapping_archive after dispatch, keeping the active tables bounded to only in-flight products so the completion checker query performance does not degrade over time.
Batch DB writes — replace per-file execute_statement calls with a single batch_execute_statement per XML label, reducing RDS API round-trips from N+1 to 2 regardless of how many files a product references.
Batch size 50 → 500 — increase the completion checker batch size and add a drain loop so each Lambda invocation processes up to 500 products at a time and continues until the queue is empty, clearing 10× more products per minute.
DISPATCHING status — introduce an intermediate DISPATCHING state so products are stamped before the DAG is triggered and reset to INCOMPLETE on failure, preventing duplicate DAG runs when two Lambda invocations overlap or when the trigger fails mid-execution.