Release 25.10#101
Merged
Merged
Conversation
+ Use Amazon Linux 2023
- S3 Loader RAW - Postgres Loader (good + bad) - Snowflake Loader (RDB -> replaced with Streaming)
There was a problem hiding this comment.
Pull Request Overview
This PR updates AWS Terraform modules to their latest versions and removes deprecated PostgreSQL and legacy Snowflake loader support. The changes prepare the infrastructure for the upcoming Amazon Linux 2023 migration ahead of next year's deprecation date.
Key changes:
- Updated core pipeline modules (collector, enrichment, loaders) to latest versions
- Removed PostgreSQL database loader support and related configurations
- Removed legacy Snowflake loader in favor of Snowflake Streaming loader
- Removed S3 raw data loader functionality
Reviewed Changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| terraform/aws/pipeline/secure/variables.tf | Removed variables for PostgreSQL, legacy Snowflake, and S3 raw loader configurations |
| terraform/aws/pipeline/secure/terraform.tfvars | Removed configuration values for deprecated PostgreSQL, legacy Snowflake, and S3 raw loader features |
| terraform/aws/pipeline/secure/target_snowflake_streaming.tf | Updated Snowflake Streaming loader module from 0.1.0 to 0.2.1 |
| terraform/aws/pipeline/secure/target_snowflake.tf | Removed entire legacy Snowflake loader implementation |
| terraform/aws/pipeline/secure/target_redshift.tf | Updated Redshift transformer (0.4.0→0.5.0) and loader (0.2.0→0.4.0) modules, added explicit instance types |
| terraform/aws/pipeline/secure/target_postgres.tf | Removed entire PostgreSQL loader implementation including RDS module |
| terraform/aws/pipeline/secure/target_databricks.tf | Updated Databricks transformer (0.4.0→0.5.0) and loader (0.2.0→0.3.0) modules, added explicit instance types |
| terraform/aws/pipeline/secure/target_amazon_s3.tf | Removed S3 raw loader module, updated S3 loader modules from 0.5.0 to 0.6.0 |
| terraform/aws/pipeline/secure/outputs.tf | Removed PostgreSQL-related output variables |
| terraform/aws/pipeline/secure/main.tf | Updated collector (0.9.0→0.10.1) and enrichment (0.6.0→0.7.1) modules, removed app_version pins |
| terraform/aws/pipeline/secure/README.md | Updated documentation reflecting module version changes and removed deprecated features |
| terraform/aws/pipeline/default/* | Applied identical changes as secure pipeline for default configuration |
| terraform/aws/iglu_server/secure/main.tf | Updated Iglu Server module from 0.5.0 to 0.6.1, removed app_version pin |
| terraform/aws/iglu_server/default/main.tf | Updated Iglu Server module from 0.5.0 to 0.6.1, removed app_version pin |
| terraform/aws/iglu_server/*/README.md | Updated documentation to reflect RDS module upgrade to 0.5.0 and Iglu Server to 0.6.1 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This module is now deprecated and we will no longer be promoting it.
Collector v3.7.0 Enrich v6.2.1 BigQuery loader v2.0.1
Updates to support Postgres Servers in Flexible mode
Also updates to have a valid backend Postgres Server
Use Ubuntu 24.04 Collector v3.7.0 Enrich v6.2.1 Lake Loader v0.7.0
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 56 out of 56 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates all cloud modules to the latest available versions, enables Collector Compression, removes deprecated loaders (RDB Snowflake + Postgres), updates all remaining loaders to latest available versions.
This also sees us ensure new base images are used (Amazon Linux 2023 for AWS and Ubuntu 24.04 for Azure + GCP).
Nick (@stanch) let me know when the docs are ready to rock and roll and I will review them so we can merge them together!