Standardize unioning macros to use fivetran_utils versions

CHANGELOG.md

@@ -1,3 +1,21 @@
+# dbt_amazon_ads v1.3.0
+
+[PR #49](https://github.com/fivetran/dbt_amazon_ads/pull/49) includes the following updates:
+
+## Schema/Data Changes (--full-refresh required after upgrading)
+**1 total change • 1 possible breaking change**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| All models | `source_relation` column (when using a single amazon_ads schema) | Empty string (`''`) | `<database>.<schema>` |  |
+
+## Feature Updates
+- Introduces the new (recommended) `amazon_ads_sources` variable for more robust union data configuration. The old`amazon_ads_union_schemas` and `amazon_ads_union_databases` variables will still be supported. See the [README](https://github.com/fivetran/dbt_amazon_ads/tree/main#define-database-and-schema-variables) for specific details.
+
+## Under the Hood
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_amazon_ads/#source-casing-for-case-sensitive-destinations) section of the README for details.
+- Introduces `fivetran_utils.partition_by_source_relation` to conditionally include `source_relation` in partition clauses only when multiplesources are configured.
+
 # dbt_amazon_ads v1.2.0
 
 [PR #32](https://github.com/fivetran/dbt_amazon_ads/pull/32) includes the following updates:
@@ -125,4 +143,4 @@ If you are using dbt Core < 1.9.6 and want to continue running Amazon Ads freshn
 - Provides insight into your ad performance across the following grains:
   - Account, portfolio, campaign, ad group, ad, keyword, and search term
 - Materializes output models designed to work simultaneously with our [multi-platform Ad Reporting package](https://github.com/fivetran/dbt_ad_reporting).
-- Generates a comprehensive data dictionary of your source and modeled Amazon Ads data through the [dbt docs site](https://fivetran.github.io/dbt_amazon_ads/).
+- Generates a comprehensive data dictionary of your source and modeled Amazon Ads data through the [dbt docs site](https://fivetran.github.io/dbt_amazon_ads/).

README.md

@@ -74,7 +74,7 @@ Include the following amazon_ads package version in your `packages.yml` file _if
 ```yaml
 packages:
   - package: fivetran/amazon_ads
-    version: [">=1.2.0", "<1.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/amazon_ads_source` in your `packages.yml` since this package has been deprecated.
@@ -88,36 +88,51 @@ dispatch:
 ```
 
 ### Define database and schema variables
-By default, this package uses your destination and the `amazon_ads` schema. If your Amazon Ads data is in a different database or schema (for example, if your Amazon Ads schema is named `amazon_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
+#### Option A: Single connection
+By default, this package runs using your destination and the `amazon_ads` schema. If this is not where your Amazon Ads data is (for example, if your Amazon Ads schema is named `amazon_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
     amazon_ads_database: your_destination_name
-    amazon_ads_schema: your_schema_name 
+    amazon_ads_schema: your_schema_name
 ```
 
-### Disable models for non-existent sources
-Your Amazon Ads connection may not sync every table that this package expects. If you do not have the `PORTFOLIO_HISTORY` table synced, add the following variable to your root `dbt_project.yml` file:
+#### Option B: Union multiple connections
+If you have multiple Amazon Ads connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.
+
+To use this functionality, you will need to set the `amazon_ads_sources` variable in your root `dbt_project.yml` file:
 
 ```yml
+# dbt_project.yml
+
 vars:
-    amazon_ads__portfolio_history_enabled: False   # Disable if you do not have the portfolio table. Default is True.
+  amazon_ads:
+    amazon_ads_sources:
+      - database: connection_1_destination_name # Required
+        schema: connection_1_schema_name # Required
+        name: connection_1_source_name # Required only if following the step in the following subsection
+
+      - database: connection_2_destination_name
+        schema: connection_2_schema_name
+        name: connection_2_source_name
 ```
 
-### (Optional) Additional configurations
-<details open><summary>Expand/Collapse details</summary>
+> Previous versions of this package employed two separate, mutually exclusive variables for unioning: `amazon_ads_union_schemas` and `amazon_ads_union_databases`. While these variables are still supported, `amazon_ads_sources` is the recommended variable to configure.
+
+#### Optional: Incorporate unioned sources into DAG
 
-#### Union multiple connections
-If you have multiple amazon_ads connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `amazon_ads_union_schemas` OR `amazon_ads_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Amazon Ads connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_amazon_ads/blob/main/models/staging/src_amazon_ads.yml). Set the variable `has_defined_sources: true` under the Amazon Ads namespace in your `dbt_project.yml`. Otherwise, your Amazon Ads connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
+
+### Disable models for non-existent sources
+Your Amazon Ads connection may not sync every table that this package expects. If you do not have the `PORTFOLIO_HISTORY` table synced, add the following variable to your root `dbt_project.yml` file:
 
 ```yml
 vars:
-    amazon_ads_union_schemas: ['amazon_ads_usa','amazon_ads_canada'] # use this if the data is in different schemas/datasets of the same database/project
-    amazon_ads_union_databases: ['amazon_ads_usa','amazon_ads_canada'] # use this if the data is in different databases/projects but uses the same schema name
+    amazon_ads__portfolio_history_enabled: False   # Disable if you do not have the portfolio table. Default is True.
 ```
-> NOTE: The native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
 
-To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
+### (Optional) Additional configurations
+<details open><summary>Expand/Collapse details</summary>
 
 #### Passing Through Additional Metrics
 By default, this package will select `clicks`, `impressions`, `cost`, `purchases_30_d`, and `sales_30_d` from the source reporting tables to store into the staging and end models. If you would like to pass through additional metrics to the package models, add the following configurations to your `dbt_project.yml` file. These variables allow the pass-through fields to be aliased (`alias`) if desired, but not required. Use the following format for declaring the respective pass-through variables:
@@ -165,6 +180,14 @@ vars:
     amazon_ads_<default_source_table_name>_identifier: your_table_name 
 ```
 
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
+
 </details>
 
 ### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'amazon_ads'
-version: '1.2.0'
+version: '1.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 models:

integration_tests/ci/test_scenarios.yml

@@ -20,4 +20,10 @@ test_scenarios:
   - name: "portfolio history disabled"
     vars:
       amazon_ads__portfolio_history_enabled: false
-    include_incremental: false
+    include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'amazon_ads_integration_tests'
-version: '1.2.0'
+version: '1.3.0'
 
 profile: 'integration_tests' 
 
@@ -39,7 +39,7 @@ dispatch:
 seeds:
   +docs:
     show: False
-  +quote_columns: "{{ true if target.type == 'redshift' else false }}"
+  +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
   amazon_ads_integration_tests:
     ad_group_level_report_data:
       +column_types:

integration_tests/seeds/advertised_product_report_data.csv

@@ -1,4 +1,4 @@
-ad_group_id,ad_id,campaign_id,date,_fivetran_synced,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions, advertised_asin, advertised_sku
+ad_group_id,ad_id,campaign_id,date,_fivetran_synced,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions,advertised_asin,advertised_sku
 501,8108,5555,2022-12-10,2022-12-13 17:17:13.167000,250.0,USD,DAILY_BUDGET,2,2.53,166,,
 501,8108,5555,2022-12-06,2022-12-13 17:17:13.017000,250.0,USD,DAILY_BUDGET,3,5.29,75,,
 421,77,2187,2022-08-20,2022-10-14 11:39:03.648000,2000.0,USD,DAILY_BUDGET,3,3.79,1310,,

models/staging/src_amazon_ads.yml

@@ -9,7 +9,7 @@ sources:
 
     config:
       loaded_at_field: _fivetran_synced
-      enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) }}"
+      enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) and var('amazon_ads_sources', []) == [] }}"
       freshness: 
         warn_after: {count: 48, period: hour}
         error_after: {count: 168, period: hour}
@@ -203,7 +203,7 @@ sources:
         description: Each record represents a portfolio in Amazon Ads.
         identifier: "{{ var('amazon_ads_portfolio_history_identifier', 'portfolio_history') }}"
         config:
-          enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) and var('amazon_ads__portfolio_history_enabled', true) }}"
+          enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) and var('amazon_ads__portfolio_history_enabled', true) and var('amazon_ads_sources', []) == [] }}"
         columns:
           - name: _fivetran_synced
             description: "{{ doc('_fivetran_synced') }}"
@@ -366,4 +366,4 @@ sources:
           - name: purchases_30_d
             description: "{{ doc('purchases_30_d') }}" 
           - name: sales_30_d
-            description: "{{ doc('sales_30_d') }}"
+            description: "{{ doc('sales_30_d') }}"

models/staging/stg_amazon_ads__ad_group_history.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -36,9 +33,9 @@ final as (
         name as ad_group_name,
         serving_status,
         state,
-        row_number() over (partition by source_relation, id order by last_updated_date desc) = 1 as is_most_recent_record
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='amazon_ads') }} order by last_updated_date desc) = 1 as is_most_recent_record
     from fields
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__ad_group_level_report.sql

@@ -15,10 +15,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -42,4 +39,4 @@ final as (
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__advertised_product_report.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -49,4 +46,4 @@ final as (
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__campaign_history.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -43,9 +40,9 @@ final as (
         budget,
         budget_type,
         effective_budget,
-        row_number() over (partition by source_relation, id order by last_updated_date desc) = 1 as is_most_recent_record
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='amazon_ads') }} order by last_updated_date desc) = 1 as is_most_recent_record
     from fields
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__campaign_level_report.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -49,4 +46,4 @@ final as (
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__keyword_history.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -40,9 +37,9 @@ final as (
         serving_status,
         state,
         native_language_locale,
-        row_number() over (partition by source_relation, id order by last_updated_date desc) = 1 as is_most_recent_record
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='amazon_ads') }} order by last_updated_date desc) = 1 as is_most_recent_record
     from fields
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__portfolio_history.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -41,9 +38,9 @@ final as (
         cast(profile_id as {{ dbt.type_string() }}) as profile_id,
         serving_status,
         state,
-        row_number() over (partition by source_relation, id order by last_updated_date desc) = 1 as is_most_recent_record
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='amazon_ads') }} order by last_updated_date desc) = 1 as is_most_recent_record
     from fields
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__product_ad_history.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -37,9 +34,9 @@ final as (
         serving_status,
         sku,
         state,
-        row_number() over (partition by source_relation, id order by last_updated_date desc) = 1 as is_most_recent_record
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='amazon_ads') }} order by last_updated_date desc) = 1 as is_most_recent_record
     from fields
 )
 
 select *
-from final
+from final

models/staging/stg_amazon_ads__profile.sql

@@ -16,10 +16,7 @@ fields as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='amazon_ads_union_schemas', 
-            union_database_variable='amazon_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='amazon_ads') }}
 
     from base
 ),
@@ -44,4 +41,4 @@ final as (
 )
 
 select *
-from final
+from final