Add params to allow alert routing to Teams via logic app

[rgwenter-mobi]

Context

migrate azure monitoring to teams

We use Azure monitor for postgres and redis alerting, which goes to an infra email address, we want the alerts to go to the relevant services alert teams channel.

Changes proposed in this pull request

This change is required for PR Migrate Azure Monitoring Notifications to Teams, it adds parameters to the alert payload to allow routing within a logic app to the appropriate teams channel. This PR will generate the parameters for each service so they are included in payloads but it will not have any effect until they are consumed by the logic app (or other services).

Guidance to review

To validate, ensure that terraform plans show the addition of the correct parameters to alerts, namely the service short code and environment name.

Edit global_config/.sh so TERRAFORM_MODULES_TAG=2677-logicapp-for-teams-notifications

make <environment> terraform-plan

Ensure that changes are to alerts and payload, the relevant services short code is used and correct environment.

[rgwenter-mobi]

Test plan for apply-for-teacher-training production:

For each Azure alert, action is updated to include webhook_properties as per below (requires replacement of action),

• Ensure Action group ID remains the same.
• target_channels set to "att"
• environment set to "production"

Terraform will perform the following actions:

  # module.postgres.azurerm_monitor_metric_alert.cpu[0] will be updated in-place
  ~ resource "azurerm_monitor_metric_alert" "cpu" {
        id                       = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-pd-rg/providers/Microsoft.Insights/metricAlerts/s189p01-att-production-psql-cpu"
        name                     = "s189p01-att-production-psql-cpu"
        tags                     = {
            "Environment"      = "Prod"
            "Product"          = "Apply for postgraduate teacher training"
            "Service Offering" = "Apply for postgraduate teacher training"
        }
        # (10 unchanged attributes hidden)

      - action {
          - action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply" -> null
          - webhook_properties = {} -> null
        }
      + action {
          + action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply"
          + webhook_properties = {
              + "environment"     = "production"
              + "target_channels" = "att"
            }
        }

        # (1 unchanged block hidden)
    }

  # module.postgres.azurerm_monitor_metric_alert.memory[0] will be updated in-place
  ~ resource "azurerm_monitor_metric_alert" "memory" {
        id                       = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-pd-rg/providers/Microsoft.Insights/metricAlerts/s189p01-att-production-psql-memory"
        name                     = "s189p01-att-production-psql-memory"
        tags                     = {
            "Environment"      = "Prod"
            "Product"          = "Apply for postgraduate teacher training"
            "Service Offering" = "Apply for postgraduate teacher training"
        }
        # (10 unchanged attributes hidden)

      - action {
          - action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply" -> null
          - webhook_properties = {} -> null
        }
      + action {
          + action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply"
          + webhook_properties = {
              + "environment"     = "production"
              + "target_channels" = "att"
            }
        }

        # (1 unchanged block hidden)
    }

  # module.postgres.azurerm_monitor_metric_alert.storage[0] will be updated in-place
  ~ resource "azurerm_monitor_metric_alert" "storage" {
        id                       = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-pd-rg/providers/Microsoft.Insights/metricAlerts/s189p01-att-production-psql-storage"
        name                     = "s189p01-att-production-psql-storage"
        tags                     = {
            "Environment"      = "Prod"
            "Product"          = "Apply for postgraduate teacher training"
            "Service Offering" = "Apply for postgraduate teacher training"
        }
        # (10 unchanged attributes hidden)

      - action {
          - action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply" -> null
          - webhook_properties = {} -> null
        }
      + action {
          + action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply"
          + webhook_properties = {
              + "environment"     = "production"
              + "target_channels" = "att"
            }
        }

        # (1 unchanged block hidden)
    }

  # module.redis-cache.azurerm_monitor_metric_alert.memory[0] will be updated in-place
  ~ resource "azurerm_monitor_metric_alert" "memory" {
        id                       = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-pd-rg/providers/Microsoft.Insights/metricAlerts/s189p01-att-production-redis-cache-memory"
        name                     = "s189p01-att-production-redis-cache-memory"
        tags                     = {
            "Environment"      = "Prod"
            "Product"          = "Apply for postgraduate teacher training"
            "Service Offering" = "Apply for postgraduate teacher training"
        }
        # (10 unchanged attributes hidden)

      - action {
          - action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply" -> null
          - webhook_properties = {} -> null
        }
      + action {
          + action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply"
          + webhook_properties = {
              + "environment"     = "production"
              + "target_channels" = "att"
            }
        }

        # (1 unchanged block hidden)
    }

  # module.redis-queue.azurerm_monitor_metric_alert.memory[0] will be updated in-place
  ~ resource "azurerm_monitor_metric_alert" "memory" {
        id                       = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-pd-rg/providers/Microsoft.Insights/metricAlerts/s189p01-att-production-redis-queue-memory"
        name                     = "s189p01-att-production-redis-queue-memory"
        tags                     = {
            "Environment"      = "Prod"
            "Product"          = "Apply for postgraduate teacher training"
            "Service Offering" = "Apply for postgraduate teacher training"
        }
        # (10 unchanged attributes hidden)

      - action {
          - action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply" -> null
          - webhook_properties = {} -> null
        }
      + action {
          + action_group_id    = "/subscriptions/3c033a0c-7a1c-4653-93cb-0f2a9f57a391/resourceGroups/s189p01-att-mn-rg/providers/Microsoft.Insights/actionGroups/s189p01-apply"
          + webhook_properties = {
              + "environment"     = "production"
              + "target_channels" = "att"
            }
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

[RMcVelia]

why not just set alert_target_channels to var.service_short?

[rgwenter-mobi]

over engineering, it was going to allow us to manage multiple targets more easily (eg alert to Infra and service) but that was going to be more work than wanted for this. Will change to use var directly.