[Incident] NSG Security Rule Change — allow-web-inbound Opens Internet Access to Private AKS Subnet (nsg-aks-auf6gw) #2

Summary

An Azure Monitor alert (Zava-nsg-change) fired at 2026-06-22T18:39:02 UTC indicating that a new NSG security rule allow-web-inbound was created or modified on nsg-aks-auf6gw in rg-zava-aks-postgres. This rule opens TCP ports 80 and 443 from the Internet to the entire AKS subnet (10.0.0.0/16), which contradicts the private cluster design of aks-Zava-auf6gw.

Risk Level: HIGH — This change reduces the defense-in-depth security posture of a private AKS cluster by allowing unsolicited inbound Internet traffic to the node subnet.

Impact

Timeline

Evidence

NSG Rule Details (Current State)

New Rule: allow-web-inbound

Previous State (from memory/prior incidents): The NSG nsg-aks-auf6gw had only ONE custom rule: allow-all-outbound (priority 1000, allow all outbound). All inbound traffic from the Internet was denied by the default DenyAllInBound rule (priority 65500).

Complete NSG Rule Table (After Change)

AKS Cluster Configuration

Network Architecture

• Service Endpoint: Microsoft.Storage on aks-subnet
• AKS Load Balancer: Standard SKU, 1 managed outbound public IP

Application Health at Time of Alert

Kubernetes Events (Last 30m)

• ingress-nginx-controller pod has FailedMount for webhook-cert secret (pre-existing, unrelated to NSG change)
• No other failure events detected

Caller Identification

Unable to determine — AzureActivity logs are not configured for ingestion into the Log Analytics workspace (69f6cc9c-c59f-4624-8b95-e12b93220ece). The workspace contains Kubernetes telemetry (KubePodInventory, ContainerLog, AppRequests, etc.) but no AzureActivity table. This is a significant observability gap.

Root Cause

A new NSG security rule allow-web-inbound was created on nsg-aks-auf6gw, attached to the AKS subnet (aks-subnet, 10.0.0.0/16). This rule allows inbound TCP traffic on ports 80 and 443 from the Internet service tag to all destinations (*) within the subnet.

Key security concern: The AKS cluster aks-Zava-auf6gw is designed as a private cluster — the API server is only accessible via private endpoint, and nodes have no public IPs. The new NSG rule contradicts this security design by opening the AKS node subnet to internet-sourced traffic on web ports.

Mitigating factors:

• AKS nodes do not have public IP addresses (enableNodePublicIP: false)
• Azure Network Policy is enabled at the cluster level
• Microsoft Defender for Containers is active
• No immediate application impact observed (zero 5xx errors)

Risk factors:

• The ingress-nginx controller is deployed — if a LoadBalancer Service with a public IP exists, the NSG rule allows internet traffic through to nginx pods
• Destination * is overly broad — applies to ALL resources in the /16 subnet
• Source Internet allows traffic from any public IP worldwide
• This change was not coordinated through known deployment pipelines

Remediation

Immediate Actions

1. Identify the change author — Enable AzureActivity diagnostic settings to the Log Analytics workspace, then query for the correlationId of this NSG rule creation
2. If unauthorized: Remove the rule:

   az network nsg rule delete \
     --resource-group rg-zava-aks-postgres \
     --nsg-name nsg-aks-auf6gw \
     --name allow-web-inbound \
     --subscription f627598e-05c5-4093-8667-5730c4026ea3

3. If legitimate: Restrict the source to specific IP ranges (corporate CIDR, CDN) and scope the destination to specific load balancer IPs

Long-Term Actions

1. Enable AzureActivity log ingestion to the Log Analytics workspace for full audit trail
2. Implement Azure Policy to deny NSG rules with source Internet on private AKS subnets
3. Review ingress architecture — ensure traffic flows through a controlled path (Application Gateway, Azure Front Door) rather than direct NSG rules
4. Enable NSG Flow Logs on nsg-aks-auf6gw for traffic auditing

Action Items

• Identify the actor who created allow-web-inbound (requires AzureActivity diagnostic setting)
• Determine if ports 80/443 inbound are a legitimate requirement for this environment
• If legitimate: restrict source IPs and destination addresses
• If unauthorized: delete the rule immediately
• Configure AzureActivity diagnostic settings for rg-zava-aks-postgres
• Evaluate Azure Policy for NSG rule governance on private AKS subnets
• Enable NSG Flow Logs on nsg-aks-auf6gw
• Resolve pre-existing ingress-nginx-controller FailedMount (webhook-cert secret missing)

References

• Alert ID: 1a5e8b12-9642-671d-8938-da40909af000
• Alert Rule: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/microsoft.insights/activityLogAlerts/Zava-nsg-change
• NSG Resource ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/Microsoft.Network/networkSecurityGroups/nsg-aks-auf6gw
• NSG Rule Resource ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/Microsoft.Network/networkSecurityGroups/nsg-aks-auf6gw/securityRules/allow-web-inbound
• AKS Resource ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/Microsoft.ContainerService/managedClusters/aks-Zava-auf6gw
• VNet Resource ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/Microsoft.Network/virtualNetworks/vnet-Zava-auf6gw
• Log Analytics Workspace ID: 69f6cc9c-c59f-4624-8b95-e12b93220ece
• Log Analytics ARM ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-zava-aks-postgres/providers/Microsoft.OperationalInsights/workspaces/law-Zava-auf6gw
• App Insights: ai-Zava-auf6gw (rg-zava-aks-postgres)
• Subscription: f627598e-05c5-4093-8667-5730c4026ea3
• Resource Group: rg-zava-aks-postgres
• Region: Sweden Central
• Azure Portal Alert: View Alert

This issue was created by sre-agent-jkazytu5kl5py--70975bf6
Tracked by the SRE agent here