diff --git a/azure/README.md b/azure/README.md index 0dfd3fd..681ff5c 100644 --- a/azure/README.md +++ b/azure/README.md @@ -580,6 +580,41 @@ onboarding turns on. You may start seeing 403s from Network Watcher in the accou once the firewall is on, which Azure documents as its normal behavior: it tries a key first, and falls back to a token when that is refused. +### Reading them from inside the region + +The list above cannot help a reader in the storage account's own region, and a private endpoint +is how you admit one. It gives the account an address inside this landing pad's VNet, so a +collector or a virtual machine there reaches the records over Azure's backbone and never over the +public endpoint the firewall guards: + +```hcl +create_flow_log_private_endpoint = true +``` + +On the ARM path that is `CREATE_FLOW_LOG_PRIVATE_ENDPOINT=true`. It is off by default because a +private endpoint bills hourly plus per GB, where everything else in this section is free. + +With it on you get the endpoint, the private DNS zone `privatelink.blob.core.windows.net` linked +to this VNet, and the zone group that keeps the account's blob name resolving to the endpoint's +address. All three go in the Network Watcher's resource group, beside the storage account. + +The subnet they need is separate, and it is carved as soon as you turn flow logs on, whether or +not you ask for an endpoint. It takes the `/24` above the gateway range, it is empty until an +endpoint lands in it, and Azure bills nothing for an empty subnet. The endpoint gets a subnet of +its own rather than sharing the workstations' one, because everything in a subnet can reach a +private endpoint in it at the network layer, and these records are the audit trail of what those +workstations did. + +Turn it on if a compliance scan asks for private endpoints on storage accounts by name, which +several do. + +**Turning it off again does not remove the endpoint.** The endpoint is deployed by an ARM +template, and ARM's incremental mode never deletes a resource it stops declaring, so setting +`create_flow_log_private_endpoint = false` leaves the endpoint, the zone and the link in place and +still billing. Delete them yourself, and delete the zone only once nothing else resolves through +it. Setting `create_flow_logs = false` is the clean path: that removes the whole deployment, the +endpoint with it, and then the storage account and every record in it. + ## A second region: name it, do not renumber it An Azure VNet is regional. A second region means a second VNet joined by **global VNet @@ -617,11 +652,15 @@ cannot read the other region's state. Keep one map, shared. Every subnet is carved out of whichever `/16` the VNet took, so there is nothing else to keep in step: -| index | region | `vnet_address_space` | `subnet_prefix` | `governed_subnet_prefix` | `gateway_subnet_prefix` | -|---|---|---|---|---|---| -| `0` | first | `10.70.0.0/16` | `10.70.1.0/24` | `10.70.224.0/20` | `10.70.240.0/24` | -| `1` | second | `10.71.0.0/16` | `10.71.1.0/24` | `10.71.224.0/20` | `10.71.240.0/24` | -| `2` | third | `10.72.0.0/16` | `10.72.1.0/24` | `10.72.224.0/20` | `10.72.240.0/24` | +| index | region | `vnet_address_space` | `subnet_prefix` | `governed_subnet_prefix` | `gateway_subnet_prefix` | `flow_log_private_endpoint_subnet_prefix` | +|---|---|---|---|---|---|---| +| `0` | first | `10.70.0.0/16` | `10.70.1.0/24` | `10.70.224.0/20` | `10.70.240.0/24` | `10.70.241.0/24` | +| `1` | second | `10.71.0.0/16` | `10.71.1.0/24` | `10.71.224.0/20` | `10.71.240.0/24` | `10.71.241.0/24` | +| `2` | third | `10.72.0.0/16` | `10.72.1.0/24` | `10.72.224.0/20` | `10.72.240.0/24` | `10.72.241.0/24` | + +The last column is carved whenever `create_flow_logs` is set, and stays empty until you ask for a +private endpoint. The range is reserved either way, so turning that switch on later renumbers +nothing. Index `0` is what this module has always created, so an existing single-region landing pad plans as a **no-op** once you name its location at `0` — nothing is renumbered by adopting @@ -635,7 +674,7 @@ individual subnet still works too. **The ARM template enforces the same allocation**, through a `regionIndex` parameter that has no default: the deployment is refused with `Missing input parameters: regionIndex` until you -say which region this is, and the four CIDR parameters become overrides that derive from it +say which region this is, and the five CIDR parameters become overrides that derive from it when unset. The two paths produce byte-identical ranges for the same index, so the table above is the authority for both. An existing deployment keeps every range it has by passing `regionIndex=0`. diff --git a/azure/arm/README.md b/azure/arm/README.md index bace151..4277b48 100644 --- a/azure/arm/README.md +++ b/azure/arm/README.md @@ -171,7 +171,18 @@ Add `CREATE_FLOW_LOGS=true` and `deploy.sh` also deploys `azuredeploy-flowlogs.j Network Watcher's resource group. It passes the VNet's id and region, the watcher's name, `FLOW_LOG_RETENTION_DAYS` (365), and `FLOW_LOG_READER_IPS` (none), the addresses allowed to read the blobs. `deploy.sh` checks each of those is shaped like an IPv4 address or a CIDR range; Azure -refuses the rest. The watcher defaults to `NetworkWatcher_` in +refuses the rest. + +`CREATE_FLOW_LOG_PRIVATE_ENDPOINT=true` additionally reaches that account over a private endpoint +in the VNet, with its private DNS zone and the link. Setting it without `CREATE_FLOW_LOGS` is +refused, since there would be no account to reach. + +The endpoint's subnet is carved by the network deployment whenever flow logs are on, not when the +endpoint is, and `deploy.sh` passes its id from that deployment's `flowLogPrivateEndpointSubnetId` +output into the flow log one. Both halves of that matter. A subnet dropped from the network +template is deleted, Azure refuses to delete one holding a private endpoint's interface, and ARM +leaves the endpoint in place when its condition turns false, so a subnet that followed the +endpoint's own switch would fail the very apply that turned it off. The watcher defaults to `NetworkWatcher_` in `NetworkWatcherRG`, and `NETWORK_WATCHER_NAME` and `NETWORK_WATCHER_RG` override it. `deploy.sh` looks the watcher up before it deploys. If the watcher does not exist or is in another region, it stops and names the watcher. It also refuses `NETWORK_WATCHER_RG=$RG`, because the role reaches that diff --git a/azure/arm/azuredeploy-flowlogs.json b/azure/arm/azuredeploy-flowlogs.json index 66afed6..0f901ee 100644 --- a/azure/arm/azuredeploy-flowlogs.json +++ b/azure/arm/azuredeploy-flowlogs.json @@ -32,11 +32,27 @@ "metadata": { "description": "Public IPv4 addresses or CIDR ranges allowed to read the flow log blobs. Empty admits no reader: the firewall denies every network except Azure trusted services, which is the exception Network Watcher writes through." } + }, + "createFlowLogPrivateEndpoint": { + "type": "Bool", + "defaultValue": false, + "metadata": { + "description": "Reach the flow log storage account over a private endpoint in the landing pad's VNet. Off by default: a private endpoint bills hourly plus per GB. Needs createFlowLogPrivateEndpointSubnet on the network template." + } + }, + "privateEndpointSubnetId": { + "type": "String", + "defaultValue": "", + "metadata": { + "description": "Resource id of the subnet the private endpoint goes in, from the network template's flowLogPrivateEndpointSubnetId output. Required when createFlowLogPrivateEndpoint is set." + } } }, "variables": { "storageAccountName": "[concat('flowlogs', uniqueString(parameters('vnetId')))]", - "flowLogName": "[concat(last(split(parameters('vnetId'), '/')), '-', uniqueString(parameters('vnetId')))]" + "flowLogName": "[concat(last(split(parameters('vnetId'), '/')), '-', uniqueString(parameters('vnetId')))]", + "privateEndpointName": "[concat(variables('storageAccountName'), '-pe')]", + "blobPrivateDnsZoneName": "privatelink.blob.core.windows.net" }, "resources": [ { @@ -81,6 +97,75 @@ "version": 2 } } + }, + { + "condition": "[parameters('createFlowLogPrivateEndpoint')]", + "type": "Microsoft.Network/privateEndpoints", + "apiVersion": "2024-05-01", + "name": "[variables('privateEndpointName')]", + "location": "[parameters('location')]", + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]" + ], + "properties": { + "subnet": { + "id": "[parameters('privateEndpointSubnetId')]" + }, + "privateLinkServiceConnections": [ + { + "name": "[variables('privateEndpointName')]", + "properties": { + "privateLinkServiceId": "[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]", + "groupIds": [ + "blob" + ] + } + } + ] + } + }, + { + "condition": "[parameters('createFlowLogPrivateEndpoint')]", + "type": "Microsoft.Network/privateDnsZones", + "apiVersion": "2024-06-01", + "name": "[variables('blobPrivateDnsZoneName')]", + "location": "global" + }, + { + "condition": "[parameters('createFlowLogPrivateEndpoint')]", + "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks", + "apiVersion": "2024-06-01", + "name": "[concat(variables('blobPrivateDnsZoneName'), '/', last(split(parameters('vnetId'), '/')))]", + "location": "global", + "dependsOn": [ + "[resourceId('Microsoft.Network/privateDnsZones', variables('blobPrivateDnsZoneName'))]" + ], + "properties": { + "virtualNetwork": { + "id": "[parameters('vnetId')]" + }, + "registrationEnabled": false + } + }, + { + "condition": "[parameters('createFlowLogPrivateEndpoint')]", + "type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups", + "apiVersion": "2024-05-01", + "name": "[concat(variables('privateEndpointName'), '/default')]", + "dependsOn": [ + "[resourceId('Microsoft.Network/privateEndpoints', variables('privateEndpointName'))]", + "[resourceId('Microsoft.Network/privateDnsZones', variables('blobPrivateDnsZoneName'))]" + ], + "properties": { + "privateDnsZoneConfigs": [ + { + "name": "blob", + "properties": { + "privateDnsZoneId": "[resourceId('Microsoft.Network/privateDnsZones', variables('blobPrivateDnsZoneName'))]" + } + } + ] + } } ] } diff --git a/azure/arm/azuredeploy-network.json b/azure/arm/azuredeploy-network.json index c7b5d81..1e331db 100644 --- a/azure/arm/azuredeploy-network.json +++ b/azure/arm/azuredeploy-network.json @@ -100,6 +100,20 @@ "description": "Optional override for the governed subnet. Empty derives the 15th /20 of the VNet range (10.70.224.0/20 at index 0), immediately below the gateway /24 at the top of the VNet, so the two egress-control ranges stay together and the workstations prefix is free to grow. Set it only to override, and then it must sit inside the VNet range." } }, + "createFlowLogPrivateEndpointSubnet": { + "type": "Bool", + "defaultValue": false, + "metadata": { + "description": "Carve the subnet the flow log storage account's private endpoint goes in. Set it whenever flow logs are on, not only when the endpoint is: a subnet dropped from this template is deleted, and Azure refuses that while a private endpoint's interface is still in it." + } + }, + "flowLogPrivateEndpointSubnetCidr": { + "type": "String", + "defaultValue": "", + "metadata": { + "description": "Range for that subnet. Empty derives the /24 above the gateway range, out of the VNet's own /16." + } + }, "additionalGovernedSubnets": { "type": "Object", "defaultValue": {}, @@ -215,8 +229,19 @@ "input": "[if(contains(variables('subnetRouteTableIds'), variables('effectiveSubnets')[copyIndex('subnetsKeepingRouteTables')].name), union(variables('effectiveSubnets')[copyIndex('subnetsKeepingRouteTables')], createObject('properties', union(variables('effectiveSubnets')[copyIndex('subnetsKeepingRouteTables')].properties, createObject('routeTable', createObject('id', variables('subnetRouteTableIds')[variables('effectiveSubnets')[copyIndex('subnetsKeepingRouteTables')].name]))))), variables('effectiveSubnets')[copyIndex('subnetsKeepingRouteTables')])]" } ], - "effectiveSubnets": "[concat(variables('workstationSubnets'), if(parameters('createGatewaySubnet'), variables('gatewaySubnets'), createArray()), if(parameters('createGovernedSubnet'), variables('governedSubnets'), createArray()), variables('additionalGovernedSubnetList'))]", - "subnetRouteTableIds": "[toObject(parameters('subnetRouteTables'), lambda('table', lambdaVariables('table').name), lambda('table', lambdaVariables('table').id))]" + "effectiveSubnets": "[concat(variables('workstationSubnets'), if(parameters('createGatewaySubnet'), variables('gatewaySubnets'), createArray()), if(parameters('createGovernedSubnet'), variables('governedSubnets'), createArray()), if(parameters('createFlowLogPrivateEndpointSubnet'), variables('flowLogPrivateEndpointSubnets'), createArray()), variables('additionalGovernedSubnetList'))]", + "subnetRouteTableIds": "[toObject(parameters('subnetRouteTables'), lambda('table', lambdaVariables('table').name), lambda('table', lambdaVariables('table').id))]", + "flowLogPrivateEndpointSubnetPrefix": "[if(empty(parameters('flowLogPrivateEndpointSubnetCidr')), cidrSubnet(variables('vnetAddressSpace'), 24, 241), parameters('flowLogPrivateEndpointSubnetCidr'))]", + "flowLogPrivateEndpointSubnetName": "flowlog-endpoint", + "flowLogPrivateEndpointSubnets": [ + { + "name": "[variables('flowLogPrivateEndpointSubnetName')]", + "properties": { + "addressPrefix": "[variables('flowLogPrivateEndpointSubnetPrefix')]", + "privateEndpointNetworkPolicies": "Disabled" + } + } + ] }, "resources": [ { @@ -402,6 +427,13 @@ "metadata": { "description": "The subnet the egress gateway VM runs in, or empty if createGatewaySubnet was false. Hand it back as spec.subnet on the Edge -- Ringleader builds no gateway until you do, because a gateway placed in the subnet it steers routes its own egress into itself. NOT governedSubnetId, which is the workstations'." } + }, + "flowLogPrivateEndpointSubnetId": { + "type": "String", + "value": "[if(parameters('createFlowLogPrivateEndpointSubnet'), resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), variables('flowLogPrivateEndpointSubnetName')), '')]", + "metadata": { + "description": "The subnet the flow log storage account's private endpoint goes in, or empty if it was not carved." + } } } } diff --git a/azure/arm/deploy.sh b/azure/arm/deploy.sh index 9acb59a..4309e51 100755 --- a/azure/arm/deploy.sh +++ b/azure/arm/deploy.sh @@ -71,6 +71,10 @@ # account's firewall denies every network except # these and Azure trusted services, so without one # nothing outside Azure can read a record (default: none) +# CREATE_FLOW_LOG_PRIVATE_ENDPOINT reach the flow log +# storage account over a private endpoint in the +# VNet, so a reader inside its region can read the +# records. Bills hourly plus per GB (default: false) # NETWORK_WATCHER_NAME the region's Network Watcher (default: NetworkWatcher_) # NETWORK_WATCHER_RG its resource group. The flow log and its # storage account are created there, never in RG (default: NetworkWatcherRG) @@ -205,12 +209,24 @@ echo ">> subject: $SUBJECT" # after the app, role and network are deployed. CREATE_FLOW_LOGS="${CREATE_FLOW_LOGS:-false}" FLOW_LOG_RETENTION_DAYS="${FLOW_LOG_RETENTION_DAYS:-365}" +CREATE_FLOW_LOG_PRIVATE_ENDPOINT="${CREATE_FLOW_LOG_PRIVATE_ENDPOINT:-false}" NETWORK_WATCHER_NAME="${NETWORK_WATCHER_NAME:-}" NETWORK_WATCHER_RG="${NETWORK_WATCHER_RG:-NetworkWatcherRG}" if [ "$CREATE_FLOW_LOGS" != "true" ] && [ "$CREATE_FLOW_LOGS" != "false" ]; then echo "CREATE_FLOW_LOGS must be true or false, not '${CREATE_FLOW_LOGS}'" >&2 exit 1 fi +if [ "$CREATE_FLOW_LOG_PRIVATE_ENDPOINT" != "true" ] && [ "$CREATE_FLOW_LOG_PRIVATE_ENDPOINT" != "false" ]; then + echo "CREATE_FLOW_LOG_PRIVATE_ENDPOINT must be true or false, not '${CREATE_FLOW_LOG_PRIVATE_ENDPOINT}'" >&2 + exit 1 +fi +# The endpoint needs the subnet, and the subnet is carved by the network deployment. Asking for one +# without the network is a request this script cannot satisfy, so it is refused here rather than +# half-applied. +if [ "$CREATE_FLOW_LOG_PRIVATE_ENDPOINT" = "true" ] && [ "$CREATE_FLOW_LOGS" != "true" ]; then + echo "CREATE_FLOW_LOG_PRIVATE_ENDPOINT is set but CREATE_FLOW_LOGS is not, so there is no flow log storage account to reach privately." >&2 + exit 1 +fi if [ "$CREATE_FLOW_LOGS" = "true" ] && [ "$CREATE_NETWORK" != "true" ]; then echo "CREATE_FLOW_LOGS=true needs CREATE_NETWORK=true: this script records flow logs only for the VNet it creates." >&2 exit 1 @@ -365,12 +381,13 @@ if [ "$CREATE_NETWORK" = "true" ]; then createGatewaySubnet="$CREATE_GATEWAY_SUBNET" \ gatewaySubnetCidr="$GATEWAY_SUBNET_CIDR" \ createGovernedSubnet="$CREATE_GOVERNED_SUBNET" \ + createFlowLogPrivateEndpointSubnet="$CREATE_FLOW_LOGS" \ governedSubnetCidr="$GOVERNED_SUBNET_CIDR" \ additionalGovernedSubnets="$ADDITIONAL_GOVERNED_SUBNETS_JSON" \ subnetRouteTables="$SUBNET_ROUTE_TABLES" \ workstationsNsgExists="$WORKSTATIONS_NSG_EXISTS" \ gatewayNsgExists="$GATEWAY_NSG_EXISTS" \ - --query '[properties.outputs.subnetId.value, properties.outputs.governedSubnetId.value, properties.outputs.gatewaySubnetId.value]' -o tsv)" + --query '[properties.outputs.subnetId.value, properties.outputs.governedSubnetId.value, properties.outputs.gatewaySubnetId.value, properties.outputs.flowLogPrivateEndpointSubnetId.value]' -o tsv)" SUBNET_ID="$(echo "$NETWORK_OUTPUTS" | sed -n 1p)" GOVERNED_SUBNET_ID="$(echo "$NETWORK_OUTPUTS" | sed -n 2p)" # The gateway subnet is printed for the same reason the other two are: it is a value the @@ -378,6 +395,9 @@ if [ "$CREATE_NETWORK" = "true" ]; then # out, an operator on this path never learns the id -- and Ringleader builds no gateway VM at # all until an Edge names it. GATEWAY_SUBNET_ID="$(echo "$NETWORK_OUTPUTS" | sed -n 3p)" + # The subnet the flow log storage account's private endpoint goes in. Carved whenever flow logs + # are on, so it is here to pass whether or not an endpoint was asked for. + FLOW_LOG_PE_SUBNET_ID="$(echo "$NETWORK_OUTPUTS" | sed -n 4p)" ADDITIONAL_GOVERNED_SUBNET_IDS="$(az deployment group show \ --resource-group "$RG" \ --name ringleader-onboarding-network \ @@ -415,6 +435,8 @@ if [ "$CREATE_FLOW_LOGS" = "true" ]; then networkWatcherName="$NETWORK_WATCHER_NAME" \ retentionDays="$FLOW_LOG_RETENTION_DAYS" \ logReaderIpRules="$FLOW_LOG_READER_IPS_JSON" \ + createFlowLogPrivateEndpoint="$CREATE_FLOW_LOG_PRIVATE_ENDPOINT" \ + privateEndpointSubnetId="${FLOW_LOG_PE_SUBNET_ID:-}" \ --query 'properties.provisioningState' -o tsv fi diff --git a/azure/terraform/README.md b/azure/terraform/README.md index 2092c6a..25bfe39 100644 --- a/azure/terraform/README.md +++ b/azure/terraform/README.md @@ -42,6 +42,8 @@ Terraform. A ready-to-apply root is in [`examples/standalone/`](examples/standal | `create_flow_logs` | `false` | Record VNet flow logs for the VNet this module creates, into a storage account created for them. **Off by default**, because in most regions Azure bills per GB collected beyond 5 GB a month, plus storage. The flow log and its storage account go in the **Network Watcher's resource group**. That is outside the resource group the role is scoped to, so Ringleader can neither read nor delete them. Whoever applies needs to create resources there. Turning it off again deletes the storage account and every record in it. Needs `create_network`. See `azure/README.md`. | | `flow_log_retention_days` | `365` | How many days each flow log record is kept, 1 to 365. | | `flow_log_reader_ip_rules` | `[]` | Public IPv4 addresses or CIDR ranges allowed to read the flow log blobs. The storage account's firewall denies every other network, except Azure trusted services, which is the exception Network Watcher writes through. Empty admits no reader at all. Add an address here rather than on the account, because a rule added by hand is removed on the next apply. See `azure/README.md`. | +| `create_flow_log_private_endpoint` | `false` | Reach the flow log storage account over a private endpoint in this landing pad's VNet, with the private DNS zone and its link. **Off by default**, because a private endpoint bills hourly plus per GB. Turn it on when a reader runs inside the storage account's region, which `flow_log_reader_ip_rules` cannot admit, or when a compliance scan asks for private endpoints by name. Needs `create_flow_logs`, and the plan refuses it without. Turning it back off leaves the endpoint in place: see `azure/README.md`. | +| `flow_log_private_endpoint_subnet_prefix` | `null` → the `/24` above the gateway range | Range for that endpoint's subnet, which is carved whenever `create_flow_logs` is set and stays empty until an endpoint lands in it. Derived from the VNet's own `/16`, so a region that takes the derived ranges needs nothing here. | | `network_watcher_name` | `null` → `NetworkWatcher_` | The region's Network Watcher. Set it if yours has another name, for example `-watcher` from the Azure CLI. It must exist and be in `location`, or the plan or apply fails naming it. | | `network_watcher_resource_group_name` | `NetworkWatcherRG` | The watcher's resource group. The flow log and its storage account are created here, and the plan refuses `resource_group_name`. | | `additional_governed_subnets` | `{}` | One more governed subnet per namespace that runs its own proxy, as a map of your label to a prefix you write out. Each is built like the governed subnet and named `governed-