feat: Enhance lab automation with region fallback - Adaptive Apps

03-Azure/01-01-App Innovation/04-adaptive-apps/labautomation/README.md

@@ -16,8 +16,9 @@ region, and participant object IDs. The participant already has Owner on that
 group. There is no interactive login, Graph application/service-principal creation,
 confidential computing, Azure Local, or shared participant Kubernetes cluster here.
 
-Default scope: **resourcegroup**, **five labs per subscription**, in **Sweden Central**
-(`swedencentral`). Organizers may override the event region before creating lab
+Default scope: **resourcegroup**, **five labs per subscription**, with **Sweden Central**
+first and **Spain Central** as fallback (`swedencentral,spaincentral`).
+Organizers may override the ordered regions before creating lab
 scopes; the shared preflight must pass for that subscription and region. Every participant
 gets a dedicated two-node AKS cluster and a separate private K3s VM. The organizer
 needs a **Linux runner**, PowerShell 7, current Az.Accounts/Az.Resources/Az.Network,
@@ -60,7 +61,11 @@ prerequisites. No GitHub or AI API credentials are provisioned by these hooks.
 `shared-deploy-lab.ps1` runs once per subscription:
 
 1. Verify both authenticated contexts target the supplied subscription.
-2. Register and wait for Compute, Network, ContainerService, ContainerRegistry,
+2. Register `Microsoft.Network/AllowBringYourOwnPublicIpAddress`, as required
+   by these hosted workshop subscriptions for public IP allocation (also used by
+   LocalBox). Wait up to 15 minutes for `Registered`; pending approval, failure,
+   or timeout blocks participant provisioning. Then re-register Network so the
+   feature takes effect. Register and wait for Compute, Network, ContainerService, ContainerRegistry,
    ManagedIdentity, DBforPostgreSQL, OperationalInsights, PolicyInsights, Sql,
    KeyVault and Storage. SQL supports the published SQL recipe; Key Vault and
    Storage preserve the manual challenge 01 subscription prerequisites.
@@ -89,14 +94,27 @@ regional service limits, policy and actual SKU capacity can still fail deploymen
 The stage fails explicitly rather than silently choosing a different VM size.
 Quota failure messages identify the required capacity and region.
 
-Console creates participant groups in the **first preferred region**. The
-PostgreSQL recipe defaults to `resourceGroup().location`, so the shared stage
-requires that first region to pass even if an alternative is eligible. The
-participant hook deploys everything in the **supplied RG's actual region**, and
-rejects any region absent from the shared metadata. To use an alternative,
-reconfigure the event location and explicitly recreate empty scopes through
-Console; never move resource deployment independently of the RG metadata.
-There is no destructive region fallback, automatic RG deletion or AKS recreation.
+Console creates participant groups in the **first preferred region**. Shared
+preflight retains eligible regions in preference order and fails if none pass.
+For **initial infrastructure provisioning only**, the participant hook uses
+`Invoke-MhhDeploymentWithRegionFallback` with those validated regions. The helper
+classifies deployment failures, retries supported capacity/region failures, and
+fails immediately for fatal errors rather than treating all errors as retryable.
+It can **delete the participant RG and all its contents**, retry the same region,
+or recreate it in the next region. Participant Owner access is restored on every
+attempt. Use initial provisioning only in dedicated, disposable lab groups.
+
+After infrastructure succeeds, `adaptiveAppsPreserve=true` prevents subsequent
+runs from invoking destructive fallback, including when Bash bootstrap failed.
+Existing labs with `adaptiveAppsReady=true` also receive this persistent guard
+before readiness is cleared. Do not remove the preservation tag to troubleshoot
+a working lab. Such labs receive incremental deployment in place; if their region
+fails preflight, the hook stops rather than relocating them.
+
+The parameter file deliberately omits `location`, letting the template follow
+`resourceGroup().location`. After fallback the hook re-reads the RG location for
+all bootstrap phases and credentials, keeping the PostgreSQL recipe in the same
+region. These checks do not guarantee AKS capacity or region eligibility.
 
 ## Participant infrastructure and execution
 
@@ -131,8 +149,9 @@ record its value in deployment history. Do not print or publish the parameter
 file. Stage messages identify context validation, parameter preparation and
 deployment submission without including credentials.
 
-The participant hook uses an **incremental deployment**, retaining resources and
-failed deployment records for diagnosis. It grants each supplied participant
+The participant hook uses an **incremental deployment**; initial-provisioning
+fallback may recycle failed attempts as described above. Protected labs retain
+resources and failed deployment records for diagnosis. It grants each supplied participant
 **Reader on this AKS node RG only**, so the PostgreSQL recipe can discover AKS
 egress IPs. Role operations use object IDs and explicitly disable Graph name
 resolution. No subscription-wide participant role is added.
@@ -267,12 +286,12 @@ the managed-cluster API requires the profile when `mode` is `Istio`, and Azure
 selects its default supported revision when no revision is specified.
 
 If a live deployment reports `AKSCapacityHeavyUsage`, quota/SKU validation has
-not reserved AKS capacity. Resolve the capacity issue or explicitly select another
-event region before retrying. A `SubscriptionNotRegisteredForFeature` error
-mentioning `AllowBringYourOwnPublicIpAddress` on these ordinary Standard public
-IPs needs subscription/provider investigation; the template does not request
-BYOIP or a custom IP prefix. Do not enable an unrelated preview feature merely
-to suppress that error. Preserve the deployment operation errors for support.
+not reserved AKS capacity. Initial provisioning can use the validated fallback;
+protected labs require organizer investigation. If
+`SubscriptionNotRegisteredForFeature` still mentions
+`AllowBringYourOwnPublicIpAddress`, verify shared setup completed both feature
+registration and subsequent Network provider registration. The template itself
+does not request BYOIP or a custom IP prefix.
 
 ## Hosted policy provenance
 

03-Azure/01-01-App Innovation/04-adaptive-apps/labautomation/deploy-lab.ps1

@@ -39,38 +39,57 @@ try {
     $group = Get-AzResourceGroup -Name $ResourceGroupName -ErrorAction Stop
     $region = $group.Location.ToLowerInvariant()
     $metadata = (Get-AzTag -ResourceId "/subscriptions/$SubscriptionId" -ErrorAction Stop).Properties.TagsProperty
-    if (-not $metadata -or $region -notin ($metadata['microhack-adaptive-regions'] -split ',')) {
-        throw "RG location '$region' was not validated by shared-deploy-lab.ps1. Run the shared hook for this region; do not relocate resources independently of their RG."
+    if (-not $metadata) { throw 'Shared region metadata is missing. Run shared-deploy-lab.ps1 first.' }
+    $regions = @($metadata['microhack-adaptive-regions'] -split ',' | Where-Object { $_ })
+    $preserve = $group.Tags -and ($group.Tags['adaptiveAppsReady'] -eq 'true' -or $group.Tags['adaptiveAppsPreserve'] -eq 'true')
+    if ($regions.Count -eq 0 -or ($preserve -and $region -notin $regions)) {
+        throw "No validated region is available for this lab. Run shared-deploy-lab.ps1 for the lab's region; an established lab will not be relocated."
     }
-    Update-AzTag -ResourceId $group.ResourceId -Operation Merge -Tag @{
+    $deploymentTags = @{
         adaptiveAppsReady = 'false'
         SecurityControl = 'Ignore'
         CostControl = 'Ignore'
-    } -ErrorAction Stop | Out-Null
+    }
+    if ($preserve) { $deploymentTags.adaptiveAppsPreserve = 'true' }
+    Update-AzTag -ResourceId $group.ResourceId -Operation Merge -Tag $deploymentTags -ErrorAction Stop | Out-Null
 
     Write-Host 'Preparing private infrastructure deployment parameters...'
     $parameterFile = Join-Path $scratch 'main.parameters.json'
     @{
         '$schema' = 'https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#'
         contentVersion = '1.0.0.0'
         parameters = @{
-            location = @{ value = $region }
             adminPassword = @{ value = (New-MhhStablePassword -Purpose 'adaptive-apps-k3s-admin' -Length 24) }
         }
     } | ConvertTo-Json -Depth 4 | Set-Content -LiteralPath $parameterFile -Encoding utf8
     & chmod 600 $parameterFile
     if ($LASTEXITCODE -ne 0) { throw 'Unable to secure infrastructure parameter file.' }
     Write-Host 'Submitting incremental Adaptive Apps infrastructure deployment...'
     try {
-        $deployment = New-AzResourceGroupDeployment -Name 'adaptive-apps-console' `
-            -ResourceGroupName $ResourceGroupName -TemplateFile (Join-Path $PSScriptRoot 'main.bicep') `
-            -TemplateParameterFile $parameterFile -Mode Incremental -ErrorAction Stop
+        if ($preserve) {
+            $deployment = New-AzResourceGroupDeployment -Name 'adaptive-apps-console' `
+                -ResourceGroupName $ResourceGroupName -TemplateFile (Join-Path $PSScriptRoot 'main.bicep') `
+                -TemplateParameterFile $parameterFile -Mode Incremental -ErrorAction Stop
+        } else {
+            Write-Warning "Initial provisioning may delete and recreate '$ResourceGroupName' on retryable infrastructure failures. Validated regions: $($regions -join ', ')."
+            $result = Invoke-MhhDeploymentWithRegionFallback -PreferredLocations $regions `
+                -ResourceGroupName $ResourceGroupName -RgOwnerEntraObjectIds $AllowedEntraUserIds `
+                -TemplateFile (Join-Path $PSScriptRoot 'main.bicep') -TemplateParameterFile $parameterFile `
+                -DeploymentNamePrefix 'adaptive-apps-console' -Tag $deploymentTags -ErrorAction Stop
+            if (-not $result.Success) { throw 'Initial infrastructure fallback did not report success.' }
+            $deployment = $result.DeploymentResult
+        }
     } finally {
         Remove-Item -LiteralPath $parameterFile -Force
     }
     if ($deployment.ProvisioningState -ne 'Succeeded') {
-        throw "Infrastructure deployment failed: $($deployment.ProvisioningState). Resources are retained for diagnosis; no destructive fallback is attempted."
+        throw "Infrastructure deployment failed: $($deployment.ProvisioningState). Bootstrap will not run."
     }
+    $group = Get-AzResourceGroup -Name $ResourceGroupName -ErrorAction Stop
+    $region = $group.Location.ToLowerInvariant()
+    if ($region -notin $regions) { throw "Deployment used unvalidated region '$region'." }
+    # Keep this marker through failed reconciliations; readiness alone is not a deletion guard.
+    Update-AzTag -ResourceId $group.ResourceId -Operation Merge -Tag @{ adaptiveAppsPreserve = 'true' } -ErrorAction Stop | Out-Null
     $acrName = $deployment.Outputs.acrName.Value
     $nodeGroup = $deployment.Outputs.nodeResourceGroup.Value
     if (-not $acrName -or -not $nodeGroup) { throw 'Infrastructure deployment omitted ACR or AKS node resource group outputs.' }

03-Azure/01-01-App Innovation/04-adaptive-apps/labautomation/lab-defaults.json

@@ -2,7 +2,7 @@
   "$schema": "https://raw.githubusercontent.com/microsoft/MicroHack/refs/heads/main/lab-defaults-schema.json",
   "deploymentType": "resourcegroup",
   "labsPerSubscription": 5,
-  "preferredLocation": "swedencentral",
+  "preferredLocation": "swedencentral,spaincentral",
   "estimatedDailyCostsUsd": 35.0,
   "estimatedSharedDeploymentDailyCostsUsd": 0.0
 }

03-Azure/01-01-App Innovation/04-adaptive-apps/labautomation/shared-deploy-lab.ps1

@@ -18,6 +18,27 @@ $locations = @($PreferredLocation | ForEach-Object { $_ -split ',' } | ForEach-O
 if ($locations.Count -eq 0) { throw 'At least one preferred location is required.' }
 $labCount = @($AllowedEntraUserIds | Where-Object { $_ } | Select-Object -Unique).Count
 if ($labCount -eq 0) { $labCount = $defaults.labsPerSubscription }
+$feature = 'AllowBringYourOwnPublicIpAddress'
+$featureState = (Invoke-AdaptiveAz -Arguments @('feature', 'show', '--namespace', 'Microsoft.Network', '--name', $feature)).properties.state
+if ($featureState -ne 'Registered') {
+    if ($featureState -notin @('Registering', 'Pending')) {
+        Invoke-AdaptiveAz -Arguments @('feature', 'register', '--namespace', 'Microsoft.Network', '--name', $feature) | Out-Null
+    }
+    for ($attempt = 0; $attempt -lt 90; $attempt++) {
+        Start-Sleep -Seconds 10
+        Update-MhhToken | Out-Null
+        $featureState = (Invoke-AdaptiveAz -Arguments @('feature', 'show', '--namespace', 'Microsoft.Network', '--name', $feature)).properties.state
+        if ($featureState -eq 'Registered') { break }
+        if ($featureState -notin @('Registering', 'Pending')) {
+            throw "Microsoft.Network/$feature registration ended in state '$featureState'."
+        }
+        Write-Host "Waiting for Microsoft.Network/$feature registration ($($attempt + 1)/90): $featureState"
+    }
+}
+if ($featureState -ne 'Registered') {
+    throw "Microsoft.Network/$feature did not reach Registered within 15 minutes (state: '$featureState'). Resolve feature registration before retrying."
+}
+# Re-register Network after the feature is registered so public IP allocation sees it.
 $providers = @('Microsoft.Compute', 'Microsoft.Network', 'Microsoft.ContainerService',
     'Microsoft.ContainerRegistry', 'Microsoft.ManagedIdentity', 'Microsoft.DBforPostgreSQL',
     'Microsoft.OperationalInsights', 'Microsoft.PolicyInsights', 'Microsoft.Sql',
@@ -48,14 +69,12 @@ foreach ($location in $locations) {
         Write-Warning $failures[-1]
     }
 }
-# Console pre-creates RGs in the first preferred location. Recipes inherit that location.
-# Do not report shared success merely because a different candidate passed.
-if ($locations[0] -notin $ready) {
-    throw "The supplied participant RG region '$($locations[0])' is not ready. Eligible alternatives: $($ready -join ', '). Reconfigure the event location and recreate EMPTY lab scopes explicitly; no automatic relocation/deletion is performed. $($failures -join '; ')"
+if ($ready.Count -eq 0) {
+    throw "No preferred region passed Adaptive Apps preflight. $($failures -join '; ')"
 }
-Initialize-MhhHostedTagPolicy -SubscriptionId $SubscriptionId -Location $locations[0]
+Initialize-MhhHostedTagPolicy -SubscriptionId $SubscriptionId -Location $ready[0]
 Update-AzTag -ResourceId "/subscriptions/$SubscriptionId" -Operation Merge -Tag @{
-    'microhack-adaptive-location' = $locations[0]
+    'microhack-adaptive-location' = $ready[0]
     'microhack-adaptive-regions' = $ready -join ','
     'microhack-adaptive-lab-count' = "$labCount"
 } -ErrorAction Stop | Out-Null