diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/Readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/Readme.md index 85af4a2cf..870ff44a1 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/Readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/Readme.md @@ -48,7 +48,7 @@ After completing this Hackathon you will: | 1 | Using Azure native platform controls (e.g. Policy, RBAC etc) to enforce sovereign controls in the public cloud | [Challenge](./challenges/challenge-01.md) | [Solution](./walkthrough/challenge-01/solution-01.md) | 45 min | Jan Egil Ring | | 2 | Encryption at rest with Customer Managed Keys in Azure Key Vault | [Challenge](./challenges/challenge-02.md) | [Solution](./walkthrough/challenge-02/solution-02.md) | 30 min | Ye Zhang | | 3 | Encryption in transit - enforcing TLS | [Challenge](./challenges/challenge-03.md) | [Solution](./walkthrough/challenge-03/solution-03.md) | 30 min | Ye Zhang | -| 4 | Encryption in use with Azure Confidential Compute - VM | [Challenge](./challenges/challenge-04.md) | [Solution](./walkthrough/challenge-04/solution-04.md) | 30 min | Murali Rao Yelamanchili / Ye Zhang | +| 4 | Runtime attestation with Confidential and Standard ACI | [Challenge](./challenges/challenge-04.md) | [Solution](./walkthrough/challenge-04/solution-04.md) | 30-45 min | Murali Rao Yelamanchili / Ye Zhang | | 5 | Encryption in use with Confidential VMs/Node Pools in Azure Kubernetes Service (AKS) | [Challenge](./challenges/challenge-05.md) | [Solution](./walkthrough/challenge-05/solution-05.md) | 30 min | Murali Rao Yelamanchili / Ye Zhang | | 6 | Operating Sovereign in a hybrid environment with Azure Local and Azure Arc | [Challenge](./challenges/challenge-06.md) | [Solution](./walkthrough/challenge-06/solution-06.md) | 60-90 min | Jan Egil Ring / Thomas Maurer | | 7 | Adaptive Apps across sovereign Azure and private-cloud environments with Radius | [Challenge](./challenges/challenge-07.md) | [Solution](./walkthrough/challenge-07/solution-07.md) | 60 min | Dylan de Jong / Jan Egil Ring / Wesley Backelant | @@ -68,10 +68,14 @@ In order to use the Hackathon time most effectively, the following tasks should 4. [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli). **Hint:** Make sure to use the latest version available. 5. Challenge 7: `kubectl`, `jq`, OpenSSL, the [Radius CLI](https://docs.radapp.io/getting-started/install/), and the Azure CLI `bastion` extension -The shared Challenge 4/5/7 platform requires **4 AMD SEV-SNP confidential-family vCPUs per participant**. Automation evaluates `Standard_DC2as_v5` and `Standard_DC2as_v6` across the configured regions, preferring the generally available v5 family. With the default two labs per subscription, prepare a minimum family quota of 8 vCPUs in at least one configured region. For example: +The shared Challenge 5/7 AKS platform requires **4 AMD SEV-SNP confidential-family vCPUs per participant** for two Ubuntu nodes, plus **12 DSv5 vCPUs** for two system nodes and one K3s VM: **16 regional vCPUs** in total. There is no standalone Confidential VM. Automation evaluates `Standard_DC2as_v5` and `Standard_DC2as_v6` across the configured regions. With two labs per subscription, allow 8 confidential-family, 24 DSv5 and 32 regional vCPUs before deployment. These are steady-state requirements; also reserve upgrade surge headroom (for example one additional node per pool: 2 confidential plus 4 DSv5 vCPUs per concurrent cluster upgrade). The default candidate order is Sweden Central/v5, Spain Central/v5, Sweden Central/v6, then Spain Central/v6. Shared preparation persists the first candidate with supported VM sizes and sufficient confidential-family, DSv5-family, and regional quota so every participant deployment in that subscription uses the same selection. +On 2026-09-21, read-only `az vm list-skus --all` checks in the validation subscription returned `Standard_DC2as_v6` in **both Sweden Central and Spain Central**, with no restrictions; `Standard_DC2as_v5` was not listed in either. This is a dated subscription-specific observation, not a capacity guarantee or proof of AKS deployment success. Recheck SKUs, quota and AKS support for each event. Ubuntu CVM pools are supported as described in [AKS confidential VM guidance](https://learn.microsoft.com/azure/aks/use-cvm). + +Challenge 4 independently deploys ACI/ACR in **North Europe**, the walkthrough's validated Confidential ACI region. It does not use VM-family quota, the old standalone CVM, or a custom attestation provider. Confirm Confidential ACI quota/capacity separately; VM availability in Sweden/Spain does not establish Confidential ACI availability there. Shared automation registers `Microsoft.ContainerInstance` and `Microsoft.ContainerRegistry`. Challenge 5 uses the actual Console AKS region, not Challenge 4's `LOCATION` value. + ```powershell ./resources/manual-setup/subscription-preparations/2-vcpu-quotas.ps1 -Region swedencentral -NumberOfLabUsers 2 -ConfidentialVmGeneration v5 -SubmitQuotaRequests ``` @@ -94,17 +98,23 @@ Use the AZQR report for planning and preferred-region ordering, not as the deplo The main cost driver for this Hackathon is virtual machines: -- **LocalBox** cost is approximately 100-110 USD per subscription per day. For hosted events, schedule provisioning at least 24 hours ahead and budget for preparation time; avoid leaving it running for a week by default. Follow the [hosted-event budget guidance](./resources/hosted-events/readme.md). -- **Challenges 4, 5, and 7** share one pre-provisioned participant platform: a two-node AKS system pool, one Confidential VM AKS node, one standalone Confidential VM, one K3s VM, Azure Bastion Standard, and a NAT Gateway. Budget approximately 45-55 USD per participant per day, depending on region and data transfer. The platform starts during lab deployment so participants can focus on validation rather than waiting for capacity-sensitive resources. -- Plan subscription quotas for at least 12 general-purpose vCPUs and 4 DCasv5- or DCasv6-family confidential vCPUs per participant. Keep the default maximum of two participants per subscription unless larger aggregate increases have been approved in advance. +- **LocalBox** cost is approximately 100-110 USD per subscription per day. Console's shared-deployment estimate is USD 110/day per subscription, separate from participant costs. For hosted events, schedule provisioning at least 24 hours ahead and budget for preparation time; avoid leaving it running for a week by default. Follow the [hosted-event budget guidance](./resources/hosted-events/readme.md). +- **Challenges 5 and 7** share one pre-provisioned cluster: two `Standard_D4s_v5` system nodes and two Ubuntu `Standard_DC2as_v5/v6` confidential nodes, plus a `Standard_D4s_v5` K3s VM, Bastion Standard and NAT Gateway. The removed standalone CVM is replaced by the second confidential node, so baseline confidential compute remains four vCPUs; sharing also avoids a second AKS system pool. +- Example Linux PAYG compute in Spain Central from the [Azure Retail Prices API](https://prices.azure.com/api/retail/prices), checked 2026-09-21: three D4s v5 at USD 0.214/hour plus two DC2as v6 at USD 0.117/hour equals **USD 21.02/day** for compute alone. Disks, Bastion, NAT, public IPs, LoadBalancers, storage and traffic are additional. +- Use **USD 35-45 per participant per day** as a planning range including a modest allowance for those services and Challenge 4's two 1-vCPU/2-GiB ACI groups, Basic ACR and image build. The Console estimate is USD 45/day, not a pricing quote. ACI duration, confidential pricing, builds and egress can change the total; validate a regional Pricing Calculator estimate before approval. +- Plan subscription quotas for at least 12 general-purpose vCPUs and 4 confidential vCPUs per participant, plus upgrade headroom. Keep two participants per subscription unless larger aggregate increases have been approved. Challenge 5 application cleanup does not release node quota because Challenge 7 still uses the cluster. -For a 50-participant event, the shared participant platforms cost approximately 2,250-2,750 USD per day while deployed, in addition to the optional LocalBox environment. +For a 50-participant event, this planning range is USD 1,750-2,250 per day, plus LocalBox for each participating subscription if Challenge 6 is enabled. There will also be smaller costs for services such as Key Vault, storage, and monitoring. Budget separately for the approved Defender for Servers plan used in Challenge 6. An Azure Pricing Calculator estimate is available [here](https://azure.com/e/1a7aec76a3e049cba57cda6742025373). This estimate can be adjusted for fewer/more students, running the VMs shorter/longer and adding additional services if desired. If you plan to run this Hackathon in your own subscription on a limited budget, skip the optional Challenge 6 environment and remove the participant resource group immediately after finishing the event. +### Existing event migration + +The revised template is intended for fresh provisioning. Existing AzureLinux confidential pools are not automatically converted to Ubuntu. Removing the old VM, NIC/subnet or attestation declarations does not delete deployed resources or stored Console credentials; an incremental redeployment can also fail if the removed subnet is still in use. Use a fresh event or have the organizer plan workload-safe pool replacement and obsolete-resource/credential cleanup explicitly. Do not rerun the region-fallback deployment on an active lab as a migration shortcut: it may recycle the resource group. No automatic teardown is added by these changes. + ## Contributors - Thomas Maurer [GitHub](https://github.com/thomasmaurer); [LinkedIn](https://www.linkedin.com/in/thomasmaurer2/) diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-05.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-05.md index 48cbe697a..97bb98653 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-05.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-05.md @@ -8,12 +8,12 @@ VM node pool and verify the execution environment with a fresh MAA token. ## Actions -- Deploy an AKS cluster with managed identity and automatic upgrade channels. -- Add and label an AMD SEV-SNP Confidential VM node pool. +- Connect to the provided AKS cluster shared with Challenge 7. +- Validate its two Ubuntu AMD SEV-SNP nodes labelled `workload=confidential`. - Deploy the Azure Voting App to the confidential nodes. - Deploy the newer runtime attestation UI to the same node pool. - Verify pod placement and inspect the attestation claims. -- Remove the AKS cluster without deleting the shared attendee resource group. +- Remove only Challenge 5 applications from `challenge-05`, retaining AKS and its pools for Challenge 7. ## Success criteria diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-06.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-06.md index b6eb5d46d..9d9359a4f 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-06.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-06.md @@ -11,6 +11,7 @@ Your organization must run workloads in a sovereign cloud while still leveraging ## Actions * Explore the LocalBox hybrid infrastructure in the Azure Portal +* Deploy a sample application to AKS on Azure Local * Deploy your own VM on Azure Local using Azure Arc VM management and verify that guest management is connected * Verify Microsoft Defender for Cloud coverage and review security recommendations for the VM you provisioned * Use Azure Update Manager to assess OS updates on the VM you provisioned @@ -21,6 +22,7 @@ Your organization must run workloads in a sovereign cloud while still leveraging * You have deployed your own VM on Azure Local via the Azure Portal and verified that guest management is Enabled (Connected) * You have verified Defender for Servers coverage for your VM and reviewed its available recommendations, or identified that its assessment is still pending * You have completed an Azure Update Manager assessment for your VM and reviewed the results, including when no updates are pending +* You have deployed a sample application to AKS on Azure Local * You understand how Azure Arc provides a unified control plane for sovereign hybrid scenarios ## Learning resources diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/deploy-lab.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/deploy-lab.ps1 index 530d5871a..ef1c3dbe1 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/deploy-lab.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/deploy-lab.ps1 @@ -70,11 +70,7 @@ if(-not $selectedCandidate) { $confidentialVmSize = $selectedCandidate.VmSize $confidentialQuotaName = $selectedCandidate.QuotaName $deploymentLocations = @($selectedCandidate.Location) -$quotaRequirements = @( - @{ Name = $confidentialQuotaName; Required = 4 } - @{ Name = 'StandardDSv5Family'; Required = 12 } - @{ Name = 'cores'; Required = 16 } -) +$quotaRequirements = @(Get-MhhSovereignComputeRequirements -ConfidentialQuotaName $confidentialQuotaName) $regionReadinessSummary = @() foreach($candidateLocation in $deploymentLocations) { $requiredVmSizes = @($confidentialVmSize, 'Standard_D4s_v5') @@ -102,8 +98,8 @@ foreach($candidateLocation in $deploymentLocations) { $available = $limit - $current $regionReadinessSummary += "${candidateLocation}/$($requirement.Name): ${current}/${limit} vCPUs used" - if($available -lt $requirement.Required) { - throw "The shared compute selection is no longer ready ($($regionReadinessSummary -join '; ')). $($requirement.Required) available $($requirement.Name) vCPUs are required per participant." + if($available -lt $requirement.PerParticipant) { + throw "The shared compute selection is no longer ready ($($regionReadinessSummary -join '; ')). $($requirement.PerParticipant) available $($requirement.Name) vCPUs are required per participant." } } } @@ -118,8 +114,9 @@ $effectiveLocation = $deploymentLocations[0] # Provider registration is initiated once per subscription by shared-deploy-lab.ps1. # Registration is asynchronous, so verify the providers required by the shared lab. $sovereignLabProviders = @( - 'Microsoft.Attestation', 'Microsoft.Compute', + 'Microsoft.ContainerInstance', + 'Microsoft.ContainerRegistry', 'Microsoft.ContainerService', 'Microsoft.ManagedIdentity', 'Microsoft.Network' @@ -145,9 +142,8 @@ foreach($providerNamespace in $sovereignLabProviders) { # on retries, so subscription and resource-group role assignments are applied after it. $nameSuffix = $stableHash.Substring(0, 8) $k3sAdminPassword = New-MhhStablePassword -Purpose 'adaptive-apps-k3s-admin' -Length 24 -$cvmAdminPassword = New-MhhStablePassword -Purpose 'sovereign-cvm-admin' -Length 24 -Write-Host "Deploying the shared Sovereign Cloud platform for Challenges 4, 5, and 7..." +Write-Host "Deploying the shared Sovereign Cloud platform for Challenges 5 and 7..." $sovereignLabResult = Invoke-MhhDeploymentWithRegionFallback ` -PreferredLocations $deploymentLocations ` -ResourceGroupName $ResourceGroupName ` @@ -156,13 +152,12 @@ $sovereignLabResult = Invoke-MhhDeploymentWithRegionFallback ` -TemplateParameterObject @{ nameSuffix = $nameSuffix adminPassword = $k3sAdminPassword - cvmAdminPassword = $cvmAdminPassword confidentialVmSize = $confidentialVmSize } ` -DeploymentNamePrefix 'sovereign-lab' ` -Tag @{ workload = 'sovereign-lab' - challenges = '4,5,7' + challenges = '5,7' CostControl = 'Ignore' } @@ -181,13 +176,9 @@ New-AzSubscriptionDeployment ` } | Out-Null @{"HackboxCredential" = @{ name = "Sovereign Lab Region"; value = $sovereignLabResult.LocationUsed; note = "Region selected after capacity checks" }} -@{"HackboxCredential" = @{ name = "Confidential VM Size"; value = $confidentialVmSize; note = "AMD SEV-SNP size selected during shared preparation" }} +@{"HackboxCredential" = @{ name = "AKS Confidential Node Size"; value = $confidentialVmSize; note = "Two Ubuntu AMD SEV-SNP nodes; four confidential vCPUs per participant" }} @{"HackboxCredential" = @{ name = "Sovereign Lab AKS Cluster"; value = $sovereignLabResult.Outputs.aksClusterName; note = "Shared by Challenges 5 and 7" }} @{"HackboxCredential" = @{ name = "AKS Confidential Node Pool"; value = $sovereignLabResult.Outputs.confidentialNodePoolName; note = "Challenge 5" }} -@{"HackboxCredential" = @{ name = "Confidential VM"; value = $sovereignLabResult.Outputs.confidentialVmName; note = "Challenge 4" }} -@{"HackboxCredential" = @{ name = "Confidential VM Admin Username"; value = "azureuser"; note = $sovereignLabResult.Outputs.confidentialVmName }} -@{"HackboxCredential" = @{ name = "Confidential VM Admin Password"; value = $cvmAdminPassword; note = $sovereignLabResult.Outputs.confidentialVmName }} -@{"HackboxCredential" = @{ name = "Attestation Provider"; value = $sovereignLabResult.Outputs.attestationProviderName; note = $sovereignLabResult.Outputs.attestationProviderUri }} @{"HackboxCredential" = @{ name = "Sovereign Lab K3s VM"; value = $sovereignLabResult.Outputs.k3sVmName; note = "Private local/edge execution environment" }} @{"HackboxCredential" = @{ name = "Sovereign Lab Bastion"; value = $sovereignLabResult.Outputs.bastionName; note = "Shared private access path" }} @{"HackboxCredential" = @{ name = "K3s VM Admin Username"; value = "azureuser"; note = $sovereignLabResult.Outputs.k3sVmName }} diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/lab-defaults.json b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/lab-defaults.json index bfb493dc3..e650ed256 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/lab-defaults.json +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/lab-defaults.json @@ -4,5 +4,6 @@ "deploymentType": "resourcegroup", "labsPerSubscription": 2, "preferredLocation": "swedencentral, spaincentral", - "estimatedDailyCostsUsd": 55.0 + "estimatedDailyCostsUsd": 45.0, + "estimatedSharedDeploymentDailyCostsUsd": 110.0 } diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/quota-helpers.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/quota-helpers.ps1 index 743f4fd52..e5a9c57f4 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/quota-helpers.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/quota-helpers.ps1 @@ -8,6 +8,18 @@ function Get-MhhResponseHeaderValue { return @($header.Value)[0] } +function Get-MhhSovereignComputeRequirements { + param([Parameter(Mandatory)][string]$ConfidentialQuotaName) + + $generalVcpus = (2 + 1) * 4 + $confidentialVcpus = 2 * 2 + @( + @{ Name = $ConfidentialQuotaName; PerParticipant = $confidentialVcpus } + @{ Name = 'StandardDSv5Family'; PerParticipant = $generalVcpus } + @{ Name = 'cores'; PerParticipant = $generalVcpus + $confidentialVcpus } + ) +} + function Get-MhhConfidentialComputeCandidates { param( [string[]]$PreferredLocation diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/resource-providers.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/resource-providers.ps1 index 6b59e9c21..f8a79f8f9 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/resource-providers.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/resource-providers.ps1 @@ -86,6 +86,8 @@ $providers = @( "Microsoft.Kubernetes", "Microsoft.KubernetesConfiguration", "Microsoft.ContainerService", + "Microsoft.ContainerInstance", + "Microsoft.ContainerRegistry", # Extended Location (for Azure Local) "Microsoft.ExtendedLocation" @@ -156,37 +158,6 @@ if ($networkProvider.RegistrationState -ne 'Registered') { throw "Resource provider Microsoft.Network did not return to Registered state within 5 minutes." } -$aksFeatureName = "AzureLinuxCVMPreview" -$aksFeature = Get-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop - -if ($aksFeature.RegistrationState -ne 'Registered') { - Write-Host " [REGISTERING] Microsoft.ContainerService/$aksFeatureName..." -ForegroundColor Yellow - Register-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop | Out-Null - - for ($attempt = 1; $attempt -le 90; $attempt++) { - Start-Sleep -Seconds 10 - $aksFeature = Get-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop - if ($aksFeature.RegistrationState -eq 'Registered') { - break - } - Write-Host " Waiting for feature registration ($attempt/90): $($aksFeature.RegistrationState)" -ForegroundColor Gray - } -} - -if ($aksFeature.RegistrationState -ne 'Registered') { - throw "Provider feature Microsoft.ContainerService/$aksFeatureName did not reach Registered state within 15 minutes (current state: $($aksFeature.RegistrationState))." -} - -Write-Host " [REGISTERED] Microsoft.ContainerService/$aksFeatureName" -ForegroundColor Green Register-AzResourceProvider -ProviderNamespace "Microsoft.ContainerService" -ErrorAction Stop | Out-Null for ($attempt = 1; $attempt -le 30; $attempt++) { diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/shared-deploy-lab.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/shared-deploy-lab.ps1 index bd985c5e7..9a23caae8 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/shared-deploy-lab.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/shared-deploy-lab.ps1 @@ -99,11 +99,7 @@ foreach($candidate in $confidentialCandidates) { $regionalUsage = Get-AzVMUsage -Location $candidateLocation -ErrorAction Stop $locationReady = $true - $quotaRequirements = @( - @{ Name = $candidate.QuotaName; PerParticipant = 4 } - @{ Name = 'StandardDSv5Family'; PerParticipant = 12 } - @{ Name = 'cores'; PerParticipant = 16 } - ) + $quotaRequirements = @(Get-MhhSovereignComputeRequirements -ConfidentialQuotaName $candidate.QuotaName) foreach($requirement in $quotaRequirements) { $quotaName = $requirement.Name $quotaRequired = $requirement.PerParticipant * $participantCount diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/sovereign-lab.bicep b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/sovereign-lab.bicep index 9f957ecd7..31c810d9e 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/sovereign-lab.bicep +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/sovereign-lab.bicep @@ -15,37 +15,33 @@ param adminUsername string = 'azureuser' @secure() param adminPassword string -@description('Local administrator password for the Confidential VM.') -@secure() -param cvmAdminPassword string - @description('AKS system-pool VM size deployed in the region selected during shared preparation.') param aksNodeVmSize string = 'Standard_D4s_v5' @description('Private K3s VM size deployed in the region selected during shared preparation.') param k3sVmSize string = 'Standard_D4s_v5' -@description('AMD SEV-SNP VM size used by Challenge 4 and the Challenge 5 AKS node pool.') +@description('AMD SEV-SNP VM size for the shared Challenge 5/7 confidential Ubuntu node pool.') +@allowed([ + 'Standard_DC2as_v5' + 'Standard_DC2as_v6' +]) param confidentialVmSize string = 'Standard_DC2as_v5' var aksName = 'aks-sovereign-${nameSuffix}' var vmName = 'vm-k3s-${nameSuffix}' -var cvmName = 'vm-cvm-${nameSuffix}' var vnetName = 'vnet-sovereign-${nameSuffix}' var nsgName = 'nsg-sovereign-${nameSuffix}' var nicName = 'nic-k3s-${nameSuffix}' -var cvmNicName = 'nic-cvm-${nameSuffix}' var bastionName = 'bas-sovereign-${nameSuffix}' var bastionPublicIpName = 'pip-bastion-${nameSuffix}' var natPublicIpName = 'pip-nat-${nameSuffix}' var natGatewayName = 'nat-sovereign-${nameSuffix}' -var attestationProviderName = 'attest${nameSuffix}' var k3sPrivateIp = '10.42.0.4' -var cvmPrivateIp = '10.42.2.4' var tags = { workload: 'sovereign-lab' - challenges: '4,5,7' + challenges: '5,7' CostControl: 'Ignore' } @@ -171,16 +167,6 @@ resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' = { addressPrefix: '10.42.1.0/26' } } - { - name: 'snet-cvm' - properties: { - addressPrefix: '10.42.2.0/24' - defaultOutboundAccess: true - networkSecurityGroup: { - id: nsg.id - } - } - } ] } } @@ -195,11 +181,6 @@ resource bastionSubnet 'Microsoft.Network/virtualNetworks/subnets@2024-05-01' ex parent: vnet } -resource cvmSubnet 'Microsoft.Network/virtualNetworks/subnets@2024-05-01' existing = { - name: 'snet-cvm' - parent: vnet -} - resource nic 'Microsoft.Network/networkInterfaces@2024-05-01' = { name: nicName location: location @@ -223,29 +204,6 @@ resource nic 'Microsoft.Network/networkInterfaces@2024-05-01' = { } } -resource cvmNic 'Microsoft.Network/networkInterfaces@2024-05-01' = { - name: cvmNicName - location: location - tags: tags - properties: { - networkSecurityGroup: { - id: nsg.id - } - ipConfigurations: [ - { - name: 'ipconfig1' - properties: { - privateIPAllocationMethod: 'Static' - privateIPAddress: cvmPrivateIp - subnet: { - id: cvmSubnet.id - } - } - } - ] - } -} - resource k3sVm 'Microsoft.Compute/virtualMachines@2024-11-01' = { name: vmName location: location @@ -303,69 +261,6 @@ resource k3sExtension 'Microsoft.Compute/virtualMachines/extensions@2024-11-01' } } -resource confidentialVm 'Microsoft.Compute/virtualMachines@2024-11-01' = { - name: cvmName - location: location - tags: tags - identity: { - type: 'SystemAssigned' - } - properties: { - hardwareProfile: { - vmSize: confidentialVmSize - } - securityProfile: { - securityType: 'ConfidentialVM' - uefiSettings: { - secureBootEnabled: true - vTpmEnabled: true - } - } - osProfile: { - computerName: cvmName - adminUsername: adminUsername - adminPassword: cvmAdminPassword - linuxConfiguration: { - disablePasswordAuthentication: false - provisionVMAgent: true - } - } - storageProfile: { - imageReference: { - publisher: 'Canonical' - offer: '0001-com-ubuntu-confidential-vm-jammy' - sku: '22_04-lts-cvm' - version: 'latest' - } - osDisk: { - createOption: 'FromImage' - managedDisk: { - storageAccountType: 'Premium_LRS' - securityProfile: { - securityEncryptionType: 'VMGuestStateOnly' - } - } - } - } - networkProfile: { - networkInterfaces: [ - { - id: cvmNic.id - } - ] - } - } -} - -resource attestationProvider 'Microsoft.Attestation/attestationProviders@2021-06-01' = { - name: attestationProviderName - location: location - tags: tags - properties: { - publicNetworkAccess: 'Enabled' - } -} - resource bastionPublicIp 'Microsoft.Network/publicIPAddresses@2024-05-01' = { name: bastionPublicIpName location: location @@ -415,6 +310,10 @@ resource aks 'Microsoft.ContainerService/managedClusters@2024-10-01' = { properties: { dnsPrefix: aksName enableRBAC: true + autoUpgradeProfile: { + upgradeChannel: 'stable' + nodeOSUpgradeChannel: 'NodeImage' + } oidcIssuerProfile: { enabled: true } @@ -454,11 +353,15 @@ resource confidentialNodePool 'Microsoft.ContainerService/managedClusters/agentP name: 'cvmnodepool' parent: aks properties: { - count: 1 + count: 2 tags: tags vmSize: confidentialVmSize osType: 'Linux' - osSKU: 'AzureLinux' + osSKU: 'Ubuntu' + nodeLabels: { + workload: 'confidential' + sku: 'amd-sev-snp' + } mode: 'User' type: 'VirtualMachineScaleSets' osDiskType: 'Managed' @@ -467,10 +370,6 @@ resource confidentialNodePool 'Microsoft.ContainerService/managedClusters/agentP output aksClusterName string = aks.name output confidentialNodePoolName string = confidentialNodePool.name -output confidentialVmName string = confidentialVm.name -output confidentialVmPrivateIp string = cvmPrivateIp -output attestationProviderName string = attestationProvider.name -output attestationProviderUri string = attestationProvider.properties.attestUri output bastionName string = bastion.name output k3sVmName string = k3sVm.name output k3sVmResourceId string = k3sVm.id diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md index 3661ac208..0293946ed 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md @@ -54,6 +54,8 @@ Azure Local may append generated suffixes to its `UserStorage1` and `UserStorage On Windows, the script invokes Azure CLI through its bundled Python executable rather than `az.cmd`. This preserves arguments such as `ConvergedSwitch(compute_management)` that the batch wrapper otherwise interprets as command syntax. Keep the standard Azure CLI installation layout intact. +The `aksarc` extension can print a first-use privacy notice even when JSON output is requested. Preparation separates stderr from JSON stdout and checks AKS validation/creation by exit code, then verifies the cluster through ARM reads. If an older script fails with `Conversion from JSON failed` after network preparation, use the corrected script and rerun with the same parameters after inspecting resource status. Matching resources are reused; do not delete them to fix an output-parsing error. + Use `-ImageVersion` to pin a Marketplace version, `-KubernetesVersion` for a supported AKS version, and size/count overrides when required. Images can take several hours; `-TimeoutMinutes` defaults to 360 per long operation. A timed-out local command does not cancel an already submitted Azure operation. Inspect status before rerunning. Matching resources are reused; incompatible resources produce an error, never automatic replacement. If an image was imported manually before preparation under the documented name but on another storage path, preparation stops before disk expansion. The facilitator must review its dependencies; only if the image is confirmed unused should it be removed manually so preparation can import it on `UserStorage1` under the same documented name. Otherwise, resolve placement with the Azure Local administrator before proceeding. Do not change image names or use `-RemoveUserStorage2` to work around this conflict. Automation never deletes or moves an existing image. diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/readme.md index b1735a8ff..21a57c85e 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/readme.md @@ -12,7 +12,9 @@ Install PowerShell 7, Azure CLI and the Az modules required by each script. Auth ## Deploy participant infrastructure -The [Sovereign lab Bicep template](../../labautomation/sovereign-lab.bicep) provisions the Azure AKS cluster and confidential node pool, confidential VM, K3s VM, attestation, Bastion and networking used across Challenges 4, 5 and 7. Deploy it into your chosen resource group with your selected region, a unique `nameSuffix`, and secure `adminPassword`/`cvmAdminPassword` inputs using Azure's deployment tooling. Do not put passwords in source control or shell history. Retain the successful deployment name and outputs for health testing. +The [Sovereign lab Bicep template](../../labautomation/sovereign-lab.bicep) provisions the shared Challenge 5/7 AKS cluster with two system and two Ubuntu confidential nodes, K3s VM, Bastion and networking. Deploy it with your selected region, a unique `nameSuffix`, a supported `confidentialVmSize`, and a secure `adminPassword` for K3s. No standalone CVM password or attestation provider is required. Do not put passwords in source control or shell history. Retain the successful deployment name and outputs for health testing; provide `aksClusterName` and `confidentialNodePoolName` to participants. + +Challenge 4 provisions its own ACI/ACR comparison in North Europe. Challenge 5 consumes the shared cluster in its actual region; its cleanup deletes only applications. See [regional findings, quotas and migration guidance](../../Readme.md#general-prerequisites) before using an existing lab. For organizers reproducing the full Console workflow locally, follow the [template's local testing instructions](../../../../../99-MicroHack-Template/labautomation/README.md#local-testing), including its helper container and shared preparation before per-lab deployment. The Console hooks are not general standalone deployment scripts. diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/1-resource-providers.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/1-resource-providers.ps1 index 94b321443..1d9010da6 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/1-resource-providers.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/1-resource-providers.ps1 @@ -115,6 +115,8 @@ $providers = @( "Microsoft.Kubernetes", "Microsoft.KubernetesConfiguration", "Microsoft.ContainerService", + "Microsoft.ContainerInstance", + "Microsoft.ContainerRegistry", # Extended Location (for Azure Local) "Microsoft.ExtendedLocation" @@ -195,37 +197,6 @@ foreach ($subscription in $subscriptions) { throw "Resource provider Microsoft.Network did not return to Registered state within 5 minutes." } - $aksFeatureName = "AzureLinuxCVMPreview" - $aksFeature = Get-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop - - if ($aksFeature.RegistrationState -ne 'Registered') { - Write-Host " [REGISTERING] Microsoft.ContainerService/$aksFeatureName..." -ForegroundColor Yellow - Register-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop | Out-Null - - for ($attempt = 1; $attempt -le 90; $attempt++) { - Start-Sleep -Seconds 10 - $aksFeature = Get-AzProviderFeature ` - -ProviderNamespace "Microsoft.ContainerService" ` - -FeatureName $aksFeatureName ` - -ErrorAction Stop - if ($aksFeature.RegistrationState -eq 'Registered') { - break - } - Write-Host " Waiting for feature registration ($attempt/90): $($aksFeature.RegistrationState)" -ForegroundColor Gray - } - } - - if ($aksFeature.RegistrationState -ne 'Registered') { - throw "Provider feature Microsoft.ContainerService/$aksFeatureName did not reach Registered state within 15 minutes (current state: $($aksFeature.RegistrationState))." - } - - Write-Host " [REGISTERED] Microsoft.ContainerService/$aksFeatureName" -ForegroundColor Green Register-AzResourceProvider -ProviderNamespace "Microsoft.ContainerService" -ErrorAction Stop | Out-Null for ($attempt = 1; $attempt -le 30; $attempt++) { diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/2-vcpu-quotas.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/2-vcpu-quotas.ps1 index c69c51e58..017f94cf2 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/2-vcpu-quotas.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/manual-setup/subscription-preparations/2-vcpu-quotas.ps1 @@ -6,11 +6,14 @@ This script checks current vCPU quota usage in a specified Azure region and calculates the required quotas for running the MicroHack lab based on the number of lab users. - The shared Challenge 4/5/7 platform requires per participant: - - 12 Standard DSv5 Family vCPUs - - 4 Standard DCasv5 or DCasv6 Family vCPUs + The shared Challenge 5/7 platform requires per participant: + - 12 Standard DSv5 Family vCPUs (two AKS system nodes and one K3s VM) + - 4 Standard DCasv5 or DCasv6 Family vCPUs (two Ubuntu AKS confidential nodes) - 16 Total Regional vCPUs + These are steady-state totals; reserve additional upgrade surge headroom separately. + Challenge 4 ACI capacity/quota is separate from these VM-family quotas. + The script can optionally submit quota increase requests using the Azure Quota REST API. .PARAMETER Region @@ -470,7 +473,7 @@ foreach ($subscription in $selectedSubscriptions) { # Get number of lab users $NumberOfLabUsers = Get-LabUserCount - # Calculate requirements for the shared Challenge 4/5/7 participant platform. + # Calculate requirements for the shared Challenge 5/7 participant platform. Write-Host "`n=== Lab Requirements Calculation ===" -ForegroundColor Cyan Write-Host ("=" * 80) @@ -480,18 +483,18 @@ foreach ($subscription in $selectedSubscriptions) { $confidentialQuotaDisplayName = "Standard DCas$ConfidentialVmGeneration Family vCPUs (Confidential)" $requirements = @{ "cores" = @{ - PerUser = 16 + PerUser = ((2 + 1) * 4) + (2 * 2) Shared = 0 Name = "Total Regional vCPUs" } "StandardDSv5Family" = @{ - PerUser = 12 + PerUser = (2 + 1) * 4 Shared = 0 Name = "Standard DSv5 Family vCPUs" } } $requirements[$confidentialQuotaName] = @{ - PerUser = 4 + PerUser = 2 * 2 Shared = 0 Name = $confidentialQuotaDisplayName } diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 index d1a64cf6e..4d0d3577d 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 @@ -130,15 +130,27 @@ function Get-LocalBoxAzInvocation { } function Invoke-LocalBoxAz { - param([Parameter(Mandatory)][string[]]$Arguments, [int]$TimeoutSeconds = 300) + param([Parameter(Mandatory)][string[]]$Arguments, [int]$TimeoutSeconds = 300, [switch]$NoOutput) $invocation = Get-LocalBoxAzInvocation - $job = Start-Job -ArgumentList $invocation, (, $Arguments) -ScriptBlock { - param($Invocation, $CommandArguments) + $job = Start-Job -ArgumentList $invocation, $Arguments, ([bool]$NoOutput) -ScriptBlock { + param($Invocation, $CommandArguments, $DiscardOutput) + $PSNativeCommandUseErrorActionPreference = $false $prefix = $Invocation.Prefix - $output = & $Invocation.Executable @prefix @CommandArguments --only-show-errors --output json 2>&1 - if ($LASTEXITCODE -ne 0) { throw ($output -join "`n") } - $text = $output -join "`n" - if ($text.Trim()) { $text | ConvertFrom-Json -AsHashtable -ErrorAction Stop } + $format = if ($DiscardOutput) { 'none' } else { 'json' } + $output = @(& $Invocation.Executable @prefix @CommandArguments --only-show-errors --output $format 2>&1) + $exitCode = $LASTEXITCODE + $operation = (@($CommandArguments | Select-Object -First 2) -join ' ') + if ($CommandArguments -contains '--validate') { $operation += ' --validate' } + $stderr = @($output | Where-Object { $_ -is [System.Management.Automation.ErrorRecord] }) + $text = ($output | Where-Object { $_ -isnot [System.Management.Automation.ErrorRecord] }) -join "`n" + if ($exitCode -ne 0) { throw "Azure CLI 'az $operation' failed (exit code $exitCode): $($stderr -join "`n")" } + if ($DiscardOutput) { return } + if ($text.Trim()) { + try { $text | ConvertFrom-Json -AsHashtable -ErrorAction Stop } + catch { + throw "Azure CLI 'az $operation' returned non-JSON stdout despite --output json (exit code $exitCode). Raw output is omitted because it may contain sensitive data. Inspect the resource state before retrying; a submitted operation may still be running." + } + } } try { if (-not (Wait-Job $job -Timeout $TimeoutSeconds)) { @@ -533,8 +545,10 @@ function Invoke-LocalBoxPreparation { '--node-count', [string]$Settings.NodeCount, '--node-vm-size', $Settings.NodeVmSize, '--control-plane-count', '1', '--control-plane-vm-size', $Settings.ControlPlaneVmSize) if ($Settings.KubernetesVersion) { $arguments += @('--kubernetes-version', $Settings.KubernetesVersion) } if ($PSCmdlet.ShouldProcess($aksId, 'Validate and create AKS on Azure Local')) { - Invoke-LocalBoxAz ($arguments + @('--validate')) -TimeoutSeconds $timeout | Out-Null - Invoke-LocalBoxAz $arguments -TimeoutSeconds $timeout | Out-Null + Write-Host "Validating AKS on Azure Local $($Settings.AksClusterName)..." + Invoke-LocalBoxAz ($arguments + @('--validate')) -TimeoutSeconds $timeout -NoOutput + Write-Host "Creating AKS on Azure Local $($Settings.AksClusterName)..." + Invoke-LocalBoxAz $arguments -TimeoutSeconds $timeout -NoOutput } } if (-not $Settings.SkipAks -and -not $WhatIfPreference) { diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/readme.md index 4717c0d00..f159e2e96 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/readme.md @@ -14,7 +14,8 @@ Both paths use [LocalBox preparation](localbox/readme.md) and the [Pester health | Task | Hosted event | Manual delivery | | --- | --- | --- | | Subscriptions, users, initial lab RBAC, provider/quota preparation | Console and content automation | Subscription owner | -| Participant Azure AKS, confidential VM, K3s, attestation and networking | Console content automation | Subscription owner using the lab template | +| Participant shared Azure AKS with Ubuntu confidential pool, K3s and networking | Console content automation for Challenges 5/7 | Subscription owner using the lab template | +| Challenge 4 ACI/ACR comparison and Challenge 5 applications | Participants using walkthrough scripts | Same walkthrough scripts against the prepared scope | | Shared LocalBox deployment | Console submits once per subscription; full readiness still requires verification | Subscription owner deploys LocalBox | | Storage, Windows image, VM/AKS networks, AKS on Azure Local | Event Lead runs preparation on each LocalBox Client | Same preparation script | | AKS Local Entra admin group | Console-owned event group; use **Lab Group ObjectId** from the Credentials tab | Supply an existing security group with intended users as members | diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 index a195afb66..3e957b97b 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 @@ -172,7 +172,7 @@ Describe 'Lab cost-control tags' { $source = Get-Content "$PSScriptRoot/../../labautomation/sovereign-lab.bicep" -Raw $source | Should -Match "(?s)var tags = \{[^}]*CostControl: 'Ignore'" $source | Should -Match "(?s)name: 'system'\s+count: 2\s+tags: tags" - $source | Should -Match "(?s)resource confidentialNodePool .*?properties: \{\s+count: 1\s+tags: tags" + $source | Should -Match "(?s)resource confidentialNodePool .*?properties: \{\s+count: 2\s+tags: tags" } } @@ -373,6 +373,93 @@ Describe 'LocalBox image storage conflicts' { Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments -contains 'create' } } } +< +Describe 'CLI output streams' { + BeforeEach { + $script:cliTestExecutable = (Get-Process -Id $PID).Path + Mock Get-LocalBoxAzInvocation { + @{ Executable = $script:cliTestExecutable; Prefix = @('-NoProfile', '-NonInteractive', '-Command', ($script:cliTestCommand + "`n#")) } + } + } + It 'parses JSON stdout when a successful native command also writes progress to stderr' { + $script:cliTestCommand = { + [Console]::Error.WriteLine('Progress: completing operation') + [Console]::Out.WriteLine('{"name":"ready"}') + exit 0 + }.ToString() + (Invoke-LocalBoxAz @('resource', 'show')).name | Should -Be 'ready' + } + It 'parses array output without stderr records leaking into resource discovery' { + $script:cliTestCommand = { + [Console]::Error.WriteLine('Progress: completing discovery') + [Console]::Out.WriteLine('[{"name":"first"},{"name":"second"}]') + exit 0 + }.ToString() + $result = @(Invoke-LocalBoxAz @('resource', 'list')) + $result.Count | Should -Be 2 + $result[0].name | Should -Be 'first' + $result[1].name | Should -Be 'second' + } + It 'accepts empty successful stdout without leaking stderr into results' { + $script:cliTestCommand = { + [Console]::Error.WriteLine('Progress: completing operation') + exit 0 + }.ToString() + @(Invoke-LocalBoxAz @('account', 'set')).Count | Should -Be 0 + } + It 'accepts a first-use notice with no JSON for an exit-code-only validation command' { + $script:cliTestCommand = { + [Console]::Out.WriteLine('Privacy notice: first invocation') + exit 0 + }.ToString() + @(Invoke-LocalBoxAz @('aksarc', 'create', '--validate') -NoOutput).Count | Should -Be 0 + } + It 'discards notice and JSON output when creation results are checked through ARM reads' { + $script:cliTestCommand = { + [Console]::Out.WriteLine('Privacy notice: first invocation') + [Console]::Out.WriteLine('{"name":"cluster"}') + exit 0 + }.ToString() + @(Invoke-LocalBoxAz @('aksarc', 'create') -NoOutput).Count | Should -Be 0 + } + It 'rejects a failed exit-code-only command and retains command context and stderr' { + $script:cliTestCommand = { + [Console]::Out.WriteLine('Privacy notice: first invocation') + [Console]::Error.WriteLine('AuthorizationFailed: test failure') + exit 17 + }.ToString() + { Invoke-LocalBoxAz @('aksarc', 'create', '--validate') -NoOutput } | Should -Throw '*az aksarc create --validate*exit code 17*AuthorizationFailed*' + } + It 'rejects a nonzero exit even when stdout contains valid JSON' { + $script:cliTestCommand = { + [Console]::Out.WriteLine('{"name":"not-success"}') + [Console]::Error.WriteLine('Resource read failed') + exit 9 + }.ToString() + { Invoke-LocalBoxAz @('resource', 'show') } | Should -Throw '*az resource show*exit code 9*Resource read failed*' + } + It 'rejects malformed read output with context but without exposing raw stdout' { + $script:cliTestCommand = { + [Console]::Out.WriteLine('mock-sensitive-payload') + exit 0 + }.ToString() + $failure = $null + try { Invoke-LocalBoxAz @('resource', 'show') } + catch { $failure = $_ } + $failure | Should -Not -BeNullOrEmpty + $failure.Exception.Message | Should -Match 'az resource show.*non-JSON stdout' + $failure.Exception.Message | Should -Not -Match 'mock-sensitive-payload' + $failure.Exception.Message | Should -Match 'Inspect the resource state before retrying' + } + It 'limits non-JSON handling to AKS validate and create while retaining ARM verification' { + $source = ${function:Invoke-LocalBoxPreparation}.ToString() + $source | Should -Match 'Invoke-LocalBoxAz \(\$arguments \+ @\(''--validate''\)\) -TimeoutSeconds \$timeout -NoOutput' + $source | Should -Match 'Invoke-LocalBoxAz \$arguments -TimeoutSeconds \$timeout -NoOutput' + $source | Should -Match 'Wait-LocalBoxResource \$instanceId' + $source | Should -Match 'Assert-LocalBoxProperties \$instance \$aksExpected' + $source | Should -Match 'Assert-LocalBoxProperties \$connected' + } +} Describe 'CLI process cleanup' { BeforeEach { diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md index 3e09e8b92..4196c6736 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md @@ -92,9 +92,9 @@ For manual deployments without retained deployment records, replace `DeploymentN -Mode Full -AllowGuestRunCommand -DownloadTests -GitHubRef $ref ``` -`-AllowGuestRunCommand` permits Azure VM Run Command on the two selected VMs solely to check K3s service state, DNS and HTTPS egress. This transport is an Azure management action and requires scoped Run Command permission even though the guest commands are read-only. It can create transient Run Command execution artifacts. Without consent, that mandatory Full check fails; it is not silently skipped. +`-AllowGuestRunCommand` permits Azure VM Run Command on the selected K3s VM solely to check K3s service state, DNS and HTTPS egress. This transport is an Azure management action and requires scoped Run Command permission even though the guest commands are read-only. It can create transient Run Command execution artifacts. Without consent, that mandatory Full check fails; it is not silently skipped. -Participant checks include VM power/agent state, confidential VM security settings, K3s install and runtime health, Azure AKS/system/confidential pools with OIDC/workload identity/Istio, private networking/NSG/NAT/Bastion configuration and attestation discovery/JWKS. Bastion's interactive user access and an actual confidential-computing attestation exchange remain separate participant smoke tests, not claims made by ARM health queries. +Participant baseline checks cover K3s VM power/agent/install/runtime health, two system and two labeled Ubuntu confidential AKS nodes, OIDC/workload identity/Istio and upgrade settings, and private networking/NSG/NAT/Bastion configuration. They no longer require a standalone CVM or custom attestation provider. Challenge 4 ACI/ACR, Challenge 5 applications, Bastion interactive access and actual signed attestation exchanges are participant smoke tests, not claims made by baseline ARM health checks. For both paths together, use `-Scope All`, supply the participant inventory and LocalBox manifest/kubeconfig/Windows credential. Use an operator context authorized for all specified scopes, not a newly broadened managed identity. @@ -111,6 +111,7 @@ Waits are bounded by `-TimeoutMinutes` per check. Terminal provisioning failures ```powershell Invoke-Pester ./tests/prepare-localbox.tests.ps1 -Output Detailed +Invoke-Pester ./tests/shared-aks.tests.ps1 -Output Detailed ``` -Run from the resources directory. This suite tests address and group validation, resource reuse/conflicts, partial-state polling/timeouts, destructive-operation guards, and refusal to report incomplete Kubernetes results as full readiness. It needs no Azure login or infrastructure. Do not run the entire tests folder by default: the two `*.health.tests.ps1` suites require runner-supplied data and live access. \ No newline at end of file +Run from the resources directory. These suites test LocalBox validation/reconciliation and shared-AKS workload targeting, namespace ownership, cleanup safety, pool compatibility and quota contracts. They need no Azure login or infrastructure. Do not run the entire tests folder by default: the two `*.health.tests.ps1` suites require runner-supplied data and live access. \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/shared-aks.tests.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/shared-aks.tests.ps1 new file mode 100644 index 000000000..deafaff99 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/shared-aks.tests.ps1 @@ -0,0 +1,171 @@ +BeforeAll { + $root = (Resolve-Path "$PSScriptRoot/../..").Path + $workloadScript = "$root/walkthrough/challenge-05/Deploy-VotingAppCC.ps1" + function az {} + function kubectl {} + . "$root/labautomation/quota-helpers.ps1" +} + +AfterAll { + Remove-Variable -Name SharedAks_* -Scope Global -ErrorAction SilentlyContinue +} + +Describe 'Shared AKS workload lifecycle' { + BeforeEach { + $global:SharedAks_azCalls = [Collections.Generic.List[object]]::new() + $global:SharedAks_kubeCalls = [Collections.Generic.List[object]]::new() + $global:SharedAks_pool = @{ provisioningState = 'Succeeded'; count = 2; osSKU = 'Ubuntu'; vmSize = 'Standard_DC2as_v5'; mode = 'User'; nodeLabels = @{ workload = 'confidential' } } + $global:SharedAks_namespace = @{ metadata = @{ labels = @{ 'microhack-challenge' = '05' } } } + Mock az { + $global:SharedAks_azCalls.Add(@($args)) + $global:LASTEXITCODE = 0 + if ($args[0] -eq 'account') { return 'test-subscription' } + if ($args[1] -eq 'show') { return '{"location":"spaincentral","provisioningState":"Succeeded"}' } + if ($args[1] -eq 'nodepool') { return ($global:SharedAks_pool | ConvertTo-Json -Depth 5 -Compress) } + if ($args[1] -eq 'get-credentials') { + $path = $args[[array]::IndexOf($args, '--file') + 1] + [IO.File]::WriteAllText($path, 'mock-kubeconfig') + } + } + Mock kubectl { + $global:SharedAks_kubeCalls.Add(@($args)) + $global:LASTEXITCODE = 0 + if ($args[6] -eq 'get' -and $args[7] -eq 'namespace') { + if ($global:SharedAks_namespace) { return ($global:SharedAks_namespace | ConvertTo-Json -Depth 5 -Compress) } + } + if ($args[6] -eq 'get' -and $args[7] -eq 'service') { return '192.0.2.1' } + if ($args[6] -eq 'create' -and $args[7] -eq 'configmap') { return 'apiVersion: v1' } + } + Mock Invoke-WebRequest { @{ StatusCode = 200; Content = 'Azure Voting App' } } + Mock Start-Sleep {} + } + It 'deploys only workloads into an explicit namespace and context on the provided cluster' { + & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' + @($global:SharedAks_azCalls | Where-Object { $_[1] -in @('create','delete') -or $_ -contains 'add' }).Count | Should -Be 0 + foreach ($call in $global:SharedAks_kubeCalls) { + $call[0] | Should -Be '--kubeconfig' + $call[2] | Should -Be '--context' + $call[3] | Should -Be 'challenge-05-test-subscription-aks-from-console' + $call[4] | Should -Be '--namespace' + $call[5] | Should -Be 'challenge-05' + Test-Path -LiteralPath $call[1] | Should -BeFalse + } + @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'apply' }).Count | Should -Be 3 + } + It 'cleanup removes only named applications, not cluster, pools, namespace or Radius' { + & $workloadScript -Cleanup -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' + $deletes = @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'delete' }) + $deletes.Count | Should -Be 1 + $deletes[0][5] | Should -Be 'challenge-05' + $deletes[0] | Should -Contain 'deployment/azure-vote-front' + $deletes[0] | Should -Contain 'deployment/cc-attest' + $deletes[0] | Should -Not -Contain 'namespace' + @($global:SharedAks_azCalls | Where-Object { $_ -contains 'delete' }).Count | Should -Be 0 + } + It 'cleanup is a no-op if its namespace is absent' { + $manifestFile = Join-Path $TestDrive 'caller-owned.txt' + [IO.File]::WriteAllText($manifestFile, 'Keep caller files') + $global:SharedAks_namespace = $null + & $workloadScript -Cleanup -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' + @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'delete' }).Count | Should -Be 0 + Test-Path $manifestFile | Should -BeTrue + } + It 'creates a dedicated namespace when absent and accepts the selected v6 confidential SKU' { + $global:SharedAks_namespace = $null + $global:SharedAks_pool.vmSize = 'Standard_DC2as_v6' + & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' + @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'apply' }).Count | Should -Be 4 + } + It 'stops if the provided pool cannot be read instead of provisioning another one' { + Mock az { $global:LASTEXITCODE = 1 } -ParameterFilter { $args[1] -eq 'nodepool' } + { & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*pool is missing*' + $global:SharedAks_kubeCalls.Count | Should -Be 0 + } + It 'cleans the private kubeconfig when application cleanup fails' { + Mock kubectl { $global:LASTEXITCODE = 1 } -ParameterFilter { $args[6] -eq 'delete' } + { & $workloadScript -Cleanup -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*application cleanup failed*' + Test-Path $global:SharedAks_kubeCalls[0][1] | Should -BeFalse + } + It 'refuses application deployment into an Istio-injected namespace' { + $global:SharedAks_namespace.metadata.labels.'istio.io/rev' = 'asm-1-24' + { & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*without Istio sidecar injection*' + @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'apply' }).Count | Should -Be 0 + } + It 'rejects a confidential pool without the workload selector label' { + $global:SharedAks_pool.nodeLabels = @{} + { & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*labelled workload=confidential*' + $global:SharedAks_kubeCalls.Count | Should -Be 0 + } + It 'refuses to delete resources in an unowned namespace' { + $global:SharedAks_namespace.metadata.labels = @{} + { & $workloadScript -Cleanup -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*ownership label*' + @($global:SharedAks_kubeCalls | Where-Object { $_[6] -eq 'delete' }).Count | Should -Be 0 + Test-Path $global:SharedAks_kubeCalls[0][1] | Should -BeFalse + } + It 'rejects incompatible pools without mutating infrastructure' -TestCases @( + @{ Property = 'osSKU'; Value = 'AzureLinux' } + @{ Property = 'count'; Value = 1 } + @{ Property = 'vmSize'; Value = 'Standard_D4s_v5' } + @{ Property = 'provisioningState'; Value = 'Failed' } + ) { + param($Property, $Value) + $global:SharedAks_pool[$Property] = $Value + { & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName 'aks-from-console' } | Should -Throw '*Expected two ready Ubuntu*' + $global:SharedAks_kubeCalls.Count | Should -Be 0 + } + It 'requires an explicit cluster rather than deriving one from attendee names' { + { & $workloadScript -Deploy -ResourceGroup 'lab-test' -ClusterName '' } | Should -Throw '*AKS_CLUSTER*' + $global:SharedAks_azCalls.Count | Should -Be 0 + } + It 'has no unscoped kubectl or infrastructure mutation commands' { + $errors = $null + $ast = [Management.Automation.Language.Parser]::ParseFile($workloadScript, [ref]$null, [ref]$errors) + $errors.Count | Should -Be 0 + $commands = $ast.FindAll({ param($node) $node -is [Management.Automation.Language.CommandAst] }, $true) + foreach ($command in $commands) { + if ($command.GetCommandName() -eq 'kubectl') { $command.Extent.Text | Should -Match 'kubectl @kubectlScope' } + if ($command.GetCommandName() -eq 'az') { $command.Extent.Text | Should -Not -Match 'az (aks (create|delete|nodepool (add|delete))|group delete)' } + } + } +} + +Describe 'Shared AKS infrastructure contract' { + It 'documents workload-only deployment and cleanup on the provided cluster' { + $guide = Get-Content "$root/walkthrough/challenge-05/solution-05.md" -Raw + $guide | Should -Not -Match 'Cleanup deletes the cluster|two separate sign-ins|Set-AzContext|CcNodeCount' + $guide | Should -Match 'AKS_CLUSTER' + $guide | Should -Match 'challenge-05' + } + It 'reserves four confidential and twelve standard vCPUs without a standalone VM' { + $requirements = @(Get-MhhSovereignComputeRequirements -ConfidentialQuotaName 'standardDCasv6Family') + ($requirements | Where-Object Name -eq 'standardDCasv6Family').PerParticipant | Should -Be 4 + ($requirements | Where-Object Name -eq 'StandardDSv5Family').PerParticipant | Should -Be 12 + ($requirements | Where-Object Name -eq 'cores').PerParticipant | Should -Be 16 + foreach ($scriptName in @('deploy-lab.ps1', 'shared-deploy-lab.ps1')) { + Get-Content "$root/labautomation/$scriptName" -Raw | Should -Match 'Get-MhhSovereignComputeRequirements' + } + $defaults = Get-Content "$root/labautomation/lab-defaults.json" -Raw | ConvertFrom-Json + $defaults.estimatedDailyCostsUsd | Should -Be 45 + $defaults.estimatedSharedDeploymentDailyCostsUsd | Should -Be 110 + } + It 'removes obsolete resources and credentials while retaining two labeled Ubuntu confidential nodes' { + $template = Get-Content "$root/labautomation/sovereign-lab.bicep" -Raw + $template | Should -Not -Match 'cvmAdminPassword|resource confidentialVm |resource attestationProvider|cvmNic|snet-cvm' + $template | Should -Match "(?s)resource confidentialNodePool .*?count: 2.*?osSKU: 'Ubuntu'.*?workload: 'confidential'" + $deployer = Get-Content "$root/labautomation/deploy-lab.ps1" -Raw + $deployer | Should -Not -Match 'cvmAdminPassword|confidentialVmName|attestationProviderName|Confidential VM Admin' + $health = Get-Content "$root/resources/tests/sovereign-lab.health.tests.ps1" -Raw + $health | Should -Not -Match 'confidentialVmName|attestationProviderName' + $health | Should -Match 'Test-SovereignKubernetes \$Lab.AksKubeconfig 4' + } + It 'registers ACI and ACR before participant deployment' { + $providers = Get-Content "$root/labautomation/resource-providers.ps1" -Raw + $providers | Should -Match 'Microsoft.ContainerInstance' + $providers | Should -Match 'Microsoft.ContainerRegistry' + $providers | Should -Not -Match 'AzureLinuxCVMPreview' + $manualProviders = Get-Content "$root/resources/manual-setup/subscription-preparations/1-resource-providers.ps1" -Raw + $manualProviders | Should -Match 'Microsoft.ContainerInstance' + $manualProviders | Should -Match 'Microsoft.ContainerRegistry' + $manualProviders | Should -Not -Match 'AzureLinuxCVMPreview' + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/sovereign-lab.health.tests.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/sovereign-lab.health.tests.ps1 index a0576414d..b3ca19638 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/sovereign-lab.health.tests.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/sovereign-lab.health.tests.ps1 @@ -4,7 +4,7 @@ BeforeAll { . "$ResourceRoot/prepare-localbox.ps1" . "$ResourceRoot/test-sovereign-cloud.ps1" -Mode $Mode -AllowGuestRunCommand:$AllowGuestRunCommand $outputs = Get-SovereignLabOutputs $Lab - foreach ($required in @('aksClusterName', 'confidentialNodePoolName', 'confidentialVmName', 'k3sVmName', 'k3sInstallStatusResource', 'attestationProviderName', 'attestationProviderUri', 'bastionName', 'natEgressIp')) { + foreach ($required in @('aksClusterName', 'confidentialNodePoolName', 'k3sVmName', 'k3sInstallStatusResource', 'bastionName', 'natEgressIp')) { if (-not $outputs[$required]) { throw "Missing deployment output $required." } } $scopeId = "/subscriptions/$($Lab.SubscriptionId)/resourceGroups/$($Lab.ResourceGroupName)" @@ -13,22 +13,14 @@ BeforeAll { } Describe 'Participant lab control plane' { - It 'has both running VMs with healthy agents' { - foreach ($name in @($outputs.k3sVmName, $outputs.confidentialVmName)) { + It 'has a running K3s VM with a healthy agent' { + foreach ($name in @($outputs.k3sVmName)) { $vm = Invoke-LocalBoxAz (@('vm', 'get-instance-view', '--name', $name) + $common) $vm.provisioningState | Should -Be 'Succeeded' $vm.instanceView.statuses.code | Should -Contain 'PowerState/running' $vm.instanceView.vmAgent.statuses.code | Should -Contain 'ProvisioningState/succeeded' } } - It 'has a confidential VM with secure boot and vTPM' { - $vm = Get-LocalBoxResource "$scopeId/providers/Microsoft.Compute/virtualMachines/$($outputs.confidentialVmName)" - $vm.properties.securityProfile.securityType | Should -Be 'ConfidentialVM' - $vm.properties.securityProfile.uefiSettings.secureBootEnabled | Should -BeTrue - $vm.properties.securityProfile.uefiSettings.vTpmEnabled | Should -BeTrue - $vm.properties.storageProfile.osDisk.managedDisk.securityProfile.securityEncryptionType | Should -Be 'VMGuestStateOnly' - $vm.properties.hardwareProfile.vmSize | Should -Match '^Standard_DC\d+as_v[56]$' - } It 'has completed the K3s install extension' { Wait-LocalBoxResource $outputs.k3sInstallStatusResource -TimeoutSeconds $TimeoutSeconds | Should -Not -BeNullOrEmpty } @@ -40,27 +32,30 @@ Describe 'Participant lab control plane' { $aks.properties.securityProfile.workloadIdentity.enabled | Should -BeTrue $aks.properties.serviceMeshProfile.mode | Should -Be 'Istio' $aks.properties.networkProfile.networkPluginMode | Should -Be 'overlay' + $aks.properties.autoUpgradeProfile.upgradeChannel | Should -Be 'stable' + $aks.properties.autoUpgradeProfile.nodeOSUpgradeChannel | Should -Be 'NodeImage' $system = Wait-LocalBoxResource "$aksId/agentPools/system" -TimeoutSeconds $TimeoutSeconds $system.properties.count | Should -Be 2 $system.properties.mode | Should -Be 'System' $system.properties.vmSize | Should -Be 'Standard_D4s_v5' $pool = Wait-LocalBoxResource "$aksId/agentPools/$($outputs.confidentialNodePoolName)" -TimeoutSeconds $TimeoutSeconds - $pool.properties.count | Should -Be 1 - $pool.properties.osSKU | Should -Be 'AzureLinux' - $pool.properties.vmSize | Should -Match '^Standard_DC\d+as_v[56]$' + $pool.properties.count | Should -Be 2 + $pool.properties.osSKU | Should -Be 'Ubuntu' + $pool.properties.mode | Should -Be 'User' + $pool.properties.nodeLabels.workload | Should -Be 'confidential' + $pool.properties.vmSize | Should -Match '^Standard_DC2as_v[56]$' } It 'has all expected network components and private VM NICs' { foreach ($type in @('Microsoft.Network/virtualNetworks', 'Microsoft.Network/networkSecurityGroups', 'Microsoft.Network/networkInterfaces', 'Microsoft.Network/natGateways', 'Microsoft.Network/publicIPAddresses', 'Microsoft.Network/bastionHosts')) { @($resources | Where-Object type -ieq $type).Count | Should -BeGreaterThan 0 -Because "$type must exist" } - foreach ($name in @($outputs.k3sVmName, $outputs.confidentialVmName)) { + foreach ($name in @($outputs.k3sVmName)) { $vm = Get-LocalBoxResource "$scopeId/providers/Microsoft.Compute/virtualMachines/$name" $nicId = $vm.properties.networkProfile.networkInterfaces[0].id $nic = Wait-LocalBoxResource $nicId -TimeoutSeconds $TimeoutSeconds $nic.properties.networkSecurityGroup.id | Should -Not -BeNullOrEmpty $nic.properties.ipConfigurations[0].properties.publicIPAddress | Should -BeNullOrEmpty - $expectedIp = if ($name -eq $outputs.k3sVmName) { '10.42.0.4' } else { '10.42.2.4' } - $nic.properties.ipConfigurations[0].properties.privateIPAddress | Should -Be $expectedIp + $nic.properties.ipConfigurations[0].properties.privateIPAddress | Should -Be '10.42.0.4' $subnetId = $nic.properties.ipConfigurations[0].properties.subnet.id $subnet = Get-LocalBoxResource $subnetId $subnet.properties.networkSecurityGroup.id | Should -Be $nic.properties.networkSecurityGroup.id @@ -82,24 +77,20 @@ Describe 'Participant lab control plane' { $bastionSubnet = Get-LocalBoxResource $bastion.properties.ipConfigurations[0].properties.subnet.id $bastionSubnet.properties.addressPrefix | Should -Be '10.42.1.0/26' } - It 'has a provisioned attestation provider and valid discovery endpoint' { - Wait-LocalBoxResource "$scopeId/providers/Microsoft.Attestation/attestationProviders/$($outputs.attestationProviderName)" -TimeoutSeconds $TimeoutSeconds | Should -Not -BeNullOrEmpty - $uri = [uri]$outputs.attestationProviderUri - $uri.Scheme | Should -Be 'https' - $discovery = Invoke-RestMethod -Uri "$($uri.AbsoluteUri.TrimEnd('/'))/.well-known/openid-configuration" -TimeoutSec 30 - $discovery.jwks_uri | Should -Not -BeNullOrEmpty - ([uri]$discovery.jwks_uri).Host | Should -Be $uri.Host - @((Invoke-RestMethod -Uri $discovery.jwks_uri -TimeoutSec 30).keys).Count | Should -BeGreaterThan 0 - } } Describe 'Participant lab runtime health' -Skip:($Mode -ne 'Full') { It 'has healthy Azure AKS nodes, confidential nodes and system workloads' { Wait-SovereignCheck -TimeoutSeconds $TimeoutSeconds -Check { - Test-SovereignKubernetes $Lab.AksKubeconfig 3 + Test-SovereignKubernetes $Lab.AksKubeconfig 4 $nodes = Invoke-SovereignKubectl $Lab.AksKubeconfig @('get', 'nodes') @($nodes.items | Where-Object { $_.metadata.labels.agentpool -eq 'system' }).Count | Should -Be 2 - @($nodes.items | Where-Object { $_.metadata.labels.agentpool -eq $outputs.confidentialNodePoolName }).Count | Should -Be 1 + $confidentialNodes = @($nodes.items | Where-Object { $_.metadata.labels.agentpool -eq $outputs.confidentialNodePoolName }) + $confidentialNodes.Count | Should -Be 2 + foreach ($node in $confidentialNodes) { + $node.metadata.labels.workload | Should -Be 'confidential' + $node.status.nodeInfo.osImage | Should -Match 'Ubuntu' + } } } It 'has a reachable healthy K3s API and system workloads' { @@ -107,7 +98,7 @@ Describe 'Participant lab runtime health' -Skip:($Mode -ne 'Full') { } It 'has working guest DNS and outbound connectivity, and an active K3s service' { $AllowGuestRunCommand | Should -BeTrue -Because 'Full guest probes require explicit -AllowGuestRunCommand; no resources are installed or changed' - foreach ($name in @($outputs.k3sVmName, $outputs.confidentialVmName)) { + foreach ($name in @($outputs.k3sVmName)) { $script = "set -eu`ngetent hosts management.azure.com >/dev/null`ncurl --fail --silent --show-error --max-time 30 'https://management.azure.com/metadata/endpoints?api-version=2020-06-01' >/dev/null`n" if ($name -eq $outputs.k3sVmName) { $script += "systemctl is-active --quiet k3s`n" } $script += "printf '\nMICROHACK_GUEST_HEALTH_OK\n'" diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-04/solution-04.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-04/solution-04.md index 62392a64e..376aeae0e 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-04/solution-04.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-04/solution-04.md @@ -115,6 +115,12 @@ $env:HASH_SUFFIX = [Convert]::ToHexString($hash).Substring(0, 8).ToLowerInvarian Confirm the target resource group and subscription: +`LOCATION` here selects the ACI/ACR deployment region only. Keep North Europe +for this validated exercise even if Console created the resource group and +shared AKS cluster in Sweden Central or Spain Central. A resource group's +metadata location does not force every resource into that region. In Challenge 5, +use the existing Console AKS cluster and its actual region instead. + ```powershell az account show --query "{subscription:name, id:id}" --output table az group show --name $env:RESOURCE_GROUP --query "{name:name, location:location}" --output table diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/Deploy-VotingAppCC.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/Deploy-VotingAppCC.ps1 index 8f301687b..ea764efff 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/Deploy-VotingAppCC.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/Deploy-VotingAppCC.ps1 @@ -4,15 +4,13 @@ .DESCRIPTION Adapts the upstream Azure Voting App confidential-computing sample to the - MicroHack environment variables and existing attendee resource group. + provided MicroHack AKS cluster shared with Challenge 7. Only Challenge 5 + workloads in its owned namespace are deployed or removed. .EXAMPLE ./Deploy-VotingAppCC.ps1 -Deploy ./Deploy-VotingAppCC.ps1 -Cleanup #> -# Hands-off script to build a small AKS cluster with an AMD SEV-SNP Confidential Computing node pool -# and deploy the public Microsoft "Azure Voting App" multi-container sample to it, exposed via a public -# LoadBalancer. Derived from the upstream sample with MicroHack naming and resource-group lifecycle. # # Simon Gallagher, ACC Product Group # Use at your own risk, no warranties implied, test in a non-production environment first @@ -25,25 +23,20 @@ # - https://github.com/Azure-Samples/azure-voting-app-redis (the demo app) # # Usage: -# ./Deploy-VotingAppCC.ps1 -Deploy [-SkipSkuPreflight] +# ./Deploy-VotingAppCC.ps1 -Deploy -ClusterName # ./Deploy-VotingAppCC.ps1 -Cleanup # # Requirements: -# - Azure PowerShell (Az) module + Azure CLI (az), both logged in to the same subscription -# - kubectl on PATH (az aks install-cli will install it if missing) +# - Azure CLI signed in to the subscription containing the provided AKS cluster +# - kubectl installed on PATH [CmdletBinding(DefaultParameterSetName='Help')] param ( [Parameter(ParameterSetName='Deploy')][switch]$Deploy, [Parameter(ParameterSetName='Cleanup')][switch]$Cleanup, - [Parameter(ParameterSetName='Deploy')][switch]$SkipSkuPreflight, [string]$ResourceGroup = $env:RESOURCE_GROUP, - [string]$AttendeeId = $env:ATTENDEE_ID, - [string]$HashSuffix = $env:HASH_SUFFIX, - [string]$Location = $env:LOCATION, - [string]$CcVmSize = "Standard_DC2as_v5", # smallest AMD SEV-SNP CVM (2 vCPU / 8 GiB) - [string]$SystemVmSize = "Standard_D2as_v6", # tiny non-CC system pool (AMD, allowed by typical Allowed-VM-SKUs policies) - [ValidateRange(1, 10)][int]$CcNodeCount = 2 + [string]$ClusterName = $env:AKS_CLUSTER, + [string]$ConfidentialNodePool = 'cvmnodepool' ) if ($PSCmdlet.ParameterSetName -eq 'Help') { @@ -51,223 +44,80 @@ if ($PSCmdlet.ParameterSetName -eq 'Help') { return } +$ErrorActionPreference = 'Stop' +$PSNativeCommandUseErrorActionPreference = $false + foreach ($requiredValue in @{ - RESOURCE_GROUP = $ResourceGroup - ATTENDEE_ID = $AttendeeId - HASH_SUFFIX = $HashSuffix - LOCATION = $Location + RESOURCE_GROUP = $ResourceGroup + AKS_CLUSTER = $ClusterName }.GetEnumerator()) { if ([string]::IsNullOrWhiteSpace($requiredValue.Value)) { throw "$($requiredValue.Key) is not set. Define the MicroHack environment variables before running this script." } } -$nameSuffix = (($HashSuffix -replace '[^a-zA-Z0-9]', '').ToLowerInvariant()) -if ($nameSuffix.Length -gt 10) { $nameSuffix = $nameSuffix.Substring(0, 10) } -if (-not $nameSuffix) { throw 'HASH_SUFFIX must contain at least one letter or number.' } - $subsID = (az account show --query id --output tsv 2>$null) if ($LASTEXITCODE -ne 0 -or -not $subsID) { throw 'Azure CLI is not signed in. Run az login.' } -$basename = "cvmcluster$nameSuffix" -$description = "MicroHack Challenge 5" -$region = $Location -$ccVmSize = $CcVmSize -$systemVmSize = $SystemVmSize -$ccNodeCount = $CcNodeCount -$smoketest = $false $sampleDirectory = Join-Path $PSScriptRoot 'resources/azure-voting-app' - $startTime = Get-Date -$scriptName = $MyInvocation.MyCommand.Name - -# Get GitHub repository URL from git remote (used as a tag) -$gitRemoteUrl = "" -try { $gitRemoteUrl = (git remote get-url origin) -replace "\.git$","" } catch {} -if (-not $gitRemoteUrl) { $gitRemoteUrl = "[Originally from] https://github.com/Microsoft/confidential-computing" } - -# ACR names cannot contain hyphens, AKS cluster names should be conservative as well. -if ($basename -match '[^a-z0-9]') { - write-host "basename must contain only lowercase letters and digits (no hyphens, no uppercase). ACR does not allow hyphens." -ForegroundColor Red - exit 1 -} - -# MicroHack uses stable names so deployment and selective cleanup address the same cluster. -$resgrp = $ResourceGroup -$aksName = "aks-cvmcluster$nameSuffix" -$acrName = $basename + "acr" -$ccPoolName = "ccpool" # 12-char max, lowercase, AMD SEV-SNP node pool -$systemPool = "syspool" - -write-host "----------------------------------------------------------------------------------------------------------------" -write-host "Building AKS cluster '$aksName' with AMD SEV-SNP CC node pool in '$region' (subscription $subsID)" -write-host " System pool : $systemPool 1x $systemVmSize" -write-host " CC pool : $ccPoolName ${ccNodeCount}x $ccVmSize (AMD SEV-SNP)" -write-host " Resource Gp : $resgrp" -if ($smoketest) { write-host "SMOKETEST MODE: Resources will be auto-deleted after the front-end is verified" -ForegroundColor Yellow } -write-host "Script: $scriptName" -write-host "Repository URL: $gitRemoteUrl" -write-host "----------------------------------------------------------------------------------------------------------------" - -# Set subscription context for both Az and az CLI -Set-AzContext -SubscriptionId $subsID | Out-Null -if (!$?) { write-host "Failed to Set-AzContext to $subsID" -ForegroundColor Red; exit 1 } -az account set --subscription $subsID | Out-Null -if (!$?) { write-host "Failed to az account set --subscription $subsID" -ForegroundColor Red; exit 1 } - -$tmp = Get-AzContext -$ownername = $AttendeeId - -if ($Cleanup) { - az group show --name $resgrp --output none - if ($LASTEXITCODE -ne 0) { throw "Resource group '$resgrp' was not found." } - - az aks show --resource-group $resgrp --name $aksName --output none 2>$null - if ($LASTEXITCODE -eq 0) { - write-host "Deleting AKS cluster '$aksName'..." -ForegroundColor Cyan - az aks delete --resource-group $resgrp --name $aksName --yes --no-wait --output none - if ($LASTEXITCODE -ne 0) { throw "Failed to delete AKS cluster '$aksName'." } +$resgrp = $ResourceGroup +$aksName = $ClusterName +$ccPoolName = $ConfidentialNodePool +$namespace = 'challenge-05' +$contextName = "challenge-05-$subsID-$aksName" +$basename = [guid]::NewGuid().ToString('N') +$kubeconfigPath = Join-Path ([IO.Path]::GetTempPath()) "challenge-05-$basename.kubeconfig" +$manifestFile = $null +$cmFile = $null +$attestManifestFile = $null +$kubectlScope = @('--kubeconfig', $kubeconfigPath, '--context', $contextName, '--namespace', $namespace) +$clusterJson = az aks show --subscription $subsID --resource-group $resgrp --name $aksName --output json --only-show-errors +if ($LASTEXITCODE -ne 0) { throw 'Cannot read the provided AKS cluster. Verify the Console cluster name and subscription.' } +$cluster = ($clusterJson -join "`n") | ConvertFrom-Json +if ($cluster.provisioningState -ne 'Succeeded') { throw 'The provided AKS cluster is not ready. Ask the facilitator to finish provisioning.' } +if ($Deploy) { + $poolJson = az aks nodepool show --subscription $subsID --resource-group $resgrp --cluster-name $aksName --name $ccPoolName --output json --only-show-errors + if ($LASTEXITCODE -ne 0) { throw 'The provided confidential pool is missing. This script does not create node pools.' } + $pool = ($poolJson -join "`n") | ConvertFrom-Json + if ($pool.provisioningState -ne 'Succeeded' -or $pool.count -ne 2 -or $pool.osSKU -ne 'Ubuntu' -or + $pool.vmSize -notmatch '^Standard_DC2as_v[56]$' -or $pool.mode -ne 'User' -or + $pool.nodeLabels.workload -ne 'confidential') { + throw 'Expected two ready Ubuntu Standard_DC2as_v5/v6 user nodes labelled workload=confidential. Ask the facilitator to align the shared pool; no infrastructure was changed.' } - write-host "Cleanup initiated. Resource group '$resgrp' was retained." -ForegroundColor Green - return + $ccVmSize = $pool.vmSize + $ccNodeCount = $pool.count } - -# ---------- Pre-flight: SKU + quota check for the CC pool ----------------------------------------- -if ($SkipSkuPreflight) { - write-host "Pre-flight check SKIPPED (-SkipSkuPreflight)." -ForegroundColor Yellow -} else { - write-host "Pre-flight: confirming '$ccVmSize' is available in '$region' with sufficient AMD CVM vCPU quota..." -ForegroundColor Cyan - - # Hard-fail on Intel SGX SKUs - this script targets full-VM CC (SEV-SNP) - if ($ccVmSize -match '^Standard_DC\d+s_v[23]$') { - write-host "ERROR: '$ccVmSize' is an Intel SGX SKU; this script targets AMD SEV-SNP Confidential VM nodes." -ForegroundColor Red - exit 1 - } - if ($ccVmSize -notmatch '^Standard_(DC|EC)\d+a[a-z]*_v\d+$') { - write-host "Warning: '$ccVmSize' does not look like an AMD SEV-SNP CVM SKU (expected DCa*/ECa*v5 family)." -ForegroundColor Yellow - } - - $skuInfo = $null - try { - $skuInfo = Get-AzComputeResourceSku -Location $region -ErrorAction Stop | - Where-Object { $_.ResourceType -eq 'virtualMachines' -and $_.Name -eq $ccVmSize } | - Select-Object -First 1 - } catch { - write-host "Warning: Get-AzComputeResourceSku failed: $($_.Exception.Message)" -ForegroundColor Yellow - } - - if ($null -eq $skuInfo) { - write-host "ERROR: '$ccVmSize' is not offered in '$region'." -ForegroundColor Red - write-host "Find available regions: Get-AzComputeResourceSku | ? { `$_.Name -eq '$ccVmSize' -and -not `$_.Restrictions } | Select Locations" -ForegroundColor Gray - exit 1 - } - - $subRestriction = $skuInfo.Restrictions | Where-Object { - $_.ReasonCode -eq 'NotAvailableForSubscription' -or - ($_.RestrictionInfo -and $_.RestrictionInfo.Locations -contains $region) - } - if ($subRestriction) { - $reason = ($skuInfo.Restrictions | ForEach-Object { $_.ReasonCode }) -join ', ' - write-host "ERROR: '$ccVmSize' is restricted for this subscription in '$region' (reason: $reason)." -ForegroundColor Red - exit 1 - } - - $skuVCpus = ($skuInfo.Capabilities | Where-Object Name -eq 'vCPUs' | Select-Object -First 1).Value -as [int] - if (-not $skuVCpus) { $skuVCpus = 2 } - $needed = $skuVCpus * $ccNodeCount - $skuFamily = $skuInfo.Family - try { - $usage = Get-AzVMUsage -Location $region -ErrorAction Stop | - Where-Object { $_.Name.Value -eq $skuFamily } | Select-Object -First 1 - if ($usage) { - $available = [int]$usage.Limit - [int]$usage.CurrentValue - write-host ("Quota for {0} in {1}: {2}/{3} used, {4} vCPUs available, this pool needs {5}." -f ` - $skuFamily, $region, $usage.CurrentValue, $usage.Limit, $available, $needed) -ForegroundColor Cyan - if ($available -lt $needed) { - write-host "ERROR: Insufficient AMD CVM vCPU quota in '$skuFamily' / '$region' ($needed needed, $available available)." -ForegroundColor Red - exit 1 - } - } - } catch { - write-host "Warning: Get-AzVMUsage failed: $($_.Exception.Message). Continuing." -ForegroundColor Yellow - } - write-host "Pre-flight passed: '$ccVmSize' available with quota in '$region'." -ForegroundColor Green +Write-Host "Using provided AKS cluster '$aksName' in '$($cluster.location)' (subscription $subsID), namespace $namespace." +if (-not (Get-Command kubectl -ErrorAction SilentlyContinue)) { + throw 'kubectl is required. Install it before running Challenge 5.' } - -# ---------- Resource group ------------------------------------------------------------------------ -$rgTags = @{ - owner = $ownername - BuiltBy = $scriptName - GitRepo = $gitRemoteUrl - Workload = "azure-voting-app" - CCType = "AMD-SEV-SNP" +try { +az aks get-credentials --subscription $subsID --resource-group $resgrp --name $aksName --file $kubeconfigPath --context $contextName --overwrite-existing --only-show-errors | Out-Null +if ($LASTEXITCODE -ne 0) { throw 'az aks get-credentials failed.' } +$namespaceJson = kubectl @kubectlScope get namespace $namespace --ignore-not-found -o json +if ($LASTEXITCODE -ne 0) { throw 'Cannot read the Challenge 5 namespace.' } +$existingNamespace = if ($namespaceJson) { ($namespaceJson -join "`n") | ConvertFrom-Json } +if ($existingNamespace -and $existingNamespace.metadata.labels.'microhack-challenge' -ne '05') { + throw 'The challenge-05 namespace exists without the expected ownership label. No changes were made.' } -if ($description -ne "") { $rgTags.Add("description", $description) } -if ($smoketest) { $rgTags.Add("smoketest", "true") } - -az group show --name $resgrp --output none -if ($LASTEXITCODE -ne 0) { throw "Resource group '$resgrp' was not found." } - -# ---------- AKS cluster --------------------------------------------------------------------------- -# Auto-patching strategy (no preview features required, safe defaults that reflect the recommended -# Azure Policies "Kubernetes clusters should have auto-upgrade enabled" and node-image auto-upgrade): -# --auto-upgrade-channel stable cluster K8s version auto-upgrades to stable -# --node-os-upgrade-channel NodeImage node OS images auto-upgrade weekly -# --enable-managed-identity system-assigned MI for the cluster -# --tier standard uptime SLA + financially-backed (cheap insurance) -# We deliberately keep local accounts enabled so 'az aks get-credentials' just works for the demo. -az aks show --resource-group $resgrp --name $aksName --output none 2>$null -if ($LASTEXITCODE -ne 0) { - write-host "Creating AKS cluster '$aksName' (this takes ~5 minutes)..." -ForegroundColor Cyan - az aks create ` - --resource-group $resgrp ` - --name $aksName ` - --location $region ` - --node-count 1 ` - --nodepool-name $systemPool ` - --node-vm-size $systemVmSize ` - --os-sku Ubuntu ` - --enable-managed-identity ` - --generate-ssh-keys ` - --auto-upgrade-channel stable ` - --node-os-upgrade-channel NodeImage ` - --tier standard ` - --network-plugin azure ` - --tags owner=$ownername BuiltBy=$scriptName Workload=azure-voting-app Challenge=05 ` - --only-show-errors - if ($LASTEXITCODE -ne 0) { write-host "az aks create failed" -ForegroundColor Red; exit 1 } +if ($Cleanup) { + if ($existingNamespace) { + kubectl @kubectlScope delete deployment/azure-vote-back deployment/azure-vote-front deployment/cc-attest service/azure-vote-back service/azure-vote-front service/cc-attest configmap/cc-attest-app --ignore-not-found --timeout=5m + if ($LASTEXITCODE -ne 0) { throw 'Challenge 5 application cleanup failed.' } + } + Write-Host 'Challenge 5 applications removed. Shared AKS, node pools, namespace and Challenge 7 resources were retained.' + return } - -# ---------- AMD SEV-SNP Confidential Computing node pool ------------------------------------------ -# AMD SEV-SNP CVM node pools require Ubuntu and a DCa*/ECa* v5 SKU. Secure Boot + vTPM are enabled -# implicitly by the platform when a CVM SKU is selected; no extra flags are needed. -az aks nodepool show --resource-group $resgrp --cluster-name $aksName --name $ccPoolName --output none 2>$null -if ($LASTEXITCODE -ne 0) { - write-host "Adding AMD SEV-SNP CC node pool '$ccPoolName' (${ccNodeCount}x $ccVmSize)..." -ForegroundColor Cyan - az aks nodepool add ` - --resource-group $resgrp ` - --cluster-name $aksName ` - --name $ccPoolName ` - --node-count $ccNodeCount ` - --node-vm-size $ccVmSize ` - --os-sku Ubuntu ` - --mode User ` - --labels workload=confidential sku=amd-sev-snp ` - --tags owner=$ownername CCType=AMD-SEV-SNP ` - --only-show-errors - if ($LASTEXITCODE -ne 0) { write-host "az aks nodepool add failed for CC pool" -ForegroundColor Red; exit 1 } +if ($existingNamespace -and ($existingNamespace.metadata.labels.'istio.io/rev' -or + $existingNamespace.metadata.labels.'istio-injection' -eq 'enabled')) { + throw 'Challenge 5 requires its namespace without Istio sidecar injection. Ask the facilitator to review the namespace labels.' } - -# ---------- kubectl access ------------------------------------------------------------------------ -if (-not (Get-Command kubectl -ErrorAction SilentlyContinue)) { - write-host "kubectl not found - installing via 'az aks install-cli'" -ForegroundColor Yellow - az aks install-cli --only-show-errors | Out-Null +if (-not $existingNamespace) { + @{ apiVersion = 'v1'; kind = 'Namespace'; metadata = @{ name = $namespace; labels = @{ 'microhack-challenge' = '05'; 'istio-injection' = 'disabled' } } } | + ConvertTo-Json -Depth 5 | kubectl @kubectlScope apply -f - + if ($LASTEXITCODE -ne 0) { throw 'Cannot create the Challenge 5 namespace.' } } -write-host "Fetching cluster credentials..." -ForegroundColor Cyan -az aks get-credentials --resource-group $resgrp --name $aksName --overwrite-existing --only-show-errors | Out-Null -if ($LASTEXITCODE -ne 0) { write-host "az aks get-credentials failed" -ForegroundColor Red; exit 1 } - -# Sanity: list nodes -kubectl get nodes -o wide +kubectl @kubectlScope get nodes -o wide if ($LASTEXITCODE -ne 0) { write-host "kubectl get nodes failed - cluster not reachable" -ForegroundColor Red; exit 1 } # ---------- Deploy public Azure Voting App (multi-container) -------------------------------------- @@ -391,31 +241,31 @@ spec: app: azure-vote-front '@ -$manifestFile = Join-Path $env:TEMP "azure-vote-$basename.yaml" +$manifestFile = Join-Path ([IO.Path]::GetTempPath()) "azure-vote-$basename.yaml" $votingManifest | Out-File -FilePath $manifestFile -Encoding utf8 -Force write-host "Applying voting-app manifest from $manifestFile..." -ForegroundColor Cyan -kubectl apply -f $manifestFile +kubectl @kubectlScope apply -f $manifestFile if ($LASTEXITCODE -ne 0) { write-host "kubectl apply failed" -ForegroundColor Red; exit 1 } # ---------- Wait for rollouts --------------------------------------------------------------------- write-host "Waiting for deployments to become available..." -ForegroundColor Cyan -kubectl rollout status deployment/azure-vote-back --timeout=5m -if ($LASTEXITCODE -ne 0) { write-host "azure-vote-back rollout failed" -ForegroundColor Red; kubectl describe deployment azure-vote-back; exit 1 } -kubectl rollout status deployment/azure-vote-front --timeout=10m -if ($LASTEXITCODE -ne 0) { write-host "azure-vote-front rollout failed" -ForegroundColor Red; kubectl describe deployment azure-vote-front; kubectl get pods -l app=azure-vote-front -o wide; exit 1 } +kubectl @kubectlScope rollout status deployment/azure-vote-back --timeout=5m +if ($LASTEXITCODE -ne 0) { write-host "azure-vote-back rollout failed" -ForegroundColor Red; kubectl @kubectlScope describe deployment azure-vote-back; exit 1 } +kubectl @kubectlScope rollout status deployment/azure-vote-front --timeout=10m +if ($LASTEXITCODE -ne 0) { write-host "azure-vote-front rollout failed" -ForegroundColor Red; kubectl @kubectlScope describe deployment azure-vote-front; kubectl @kubectlScope get pods -l app=azure-vote-front -o wide; exit 1 } # ---------- Wait for LoadBalancer external IP ----------------------------------------------------- write-host "Waiting for LoadBalancer to allocate a public IP (up to 5 minutes)..." -ForegroundColor Cyan $externalIP = $null for ($i = 1; $i -le 30; $i++) { - $externalIP = kubectl get service azure-vote-front -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>$null + $externalIP = kubectl @kubectlScope get service azure-vote-front -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>$null if ($externalIP) { break } Start-Sleep -Seconds 10 write-host " ...still waiting for external IP (attempt $i/30)" } if (-not $externalIP) { write-host "Timed out waiting for external IP" -ForegroundColor Red - kubectl describe service azure-vote-front + kubectl @kubectlScope describe service azure-vote-front exit 1 } write-host "Voting app external IP: $externalIP" -ForegroundColor Green @@ -435,8 +285,8 @@ for ($i = 1; $i -le 30; $i++) { } if (-not $ok) { write-host "Front-end did not respond with expected content within 5 minutes" -ForegroundColor Red - kubectl get pods -o wide - kubectl logs -l app=azure-vote-front --tail=50 + kubectl @kubectlScope get pods -o wide + kubectl @kubectlScope logs -l app=azure-vote-front --tail=50 exit 1 } @@ -445,7 +295,7 @@ write-host "SUCCESS: Azure Voting App is live at http://$externalIP/" -Foregrou write-host "Cluster : $aksName" write-host "Resource group : $resgrp" write-host "CC node pool : $ccPoolName (${ccNodeCount}x $ccVmSize - AMD SEV-SNP)" -write-host "Auto-upgrade : cluster=stable node-os=NodeImage" +write-host "Namespace : $namespace" write-host "----------------------------------------------------------------------------------------------------------------" # ---------- Deploy the runtime-attestation web UI ------------------------------------------------- @@ -456,9 +306,7 @@ write-host "Deploying CC runtime-attestation web UI..." -ForegroundColor Cyan $attestDir = Join-Path $sampleDirectory 'attestation' foreach ($f in @('app.py','config_snp.json','templates/index.html')) { if (-not (Test-Path (Join-Path $attestDir $f))) { - write-host "Missing $f under $attestDir - skipping attestation deployment." -ForegroundColor Yellow - $attestDir = $null - break + throw "Missing attestation asset $f under $attestDir. Use a complete repository checkout." } } @@ -467,15 +315,16 @@ if ($attestDir) { $attestAppPath = Join-Path $attestDir 'app.py' $attestConfigPath = Join-Path $attestDir 'config_snp.json' $attestTemplatePath = Join-Path $attestDir 'templates/index.html' - $cmYaml = kubectl create configmap cc-attest-app ` + $cmYaml = kubectl @kubectlScope create configmap cc-attest-app ` "--from-file=app.py=$attestAppPath" ` "--from-file=config_snp.json=$attestConfigPath" ` "--from-file=index.html=$attestTemplatePath" ` --dry-run=client -o yaml if ($LASTEXITCODE -ne 0) { write-host "Failed to build attestation ConfigMap" -ForegroundColor Red; exit 1 } - $cmFile = Join-Path $env:TEMP "cc-attest-cm-$basename.yaml" + $cmFile = Join-Path ([IO.Path]::GetTempPath()) "cc-attest-cm-$basename.yaml" $cmYaml | Out-File -FilePath $cmFile -Encoding utf8 -Force - kubectl apply -f $cmFile | Out-Null + kubectl @kubectlScope apply -f $cmFile | Out-Null + if ($LASTEXITCODE -ne 0) { throw 'Cannot apply the attestation ConfigMap.' } $attestManifest = @' apiVersion: apps/v1 @@ -566,24 +415,26 @@ spec: targetPort: 80 '@ - $attestManifestFile = Join-Path $env:TEMP "cc-attest-$basename.yaml" + $attestManifestFile = Join-Path ([IO.Path]::GetTempPath()) "cc-attest-$basename.yaml" $attestManifest | Out-File -FilePath $attestManifestFile -Encoding utf8 -Force - kubectl apply -f $attestManifestFile + kubectl @kubectlScope apply -f $attestManifestFile if ($LASTEXITCODE -ne 0) { write-host "kubectl apply failed for attestation manifest" -ForegroundColor Red; exit 1 } # Restart deployment so any ConfigMap changes from re-runs are picked up. - kubectl rollout restart deployment/cc-attest | Out-Null + kubectl @kubectlScope rollout restart deployment/cc-attest | Out-Null + if ($LASTEXITCODE -ne 0) { throw 'Cannot restart the attestation deployment.' } write-host "Waiting for cc-attest rollout (first run installs tpm2-tools + clones upstream)..." -ForegroundColor Cyan - kubectl rollout status deployment/cc-attest --timeout=10m + kubectl @kubectlScope rollout status deployment/cc-attest --timeout=10m if ($LASTEXITCODE -ne 0) { write-host "cc-attest rollout failed" -ForegroundColor Red - kubectl describe deployment cc-attest - kubectl logs -l app=cc-attest --tail=80 + kubectl @kubectlScope describe deployment cc-attest + kubectl @kubectlScope logs -l app=cc-attest --tail=80 + throw 'Attestation rollout failed.' } else { # Wait for the attestation LoadBalancer. $attestIP = $null for ($i = 1; $i -le 30; $i++) { - $attestIP = kubectl get service cc-attest -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>$null + $attestIP = kubectl @kubectlScope get service cc-attest -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>$null if ($attestIP) { break } Start-Sleep -Seconds 10 write-host " ...still waiting for cc-attest external IP (attempt $i/30)" @@ -594,7 +445,7 @@ spec: write-host " Click 'Attest' to fetch a fresh MAA-signed SEV-SNP token with every claim explained." write-host "----------------------------------------------------------------------------------------------------------------" } else { - write-host "cc-attest LoadBalancer did not get an external IP in time." -ForegroundColor Yellow + throw 'Attestation LoadBalancer did not get an external IP in time.' } } } @@ -606,3 +457,11 @@ write-host "To clean up: ./Deploy-VotingAppCC.ps1 -Cleanup" $myTimeSpan = New-TimeSpan -Start $startTime -End (Get-Date) Write-Output ("Execution time was {0} minutes and {1} seconds." -f $myTimeSpan.Minutes, $myTimeSpan.Seconds) +} +finally { + foreach ($temporaryFile in @($kubeconfigPath, $manifestFile, $cmFile, $attestManifestFile)) { + if ($temporaryFile -and (Test-Path -LiteralPath $temporaryFile)) { + Remove-Item -LiteralPath $temporaryFile -Force + } + } +} diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/UPSTREAM-SOURCE.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/UPSTREAM-SOURCE.md index 2320a088a..83eea2cd2 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/UPSTREAM-SOURCE.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/UPSTREAM-SOURCE.md @@ -20,17 +20,12 @@ the workshop adaptations. ## MicroHack adaptations -1. Read `RESOURCE_GROUP`, `ATTENDEE_ID`, `HASH_SUFFIX`, and `LOCATION` from the - established MicroHack environment variables. -2. Derive the subscription from the active Azure CLI account and retain the - upstream requirement for the Az PowerShell context to target that same - subscription. -3. Reuse the attendee resource group instead of creating or deleting a resource - group. -4. Derive a deterministic AKS cluster name from `HASH_SUFFIX` and make cluster - and confidential node-pool creation rerunnable. +1. Read `RESOURCE_GROUP` and the Console-provided `AKS_CLUSTER`, with explicit parameter overrides. +2. Use the active Azure CLI subscription; no Az PowerShell session or unused-quota preflight is required for application deployment. +3. Validate the existing shared AKS cluster and its two labeled Ubuntu confidential nodes; never create, scale or delete infrastructure. +4. Use a temporary kubeconfig, explicit context and owned `challenge-05` namespace for every Kubernetes command. Preserve Challenge 7's Radius namespaces. 5. Resolve attestation assets from the unchanged local snapshot. -6. Delete only the Challenge 5 AKS cluster during cleanup. +6. Delete only the named Challenge 5 deployments, services and ConfigMap during cleanup. Retain the cluster and node pools. When updating the sample, replace the complete snapshot from one upstream commit, update the commit above, and review the top-level script diff against diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/solution-05.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/solution-05.md index 0af131544..34efa4388 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/solution-05.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-05/solution-05.md @@ -4,62 +4,41 @@ ## Objective -Use the newer Azure Voting App automation to deploy an AKS cluster with an AMD -SEV-SNP node pool. The script also deploys a runtime attestation UI, allowing -you to verify the node from inside a pod before trusting the workload. +Deploy and validate the Azure Voting App and runtime attestation UI on the +provided AKS cluster shared with Challenge 7. Console provisions two Ubuntu +AMD SEV-SNP nodes labelled `workload=confidential`. This script deploys only +applications and never creates, scales or deletes the cluster or its pools. > [!IMPORTANT] > **Execution environment.** Like Challenge 4, this challenge runs in a **local > PowerShell 7+ session**, not in Azure Cloud Shell. It requires `kubectl` access -> and both Azure command-line tools signed in locally. +> and Azure CLI signed in locally. ## Prerequisites - The [general MicroHack prerequisites](../../Readme.md#general-prerequisites). - PowerShell 7 or later, running locally. - Azure CLI signed in to the target subscription. -- Azure PowerShell (`Az`) signed in to the same subscription. - Contributor access to the attendee resource group. -- At least 4 available `standardDCASv5Family` vCPUs in the selected region for the default two-node pool. -- Permission to create AKS clusters and public LoadBalancer services. -- `kubectl` on PATH. The script installs it with `az aks install-cli` if missing. +- A provided AKS cluster and `cvmnodepool` with two Ubuntu `Standard_DC2as_v5/v6` nodes. The four confidential vCPUs are already allocated; application deployment needs no additional free VM quota. +- Permission to retrieve AKS credentials and deploy workloads/public LoadBalancer services in `challenge-05`. +- `kubectl` installed on PATH. The current confidential AKS capability is GA. The old `aks-preview` extension and `AzureLinuxCVMPreview` feature registration are not used. -> [!WARNING] -> **This challenge needs two separate sign-ins.** The Azure CLI (`az`) and the -> Az PowerShell module maintain **independent** credential stores. The script -> calls `az account show` to read the subscription **and** `Set-AzContext` to set -> the PowerShell context, so `az login` on its own is not sufficient. -> -> If you completed Challenges 1-4 you are signed in to the Azure CLI only, and -> the script fails at `Set-AzContext` with -> `SharedTokenCacheCredential authentication unavailable`. - -### Sign in to both tools and verify +### Sign in and verify ```powershell -# 1. Install the Az PowerShell module if it is not already present -Get-Module -ListAvailable Az.Accounts, Az.Compute -Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force - -# 2. Sign in to Az PowerShell. This is separate from 'az login'. -# Pass -TenantId when your machine has signed in to more than one tenant. -Connect-AzAccount -TenantId - -# 3. Point both tools at the same subscription -Set-AzContext -SubscriptionId -az account set --subscription - -# 4. Confirm the subscription IDs match -Get-AzContext | Select-Object Name, Subscription, Tenant +az login --tenant '' +az account set --subscription '' az account show --query "{sub:name, id:id, tenant:tenantId}" --output table +kubectl version --client ``` > [!NOTE] -> The script uses `Get-AzComputeResourceSku` and `Get-AzVMUsage` for its SKU and -> quota preflight, which is why the Az module is required rather than optional. +> Only the Azure CLI sign-in is needed. Console handles SKU/quota checks and +> infrastructure provisioning; the application script does not call Az PowerShell. ### Check for a conflicting `aks-preview` extension @@ -85,30 +64,26 @@ az aks list --output table > [when Confidential VM Orchestrator is required](CVM-ORCHESTRATOR.md). > [!IMPORTANT] -> The automation creates the AKS cluster in the existing attendee resource -> group. Cleanup deletes the cluster, not the resource group. +> This cluster is shared with Challenge 7. Cleanup removes only Challenge 5's +> named applications. Do not delete the cluster or node pools between challenges. ## Task 1: Configure the MicroHack environment -Challenge 5 uses the same PowerShell environment variables as Challenge 4. If -the session still contains them, reuse the existing values. +Use a full repository checkout and run from the Challenge 5 walkthrough directory. +Set the **Sovereign Lab AKS Cluster** name from Console's Credentials tab. For +manual delivery, use the shared lab template's `aksClusterName` output. ```powershell $env:RESOURCE_GROUP = "labuser-xx" -$env:ATTENDEE_ID = $env:RESOURCE_GROUP -$env:LOCATION = "northeurope" - -# Run only when HASH_SUFFIX is not already set. -$seed = "$($env:ATTENDEE_ID)-$(Get-Random)-$(Get-Random)" -$bytes = [Text.Encoding]::UTF8.GetBytes($seed) -$hash = [Security.Cryptography.MD5]::HashData($bytes) -$env:HASH_SUFFIX = [Convert]::ToHexString($hash).Substring(0, 8).ToLowerInvariant() +$env:AKS_CLUSTER = "" +az aks show --resource-group $env:RESOURCE_GROUP --name $env:AKS_CLUSTER --query '{name:name,location:location,state:provisioningState}' --output table ``` > [!NOTE] -> These are PowerShell environment variables (`$env:` prefix), not the Bash -> variables used in Challenges 1-3. The script exits immediately if any of -> `RESOURCE_GROUP`, `ATTENDEE_ID`, `LOCATION`, or `HASH_SUFFIX` is missing. +> These are PowerShell environment variables (`$env:` prefix). Alternatively, +> pass `-ResourceGroup` and `-ClusterName`. `-ConfidentialNodePool` defaults to +> `cvmnodepool`. The cluster's actual region is used; Challenge 4's ACI +> `LOCATION=northeurope` does not change the provided AKS cluster's location. ## Task 2: Run the automated deployment @@ -118,41 +93,33 @@ From this walkthrough directory, run: ./Deploy-VotingAppCC.ps1 -Deploy ``` -The script starts by confirming the cluster plan, then runs a SKU and quota -preflight before creating the AKS cluster: - -![PowerShell output showing the cluster plan (system pool, CC pool, resource group), the Standard_DC2as_v5 preflight passing with quota available in northeurope, and the AKS cluster creation beginning](./images/01-deploy-start-preflight.png) - The script automatically: -1. Checks that `Standard_DC2as_v5` is available in `$env:LOCATION`. -2. Creates a managed-identity AKS cluster with stable Kubernetes and node-image upgrades. -3. Adds two `Standard_DC2as_v5` nodes labelled `workload=confidential`. -4. Acquires cluster credentials and deploys the Azure Voting App. -5. Creates the attestation ConfigMap and deploys the runtime attestation UI. -6. Waits for both rollouts and LoadBalancer IP addresses. +1. Validates the existing cluster and its two-node Ubuntu confidential pool. +2. Retrieves credentials into a temporary kubeconfig with an explicit context, leaving your default context unchanged. +3. Creates/reuses the owned `challenge-05` namespace without Istio injection, separate from Challenge 7's Radius namespaces. +4. Deploys the voting applications and attestation ConfigMap/UI in that namespace. +5. Waits for rollouts and LoadBalancer IP addresses, then removes the temporary kubeconfig. > [!NOTE] -> **Expect roughly 10 minutes.** Cluster creation alone takes about five -> minutes, the confidential node pool adds several more, and the LoadBalancer -> can take up to five minutes to receive a public IP. The script retries the +> **Expect several minutes for application startup.** The cluster already +> exists, but image downloads, package installation and LoadBalancer allocation +> take time. The script retries the > front-end smoke test up to 30 times, so early "not responding yet" messages > are expected rather than a failure. -A successful run ends with both rollouts complete and both public URLs printed: - -![PowerShell output listing three Ready nodes (two ccpool, one syspool), the voting app rollout succeeding with external IP, and the CC attestation UI URL, followed by the cleanup reminder and total execution time](./images/02-deployment-complete-urls.png) - -Use `-CcNodeCount 1` when the event design calls for a single confidential node -and the reduced resiliency is acceptable. The script prints both public URLs; -it does not open a browser automatically. +A successful run prints both public URLs. The attestation pod requires privileged +TPM access, so use only the dedicated lab cluster. Namespace separation prevents +naming and cleanup collisions, not a security boundary against privileged pods. ## Task 3: Verify workload placement ```powershell -kubectl get nodes --show-labels -kubectl get pods --output wide -kubectl get deployment azure-vote-front --output jsonpath='{.spec.template.spec.nodeSelector}' +$context = "challenge-05-$((az account show --query id -o tsv).Trim())-$env:AKS_CLUSTER" +az aks get-credentials --resource-group $env:RESOURCE_GROUP --name $env:AKS_CLUSTER --context $context --overwrite-existing +kubectl --context $context -n challenge-05 get nodes --show-labels +kubectl --context $context -n challenge-05 get pods --output wide +kubectl --context $context -n challenge-05 get deployment azure-vote-front --output jsonpath='{.spec.template.spec.nodeSelector}' ``` Confirm that `azure-vote-front` and `cc-attest` run on nodes with the @@ -188,25 +155,25 @@ the token signature, issuer, nonce, and required claims before releasing data. | Message | Cause | Fix | | --- | --- | --- | -| `SharedTokenCacheCredential authentication unavailable` followed by `Failed to Set-AzContext` | Signed in to the Azure CLI only. The Az PowerShell module has no session. | `Connect-AzAccount -TenantId `, then `Set-AzContext -SubscriptionId `. See the prerequisites above. | +| Expected two ready Ubuntu nodes | The shared pool is outdated or provisioning is incomplete. | Ask the facilitator to align the pool. Do not create another cluster. | | `ModuleNotFoundError: No module named 'msrestazure'` in a Python traceback during `az aks create` | A stale `aks-preview` CLI extension is intercepting `az aks` commands. This challenge does not need it. | `az extension remove --name aks-preview`, verify with `az aks list -o table`, then re-run. | | `Azure CLI is not signed in. Run az login.` | No Azure CLI session. | `az login`, then `az account set --subscription `. | | ` is not set. Define the MicroHack environment variables...` | Task 1 variables missing, or set with Bash syntax instead of `$env:`. | Re-run the Task 1 block in the current PowerShell session. | | `Resource group '' was not found.` | `RESOURCE_GROUP` does not match the attendee resource group. | Confirm with `az group show --name $env:RESOURCE_GROUP`. | -| Preflight reports unavailable capacity or insufficient quota | The region has no `Standard_DC2as_v5` capacity or no AMD CVM quota. | Select a supported region, request quota, or use `-CcNodeCount 1`. | +| Namespace ownership-label error | `challenge-05` exists without this script's ownership label. | Have the facilitator inspect it; do not delete or relabel unrelated workloads. | | `az aks get-credentials failed` / `kubectl get nodes failed` | Cluster not reachable or `kubectl` missing. | `az aks install-cli`, then re-run `az aks get-credentials -g $env:RESOURCE_GROUP -n `. | | Rollout or LoadBalancer timeout | Image pull, scheduling, or public IP allocation is slow. | Inspect with the commands below, then re-run `-Deploy`; the script is idempotent for existing resources. | Check rollout and bootstrap logs: ```powershell -kubectl get pods --output wide -kubectl describe deployment azure-vote-front -kubectl logs deployment/cc-attest --tail=100 +kubectl --context $context -n challenge-05 get pods --output wide +kubectl --context $context -n challenge-05 describe deployment azure-vote-front +kubectl --context $context -n challenge-05 logs deployment/cc-attest --tail=100 ``` -If the preflight reports unavailable capacity, select a region that supports -`Standard_DC2as_v5` or request AMD confidential VM quota. +Region, SKU and quota selection belong to facilitator provisioning. Rerunning +the workload script does not request or consume quota for another cluster. ## Task 5: Clean up @@ -214,13 +181,14 @@ If the preflight reports unavailable capacity, select a region that supports ./Deploy-VotingAppCC.ps1 -Cleanup ``` -AKS deletion is asynchronous. The attendee resource group and resources from -other challenges are retained. +Cleanup removes only the three Challenge 5 deployments, three services and +attestation ConfigMap. It retains the namespace, AKS cluster, both node pools, +resource group and all Challenge 7 resources. It refuses an unowned namespace. > [!IMPORTANT] -> An AKS cluster with two `Standard_DC2as_v5` nodes plus two public -> LoadBalancer services is the most expensive resource in this MicroHack. Run -> cleanup as soon as you finish the challenge. +> Clean up the applications after Challenge 5 to release their public services. +> Shared nodes continue accruing charges for Challenge 7. Console/the organizer +> removes infrastructure only after the whole event. ## Source alignment diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-07/solution-07.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-07/solution-07.md index 0dd9e8643..4e27795f3 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-07/solution-07.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/walkthrough/challenge-07/solution-07.md @@ -30,6 +30,11 @@ az account set --subscription "$AZURE_SUBSCRIPTION" > [!NOTE] > The platform creates stable resource names and credentials. Re-running the lab deployment converges on the same names and replaces the isolated resource group only when region fallback is required. +The Azure AKS cluster is the same one used in Challenge 5. Its applications +live in `challenge-05`; Radius uses `radius-system` and `env-azure-prod` here. +Challenge 5 cleanup does not delete this cluster or either node pool. Switch to +the explicit context below rather than relying on the context used in Challenge 5. + ## Task 1: Connect to and validate both platforms (10 minutes) 💡 **Start with explicit target discipline. A Kubernetes context and a Radius workspace are independent selectors.**