diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md index 0293946ed..ec8f7f9b4 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/readme.md @@ -7,7 +7,7 @@ Run [prepare-localbox.ps1](../prepare-localbox.ps1) **inside LocalBox-Client**, - Elevated PowerShell 7 on `LocalBox-Client` and an installed Azure CLI. Preparation automatically installs missing `stack-hci-vm`, `customlocation` and `aksarc` extensions, including during `-WhatIf`, verifies their versions and leaves existing versions unchanged. Internet access to the extension downloads is required; installation failure stops preparation before infrastructure changes. Record the versions printed by the script for your event. - Extensions are installed in your persistent CLI extension directory (`AZURE_EXTENSION_DIR`, if set), not the temporary authentication profile. The script does not install or upgrade the Azure CLI itself. Health checks additionally require Pester 5.7.1 or later within major version 5; an installed Pester 6 is not a substitute for the runner's supported version. - Healthy Azure Local nodes, storage, Arc Resource Bridge, `jumpstart` custom location and `hybridaksextension`. -- The Client VM's system-assigned managed identity and its existing LocalBox resource-group permissions. The script uses a temporary isolated CLI login and restores the caller's profile. No subscription-wide or Graph permissions are added. +- The Client VM's system-assigned managed identity and its existing LocalBox resource-group permissions. It needs `Microsoft.Authorization/roleAssignments/read` at the AKS connected-cluster scope and, if the group lacks the proxy role, `Microsoft.Authorization/roleAssignments/write` there. Contributor alone cannot create role assignments. An authorized access administrator must arrange this permission or preassign the group role before preparation; the script never elevates its own identity. It uses a temporary isolated CLI login and restores the caller's profile. No subscription-wide or Graph permissions are added. - An existing Entra AKS admin-group object ID. Hosted events use **Lab Group ObjectId** from the Console's **Credentials** tab, published by [shared automation](../hosted-events/readme.md#console-group-dependency). Group membership is not validated through Graph by this script. - The installed Jumpstart configuration referenced by `$env:LocalBoxConfigFile` (or `-ConfigPath`), containing `SDNDomainFQDN` and `SDNAdminPassword`. Preparation constructs the nested administrator credential locally without printing it. Supply `-NodeCredential` to override it if the nested password/account has changed. Azure managed identity cannot authenticate Windows PowerShell Direct inside nested nodes. - `Microsoft.EdgeMarketplace` registered by the subscription owner. The Marketplace image workflow also requires the Azure Connected Machine Resource Manager role for the `Microsoft.AzureStackHCI` resource-provider identity on the image resource group; ask the deployment owner to verify this prerequisite. No automatic privilege escalation is attempted. @@ -34,7 +34,7 @@ The script prompts for the Entra group object ID, or accepts `-AksAdminGroupObje If the Console-owned group is not available yet, `-SkipAks` explicitly prepares only storage, the VM image and both networks. It does not fabricate a group or grant anyone cluster access. The manifest records this as a partial run, and full health checks fail until preparation is rerun without `-SkipAks` using a valid admin-group ID. -`-WhatIf` installs missing CLI extensions as a deliberate local-tooling exception, then performs authenticated reads and validation. It does not provision Azure resources or change Hyper-V or storage. It can create and remove a temporary local CLI profile. It does not establish that later resource creation will succeed. +`-WhatIf` installs missing CLI extensions as a deliberate local-tooling exception, then performs authenticated reads and validation. It does not provision Azure resources, assign roles or change Hyper-V or storage. The RBAC step only reports the planned group/cluster assignment, without querying a cluster that may not exist yet; it does not validate role-assignment permissions. It can create and remove a temporary local CLI profile. It does not establish that later resource creation will succeed. ## Prepared resources @@ -46,6 +46,7 @@ If the Console-owned group is not available yet, `-SkipAks` explicitly prepares | AKS network | `localbox-aks-lnet-vlan110`, VLAN 110, `10.10.0.0/24`, nodes `.101-.199` | | AKS control plane / service reservation | `10.10.0.5`; `.10-.100` reserved for future service VIPs; this script does not install a load balancer | | AKS cluster | `localbox-aks`, one control-plane node and three Linux workers; `Standard_A4_v2` defaults, subject to installed-version capacity/CLI validation | +| AKS proxy access | Azure Arc Enabled Kubernetes Cluster User Role for `AksAdminGroupObjectId`, scoped to the connected-cluster resource; matching direct or inherited grants are reused | | Output | `C:\LocalBox\sovereign-localbox.json`, nonsecret IDs and expected configuration for health tests | Topology defaults come from the installed Jumpstart config, except the VM pool bounds and configurable resource names/sizes. The network/broadcast addresses, gateways and Jumpstart infrastructure reservations must not be allocated. `-AddressReservationsConfirmed` is your explicit confirmation that DHCP/static reservations were reviewed; the script cannot safely infer all DHCP leases from Azure. It does not change routers or DHCP configuration. @@ -70,14 +71,64 @@ If existing workloads use the secondary volume, do not remove it. Use the [manua Follow [Pester health checks](../tests/readme.md). Provisioning success is not a health verdict. The nonsecret manifest intentionally records `FullHealthVerified = false`. -For AKS runtime access, follow [Jumpstart's AKS guide](https://jumpstart.azure.com/azure_jumpstart_localbox/AKS): use a separate CLI profile/session authenticated as a group member, and keep `az connectedk8s proxy` running with a dedicated kubeconfig. Do not log out the provisioning identity or give its managed identity broader permissions to run tests. Group membership grants Kubernetes administration; Azure proxy/connect permissions must also be in place. +For AKS runtime access, follow [Jumpstart's AKS guide](https://jumpstart.azure.com/azure_jumpstart_localbox/AKS): use a separate CLI profile/session authenticated as a group member, and keep `az connectedk8s proxy` running with the kubeconfig used for health checks (default `$HOME/.kube/config`). Do not log out the provisioning identity or give its managed identity broader permissions to run tests. + +Preparation now reconciles two separate access requirements: the cluster's Entra admin-group configuration for Kubernetes administration, and **Azure Arc Enabled Kubernetes Cluster User Role** (`00493d72-78f6-4148-b6c5-d3ce8e4799dd`) for ARM's `Microsoft.Kubernetes/connectedClusters/listClusterUserCredential/action`. New role assignments target only the `Microsoft.Kubernetes/connectedClusters/localbox-aks` resource (or the configured cluster name), never the resource group or subscription. Existing unconditional assignments for the same group/role at the cluster, resource group or subscription are reused, including a manually assigned resource-group grant. Existing assignments are never deleted or broadened; conditional grants require administrator review. + +Rerunning preparation repairs a missing proxy assignment on an existing cluster, using a stable assignment ID. `-SkipAks` skips this RBAC step as well as AKS preparation. Authorization failures stop preparation with the required permission and scope; they are not treated as success. After a new grant, allow Azure RBAC propagation and refresh the group member's sign-in/proxy session if access is initially denied. Role assignment does not prove group membership or successful proxy access; rerun the Full health checks using a group-member kubeconfig. Finally, use the [participant VM exercise](manual-preparation.md#step-6-test-the-environment) to validate actual VM guest networking, guest management, Defender coverage and update assessment. These participant-created VMs are not created by the preparation script. +## Measure participant VM capacity + +Use [test-localbox-capacity.ps1](../test-localbox-capacity.ps1) beside [prepare-localbox.ps1](../prepare-localbox.ps1) to test the Challenge 6 VM footprint: 2 vCPUs, fixed 4096 MiB RAM, the prepared Windows image, VLAN 200 and `UserStorage_1`, with guest management enabled. It does not install updates, alter Defender plans, change RBAC or resize shared infrastructure. Existing subscription policies and paid plans still apply to the test VMs. + +The script creates a uniquely named, tagged `rg-lbcap-` resource group. By default it provisions one VM at a time. Set `-BatchSize 5` to submit up to five deployments without waiting for each VM to finish; Azure provisioning overlaps, although ARM previews and request submission are serialized. The whole batch must reach ARM success, Running power state and Connected guest management before another batch starts. It stops at the requested total, the first failure/timeout, or a safety reserve. Failed provisioning is not automatically classified as capacity exhaustion, and submitted Azure operations can continue after a local timeout. No automatic rollback or deletion occurs. + +Run in PowerShell 7 with your existing Azure CLI login. The operator needs resource-group creation/deployment permissions in the selected subscription, access to the shared image/network/storage/custom location, and VM Run Command permission on `LocalBox-Client` for remote mode. No login, logout or account selection is performed by the script. Keep other preparation/capacity sessions idle. + +From your workstation, Linux/WSL or another machine, use `-RemoteClusterId`. Deployments run locally; read-only probes execute on the Client through Azure VM Run Command. You do not need to copy the scripts or retrieve the nested administrator password. The probe uses the installed configuration locally on the Client and returns only an allowlisted nonsecret manifest and capacity measurements. Run Command creates normal operational execution artifacts. Even `-WhatIf` performs these probes, but creates no VMs, resource group or journal. + +```powershell +$clusterId = '/subscriptions//resourceGroups/rg-localbox-shared/providers/Microsoft.AzureStackHCI/clusters/localboxcluster' +$statePath = Join-Path $HOME '.microhack/capacity-test/run.json' +$run = @{ RemoteClusterId = $clusterId; StatePath = $statePath } + +./test-localbox-capacity.ps1 @run -VmCount 5 -WhatIf +./test-localbox-capacity.ps1 @run -VmCount 5 -GenerateVmCredential +./test-localbox-capacity.ps1 @run -Mode Status + +# Increase the total for the same run; existing Ready VMs are retained. +./test-localbox-capacity.ps1 @run -VmCount 10 -GenerateVmCredential + +# Exercise participant-like deployment overlap in a bounded batch. +./test-localbox-capacity.ps1 @run -VmCount 15 -BatchSize 5 -GenerateVmCredential -SubmitBatch +./test-localbox-capacity.ps1 @run -Mode Status + +# Review first; the second command prompts before deletion. +./test-localbox-capacity.ps1 @run -Mode Cleanup -WhatIf +./test-localbox-capacity.ps1 @run -Mode Cleanup +``` + +Alternatively, run elevated on `LocalBox-Client` without `-RemoteClusterId`. The local defaults use `C:\LocalBox\sovereign-localbox.json` and write the journal to `C:\LocalBox\capacity-test\run.json`. Supply `-NodeCredential` only for a rotated nested account; remote probes require the installed configuration to contain the current nested credential. + +`-GenerateVmCredential` creates a random local administrator password in memory and discards it after the invocation. It is not recoverable from the report. Use `-VmCredential (Get-Credential -UserName localadmin)` instead when you need interactive guest sign-in. The password is passed via a secure ARM parameter in a temporary directory restricted to the invoking user, never as a command-line password. Temporary files are removed on normal success/failure; a forcibly killed process can leave the protected temporary directory behind. Never print or publish these files. + +Default reserves are 300 GiB on the backing `V:` volume, 150 GiB on the primary CSV and 12 GiB of free memory on **each** nested node. Before submitting a batch, the script additionally budgets 40 GiB of guest disk growth per new VM multiplied by the storage copy count on `V:`, and the entire batch's memory on each node without assuming balanced placement. For five 4-GiB VMs and two storage copies, that requires at least 700 GiB free on `V:`, 350 GiB on the primary CSV and 32 GiB free on each node before submission. These conservative stop thresholds are configurable upward or within the parameter limits, not guarantees that later workloads cannot grow. The two CSV capacities are not additional physical capacity, and the script does not place test VMs on `UserStorage_2`. + +Keep the JSON journal: it identifies the run, VMs, target, baseline/latest measurements and stop reason. The adjacent CSV records VM readiness, elapsed seconds and per-VM post-deployment headroom. Re-running with a larger `-VmCount` extends the same run only when its existing VMs are Ready; it never changes their size or recreates them. Use `-Mode Status` after an interrupted or timed-out invocation before retrying. A trailing `Submitting` entry can be retried only after Azure confirms no deployment, VM or NIC exists; partial resources require manual inspection. A new journal creates a separate run, so clean up old runs to avoid consuming capacity twice. + +For terminals that cannot maintain a long-running monitor, add `-SubmitNext` for one VM, or `-SubmitBatch -BatchSize 5` for one batch. The command returns while Azure provisions the submitted VMs. Check `-Mode Status`, then repeat once all existing VMs are Ready. `-VmCount` remains the total limit, not the number to add; a partly filled final batch will be smaller. Submitted VMs are not counted as healthy. ARM what-if must confirm exactly the three intended new resources for each VM; harmless `Ignore`/`NoChange` entries for existing resources are allowed, but modifications and deletions are rejected. + +Cleanup validates the exact journal-derived group name, subscription, cluster, group ownership tags and resource inventory. It refuses a group containing unrelated resources and never deletes the shared LocalBox group. Do not add other resources to the test group. Review the Azure Local VM list after cleanup to confirm nested workload storage was released, then rerun the Full LocalBox health checks. + +A successful test establishes a **running, guest-connected VM count**, not a maximum or a workshop performance guarantee. The journal records each VM's batch ID and readiness time; post-deployment headroom measurements apply to the completed batch. CPU is sampled, not stress-tested. Batch mode exercises overlapping deployment and guest onboarding, but Defender coverage and simultaneous Update Manager assessments still need a representative rehearsal. Node-failure reserve is not certified by this script. Leave event headroom below the demonstrated count. + ## References - [Jumpstart AKS on Azure Local](https://jumpstart.azure.com/azure_jumpstart_localbox/AKS) - [Azure Local Marketplace images](https://learn.microsoft.com/azure/azure-local/manage/virtual-machine-image-azure-marketplace) - [Logical networks for AKS on Azure Local](https://learn.microsoft.com/azure/aks-hybrid-edge/local/hyperconverged/aks-networks) - [Azure CLI image commands](https://learn.microsoft.com/cli/azure/stack-hci-vm/image) -- [Jumpstart configuration](https://github.com/microsoft/azure_arc/blob/main/azure_jumpstart_localbox/artifacts/PowerShell/LocalBox-Config.psd1) \ No newline at end of file +- [Jumpstart configuration](https://github.com/microsoft/azure_arc/blob/main/azure_jumpstart_localbox/artifacts/PowerShell/LocalBox-Config.psd1) +- [Create Azure Local VMs with ARM templates](https://learn.microsoft.com/azure/azure-local/manage/create-arc-virtual-machines) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 index 4d0d3577d..051563230 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/prepare-localbox.ps1 @@ -9,6 +9,8 @@ credential from the installed Jumpstart configuration. Passwords are not logged. Missing required Azure CLI extensions are installed without upgrading existing versions, including during WhatIf. Azure, Hyper-V and storage changes remain simulated during WhatIf. +AKS preparation grants its Entra admin group the cluster-scoped Azure Arc Enabled +Kubernetes Cluster User Role for proxy access, reusing applicable existing grants. Dot-source this file to load its functions without running provisioning. #> [CmdletBinding(SupportsShouldProcess)] @@ -264,6 +266,54 @@ function Sync-LocalBoxResource { } } +function Sync-LocalBoxAksProxyRole { + [CmdletBinding(SupportsShouldProcess)] + param([Parameter(Mandatory)][string]$ClusterId, [Parameter(Mandatory)][string]$GroupObjectId) + if ($ClusterId -notmatch '^/subscriptions/([^/]+)/resourceGroups/[^/]+/providers/Microsoft\.Kubernetes/connectedClusters/[^/]+$') { + throw 'The Arc proxy role must be scoped to an Azure Arc connected-cluster resource.' + } + $subscription = $Matches[1] + $groupId = Assert-LocalBoxGroupId $GroupObjectId + $roleId = '00493d72-78f6-4148-b6c5-d3ce8e4799dd' + $roleDefinitionId = "/subscriptions/$subscription/providers/Microsoft.Authorization/roleDefinitions/$roleId" + $action = "Assign Azure Arc Enabled Kubernetes Cluster User Role to group $groupId" + if ($WhatIfPreference) { + $null = $PSCmdlet.ShouldProcess($ClusterId, $action) + return + } + try { + $assignments = @(Invoke-LocalBoxAz @('role', 'assignment', 'list', '--subscription', $subscription, + '--scope', $ClusterId, '--include-inherited', '--fill-principal-name', 'false', '--fill-role-definition-name', 'false')) + } + catch { throw "Cannot inspect Arc proxy RBAC on $ClusterId. The Client managed identity needs Microsoft.Authorization/roleAssignments/read at this scope. $($_.Exception.Message)" } + $matching = @($assignments | Where-Object { + $_.principalId -ieq $groupId -and ([string]$_.roleDefinitionId).Split('/')[-1] -ieq $roleId -and + $_.scope -and ($_.scope -ieq $ClusterId -or $ClusterId.StartsWith(([string]$_.scope).TrimEnd('/') + '/', [StringComparison]::OrdinalIgnoreCase)) + }) + if (@($matching | Where-Object { -not $_.condition }).Count) { + Write-Host "Reusing Azure Arc Enabled Kubernetes Cluster User Role for group $groupId on $ClusterId (direct or inherited)." + return + } + if ($matching.Count) { + throw 'An existing applicable Arc proxy role assignment has a condition. Have the access administrator review it; preparation will not replace or bypass its restrictions.' + } + if ($PSCmdlet.ShouldProcess($ClusterId, $action)) { + $algorithm = [Security.Cryptography.SHA256]::Create() + try { $hash = $algorithm.ComputeHash([Text.Encoding]::UTF8.GetBytes("$ClusterId|$groupId|$roleId".ToLowerInvariant())) } + finally { $algorithm.Dispose() } + $assignmentName = [guid]::new([byte[]]$hash[0..15]).ToString() + Write-Host "Assigning Azure Arc Enabled Kubernetes Cluster User Role to group $groupId on $ClusterId..." + try { + $assignment = Invoke-LocalBoxAz @('role', 'assignment', 'create', '--subscription', $subscription, + '--name', $assignmentName, '--assignee-object-id', $groupId, '--assignee-principal-type', 'Group', + '--role', $roleDefinitionId, '--scope', $ClusterId) + } + catch { throw "Cannot assign Arc proxy RBAC on $ClusterId. Have an authorized administrator grant this role to group $groupId, or authorize the Client managed identity for Microsoft.Authorization/roleAssignments/write at this scope, then rerun. Preparation does not elevate its identity. $($_.Exception.Message)" } + Assert-LocalBoxProperties $assignment @{ scope = $ClusterId; principalId = $groupId; principalType = 'Group'; roleDefinitionId = $roleDefinitionId } + if ($assignment.condition) { throw 'The returned Arc proxy role assignment is conditional; access has not been established.' } + } +} + function Assert-LocalBoxImagePlacement { param([array]$Resources, [string]$ImageId, [string]$StorageId) if (@($Resources | Where-Object { $_.id -ieq $ImageId }).Count -eq 0) { return } @@ -563,6 +613,9 @@ function Invoke-LocalBoxPreparation { $connected = Get-LocalBoxResource $aksId } } + if (-not $Settings.SkipAks) { + Sync-LocalBoxAksProxyRole -ClusterId $aksId -GroupObjectId $groupId -WhatIf:$WhatIfPreference + } $manifest = @{ SubscriptionId = $Settings.SubscriptionId; ResourceGroupName = $Settings.ResourceGroupName; CustomLocationId = $custom.id ClusterId = $cluster[0].id; BridgeId = $bridge.id; ExtensionId = $extensions[0]; StorageId = $storageId diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-localbox-capacity.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-localbox-capacity.ps1 new file mode 100644 index 000000000..3b125e7ae --- /dev/null +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-localbox-capacity.ps1 @@ -0,0 +1,565 @@ +#Requires -Version 7.0 +<# +.SYNOPSIS +Creates bounded, disposable Azure Local VM capacity tests on LocalBox-Client. +.DESCRIPTION +Uses existing Azure CLI authentication and the preparation manifest. Does not change +shared infrastructure, authentication, Defender plans or update policy. +Dot-source to load functions without provisioning. +#> +[CmdletBinding(SupportsShouldProcess)] +param( + [ValidateSet('Deploy', 'Status', 'Cleanup')][string]$Mode = 'Deploy', + [string]$LocalBoxManifestPath = 'C:\LocalBox\sovereign-localbox.json', + [string]$StatePath = 'C:\LocalBox\capacity-test\run.json', + [string]$RemoteClusterId, + [switch]$GenerateVmCredential, + [switch]$SubmitNext, + [switch]$SubmitBatch, + [ValidateRange(1, 5)][int]$BatchSize = 1, + [ValidateRange(1, 50)][int]$VmCount = 5, + [ValidateRange(1, 8)][int]$ProcessorCount = 2, + [ValidateRange(2048, 16384)][int]$MemoryMB = 4096, + [ValidateRange(200, 2048)][int]$MinimumHostFreeGB = 300, + [ValidateRange(100, 1024)][int]$MinimumCsvFreeGB = 150, + [ValidateRange(8, 64)][int]$MinimumNodeFreeGB = 12, + [ValidateRange(32, 256)][int]$DiskBudgetGB = 40, + [ValidateRange(5, 120)][int]$TimeoutMinutes = 30, + [pscredential]$VmCredential, + [pscredential]$NodeCredential +) + +function Get-LocalBoxCapacitySnapshot { + param([hashtable]$LocalBox, [pscredential]$Credential) + if ($LocalBox.RemoteClusterId) { + return (Get-LocalBoxRemoteCapacityContext $LocalBox.RemoteClusterId $LocalBox.RemoteManifestPath).Snapshot + } + $volume = Get-Volume -DriveLetter V -ErrorAction Stop + $nodes = @(foreach ($node in $LocalBox.NodeNames) { + Invoke-Command -VMName $node -Credential $Credential -ErrorAction Stop -ScriptBlock { + $ErrorActionPreference = 'Stop' + $os = Get-CimInstance Win32_OperatingSystem + $disk = Get-VirtualDisk -FriendlyName 'UserStorage_1' + $csv = Get-Volume -FileSystemLabel 'UserStorage_1' + $pool = Get-StoragePool -FriendlyName 'SU1_Pool' + [pscustomobject]@{ + Name = $env:COMPUTERNAME + FreeMemoryGB = $os.FreePhysicalMemory / 1MB + TotalMemoryGB = $os.TotalVisibleMemorySize / 1MB + CpuPercent = (Get-CimInstance Win32_Processor | Measure-Object LoadPercentage -Average).Average + CsvFreeGB = $csv.SizeRemaining / 1GB + Copies = $disk.NumberOfDataCopies + Healthy = $disk.HealthStatus -eq 'Healthy' -and $csv.HealthStatus -eq 'Healthy' -and + $pool.HealthStatus -eq 'Healthy' -and + @(Get-ClusterNode | Where-Object State -ne 'Up').Count -eq 0 -and + @(Get-StorageJob | Where-Object JobState -ne 'Completed').Count -eq 0 + } + } + }) + @{ + Timestamp = [datetime]::UtcNow.ToString('o') + HostFreeGB = $volume.SizeRemaining / 1GB + HostHealthy = $volume.HealthStatus -eq 'Healthy' + Nodes = $nodes + } +} + +function Get-LocalBoxRemoteCapacityContext { + param([string]$ClusterId, [string]$ManifestPath) + if ($ClusterId -notmatch '^/subscriptions/([a-f0-9-]{36})/resourceGroups/([a-zA-Z0-9_.()-]+)/providers/Microsoft\.AzureStackHCI/clusters/[a-zA-Z0-9-]+$') { + throw 'RemoteClusterId must identify the exact Azure Local cluster.' + } + $subscription = $Matches[1] + $resourceGroup = $Matches[2] + $pathData = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($ManifestPath)) + $clusterData = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($ClusterId)) + $probe = "function Get-LocalBoxCapacitySnapshot {`n${function:Get-LocalBoxCapacitySnapshot}`n}`n" + + "function Resolve-LocalBoxNodeCredential {`n${function:Resolve-LocalBoxNodeCredential}`n}`n" + @' +$ErrorActionPreference = 'Stop' +try { + $manifestPath = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('__PATH__')) + $clusterId = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('__CLUSTER__')) + $manifest = Get-Content -LiteralPath $manifestPath -Raw | ConvertFrom-Json -AsHashtable + if ($manifest.ClusterId -ine $clusterId) { throw 'Target mismatch' } + $configPath = [Environment]::GetEnvironmentVariable('LocalBoxConfigFile', 'Machine') + if (-not $configPath) { $configPath = $env:LocalBoxConfigFile } + $configuration = Import-PowerShellDataFile -LiteralPath $configPath + $credential = Resolve-LocalBoxNodeCredential -Configuration $configuration + $snapshot = Get-LocalBoxCapacitySnapshot $manifest $credential + $snapshot.Nodes = @($snapshot.Nodes | Select-Object Name, FreeMemoryGB, TotalMemoryGB, CpuPercent, CsvFreeGB, Copies, Healthy) + $summary = @{} + foreach ($key in @('SubscriptionId', 'ResourceGroupName', 'ClusterId', 'CustomLocationId', 'StorageId', 'ImageId', 'NodeNames')) { $summary[$key] = $manifest[$key] } + $summary.Networks = @($manifest.Networks | ForEach-Object { @{ Name = $_.Name; Id = $_.Id } }) + 'LB_CAPACITY_BEGIN' + @{ LocalBox = $summary; Snapshot = $snapshot } | ConvertTo-Json -Depth 8 -Compress + 'LB_CAPACITY_END' +} +catch { Write-Error 'LocalBox telemetry failed. Check the manifest, installed configuration and nested-node access on the target Client; no configuration or credential values are returned.'; exit 1 } +'@ + $probe = $probe.Replace('__PATH__', $pathData).Replace('__CLUSTER__', $clusterData) + $encoded = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($probe)) + $command = "& 'C:\Program Files\PowerShell\7\pwsh.exe' -NoLogo -NoProfile -NonInteractive -EncodedCommand '$encoded'" + Write-Host "Reading LocalBox headroom through Run Command on $resourceGroup/LocalBox-Client..." + $result = Invoke-LocalBoxAz @('vm', 'run-command', 'invoke', '--subscription', $subscription, + '--resource-group', $resourceGroup, '--name', 'LocalBox-Client', '--command-id', 'RunPowerShellScript', '--scripts', $command) -TimeoutSeconds 300 + $output = ($result.value.message -join "`n") + if ($output -notmatch '(?s)LB_CAPACITY_BEGIN\s*(\{.*?\})\s*LB_CAPACITY_END') { + throw 'Remote telemetry returned no complete result. No VM will be submitted; inspect Run Command on the target Client.' + } + $context = $Matches[1] | ConvertFrom-Json -AsHashtable -ErrorAction Stop + if ($context.LocalBox.ClusterId -ine $ClusterId -or $context.LocalBox.SubscriptionId -ine $subscription -or + $context.LocalBox.ResourceGroupName -ine $resourceGroup) { throw 'Remote telemetry does not match the requested LocalBox scope.' } + $context.LocalBox.RemoteClusterId = $ClusterId + $context.LocalBox.RemoteManifestPath = $ManifestPath + return $context +} + +function Assert-LocalBoxCapacityHeadroom { + param([hashtable]$Snapshot, [hashtable]$Limits, [ValidateRange(1, 5)][int]$AdditionalVMs = 1) + if (-not $Snapshot.HostHealthy -or @($Snapshot.Nodes).Count -ne 2 -or + @($Snapshot.Nodes | Where-Object { -not $_.Healthy }).Count) { + throw 'Capacity test stopped: host storage or the two-node cluster is not healthy and idle.' + } + if (@($Snapshot.Nodes | Where-Object { $_.Copies -lt 2 }).Count) { throw 'Cannot establish the storage resiliency copy count; no VM was submitted.' } + $copies = ($Snapshot.Nodes | Measure-Object Copies -Maximum).Maximum + if ($Snapshot.HostFreeGB -lt ($Limits.MinimumHostFreeGB + $Limits.DiskBudgetGB * $copies * $AdditionalVMs)) { + throw 'Capacity test stopped at the host backing-volume reserve (including mirrored disk growth).' + } + foreach ($node in $Snapshot.Nodes) { + if ($node.CsvFreeGB -lt ($Limits.MinimumCsvFreeGB + $Limits.DiskBudgetGB * $AdditionalVMs)) { + throw 'Capacity test stopped at the UserStorage_1 free-space reserve.' + } + if ($node.FreeMemoryGB -lt ($Limits.MinimumNodeFreeGB + $Limits.MemoryMB / 1024 * $AdditionalVMs)) { + throw "Capacity test stopped at the memory reserve on $($node.Name)." + } + } +} + +function New-LocalBoxCapacityTemplate { + $machineId = "[resourceId('Microsoft.HybridCompute/machines', parameters('name'))]" + $nicId = "[resourceId('Microsoft.AzureStackHCI/networkInterfaces', concat(parameters('name'), '-nic'))]" + $extendedLocation = @{ type = 'CustomLocation'; name = "[parameters('customLocationId')]" } + @{ + '$schema' = 'https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#' + contentVersion = '1.0.0.0' + parameters = @{ + name = @{ type = 'string'; maxLength = 15 }; location = @{ type = 'string' } + customLocationId = @{ type = 'string' }; imageId = @{ type = 'string' } + networkId = @{ type = 'string' }; storageId = @{ type = 'string' } + processors = @{ type = 'int' }; memoryMB = @{ type = 'int' } + adminUsername = @{ type = 'string' }; adminPassword = @{ type = 'securestring' } + tags = @{ type = 'object' } + } + resources = @( + @{ + type = 'Microsoft.HybridCompute/machines'; apiVersion = '2023-10-03-preview' + name = "[parameters('name')]"; location = "[parameters('location')]" + kind = 'HCI'; identity = @{ type = 'SystemAssigned' }; tags = "[parameters('tags')]" + } + @{ + type = 'Microsoft.AzureStackHCI/networkInterfaces'; apiVersion = '2024-01-01' + name = "[concat(parameters('name'), '-nic')]"; location = "[parameters('location')]" + extendedLocation = $extendedLocation; tags = "[parameters('tags')]" + properties = @{ ipConfigurations = @(@{ + name = 'ipconfig1'; properties = @{ subnet = @{ id = "[parameters('networkId')]" } } + }) } + } + @{ + type = 'Microsoft.AzureStackHCI/virtualMachineInstances'; apiVersion = '2024-01-01' + name = 'default'; scope = $machineId; extendedLocation = $extendedLocation + dependsOn = @($machineId, $nicId) + properties = @{ + hardwareProfile = @{ vmSize = 'Custom'; processors = "[parameters('processors')]"; memoryMB = "[parameters('memoryMB')]" } + osProfile = @{ + adminUsername = "[parameters('adminUsername')]"; adminPassword = "[parameters('adminPassword')]" + computerName = "[parameters('name')]" + windowsConfiguration = @{ provisionVMAgent = $true; provisionVMConfigAgent = $true } + } + storageProfile = @{ vmConfigStoragePathId = "[parameters('storageId')]"; imageReference = @{ id = "[parameters('imageId')]" } } + networkProfile = @{ networkInterfaces = @(@{ id = $nicId }) } + } + } + ) + } +} + +function Save-LocalBoxCapacityState { + param([hashtable]$State, [string]$Path) + $State.UpdatedAt = [datetime]::UtcNow.ToString('o') + $fullPath = [IO.Path]::GetFullPath($Path) + [IO.Directory]::CreateDirectory([IO.Path]::GetDirectoryName($fullPath)) | Out-Null + $temporary = "$fullPath.tmp" + [IO.File]::WriteAllText($temporary, ($State | ConvertTo-Json -Depth 30)) + [IO.File]::Move($temporary, $fullPath, $true) + $State.VMs | ForEach-Object { [pscustomobject]$_ } | + Export-Csv -LiteralPath ([IO.Path]::ChangeExtension($fullPath, '.csv')) -NoTypeInformation -WhatIf:$false -Confirm:$false +} + +function Assert-LocalBoxCapacityOwnership { + param([hashtable]$State, [hashtable]$LocalBox, $Group, [array]$Resources) + if ($State.SchemaVersion -ne 1 -or $State.RunId -notmatch '^[a-f0-9]{32}$' -or + $State.ResourceGroupName -cne "rg-lbcap-$($State.RunId)" -or + $State.SubscriptionId -ine $LocalBox.SubscriptionId -or $State.ClusterId -ine $LocalBox.ClusterId -or + $State.ResourceGroupName -ieq $LocalBox.ResourceGroupName) { + throw 'Invalid capacity-test journal or mismatched LocalBox scope; refusing to manage resources.' + } + $groupId = "/subscriptions/$($State.SubscriptionId)/resourceGroups/$($State.ResourceGroupName)" + if ($Group.id -ine $groupId -or $Group.tags.MicroHackPurpose -cne 'LocalBoxCapacityTest' -or + $Group.tags.MicroHackRunId -cne $State.RunId) { + throw 'Resource-group ownership tags do not match this capacity-test journal.' + } + $allowedIds = @() + $machineIds = @() + foreach ($entry in $State.VMs) { + $index = [int]$entry.Index + if ($index -lt 1 -or $index -gt 50 -or $entry.Name -cne ('lc{0}-{1:D3}' -f $State.RunId.Substring(0, 8), $index)) { + throw 'Invalid VM name in capacity-test journal.' + } + $machineIds += "$groupId/providers/Microsoft.HybridCompute/machines/$($entry.Name)" + $allowedIds += "$groupId/providers/Microsoft.AzureStackHCI/networkInterfaces/$($entry.Name)-nic" + $allowedIds += "$groupId/providers/Microsoft.Resources/deployments/$($entry.Name)" + } + foreach ($resource in $Resources) { + $owned = $resource.id -iin ($allowedIds + $machineIds) + foreach ($machineId in $machineIds) { + if ($resource.id -ieq "$machineId/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default" -or + $resource.id.StartsWith("$machineId/extensions/", [StringComparison]::OrdinalIgnoreCase)) { $owned = $true } + } + if (-not $owned) { throw "Unrelated resource $($resource.id) found in the test group; refusing modification or cleanup." } + } +} + +function Submit-LocalBoxCapacityVM { + param([hashtable]$State, [hashtable]$Entry, [pscredential]$Credential) + $directory = Join-Path ([IO.Path]::GetTempPath()) "lbcap-$([guid]::NewGuid())" + try { + if ($IsWindows) { + New-Item -ItemType Directory -Path $directory -ErrorAction Stop | Out-Null + $acl = Get-Acl -LiteralPath $directory + $acl.SetAccessRuleProtection($true, $false) + $identity = [Security.Principal.WindowsIdentity]::GetCurrent().User + $rule = [Security.AccessControl.FileSystemAccessRule]::new($identity, 'FullControl', 'ContainerInherit, ObjectInherit', 'None', 'Allow') + $acl.SetAccessRule($rule) + Set-Acl -LiteralPath $directory -AclObject $acl -ErrorAction Stop + } + else { [IO.Directory]::CreateDirectory($directory, [IO.UnixFileMode]::UserRead -bor [IO.UnixFileMode]::UserWrite -bor [IO.UnixFileMode]::UserExecute) | Out-Null } + $templatePath = Join-Path $directory 'vm.json' + New-LocalBoxCapacityTemplate | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $templatePath -ErrorAction Stop + $values = @{ + name = $Entry.Name; location = $State.Location; customLocationId = $State.CustomLocationId + imageId = $State.ImageId; networkId = $State.NetworkId; storageId = $State.StorageId + processors = $State.ProcessorCount; memoryMB = $State.MemoryMB + adminUsername = $Credential.UserName; adminPassword = $Credential.GetNetworkCredential().Password + tags = @{ MicroHackPurpose = 'LocalBoxCapacityTest'; MicroHackRunId = $State.RunId } + } + $parameters = @{} + foreach ($key in $values.Keys) { $parameters[$key] = @{ value = $values[$key] } } + $parameterPath = Join-Path $directory 'parameters.json' + @{ '$schema' = 'https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#'; contentVersion = '1.0.0.0'; parameters = $parameters } | + ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $parameterPath -ErrorAction Stop + $arguments = @('--subscription', $State.SubscriptionId, '--resource-group', $State.ResourceGroupName, + '--name', $Entry.Name, '--template-file', $templatePath, '--parameters', "@$parameterPath") + $preview = Invoke-LocalBoxAz (@('deployment', 'group', 'what-if') + $arguments + @('--no-pretty-print', '--result-format', 'ResourceIdOnly')) + $machineId = "/subscriptions/$($State.SubscriptionId)/resourceGroups/$($State.ResourceGroupName)/providers/Microsoft.HybridCompute/machines/$($Entry.Name)" + $allowed = @($machineId, "$machineId/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default", + "/subscriptions/$($State.SubscriptionId)/resourceGroups/$($State.ResourceGroupName)/providers/Microsoft.AzureStackHCI/networkInterfaces/$($Entry.Name)-nic") + $creates = @($preview.changes | Where-Object changeType -eq 'Create') + $unsafe = @($preview.changes | Where-Object { + ($_.resourceId -iin $allowed -and $_.changeType -ne 'Create') -or + ($_.resourceId -inotIn $allowed -and $_.changeType -notin @('Ignore', 'NoChange')) + }) + if ($preview.status -ne 'Succeeded' -or $creates.Count -ne 3 -or + @($creates.resourceId | Sort-Object -Unique).Count -ne 3 -or $unsafe.Count) { + $Entry.SubmissionError = 'ARM what-if did not confirm exactly the three new test resources with no changes to existing resources.' + throw $Entry.SubmissionError + } + Invoke-LocalBoxAz (@('deployment', 'group', 'create') + $arguments + @('--no-wait')) -NoOutput + } + catch { throw "VM submission failed for $($Entry.Name). Inspect its ARM deployment; resources may still be provisioning. CLI details are withheld to protect the VM credential." } + finally { + $values = $null + $parameters = $null + if (Test-Path -LiteralPath $directory) { Remove-Item -LiteralPath $directory -Recurse -Force -ErrorAction Stop } + } +} + +function Update-LocalBoxCapacityVM { + param([hashtable]$State, [hashtable]$Entry) + $common = @('--subscription', $State.SubscriptionId, '--resource-group', $State.ResourceGroupName, '--name', $Entry.Name) + $deployment = Invoke-LocalBoxAz (@('deployment', 'group', 'show') + $common) + $Entry.DeploymentState = $deployment.properties.provisioningState + if ($Entry.DeploymentState -in @('Failed', 'Canceled', 'Deleting')) { + $Entry.Status = 'Failed' + return + } + if ($Entry.DeploymentState -ne 'Succeeded') { $Entry.Status = 'Provisioning'; return } + $machineId = "/subscriptions/$($State.SubscriptionId)/resourceGroups/$($State.ResourceGroupName)/providers/Microsoft.HybridCompute/machines/$($Entry.Name)" + $instance = Invoke-LocalBoxAz @('resource', 'show', '--ids', "$machineId/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default", '--api-version', '2024-01-01') + $machine = Invoke-LocalBoxAz @('resource', 'show', '--ids', $machineId, '--api-version', '2023-10-03-preview') + $Entry.PowerState = $instance.properties.status.powerState + $Entry.GuestStatus = $machine.properties.status + if ($instance.properties.provisioningState -eq 'Failed') { $Entry.Status = 'Failed'; return } + $Entry.Status = if ($instance.properties.provisioningState -eq 'Succeeded' -and + $Entry.PowerState -eq 'Running' -and $Entry.GuestStatus -eq 'Connected') { 'Ready' } else { 'WaitingForGuest' } + if ($Entry.Status -eq 'Ready' -and -not $Entry.ReadyAt) { + $Entry.ReadyAt = [datetime]::UtcNow.ToString('o') + $Entry.ElapsedSeconds = [math]::Round(([datetime]::UtcNow - [datetime]$Entry.SubmittedAt).TotalSeconds) + } +} + +function Remove-LocalBoxCapacityRun { + [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'High')] + param([hashtable]$State, [hashtable]$LocalBox, [string]$Path) + $exists = Invoke-LocalBoxAz @('group', 'exists', '--subscription', $State.SubscriptionId, '--name', $State.ResourceGroupName) + if (-not $exists) { Write-Host 'Test resource group is already absent.'; return } + $group = Invoke-LocalBoxAz @('group', 'show', '--subscription', $State.SubscriptionId, '--name', $State.ResourceGroupName) + $resources = @(Invoke-LocalBoxAz @('resource', 'list', '--subscription', $State.SubscriptionId, '--resource-group', $State.ResourceGroupName)) + Assert-LocalBoxCapacityOwnership $State $LocalBox $group $resources + if ($PSCmdlet.ShouldProcess($State.ResourceGroupName, 'Permanently delete all capacity-test VMs, NICs, disks and the dedicated test resource group')) { + Invoke-LocalBoxAz @('group', 'delete', '--subscription', $State.SubscriptionId, '--name', $State.ResourceGroupName, '--yes') -TimeoutSeconds 3600 -NoOutput + if (Invoke-LocalBoxAz @('group', 'exists', '--subscription', $State.SubscriptionId, '--name', $State.ResourceGroupName)) { + throw 'Test resource group still exists; cleanup is not verified.' + } + $State.Status = 'Deleted' + Save-LocalBoxCapacityState $State $Path + } +} + +function Assert-LocalBoxCapacityHost { + if (-not $IsWindows -or $env:COMPUTERNAME -ine 'LocalBox-Client') { throw 'Run in elevated PowerShell 7 on LocalBox-Client.' } + $principal = [Security.Principal.WindowsPrincipal]::new([Security.Principal.WindowsIdentity]::GetCurrent()) + if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { throw 'Run PowerShell 7 as Administrator.' } +} + +function Repair-LocalBoxCapacityPendingSubmission { + param([hashtable]$State) + $pending = @($State.VMs | Where-Object Status -eq 'Submitting') + if (-not $pending.Count) { return } + if ($pending.Count -ne 1 -or $pending[0].Name -ne $State.VMs[-1].Name) { + throw 'Ambiguous pending submissions; inspect the journal before continuing.' + } + $entry = $pending[0] + $deployments = @(Invoke-LocalBoxAz @('deployment', 'group', 'list', '--subscription', $State.SubscriptionId, '--resource-group', $State.ResourceGroupName)) + if (@($deployments | Where-Object name -ieq $entry.Name).Count) { return } + $resources = @(Invoke-LocalBoxAz @('resource', 'list', '--subscription', $State.SubscriptionId, '--resource-group', $State.ResourceGroupName)) + $scope = "/subscriptions/$($State.SubscriptionId)/resourceGroups/$($State.ResourceGroupName)" + $machineId = "$scope/providers/Microsoft.HybridCompute/machines/$($entry.Name)" + $nicId = "$scope/providers/Microsoft.AzureStackHCI/networkInterfaces/$($entry.Name)-nic" + if (@($resources | Where-Object { + $_.id -ieq $machineId -or $_.id -ieq $nicId -or + ([string]$_.id).StartsWith("$machineId/", [StringComparison]::OrdinalIgnoreCase) + }).Count) { throw 'Pending submission has resources but no deployment record; inspect manually before continuing.' } + Write-Host "Retrying unsubmitted VM $($entry.Name): Azure confirms no deployment, machine or NIC exists." + $State.UnsubmittedAttempts = @($State.UnsubmittedAttempts | Where-Object { $null -ne $_ }) + @{ + Name = $entry.Name; SubmittedAt = $entry.SubmittedAt; VerifiedAbsentAt = [datetime]::UtcNow.ToString('o') + } + $State.VMs = @($State.VMs | Where-Object Name -ine $entry.Name) +} + +function Invoke-LocalBoxCapacityRun { + [CmdletBinding(SupportsShouldProcess)] + param([hashtable]$Settings) + $ErrorActionPreference = 'Stop' + $batchSize = if ($Settings.BatchSize) { [int]$Settings.BatchSize } else { 1 } + if ($batchSize -lt 1 -or $batchSize -gt 5) { throw 'BatchSize must be between 1 and 5.' } + if ($Settings.SubmitNext -and ($batchSize -ne 1 -or $Settings.SubmitBatch)) { + throw 'SubmitNext requires BatchSize 1. Use SubmitBatch for a concurrent batch.' + } + if ($Settings.RemoteClusterId) { + $context = Get-LocalBoxRemoteCapacityContext $Settings.RemoteClusterId $Settings.LocalBoxManifestPath + $localBox = $context.LocalBox + } + else { + Assert-LocalBoxCapacityHost + $localBox = Get-Content -LiteralPath $Settings.LocalBoxManifestPath -Raw | ConvertFrom-Json -AsHashtable + } + foreach ($key in @('SubscriptionId', 'ResourceGroupName', 'ClusterId', 'CustomLocationId', 'StorageId', 'ImageId', 'NodeNames', 'Networks')) { + if (-not $localBox[$key]) { throw "Missing preparation manifest field $key." } + } + if (-not $Settings.RemoteClusterId -and ($localBox.SubscriptionId -ine $env:subscriptionId -or $localBox.ResourceGroupName -ine $env:resourceGroup)) { + throw 'Manifest does not match this LocalBox deployment.' + } + $state = if (Test-Path -LiteralPath $Settings.StatePath) { Get-Content -LiteralPath $Settings.StatePath -Raw | ConvertFrom-Json -AsHashtable } else { $null } + if ($Settings.Mode -eq 'Cleanup') { + if (-not $state) { throw 'Cleanup requires the original capacity-test journal.' } + Remove-LocalBoxCapacityRun $state $localBox $Settings.StatePath -WhatIf:$WhatIfPreference + return + } + if ($state -and $state.Status -eq 'Deleted') { throw 'This run was deleted. Use a new StatePath for a new run.' } + if ($state) { + $group = Invoke-LocalBoxAz @('group', 'show', '--subscription', $state.SubscriptionId, '--name', $state.ResourceGroupName) + $resources = @(Invoke-LocalBoxAz @('resource', 'list', '--subscription', $state.SubscriptionId, '--resource-group', $state.ResourceGroupName)) + Assert-LocalBoxCapacityOwnership $state $localBox $group $resources + if ($state.MemoryMB -ne $Settings.MemoryMB -or $state.ProcessorCount -ne $Settings.ProcessorCount) { + throw 'Do not change VM sizing within a capacity run; use the original sizing or a new StatePath.' + } + } + elseif ($Settings.Mode -eq 'Status') { throw 'Status requires an existing capacity-test journal.' } + $credential = $null + if ($Settings.RemoteClusterId) { $snapshot = $context.Snapshot } + else { + $configuration = Import-PowerShellDataFile -LiteralPath $env:LocalBoxConfigFile + $credential = Resolve-LocalBoxNodeCredential -Configuration $configuration -Credential $Settings.NodeCredential + $snapshot = Get-LocalBoxCapacitySnapshot $localBox $credential + } + if ($Settings.Mode -eq 'Status') { + foreach ($entry in $state.VMs) { Update-LocalBoxCapacityVM $state $entry } + $state.LastSnapshot = $snapshot + if (-not $WhatIfPreference) { Save-LocalBoxCapacityState $state $Settings.StatePath } + $state.VMs | ForEach-Object { [pscustomobject]$_ } | Format-Table Name, Status, DeploymentState, PowerState, GuestStatus, ElapsedSeconds + return + } + Assert-LocalBoxCapacityHeadroom $snapshot $Settings + if (-not $state) { + $custom = Get-LocalBoxResource $localBox.CustomLocationId + $image = Get-LocalBoxResource $localBox.ImageId + if ($image.properties.provisioningState -ne 'Succeeded' -or $image.properties.status.progressPercentage -ne 100 -or + $image.properties.containerId -ine $localBox.StorageId) { throw 'The prepared image is not ready on UserStorage_1.' } + $network = @($localBox.Networks | Where-Object Name -eq 'localbox-vm-lnet-vlan200') + if ($network.Count -ne 1) { throw 'Cannot uniquely identify the prepared participant VM logical network.' } + $runId = [guid]::NewGuid().ToString('N') + $state = @{ + SchemaVersion = 1; RunId = $runId; SubscriptionId = $localBox.SubscriptionId; ClusterId = $localBox.ClusterId + ResourceGroupName = "rg-lbcap-$runId"; Location = $custom.location + CustomLocationId = $localBox.CustomLocationId; ImageId = $localBox.ImageId + NetworkId = $network[0].Id; StorageId = $localBox.StorageId + ProcessorCount = $Settings.ProcessorCount; MemoryMB = $Settings.MemoryMB + Status = 'Planned'; VMs = @(); Baseline = $snapshot + } + } + if ($WhatIfPreference) { + $null = $PSCmdlet.ShouldProcess($state.ResourceGroupName, "Create up to $($Settings.VmCount) total VMs, $($Settings.ProcessorCount) vCPU / $($Settings.MemoryMB) MiB each, in batches of $batchSize with guest management") + return + } + if (-not $PSCmdlet.ShouldProcess($state.ResourceGroupName, "Run capacity test up to $($Settings.VmCount) VMs")) { return } + try { + Repair-LocalBoxCapacityPendingSubmission $state + if (-not $Settings.VmCredential -and @($state.VMs).Count -lt $Settings.VmCount) { + if ($Settings.GenerateVmCredential) { + $password = 'Lb!' + [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(30)) + '9a' + $secure = [Security.SecureString]::new() + foreach ($character in $password.GetEnumerator()) { $secure.AppendChar($character) } + $secure.MakeReadOnly() + $Settings.VmCredential = [pscredential]::new('localadmin', $secure) + $password = $null + Write-Host 'Generated a disposable test-VM password in memory. It is not saved; use a supplied VmCredential if interactive sign-in is needed.' + } + else { $Settings.VmCredential = Get-Credential -UserName 'localadmin' -Message 'Local administrator for disposable capacity-test VMs (not the LocalBox administrator)' } + } + if (@($state.VMs).Count -lt $Settings.VmCount -and -not $Settings.VmCredential) { throw 'A test-VM credential is required.' } + if ($state.Status -eq 'Planned') { + Save-LocalBoxCapacityState $state $Settings.StatePath + if (Invoke-LocalBoxAz @('group', 'exists', '--subscription', $state.SubscriptionId, '--name', $state.ResourceGroupName)) { + throw 'The new run resource group already exists; no resources were changed.' + } + Invoke-LocalBoxAz @('group', 'create', '--subscription', $state.SubscriptionId, '--name', $state.ResourceGroupName, + '--location', $state.Location, '--tags', 'MicroHackPurpose=LocalBoxCapacityTest', "MicroHackRunId=$($state.RunId)") | Out-Null + } + foreach ($entry in $state.VMs) { + Update-LocalBoxCapacityVM $state $entry + if ($entry.Status -ne 'Ready') { throw "Existing test VM $($entry.Name) is $($entry.Status). Inspect it or clean up before adding VMs." } + } + $state.Status = 'Running' + $state.StopReason = $null + $state.RequestedCount = $Settings.VmCount + $state.BatchSize = $batchSize + $state.Limits = @{ + MinimumHostFreeGB = $Settings.MinimumHostFreeGB; MinimumCsvFreeGB = $Settings.MinimumCsvFreeGB + MinimumNodeFreeGB = $Settings.MinimumNodeFreeGB; DiskBudgetGB = $Settings.DiskBudgetGB + } + while (@($state.VMs).Count -lt $Settings.VmCount) { + $batchCount = [math]::Min($batchSize, $Settings.VmCount - @($state.VMs).Count) + $snapshot = Get-LocalBoxCapacitySnapshot $localBox $credential + $state.LastSnapshot = $snapshot + Assert-LocalBoxCapacityHeadroom $snapshot $Settings -AdditionalVMs $batchCount + $batch = @() + $batchId = [guid]::NewGuid().ToString('N') + for ($batchIndex = 0; $batchIndex -lt $batchCount; $batchIndex++) { + $index = @($state.VMs).Count + 1 + $entry = @{ + Index = $index; Name = ('lc{0}-{1:D3}' -f $state.RunId.Substring(0, 8), $index) + Status = 'Submitting'; SubmittedAt = [datetime]::UtcNow.ToString('o'); BatchId = $batchId + DeploymentState = ''; PowerState = ''; GuestStatus = ''; ReadyAt = ''; ElapsedSeconds = 0 + HostFreeGBAfter = $null; MinimumNodeFreeGBAfter = $null; CsvFreeGBAfter = $null + } + $state.VMs += $entry + $batch += $entry + Save-LocalBoxCapacityState $state $Settings.StatePath + Write-Host "[$index/$($Settings.VmCount)] Submitting $($entry.Name) in batch of $batchCount; host free before batch: $([math]::Round($snapshot.HostFreeGB)) GiB..." + Submit-LocalBoxCapacityVM $state $entry $Settings.VmCredential + $entry.Status = 'Submitted' + Save-LocalBoxCapacityState $state $Settings.StatePath + } + if ($Settings.SubmitNext -or $Settings.SubmitBatch) { + $state.Status = 'AwaitingReadiness' + Write-Host "Submitted $batchCount VM(s). Azure continues provisioning; use Status before submitting another batch." + return + } + $deadline = [datetime]::UtcNow.AddMinutes($Settings.TimeoutMinutes) + do { + foreach ($entry in $batch) { + Update-LocalBoxCapacityVM $state $entry + Write-Host "$($entry.Name): $($entry.Status); deployment=$($entry.DeploymentState), power=$($entry.PowerState), guest=$($entry.GuestStatus)" + } + Save-LocalBoxCapacityState $state $Settings.StatePath + if (@($batch | Where-Object Status -eq 'Failed').Count) { throw 'A batch deployment failed. This is not necessarily capacity exhaustion; inspect its ARM deployment.' } + if (-not @($batch | Where-Object Status -ne 'Ready').Count) { break } + if ([datetime]::UtcNow -ge $deadline) { throw 'Timed out waiting for the batch. Azure operations may still be running; no additional VMs will be submitted.' } + Start-Sleep -Seconds 15 + } while ($true) + $state.LastSnapshot = Get-LocalBoxCapacitySnapshot $localBox $credential + foreach ($entry in $batch) { + $entry.HostFreeGBAfter = [math]::Round($state.LastSnapshot.HostFreeGB, 1) + $entry.MinimumNodeFreeGBAfter = [math]::Round(($state.LastSnapshot.Nodes | Measure-Object FreeMemoryGB -Minimum).Minimum, 1) + $entry.CsvFreeGBAfter = [math]::Round(($state.LastSnapshot.Nodes | Measure-Object CsvFreeGB -Minimum).Minimum, 1) + } + Save-LocalBoxCapacityState $state $Settings.StatePath + } + foreach ($entry in $state.VMs) { + Update-LocalBoxCapacityVM $state $entry + if ($entry.Status -ne 'Ready') { throw "Test VM $($entry.Name) is no longer ready. Check the final run state before choosing a participant count." } + } + $state.Status = 'TargetReached' + Write-Host "Target reached: $(@($state.VMs | Where-Object Status -eq 'Ready').Count) guest-connected VMs. This is a tested count, not a maximum or a concurrent-workshop performance guarantee." + } + catch { + $state.Status = 'Stopped' + $state.StopReason = $_.Exception.Message + throw + } + finally { + Save-LocalBoxCapacityState $state $Settings.StatePath + } +} + +function Invoke-LocalBoxCapacityTest { + [CmdletBinding(SupportsShouldProcess)] + param([hashtable]$Settings) + $lock = [Threading.Mutex]::new($false, 'Global\MicroHackLocalBoxCapacity') + $locked = $false + try { + $locked = $lock.WaitOne(0) + if (-not $locked) { throw 'Another capacity-test session is running. Deployment, status and cleanup cannot overlap.' } + Invoke-LocalBoxCapacityRun $Settings -WhatIf:$WhatIfPreference + } + finally { + if ($locked) { $lock.ReleaseMutex() } + $lock.Dispose() + } +} + +if ($MyInvocation.InvocationName -ne '.') { + if ($RemoteClusterId -and -not $PSBoundParameters.ContainsKey('StatePath')) { $StatePath = Join-Path $HOME '.microhack/capacity-test/run.json' } + $settings = @{ + Mode = $Mode; LocalBoxManifestPath = $LocalBoxManifestPath; StatePath = $StatePath + RemoteClusterId = $RemoteClusterId; GenerateVmCredential = [bool]$GenerateVmCredential + SubmitNext = [bool]$SubmitNext + SubmitBatch = [bool]$SubmitBatch; BatchSize = $BatchSize + VmCount = $VmCount; ProcessorCount = $ProcessorCount; MemoryMB = $MemoryMB + MinimumHostFreeGB = $MinimumHostFreeGB; MinimumCsvFreeGB = $MinimumCsvFreeGB + MinimumNodeFreeGB = $MinimumNodeFreeGB; DiskBudgetGB = $DiskBudgetGB + TimeoutMinutes = $TimeoutMinutes; VmCredential = $VmCredential; NodeCredential = $NodeCredential + } + . "$PSScriptRoot/prepare-localbox.ps1" -WhatIf:$WhatIfPreference + Invoke-LocalBoxCapacityTest $settings -WhatIf:$WhatIfPreference +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-sovereign-cloud.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-sovereign-cloud.ps1 index a100c53b6..879863a64 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-sovereign-cloud.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/test-sovereign-cloud.ps1 @@ -6,6 +6,9 @@ Runs Pester health checks against explicitly selected Sovereign Cloud lab scopes Uses the caller's Azure CLI login. Never changes authentication or provisions resources. Full checks require kubeconfigs and, for participant guest probes, explicit consent to invoke read-only commands using Azure VM Run Command. See tests/readme.md. +Test scripts are downloaded by default; use -DownloadTests:$false for a local checkout. +LocalBox uses the inventory kubeconfig when present, otherwise $HOME/.kube/config. +An explicit -LocalBoxKubeconfig overrides both. No credentials are retrieved automatically. .EXAMPLE ./test-sovereign-cloud.ps1 -Scope LocalBox -LocalBoxManifestPath ./sovereign-localbox.json -Mode ControlPlane .EXAMPLE @@ -17,7 +20,7 @@ param( [ValidateSet('ControlPlane', 'Full')][string]$Mode = 'Full', [string]$InventoryPath, [string]$LocalBoxManifestPath, - [string]$LocalBoxKubeconfig, + [string]$LocalBoxKubeconfig = (Join-Path $HOME '.kube/config'), [pscredential]$NodeCredential, [switch]$AllowGuestRunCommand, [ValidateRange(1, 120)][int]$TimeoutMinutes = 15, @@ -25,12 +28,13 @@ param( [ValidatePattern('^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$')] [string]$GitHubRepository = 'microsoft/MicroHack', [string]$GitHubRef = 'main', - [switch]$DownloadTests + [switch]$DownloadTests = $true ) function Invoke-SovereignKubectl { param([string]$Kubeconfig, [string[]]$Arguments) if (-not $Kubeconfig -or -not (Test-Path -LiteralPath $Kubeconfig)) { throw 'A valid, independently authenticated kubeconfig is required for Full checks.' } + Write-Host "Querying Kubernetes: kubectl $($Arguments -join ' ') (request timeout: 20s)..." $output = & kubectl --kubeconfig $Kubeconfig --request-timeout=20s @Arguments -o json 2>&1 if ($LASTEXITCODE -ne 0) { throw "kubectl failed: $($output -join ' ')" } ($output -join "`n") | ConvertFrom-Json -AsHashtable -ErrorAction Stop @@ -80,7 +84,12 @@ function Test-SovereignKubernetes { if (-not @($controllers.items).Count) { throw 'No kube-system controllers found.' } foreach ($controller in $controllers.items) { if ($controller.kind -eq 'DaemonSet') { - if ($controller.status.desiredNumberScheduled -lt 1 -or $controller.status.numberReady -lt $controller.status.desiredNumberScheduled) { throw "DaemonSet $($controller.metadata.name) is unavailable." } + $desired = $controller.status.desiredNumberScheduled + $ready = $controller.status.numberReady + if ($null -eq $desired -or $null -eq $ready -or $desired -lt 0 -or $ready -lt 0) { + throw "DaemonSet $($controller.metadata.name) has missing or invalid scheduling status." + } + if ($ready -lt $desired) { throw "DaemonSet $($controller.metadata.name) is unavailable ($ready Ready / $desired desired)." } } elseif ($controller.spec.replicas -gt 0 -and $controller.status.availableReplicas -lt $controller.spec.replicas) { throw "Deployment $($controller.metadata.name) is unavailable." @@ -91,11 +100,17 @@ function Test-SovereignKubernetes { function Wait-SovereignCheck { param([scriptblock]$Check, [int]$TimeoutSeconds = 900) $deadline = [datetime]::UtcNow.AddSeconds($TimeoutSeconds) + $attempt = 0 do { + $attempt++ try { & $Check; return } catch { if ($_.Exception.Message -match 'AuthorizationFailed|Forbidden|Unauthorized|InvalidAuthentication|not logged in|independently authenticated|provisioning failed') { throw } if ([datetime]::UtcNow -ge $deadline) { throw } + $remaining = [math]::Max(0, [math]::Ceiling(($deadline - [datetime]::UtcNow).TotalSeconds)) + $reason = ($_.Exception.Message -replace '\s+', ' ').Trim() + if ($reason.Length -gt 300) { $reason = $reason.Substring(0, 300) + '...' } + Write-Host "Health check attempt $attempt not ready (${remaining}s remaining; retry in 15s): $reason" Start-Sleep -Seconds 15 } } while ($true) @@ -126,6 +141,7 @@ if ($MyInvocation.InvocationName -ne '.') { $root = Join-Path ([IO.Path]::GetTempPath()) "microhack-health-$([guid]::NewGuid())" New-Item -ItemType Directory -Path "$root/tests" -Force | Out-Null $base = "https://raw.githubusercontent.com/$GitHubRepository/$GitHubRef/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources" + Write-Host "Downloading health tests from $GitHubRepository at ref $GitHubRef..." foreach ($file in @('prepare-localbox.ps1', 'test-sovereign-cloud.ps1', 'tests/localbox.health.tests.ps1', 'tests/sovereign-lab.health.tests.ps1')) { Invoke-WebRequest -Uri "$base/$file" -OutFile (Join-Path $root $file) } @@ -133,7 +149,11 @@ if ($MyInvocation.InvocationName -ne '.') { Import-Module Pester -MinimumVersion 5.7.1 -MaximumVersion 5.999.999 -ErrorAction Stop $inventory = if ($InventoryPath) { Get-Content -LiteralPath $InventoryPath -Raw | ConvertFrom-Json -AsHashtable } else { @{} } if ($LocalBoxManifestPath) { $inventory.LocalBox = Get-Content -LiteralPath $LocalBoxManifestPath -Raw | ConvertFrom-Json -AsHashtable } - if ($LocalBoxKubeconfig) { $inventory.LocalBox.Kubeconfig = $LocalBoxKubeconfig } + if ($Scope -in @('LocalBox', 'All') -and $inventory.LocalBox) { + if ($PSBoundParameters.ContainsKey('LocalBoxKubeconfig') -or -not $inventory.LocalBox.Kubeconfig) { + $inventory.LocalBox.Kubeconfig = $LocalBoxKubeconfig + } + } $containers = @() $data = @{ Mode = $Mode; TimeoutSeconds = $TimeoutMinutes * 60; ResourceRoot = $root } if ($Scope -in @('LocalBox', 'All')) { diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 index 3e957b97b..5de1cfaf6 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/prepare-localbox.tests.ps1 @@ -4,6 +4,589 @@ BeforeAll { . "$PSScriptRoot/../../labautomation/localbox-credentials.ps1" } +Describe 'LocalBox capacity safety' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $snapshot = @{ HostHealthy = $true; HostFreeGB = 1200; Nodes = @( + @{ Name = 'node1'; Healthy = $true; CsvFreeGB = 926; Copies = 2; FreeMemoryGB = 48 } + @{ Name = 'node2'; Healthy = $true; CsvFreeGB = 926; Copies = 2; FreeMemoryGB = 48 } + ) } + $limits = @{ MinimumHostFreeGB = 300; MinimumCsvFreeGB = 150; MinimumNodeFreeGB = 12; DiskBudgetGB = 40; MemoryMB = 4096 } + } + It 'allows one VM within all reserves' { + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Not -Throw + } + It 'reserves mirrored growth on the backing host, not just CSV free space' { + $snapshot.HostFreeGB = 379 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Throw '*backing-volume*' + } + It 'reserves disk growth for the entire in-flight batch' { + $snapshot.HostFreeGB = 699 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits -AdditionalVMs 5 } | Should -Throw '*backing-volume*' + $snapshot.HostFreeGB = 700 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits -AdditionalVMs 5 } | Should -Not -Throw + } + It 'reserves batch memory on each node without assuming balanced placement' { + $snapshot.Nodes[0].FreeMemoryGB = 31 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits -AdditionalVMs 5 } | Should -Throw '*node1*' + } + It 'reserves primary CSV space for every in-flight VM' { + $snapshot.Nodes[0].CsvFreeGB = 349 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits -AdditionalVMs 5 } | Should -Throw '*UserStorage_1*' + } + It 'protects the primary CSV' { + $snapshot.Nodes[0].CsvFreeGB = 189 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Throw '*UserStorage_1*' + } + It 'requires placement headroom on each node' { + $snapshot.Nodes[1].FreeMemoryGB = 15 + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Throw '*node2*' + } + It 'stops on an unhealthy node or storage' { + $snapshot.Nodes[1].Healthy = $false + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Throw '*not healthy*' + } + It 'does not estimate capacity without a known resiliency factor' { + $snapshot.Nodes | ForEach-Object { $_.Copies = 0 } + { Assert-LocalBoxCapacityHeadroom $snapshot $limits } | Should -Throw '*copy count*' + } +} + +Describe 'LocalBox capacity lifecycle' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $runId = '123456781234123412341234567890ab' + $groupId = "/subscriptions/test/resourceGroups/rg-lbcap-$runId" + $clusterId = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.AzureStackHCI/clusters/localboxcluster' + $entry = @{ Index = 1; Name = 'lc12345678-001'; Status = 'Submitting'; SubmittedAt = [datetime]::UtcNow.AddSeconds(-60).ToString('o') } + $state = @{ SchemaVersion = 1; RunId = $runId; SubscriptionId = 'test'; ResourceGroupName = "rg-lbcap-$runId"; ClusterId = $clusterId; VMs = @($entry) } + $localBox = @{ SubscriptionId = 'test'; ResourceGroupName = 'localbox'; ClusterId = $clusterId } + $group = @{ id = $groupId; tags = @{ MicroHackPurpose = 'LocalBoxCapacityTest'; MicroHackRunId = $runId } } + $machineId = "$groupId/providers/Microsoft.HybridCompute/machines/$($entry.Name)" + $resources = @(@{ id = $machineId }, @{ id = "$groupId/providers/Microsoft.AzureStackHCI/networkInterfaces/$($entry.Name)-nic" }) + Mock Invoke-LocalBoxAz { throw 'Unexpected CLI call' } + Mock Save-LocalBoxCapacityState {} + } + It 'accepts only matching run ownership' { + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Not -Throw + } + It 'rejects mismatched tags' { + $group.tags.MicroHackRunId = 'another-run' + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*ownership*' + } + It 'cannot target the shared resource group' { + $state.ResourceGroupName = 'localbox' + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*mismatched*' + } + It 'cannot target another LocalBox or subscription' -TestCases @( + @{ Key = 'SubscriptionId'; Value = 'other' } + @{ Key = 'ClusterId'; Value = '/subscriptions/test/resourceGroups/another-cluster' } + ) { + param($Key, $Value) + $state[$Key] = $Value + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*mismatched*' + } + It 'rejects unrelated resources even inside a tagged group' { + $resources += @{ id = "$groupId/providers/Microsoft.HybridCompute/machines/student-vm" } + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*Unrelated resource*' + } + It 'accepts only children of its own machines' { + $resources += @{ id = "$machineId/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default" } + $resources += @{ id = "$machineId/extensions/MDE.Windows" } + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Not -Throw + $resources += @{ id = "$groupId/providers/Microsoft.HybridCompute/machines/student-vm/extensions/MDE.Windows" } + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*Unrelated resource*' + } + It 'rejects journal VM names that do not belong to this run' { + $entry.Name = 'student-vm' + { Assert-LocalBoxCapacityOwnership $state $localBox $group $resources } | Should -Throw '*Invalid VM name*' + } + It 'does not delete under WhatIf' { + Mock Invoke-LocalBoxAz { $true } -ParameterFilter { $Arguments[0] -eq 'group' -and $Arguments[1] -eq 'exists' } + Mock Invoke-LocalBoxAz { $group } -ParameterFilter { $Arguments[0] -eq 'group' -and $Arguments[1] -eq 'show' } + Mock Invoke-LocalBoxAz { $resources } -ParameterFilter { $Arguments[0] -eq 'resource' -and $Arguments[1] -eq 'list' } + Remove-LocalBoxCapacityRun $state $localBox 'state.json' -WhatIf + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments -contains 'delete' } + Should -Invoke Save-LocalBoxCapacityState -Times 0 + } + It 'refuses cleanup when an unrelated resource exists' { + $resources += @{ id = "$groupId/providers/Microsoft.Storage/storageAccounts/unrelated" } + Mock Invoke-LocalBoxAz { $true } -ParameterFilter { $Arguments[1] -eq 'exists' } + Mock Invoke-LocalBoxAz { $group } -ParameterFilter { $Arguments[1] -eq 'show' } + Mock Invoke-LocalBoxAz { $resources } -ParameterFilter { $Arguments[1] -eq 'list' } + { Remove-LocalBoxCapacityRun $state $localBox 'state.json' -Confirm:$false } | Should -Throw '*Unrelated*' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments -contains 'delete' } + } + It 'deletes only the exact owned group and verifies it is gone' { + $checks = @{ Count = 0 } + Mock Invoke-LocalBoxAz { $checks.Count++; return $checks.Count -eq 1 } -ParameterFilter { $Arguments[1] -eq 'exists' } + Mock Invoke-LocalBoxAz { $group } -ParameterFilter { $Arguments[1] -eq 'show' } + Mock Invoke-LocalBoxAz { $resources } -ParameterFilter { $Arguments[1] -eq 'list' } + Mock Invoke-LocalBoxAz {} -ParameterFilter { $Arguments[1] -eq 'delete' } + Remove-LocalBoxCapacityRun $state $localBox 'state.json' -Confirm:$false + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { + $Arguments[0] -eq 'group' -and $Arguments[1] -eq 'delete' -and $Arguments -contains $state.ResourceGroupName -and $Arguments -contains 'test' + } + $state.Status | Should -Be 'Deleted' + Should -Invoke Save-LocalBoxCapacityState -Times 1 + } + It 'requires running and connected status after deployment succeeds' -TestCases @( + @{ Power = 'Running'; Guest = 'Connected'; Expected = 'Ready' } + @{ Power = 'Stopped'; Guest = 'Connected'; Expected = 'WaitingForGuest' } + @{ Power = 'Running'; Guest = 'Disconnected'; Expected = 'WaitingForGuest' } + ) { + param($Power, $Guest, $Expected) + Mock Invoke-LocalBoxAz { @{ properties = @{ provisioningState = 'Succeeded' } } } -ParameterFilter { $Arguments[0] -eq 'deployment' } + Mock Invoke-LocalBoxAz { @{ properties = @{ provisioningState = 'Succeeded'; status = @{ powerState = $Power } } } } -ParameterFilter { $Arguments -contains '2024-01-01' } + Mock Invoke-LocalBoxAz { @{ properties = @{ status = $Guest } } } -ParameterFilter { $Arguments -contains '2023-10-03-preview' } + Update-LocalBoxCapacityVM $state $entry + $entry.Status | Should -Be $Expected + if ($Expected -eq 'Ready') { $entry.ElapsedSeconds | Should -BeGreaterOrEqual 60 } + } + It 'does not count a pending or failed deployment as ready' -TestCases @( + @{ DeploymentState = 'Running'; Expected = 'Provisioning' } + @{ DeploymentState = 'Failed'; Expected = 'Failed' } + ) { + param($DeploymentState, $Expected) + Mock Invoke-LocalBoxAz { @{ properties = @{ provisioningState = $DeploymentState } } } -ParameterFilter { $Arguments[0] -eq 'deployment' } + Update-LocalBoxCapacityVM $state $entry + $entry.Status | Should -Be $Expected + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[0] -eq 'resource' } + } + It 'uses a secure password parameter and explicitly targets the prepared storage and network' { + $template = New-LocalBoxCapacityTemplate + $template.parameters.adminPassword.type | Should -Be 'securestring' + $template.resources.Count | Should -Be 3 + $vm = $template.resources | Where-Object type -eq 'Microsoft.AzureStackHCI/virtualMachineInstances' + $vm.properties.storageProfile.vmConfigStoragePathId | Should -Be "[parameters('storageId')]" + $vm.properties.storageProfile.imageReference.id | Should -Be "[parameters('imageId')]" + $vm.properties.hardwareProfile.memoryMB | Should -Be "[parameters('memoryMB')]" + $vm.properties.hardwareProfile.ContainsKey('dynamicMemoryConfig') | Should -BeFalse + $vm.properties.osProfile.windowsConfiguration.provisionVMAgent | Should -BeTrue + $vm.properties.osProfile.windowsConfiguration.provisionVMConfigAgent | Should -BeTrue + $nic = $template.resources | Where-Object type -eq 'Microsoft.AzureStackHCI/networkInterfaces' + $nic.properties.ipConfigurations[0].properties.subnet.id | Should -Be "[parameters('networkId')]" + $nic.properties.ipConfigurations[0].properties.ContainsKey('privateIPAddress') | Should -BeFalse + } +} + +Describe 'LocalBox capacity orchestration' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $oldSubscription = $env:subscriptionId + $oldGroup = $env:resourceGroup + $oldConfigPath = $env:LocalBoxConfigFile + $env:subscriptionId = 'test' + $env:resourceGroup = 'localbox' + $env:LocalBoxConfigFile = 'mock-config.psd1' + $localBox = @{ + SubscriptionId = 'test'; ResourceGroupName = 'localbox'; ClusterId = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.AzureStackHCI/clusters/localboxcluster' + CustomLocationId = 'custom-location'; StorageId = 'storage1'; ImageId = 'image'; NodeNames = @('node1', 'node2') + Networks = @(@{ Name = 'localbox-vm-lnet-vlan200'; Id = 'vm-network' }) + } + $credential = [pscredential]::new('localadmin', [Security.SecureString]::new()) + $settings = @{ + Mode = 'Deploy'; LocalBoxManifestPath = 'manifest.json'; StatePath = 'state.json'; VmCount = 2 + ProcessorCount = 2; MemoryMB = 4096; MinimumHostFreeGB = 300; MinimumCsvFreeGB = 150 + MinimumNodeFreeGB = 12; DiskBudgetGB = 40; TimeoutMinutes = 1; NodeCredential = $credential; VmCredential = $credential + } + $snapshot = @{ HostHealthy = $true; HostFreeGB = 1200; Nodes = @( + @{ Name = 'node1'; Healthy = $true; CsvFreeGB = 926; Copies = 2; FreeMemoryGB = 48 } + @{ Name = 'node2'; Healthy = $true; CsvFreeGB = 926; Copies = 2; FreeMemoryGB = 48 } + ) } + $journal = @{ State = $null } + Mock Assert-LocalBoxCapacityHost {} + Mock Import-PowerShellDataFile { @{} } + Mock Resolve-LocalBoxNodeCredential { $credential } + Mock Get-LocalBoxCapacitySnapshot { $snapshot } + Mock Test-Path { $false } + Mock Get-Content { $localBox | ConvertTo-Json -Depth 10 } + Mock Get-LocalBoxResource { @{ location = 'westeurope'; properties = @{ provisioningState = 'Succeeded'; containerId = 'storage1'; status = @{ progressPercentage = 100 } } } } + Mock Invoke-LocalBoxAz { $false } + Mock Submit-LocalBoxCapacityVM {} + Mock Update-LocalBoxCapacityVM { $Entry.Status = 'Ready'; $Entry.GuestStatus = 'Connected'; $Entry.PowerState = 'Running' } + Mock Save-LocalBoxCapacityState { $journal.State = $State } + Mock Start-Sleep { throw 'Unexpected sleep' } + Mock Get-Credential { throw 'Unexpected prompt' } + } + AfterEach { $env:subscriptionId = $oldSubscription; $env:resourceGroup = $oldGroup; $env:LocalBoxConfigFile = $oldConfigPath } + It 'creates a bounded run and records only nonsecret state' { + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 2 -Exactly + $journal.State.VMs.Count | Should -Be 2 + $journal.State.Status | Should -Be 'TargetReached' + $journal.State.ResourceGroupName | Should -Match '^rg-lbcap-[a-f0-9]{32}$' + ($journal.State | ConvertTo-Json -Depth 20) | Should -Not -Match 'Password|Credential' + } + It 'does no writes or credential prompts during WhatIf' { + $settings.VmCredential = $null + Invoke-LocalBoxCapacityRun $settings -WhatIf + Should -Invoke Submit-LocalBoxCapacityVM -Times 0 + Should -Invoke Save-LocalBoxCapacityState -Times 0 + Should -Invoke Get-Credential -Times 0 + Should -Invoke Invoke-LocalBoxAz -Times 0 + } + It 'submits only one additional VM without a long-running local monitor' { + $settings.SubmitNext = $true + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 1 -Exactly + Should -Invoke Start-Sleep -Times 0 + $journal.State.Status | Should -Be 'AwaitingReadiness' + $journal.State.VMs.Count | Should -Be 1 + $journal.State.VMs[0].Status | Should -Be 'Submitted' + } + It 'submits five VMs before checking their readiness' { + $settings.VmCount = 5 + $settings.BatchSize = 5 + $submissions = @{ Count = 0 } + Mock Submit-LocalBoxCapacityVM { $submissions.Count++ } + Mock Update-LocalBoxCapacityVM { + $submissions.Count | Should -Be 5 + $Entry.Status = 'Ready' + } + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 5 -Exactly + $journal.State.Status | Should -Be 'TargetReached' + @($journal.State.VMs.BatchId | Select-Object -Unique).Count | Should -Be 1 + } + It 'submits one bounded batch and exits without starting a second' { + $settings.VmCount = 10 + $settings.BatchSize = 5 + $settings.SubmitBatch = $true + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 5 -Exactly + Should -Invoke Start-Sleep -Times 0 + $journal.State.Status | Should -Be 'AwaitingReadiness' + $journal.State.VMs.Count | Should -Be 5 + } + It 'limits the last batch to the remaining target count' { + $settings.VmCount = 3 + $settings.BatchSize = 5 + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 3 -Exactly + $journal.State.VMs.Count | Should -Be 3 + } + It 'does not submit any of a batch that cannot fit the reserves' { + $settings.VmCount = 5 + $settings.BatchSize = 5 + $snapshot.HostFreeGB = 600 + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*backing-volume*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 0 + } + It 'does not start a second batch after a failure in the first' { + $settings.VmCount = 10 + $settings.BatchSize = 5 + Mock Update-LocalBoxCapacityVM { $Entry.Status = 'Failed' } + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*batch deployment failed*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 5 -Exactly + } + It 'stops after one submission fails and preserves the journal' { + Mock Submit-LocalBoxCapacityVM { throw 'Submission failed; inspect deployment' } + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*Submission failed*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 1 + $journal.State.Status | Should -Be 'Stopped' + $journal.State.VMs.Count | Should -Be 1 + } + It 'stops on readiness timeout without retrying VM creation' { + $settings.TimeoutMinutes = 0 + Mock Update-LocalBoxCapacityVM { $Entry.Status = 'WaitingForGuest' } + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*Timed out*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 1 + Should -Invoke Start-Sleep -Times 0 + $journal.State.VMs[0].Status | Should -Be 'WaitingForGuest' + } + It 'does not create the resource group when headroom is insufficient' { + $snapshot.HostFreeGB = 250 + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*backing-volume*' + Should -Invoke Invoke-LocalBoxAz -Times 0 + Should -Invoke Submit-LocalBoxCapacityVM -Times 0 + } + It 'rechecks headroom before each subsequent VM' { + $samples = @{ Count = 0 } + Mock Get-LocalBoxCapacitySnapshot { + $samples.Count++ + if ($samples.Count -gt 2) { $snapshot.HostFreeGB = 350 } + $snapshot + } + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*backing-volume*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 1 + $journal.State.Status | Should -Be 'Stopped' + } + It 'extends an existing ready run without recreating its VMs' { + $settings.VmCount = 1 + Invoke-LocalBoxCapacityRun $settings + $saved = $journal.State | ConvertTo-Json -Depth 20 + Mock Test-Path { $true } + Mock Get-Content { $saved } -ParameterFilter { $LiteralPath -eq 'state.json' } + Mock Assert-LocalBoxCapacityOwnership {} + $settings.VmCount = 2 + Invoke-LocalBoxCapacityRun $settings + Should -Invoke Submit-LocalBoxCapacityVM -Times 2 -Exactly + $journal.State.VMs.Count | Should -Be 2 + $journal.State.VMs[0].Name | Should -Not -Be $journal.State.VMs[1].Name + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { $Arguments[0] -eq 'group' -and $Arguments[1] -eq 'create' } + } + It 'refuses to extend a run whose existing VM is unhealthy' { + $settings.VmCount = 1 + Invoke-LocalBoxCapacityRun $settings + $saved = $journal.State | ConvertTo-Json -Depth 20 + Mock Test-Path { $true } + Mock Get-Content { $saved } -ParameterFilter { $LiteralPath -eq 'state.json' } + Mock Assert-LocalBoxCapacityOwnership {} + Mock Update-LocalBoxCapacityVM { $Entry.Status = 'Failed' } + $settings.VmCount = 2 + { Invoke-LocalBoxCapacityRun $settings } | Should -Throw '*Existing test VM*' + Should -Invoke Submit-LocalBoxCapacityVM -Times 1 + } +} + +Describe 'LocalBox capacity pending submission recovery' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $scope = '/subscriptions/test/resourceGroups/rg-test' + $pending = @{ Name = 'lc-test-002'; Status = 'Submitting'; SubmittedAt = '2026-09-21T00:00:00Z' } + $state = @{ SubscriptionId = 'test'; ResourceGroupName = 'rg-test'; VMs = @(@{ Name = 'lc-test-001'; Status = 'Ready' }, $pending) } + Mock Invoke-LocalBoxAz { @() } + } + It 'retries only a verified absent trailing submission and retains its history' { + Repair-LocalBoxCapacityPendingSubmission $state + $state.VMs.Count | Should -Be 1 + $state.VMs[0].Name | Should -Be 'lc-test-001' + $state.UnsubmittedAttempts[0].Name | Should -Be 'lc-test-002' + Should -Invoke Invoke-LocalBoxAz -Times 2 -Exactly + } + It 'does not resubmit when a deployment record exists' { + Mock Invoke-LocalBoxAz { @(@{ name = 'lc-test-002' }) } -ParameterFilter { $Arguments[0] -eq 'deployment' } + Repair-LocalBoxCapacityPendingSubmission $state + $state.VMs.Count | Should -Be 2 + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[0] -eq 'resource' } + } + It 'refuses to retry when any resource already exists' -TestCases @( + @{ Suffix = '/providers/Microsoft.HybridCompute/machines/lc-test-002' } + @{ Suffix = '/providers/Microsoft.AzureStackHCI/networkInterfaces/lc-test-002-nic' } + @{ Suffix = '/providers/Microsoft.HybridCompute/machines/lc-test-002/extensions/MDE.Windows' } + ) { + param($Suffix) + Mock Invoke-LocalBoxAz { @(@{ id = "$scope$Suffix" }) } -ParameterFilter { $Arguments[0] -eq 'resource' } + { Repair-LocalBoxCapacityPendingSubmission $state } | Should -Throw '*resources but no deployment*' + $state.VMs.Count | Should -Be 2 + } + It 'does not treat a failed read as an absent deployment' { + Mock Invoke-LocalBoxAz { throw 'AuthorizationFailed' } + { Repair-LocalBoxCapacityPendingSubmission $state } | Should -Throw '*AuthorizationFailed*' + $state.VMs.Count | Should -Be 2 + } + It 'rejects ambiguous journal entries' { + $state.VMs[0].Status = 'Submitting' + { Repair-LocalBoxCapacityPendingSubmission $state } | Should -Throw '*Ambiguous*' + Should -Invoke Invoke-LocalBoxAz -Times 0 + } +} + +Describe 'LocalBox capacity remote execution' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $cluster = '/subscriptions/a0000000-0000-0000-0000-000000000001/resourceGroups/localbox/providers/Microsoft.AzureStackHCI/clusters/cluster' + $context = @{ LocalBox = @{ ClusterId = $cluster; SubscriptionId = 'a0000000-0000-0000-0000-000000000001'; ResourceGroupName = 'localbox' }; Snapshot = @{ HostFreeGB = 1200 } } + Mock Invoke-LocalBoxAz { @{ value = @(@{ message = "[stdout]`nLB_CAPACITY_BEGIN`n$($context | ConvertTo-Json -Depth 8 -Compress)`nLB_CAPACITY_END`n[stderr]" }) } } + } + It 'targets the exact subscription and Client and returns only parsed telemetry' { + $result = Get-LocalBoxRemoteCapacityContext $cluster 'C:\LocalBox\sovereign-localbox.json' + $result.Snapshot.HostFreeGB | Should -Be 1200 + $result.LocalBox.RemoteClusterId | Should -Be $cluster + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { + $Arguments -contains 'run-command' -and $Arguments -contains 'a0000000-0000-0000-0000-000000000001' -and + $Arguments -contains 'localbox' -and $Arguments -contains 'LocalBox-Client' + } + } + It 'fails closed on truncated telemetry' { + Mock Invoke-LocalBoxAz { @{ value = @(@{ message = 'LB_CAPACITY_BEGIN {}' }) } } + { Get-LocalBoxRemoteCapacityContext $cluster 'manifest.json' } | Should -Throw '*no complete result*' + } + It 'rejects a result from another cluster' { + $context.LocalBox.ClusterId = '/some/other/cluster' + { Get-LocalBoxRemoteCapacityContext $cluster 'manifest.json' } | Should -Throw '*does not match*' + } + It 'rejects an invalid target before invoking Run Command' { + { Get-LocalBoxRemoteCapacityContext '/subscriptions/other/resourceGroups/shared' 'manifest.json' } | Should -Throw '*exact Azure Local cluster*' + Should -Invoke Invoke-LocalBoxAz -Times 0 + } +} + +Describe 'LocalBox capacity secure submission' { + BeforeAll { . "$PSScriptRoot/../test-localbox-capacity.ps1" } + BeforeEach { + $state = @{ + SubscriptionId = 'test'; ResourceGroupName = 'rg-test'; Location = 'westeurope'; RunId = 'test' + CustomLocationId = 'custom'; ImageId = 'image'; NetworkId = 'network'; StorageId = 'storage' + ProcessorCount = 2; MemoryMB = 4096 + } + $entry = @{ Name = 'lc-test-001' } + $mockPassword = [Security.SecureString]::new() + foreach ($character in 'Mock!Password9-not-real'.GetEnumerator()) { $mockPassword.AppendChar($character) } + $mockPassword.MakeReadOnly() + $credential = [pscredential]::new('localadmin', $mockPassword) + $paths = [Collections.Generic.List[string]]::new() + $machineId = '/subscriptions/test/resourceGroups/rg-test/providers/Microsoft.HybridCompute/machines/lc-test-001' + $preview = @{ status = 'Succeeded'; changes = @( + @{ resourceId = $machineId; changeType = 'Create' } + @{ resourceId = "$machineId/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default"; changeType = 'Create' } + @{ resourceId = '/subscriptions/test/resourceGroups/rg-test/providers/Microsoft.AzureStackHCI/networkInterfaces/lc-test-001-nic'; changeType = 'Create' } + ) } + Mock Invoke-LocalBoxAz { + $Arguments | Should -Not -Contain 'Mock!Password9-not-real' + $parameterFile = $Arguments[$Arguments.IndexOf('--parameters') + 1].Substring(1) + $paths.Add($parameterFile) + (Get-Content -LiteralPath $parameterFile -Raw | ConvertFrom-Json).parameters.adminPassword.value | Should -Be 'Mock!Password9-not-real' + if (-not $IsWindows) { + $directory = [IO.Path]::GetDirectoryName($parameterFile) + [int][IO.File]::GetUnixFileMode($directory) | Should -Be 448 + } + if ($Arguments[2] -eq 'what-if') { $preview } + } + } + It 'previews then submits and removes credential files' { + Submit-LocalBoxCapacityVM $state $entry $credential + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { $Arguments[2] -eq 'what-if' } + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { $Arguments[2] -eq 'create' -and $NoOutput -and $Arguments -contains '--no-wait' } + foreach ($path in $paths) { Test-Path -LiteralPath $path | Should -BeFalse } + } + It 'rejects changes outside the three new VM resources' -TestCases @( + @{ Field = 'changeType'; Value = 'Modify' } + @{ Field = 'resourceId'; Value = '/subscriptions/test/resourceGroups/shared/providers/Microsoft.HybridCompute/machines/student' } + ) { + param($Field, $Value) + $preview.changes[0][$Field] = $Value + { Submit-LocalBoxCapacityVM $state $entry $credential } | Should -Throw '*submission failed*' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + foreach ($path in $paths) { Test-Path -LiteralPath $path | Should -BeFalse } + } + It 'permits harmless existing-resource entries during incremental deployment' -TestCases @( + @{ ChangeType = 'Ignore' } + @{ ChangeType = 'NoChange' } + ) { + param($ChangeType) + $preview.changes += @{ resourceId = '/subscriptions/test/resourceGroups/rg-test/providers/Microsoft.HybridCompute/machines/earlier-vm'; changeType = $ChangeType } + Submit-LocalBoxCapacityVM $state $entry $credential + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'still rejects modifications or deletions of an existing VM' -TestCases @( + @{ ChangeType = 'Modify' } + @{ ChangeType = 'Delete' } + @{ ChangeType = 'Unsupported' } + ) { + param($ChangeType) + $preview.changes += @{ resourceId = '/subscriptions/test/resourceGroups/rg-test/providers/Microsoft.HybridCompute/machines/earlier-vm'; changeType = $ChangeType } + { Submit-LocalBoxCapacityVM $state $entry $credential } | Should -Throw '*submission failed*' + $entry.SubmissionError | Should -Match 'ARM what-if' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'rejects duplicate create entries instead of accepting a missing resource' { + $preview.changes[2] = $preview.changes[0] + { Submit-LocalBoxCapacityVM $state $entry $credential } | Should -Throw '*submission failed*' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'redacts CLI errors and cleans credential files on failure' { + Mock Invoke-LocalBoxAz { + $paths.Add($Arguments[$Arguments.IndexOf('--parameters') + 1].Substring(1)) + throw 'Mock!Password9-not-real' + } + try { Submit-LocalBoxCapacityVM $state $entry $credential; throw 'Expected failure' } + catch { + $_.Exception.Message | Should -Match 'CLI details are withheld' + $_.Exception.Message | Should -Not -Match 'Mock!Password9-not-real' + } + foreach ($path in $paths) { Test-Path -LiteralPath $path | Should -BeFalse } + } +} + +Describe 'Health runner defaults and feedback' { + BeforeAll { + function kubectl {} + } + It 'defaults to the home kubeconfig and downloads without side effects when dot-sourced' { + Mock Invoke-WebRequest { throw 'Unexpected download' } + . "$PSScriptRoot/../test-sovereign-cloud.ps1" + $LocalBoxKubeconfig | Should -Be (Join-Path $HOME '.kube/config') + [bool]$DownloadTests | Should -BeTrue + Should -Invoke Invoke-WebRequest -Times 0 + } + It 'accepts an explicit kubeconfig and disables downloads for a local checkout' { + . "$PSScriptRoot/../test-sovereign-cloud.ps1" -LocalBoxKubeconfig 'custom.kubeconfig' -DownloadTests:$false + $LocalBoxKubeconfig | Should -Be 'custom.kubeconfig' + [bool]$DownloadTests | Should -BeFalse + } + It 'preserves inventory precedence and only applies the default to LocalBox scopes' -TestCases @( + @{ TestScope = 'LocalBox'; Existing = 'inventory.kubeconfig'; Explicit = $false; Expected = 'inventory.kubeconfig' } + @{ TestScope = 'All'; Existing = 'inventory.kubeconfig'; Explicit = $true; Expected = 'chosen.kubeconfig' } + @{ TestScope = 'LocalBox'; Existing = $null; Explicit = $false; Expected = 'chosen.kubeconfig' } + @{ TestScope = 'ParticipantLabs'; Existing = 'inventory.kubeconfig'; Explicit = $true; Expected = 'inventory.kubeconfig' } + @{ TestScope = 'ParticipantLabs'; Existing = $null; Explicit = $false; Expected = $null } + ) { + param($TestScope, $Existing, $Explicit, $Expected) + $ast = [Management.Automation.Language.Parser]::ParseFile("$PSScriptRoot/../test-sovereign-cloud.ps1", [ref]$null, [ref]$null) + $assignment = $ast.Find({ param($node) + $node -is [Management.Automation.Language.IfStatementAst] -and + $node.Clauses[0].Item1.Extent.Text -eq '$Scope -in @(''LocalBox'', ''All'') -and $inventory.LocalBox' + }, $true) + $assignment | Should -Not -BeNullOrEmpty + $Scope = $TestScope + $inventory = if ($TestScope -eq 'ParticipantLabs' -and -not $Existing) { @{ Labs = @() } } else { @{ LocalBox = @{ Kubeconfig = $Existing } } } + $parameters = if ($Explicit) { @{ LocalBoxKubeconfig = 'chosen.kubeconfig' } } else { @{} } + $resolve = [scriptblock]::Create('param([string]$LocalBoxKubeconfig = ''chosen.kubeconfig'')' + "`n" + $assignment.Extent.Text) + & $resolve @parameters + $inventory.LocalBox.Kubeconfig | Should -Be $Expected + if ($TestScope -eq 'ParticipantLabs' -and -not $Existing) { $inventory.ContainsKey('LocalBox') | Should -BeFalse } + } + It 'reports the waiting condition and remaining time before retrying' { + Mock Write-Host {} + Mock Start-Sleep {} + $attempts = @{ Count = 0 } + $output = @(Wait-SovereignCheck -TimeoutSeconds 60 -Check { + $attempts.Count++ + if ($attempts.Count -eq 1) { throw 'System pod kube-system/test-pod is not healthy.' } + }) + $attempts.Count | Should -Be 2 + $output.Count | Should -Be 0 + Should -Invoke Write-Host -Times 1 -ParameterFilter { + $Object -match 'attempt 1 not ready.*remaining; retry in 15s.*test-pod' + } + Should -Invoke Start-Sleep -Times 1 -Exactly -ParameterFilter { $Seconds -eq 15 } + } + It 'does not retry terminal authorization errors' { + Mock Write-Host {} + Mock Start-Sleep {} + { Wait-SovereignCheck -Check { throw 'Forbidden: cannot list pods' } } | Should -Throw '*Forbidden*' + Should -Invoke Start-Sleep -Times 0 + Should -Invoke Write-Host -Times 0 + } + It 'throws the final failed condition at the timeout instead of claiming readiness' { + Mock Write-Host {} + Mock Start-Sleep {} + { Wait-SovereignCheck -TimeoutSeconds 0 -Check { throw 'Deployment coredns is unavailable.' } } | Should -Throw '*coredns*' + Should -Invoke Start-Sleep -Times 0 + } + It 'shows the Kubernetes query without mixing progress into JSON results' { + Mock Test-Path { $true } + Mock Write-Host {} + Mock kubectl { + $global:LASTEXITCODE = 0 + '{"items":[{"metadata":{"name":"node-1"}}]}' + } + $result = @(Invoke-SovereignKubectl 'test.kubeconfig' @('get', 'nodes')) + $result.Count | Should -Be 1 + $result[0].items[0].metadata.name | Should -Be 'node-1' + Should -Invoke Write-Host -Times 1 -ParameterFilter { $Object -match 'Querying Kubernetes: kubectl get nodes.*20s' } + Should -Invoke kubectl -Times 1 -ParameterFilter { $args -contains '--request-timeout=20s' -and $args -contains 'test.kubeconfig' } + } + It 'does not claim authentication when the default kubeconfig is absent' { + Mock Test-Path { $false } + Mock kubectl {} + { Invoke-SovereignKubectl (Join-Path $HOME '.kube/config') @('get', 'nodes') } | Should -Throw '*independently authenticated*' + Should -Invoke kubectl -Times 0 + } +} + Describe 'Console lab group credentials' { BeforeAll { function Get-MhhDefaultLabGroup { [CmdletBinding()] param() } @@ -272,6 +855,123 @@ Describe 'AKS Local resource API selection' { } } +Describe 'AKS Local Arc proxy RBAC' { + BeforeEach { + $clusterId = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.Kubernetes/connectedClusters/localbox-aks' + $groupId = 'a0000000-0000-0000-0000-000000000001' + $roleId = '/subscriptions/test/providers/Microsoft.Authorization/roleDefinitions/00493d72-78f6-4148-b6c5-d3ce8e4799dd' + $assignment = @{ scope = $clusterId; principalId = $groupId; principalType = 'Group'; roleDefinitionId = $roleId } + Mock Invoke-LocalBoxAz { @() } + Mock Invoke-LocalBoxAz { $assignment } -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'creates only the group proxy role at connected-cluster scope without Graph lookups' { + Sync-LocalBoxAksProxyRole $clusterId $groupId + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { + $Arguments[2] -eq 'list' -and $Arguments -contains '--include-inherited' -and + $Arguments[$Arguments.IndexOf('--fill-principal-name') + 1] -eq 'false' -and + $Arguments[$Arguments.IndexOf('--fill-role-definition-name') + 1] -eq 'false' + } + Should -Invoke Invoke-LocalBoxAz -Times 1 -Exactly -ParameterFilter { + $Arguments[2] -eq 'create' -and $Arguments -contains '--assignee-object-id' -and $Arguments -notcontains '--assignee' -and + $Arguments[$Arguments.IndexOf('--assignee-object-id') + 1] -eq $groupId -and + $Arguments[$Arguments.IndexOf('--assignee-principal-type') + 1] -eq 'Group' -and + $Arguments[$Arguments.IndexOf('--scope') + 1] -eq $clusterId -and + $Arguments[$Arguments.IndexOf('--role') + 1] -eq $roleId -and + $Arguments[$Arguments.IndexOf('--subscription') + 1] -eq 'test' -and + $Arguments[$Arguments.IndexOf('--name') + 1] -match '^[0-9a-f-]{36}$' + } + } + It 'reuses direct and inherited grants without writes' -TestCases @( + @{ AssignmentScope = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.Kubernetes/connectedClusters/localbox-aks' } + @{ AssignmentScope = '/subscriptions/test/resourceGroups/localbox' } + @{ AssignmentScope = '/subscriptions/test' } + ) { + param($AssignmentScope) + $assignment.scope = $AssignmentScope.ToUpperInvariant() + Mock Invoke-LocalBoxAz { @($assignment) } -ParameterFilter { $Arguments[2] -eq 'list' } + Sync-LocalBoxAksProxyRole $clusterId $groupId + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'does not accept a grant for another group, role or resource' -TestCases @( + @{ Property = 'principalId'; Value = 'b0000000-0000-0000-0000-000000000002' } + @{ Property = 'roleDefinitionId'; Value = '/subscriptions/test/providers/Microsoft.Authorization/roleDefinitions/00000000-0000-0000-0000-000000000001' } + @{ Property = 'scope'; Value = '/subscriptions/test/resourceGroups/other' } + ) { + param($Property, $Value) + $other = $assignment.Clone() + $other[$Property] = $Value + Mock Invoke-LocalBoxAz { @($other) } -ParameterFilter { $Arguments[2] -eq 'list' } + Sync-LocalBoxAksProxyRole $clusterId $groupId + Should -Invoke Invoke-LocalBoxAz -Times 1 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'uses a stable assignment name on retries regardless of ID casing' { + $names = [Collections.Generic.List[string]]::new() + Mock Invoke-LocalBoxAz { + $names.Add($Arguments[$Arguments.IndexOf('--name') + 1]) + $assignment + } -ParameterFilter { $Arguments[2] -eq 'create' } + Sync-LocalBoxAksProxyRole $clusterId $groupId + Sync-LocalBoxAksProxyRole $clusterId.ToUpperInvariant() $groupId.ToUpperInvariant() + $names.Count | Should -Be 2 + $names[0] | Should -Be $names[1] + } + It 'does not query or create RBAC in WhatIf even before the cluster exists' { + Sync-LocalBoxAksProxyRole $clusterId $groupId -WhatIf + Should -Invoke Invoke-LocalBoxAz -Times 0 + } + It 'rejects resource-group scope and malformed groups before CLI calls' { + { Sync-LocalBoxAksProxyRole '/subscriptions/test/resourceGroups/localbox' $groupId } | Should -Throw '*connected-cluster*' + { Sync-LocalBoxAksProxyRole $clusterId 'not-a-group-id' } | Should -Throw '*object ID*' + Should -Invoke Invoke-LocalBoxAz -Times 0 + } + It 'does not replace or bypass an existing conditional grant' -TestCases @( + @{ AssignmentScope = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.Kubernetes/connectedClusters/localbox-aks' } + @{ AssignmentScope = '/subscriptions/test/resourceGroups/localbox' } + ) { + param($AssignmentScope) + $assignment.scope = $AssignmentScope + $assignment.condition = 'restricted' + Mock Invoke-LocalBoxAz { @($assignment) } -ParameterFilter { $Arguments[2] -eq 'list' } + { Sync-LocalBoxAksProxyRole $clusterId $groupId } | Should -Throw '*will not replace*' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'stops on read authorization failures without attempting creation' { + Mock Invoke-LocalBoxAz { throw 'AuthorizationFailed' } -ParameterFilter { $Arguments[2] -eq 'list' } + { Sync-LocalBoxAksProxyRole $clusterId $groupId } | Should -Throw '*roleAssignments/read*AuthorizationFailed*' + Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments[2] -eq 'create' } + } + It 'reports missing write permission without elevating the managed identity' { + Mock Invoke-LocalBoxAz { throw 'AuthorizationFailed' } -ParameterFilter { $Arguments[2] -eq 'create' } + { Sync-LocalBoxAksProxyRole $clusterId $groupId } | Should -Throw '*roleAssignments/write*does not elevate*AuthorizationFailed*' + Should -Invoke Invoke-LocalBoxAz -Times 2 -Exactly + } + It 'rejects an empty or mismatched creation response' -TestCases @( + @{ Response = $null } + @{ Response = @{ scope = '/subscriptions/test/resourceGroups/localbox' } } + ) { + param($Response) + Mock Invoke-LocalBoxAz { $Response } -ParameterFilter { $Arguments[2] -eq 'create' } + { Sync-LocalBoxAksProxyRole $clusterId $groupId } | Should -Throw + } + It 'wires reconciliation for existing and new AKS clusters but skips it with SkipAks' { + $ast = [Management.Automation.Language.Parser]::ParseFile("$PSScriptRoot/../prepare-localbox.ps1", [ref]$null, [ref]$null) + $guard = $ast.Find({ param($node) + $node -is [Management.Automation.Language.IfStatementAst] -and + $node.Clauses[0].Item1.Extent.Text -eq '-not $Settings.SkipAks' + }, $true) + $guard | Should -Not -BeNullOrEmpty + $reconcile = [scriptblock]::Create($guard.Extent.Text) + $aksId = $clusterId + $Settings = @{ SkipAks = $true } + Mock Sync-LocalBoxAksProxyRole {} + & $reconcile + Should -Invoke Sync-LocalBoxAksProxyRole -Times 0 + $Settings.SkipAks = $false + & $reconcile + Should -Invoke Sync-LocalBoxAksProxyRole -Times 1 -Exactly -ParameterFilter { $ClusterId -eq $aksId -and $GroupObjectId -eq $groupId } + } +} + Describe 'Resource reconciliation' { BeforeEach { $resourceId = '/subscriptions/test/resourceGroups/localbox/providers/Microsoft.AzureStackHCI/logicalNetworks/vm-net' @@ -373,7 +1073,6 @@ Describe 'LocalBox image storage conflicts' { Should -Invoke Invoke-LocalBoxAz -Times 0 -ParameterFilter { $Arguments -contains 'create' } } } -< Describe 'CLI output streams' { BeforeEach { $script:cliTestExecutable = (Get-Process -Id $PID).Path @@ -625,6 +1324,58 @@ Describe 'Generated Azure Local storage names' { } } +Describe 'Kubernetes DaemonSet readiness' { + BeforeEach { + $controllers = @{ items = @( + @{ kind = 'DaemonSet'; metadata = @{ name = 'calico-node' }; status = @{ desiredNumberScheduled = 4; numberReady = 4 } } + @{ kind = 'DaemonSet'; metadata = @{ name = 'calico-node-windows' }; spec = @{ template = @{ spec = @{ nodeSelector = @{ 'kubernetes.io/os' = 'windows' } } } }; status = @{ desiredNumberScheduled = 0; numberReady = 0 } } + @{ kind = 'Deployment'; metadata = @{ name = 'coredns' }; spec = @{ replicas = 2 }; status = @{ availableReplicas = 2 } } + ) } + Mock Invoke-SovereignKubectl { + switch ($Arguments[1]) { + 'nodes' { return @{ items = @(@{ metadata = @{ name = 'linux-node' }; status = @{ conditions = @(@{ type = 'Ready'; status = 'True' }) } }) } } + 'pods' { return @{ items = @(@{ metadata = @{ name = 'calico-node-pod'; namespace = 'kube-system' }; status = @{ phase = 'Running'; conditions = @(@{ type = 'Ready'; status = 'True' }); containerStatuses = @(@{ name = 'calico-node'; ready = $true; state = @{ running = @{} } }) } }) } } + 'deployments,daemonsets' { return $controllers } + default { throw 'Unexpected query' } + } + } + } + It 'accepts healthy Linux controllers alongside a Windows DaemonSet with no eligible nodes' { + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Not -Throw + } + It 'uses scheduling counts rather than a special case for Windows controller names' { + $controllers.items[1].metadata.name = 'optional-agent' + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Not -Throw + } + It 'fails when a Linux DaemonSet is missing a Ready pod' { + $controllers.items[0].status.numberReady = 3 + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Throw '*calico-node is unavailable (3 Ready / 4 desired)*' + } + It 'fails when a Windows DaemonSet has eligible nodes but no Ready pods' { + $controllers.items[1].status.desiredNumberScheduled = 1 + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Throw '*calico-node-windows is unavailable (0 Ready / 1 desired)*' + } + It 'does not treat missing or invalid status as zero desired pods' -TestCases @( + @{ Status = @{} } + @{ Status = @{ desiredNumberScheduled = 0 } } + @{ Status = @{ numberReady = 0 } } + @{ Status = @{ desiredNumberScheduled = -1; numberReady = 0 } } + @{ Status = @{ desiredNumberScheduled = 0; numberReady = -1 } } + ) { + param($Status) + $controllers.items[1].status = $Status + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Throw '*missing or invalid scheduling status*' + } + It 'still fails an unavailable deployment alongside a zero-target DaemonSet' { + $controllers.items[2].status.availableReplicas = 1 + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Throw '*Deployment coredns is unavailable*' + } + It 'still rejects an empty controller collection' { + $controllers.items = @() + { Test-SovereignKubernetes 'test.kubeconfig' 1 } | Should -Throw '*No kube-system controllers found*' + } +} + Describe 'Health checks do not report false readiness' { It 'accepts one Ready control-plane node and three Ready workers' { $control = @{ metadata = @{ labels = @{ 'node-role.kubernetes.io/control-plane' = '' } }; status = @{ conditions = @(@{ type = 'Ready'; status = 'True' }) } } diff --git a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md index 4196c6736..e9df64b3b 100644 --- a/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md +++ b/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/tests/readme.md @@ -8,11 +8,11 @@ Use PowerShell 7, Pester 5.7.1 or later within major version 5, Azure CLI, and ` The runner uses the caller's existing Azure CLI login and never changes it. LocalBox control-plane checks can use its managed identity. Participant-lab checks need an operator authorized for those explicitly listed resource groups; the LocalBox identity is not automatically authorized there. Full LocalBox checks run on LocalBox-Client with a nested-node Windows credential. Do not serialize passwords or tokens in JSON. -Full Kubernetes checks take dedicated kubeconfigs that already authenticate to the intended clusters. For AKS Local, use an Entra group member in a separate CLI profile/session and the [Jumpstart proxy workflow](https://jumpstart.azure.com/azure_jumpstart_localbox/AKS). Keep the proxy/tunnel running throughout testing. For K3s, use its existing private/Bastion access path. Tests never switch the current kubeconfig or retrieve admin credentials automatically. +Full Kubernetes checks require kubeconfigs that already authenticate to the intended clusters. LocalBox defaults to `$HOME/.kube/config` (on Windows, `$HOME\.kube\config`), the usual path updated by `az connectedk8s proxy`. An explicit `-LocalBoxKubeconfig` takes precedence; otherwise a `LocalBox.Kubeconfig` value in the inventory is preserved before using the default. The selected file's current context must target `localbox-aks`; do not switch it during testing. For AKS Local, use an Entra group member in a separate CLI profile/session and the [Jumpstart proxy workflow](https://jumpstart.azure.com/azure_jumpstart_localbox/AKS). Keep the proxy/tunnel running throughout testing. For K3s, use its existing private/Bastion access path. Tests never switch the current kubeconfig or retrieve admin credentials automatically. ## Download without cloning -Download [test-sovereign-cloud.ps1](../test-sovereign-cloud.ps1) from the same trusted ref as preparation. The runner's `-DownloadTests` downloads the helper and two suites to a temporary directory; `-GitHubRef` must identify that **same ref**, preferably a commit SHA. Downloaded scripts execute under your privileges, so review and pin them. +Download [test-sovereign-cloud.ps1](../test-sovereign-cloud.ps1) from the same trusted ref as preparation. `-DownloadTests` defaults to `$true`: every run downloads the helpers and two suites to a temporary directory and prints the repository/ref being used. Internet access is required unless you pass `-DownloadTests:$false` with a complete local checkout. `-GitHubRef` must identify the **same reviewed ref** as the runner, preferably a commit SHA; it defaults to `main`. Downloaded scripts execute under your privileges, so review and pin them. Download failures stop the run rather than silently falling back to other code. For a fork, also specify `-GitHubRepository 'owner/MicroHack'`. For example, testing this branch uses `-GitHubRepository 'janegilring/MicroHack' -GitHubRef 'sov-cloud-localbox-post-automation'`. The default repository remains `microsoft/MicroHack`. @@ -21,10 +21,10 @@ $ref = 'main' $base = "https://raw.githubusercontent.com/microsoft/MicroHack/$ref/03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources" Invoke-WebRequest "$base/test-sovereign-cloud.ps1" -OutFile './test-sovereign-cloud.ps1' ./test-sovereign-cloud.ps1 -Scope LocalBox -Mode ControlPlane ` - -LocalBoxManifestPath 'C:\LocalBox\sovereign-localbox.json' -DownloadTests -GitHubRef $ref + -LocalBoxManifestPath 'C:\LocalBox\sovereign-localbox.json' -GitHubRef $ref ``` -For a checkout, run the same command without `-DownloadTests`. +For a checkout, pass `-DownloadTests:$false` explicitly to use its local helpers and suites. Omitting the switch now downloads tests. ## Full LocalBox check @@ -35,10 +35,11 @@ Keep a reviewed [prepare-localbox.ps1](../prepare-localbox.ps1) from the same re $nodeCredential = Resolve-LocalBoxNodeCredential -Configuration (Import-PowerShellDataFile -LiteralPath $env:LocalBoxConfigFile) ./test-sovereign-cloud.ps1 -Scope LocalBox -Mode Full ` -LocalBoxManifestPath 'C:\LocalBox\sovereign-localbox.json' ` - -LocalBoxKubeconfig 'C:\LocalBox\aks-local.kubeconfig' ` - -NodeCredential $nodeCredential -DownloadTests -GitHubRef $ref + -NodeCredential $nodeCredential -GitHubRef $ref ``` + This example uses the default `$HOME/.kube/config`. Use `-LocalBoxKubeconfig 'C:\LocalBox\aks-local.kubeconfig'` for a dedicated file instead. Changing the defaults does not authenticate kubectl or start the Arc proxy. + Checks cover the Client and bootstrap, cluster/Arc bridge/custom location/AKS extension, supporting Azure resources, image download/storage placement, exact network settings, expected AKS topology/admin group, nested-node/storage health and capacity, Kubernetes node readiness and system pods/controllers. Gateway/DNS/HTTPS probes run from a nested node; these do not prove connectivity from a participant VM that has not yet been created. ## Rerun from a local copy @@ -59,7 +60,7 @@ $nodeCredential = Resolve-LocalBoxNodeCredential -Configuration (Import-PowerShe .\test-sovereign-cloud.ps1 -Scope LocalBox -Mode Full ` -LocalBoxManifestPath $manifestPath ` -LocalBoxKubeconfig $kubeconfigPath ` - -NodeCredential $nodeCredential -OutputDirectory $outputDirectory + -NodeCredential $nodeCredential -OutputDirectory $outputDirectory -DownloadTests:$false Get-Content (Join-Path $outputDirectory 'health.json') ``` @@ -89,7 +90,7 @@ For manual deployments without retained deployment records, replace `DeploymentN ```powershell ./test-sovereign-cloud.ps1 -Scope ParticipantLabs -InventoryPath './inventory.json' ` - -Mode Full -AllowGuestRunCommand -DownloadTests -GitHubRef $ref + -Mode Full -AllowGuestRunCommand -GitHubRef $ref ``` `-AllowGuestRunCommand` permits Azure VM Run Command on the selected K3s VM solely to check K3s service state, DNS and HTTPS egress. This transport is an Azure management action and requires scoped Run Command permission even though the guest commands are read-only. It can create transient Run Command execution artifacts. Without consent, that mandatory Full check fails; it is not silently skipped. @@ -105,7 +106,19 @@ For both paths together, use `-Scope All`, supply the participant inventory and - Nonzero exit: a failure, or incomplete Full coverage. Missing credentials/permissions/connectivity is not a healthy result. - A successful `ControlPlane` run is explicitly partial and never sets `FullReadiness`. Full readiness covers the checks described above, not application functionality or uncreated student resources. -Waits are bounded by `-TimeoutMinutes` per check. Terminal provisioning failures and authorization errors stop promptly. Secure the reports as operational metadata; underlying tool errors in XML may include resource IDs and tenant information. Retest after resolving failures rather than mutating infrastructure from the health suite. +Readiness retries use `-TimeoutMinutes` per check (default 15 minutes). The runner prints each Kubernetes query, then reports the failed condition, attempt number and remaining time before each 15-second retry. Nodes can be Ready while a system pod, Deployment or DaemonSet is not; the same test verifies all of them. Authorization errors stop without retrying. + +DaemonSets are checked against their desired scheduling count. Windows-only DaemonSets such as `calico-node-windows` can legitimately report zero desired and zero Ready pods on a Linux-only cluster; that is not a failure. Missing scheduling status or fewer Ready pods than desired still fails. An older runner that reports a zero-target DaemonSet as unavailable needs the corrected runner and downloaded helpers from the same ref, not deletion of the DaemonSet or a longer timeout. + +Each kubectl API request uses `--request-timeout=20s`. The readiness deadline is checked between attempts, not as a hard process timeout; an external authentication plugin can take longer or require attention. If the last query stays on screen without retry messages, check the proxy terminal and authentication state. In a separate terminal, inspect the same kubeconfig without changing its context: + +```powershell +kubectl --kubeconfig "$HOME/.kube/config" --request-timeout=20s get nodes +kubectl --kubeconfig "$HOME/.kube/config" --request-timeout=20s get pods -A +kubectl --kubeconfig "$HOME/.kube/config" --request-timeout=20s get deployments,daemonsets -n kube-system +``` + +Use your explicit kubeconfig path instead when testing a dedicated file. Secure the reports as operational metadata; underlying tool errors in XML may include resource IDs and tenant information. Retest after resolving failures rather than mutating infrastructure from the health suite. ## Offline regression tests