diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/README.md b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/README.md index cc2f8ea07..957bd0156 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/README.md +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/README.md @@ -8,30 +8,44 @@ integration for the Talk-To-Your-Data MicroHack. | File | Purpose | | --- | --- | | `lab-defaults.json` | Platform sizing: `resourcegroup`, 42 labs/subscription. | -| `shared-deploy-lab.ps1` | Runs once per subscription → `rg-shared`. Shared VNet, SQL MI, Fabric F32 capacity + gateway, backup storage, shared webshop, demo databases, stored proc, product, Agent job. | -| `deploy-lab.ps1` | Runs once per attendee. Two databases in the shared MI, the attendee's login + Fabric workspace, a per-attendee CSV storage account. | -| `shared.bicep` / `main.bicep` | ARM templates for the shared stack and the per-attendee resources (reuse `infra/modules`). | +| `shared-deploy-lab.ps1` | Runs once per subscription → `rg-shared`. Shared VNet, SQL MI, Fabric F32 capacity + gateway, backup storage, shared CSV storage, shared webshop, demo databases, stored proc, product, Agent job. | +| `deploy-lab.ps1` | Runs once per attendee. Two databases in the shared MI, the attendee's login + Fabric workspace, and CSV upload into the attendee's shared storage container. | +| `shared.bicep` | ARM template for the shared stack (reuses `infra/modules`). There is no per-attendee ARM deployment. | | `sql/` | `StoredProcedure.sql`, `Jobs.sql`, `InsertProduct.sql` (discover per-attendee DBs by name pattern). | ## Split model - **Shared, once per subscription:** one SQL Managed Instance and one Fabric F32 - capacity. The webshop and the `usp_PurchaseSpaceRanger` proc / Agent job are shared - and fan out to every attendee database that exists. -- **Per attendee:** `TailspinToys_` + `TailspinToysFeedback_` in the - shared MI, one Fabric workspace on the shared capacity, one CSV storage account in - the attendee resource group. + capacity. Every lab user is a Fabric capacity administrator. The webshop and the + `usp_PurchaseSpaceRanger` proc / Agent job are shared and fan out to every attendee + database that exists. Employee CSV files live in one + shared `employeedata...` storage account with per-user containers such as `container0001` + for lab users or `containerx001` for local tests with regular tenant accounts. +- **Per attendee:** `TailspinToys_User####` + `TailspinToysFeedback_User####` in the + shared MI, one Fabric workspace on the shared capacity, and one shared-storage + container named `container####`. Local tests with regular tenant accounts use + `TailspinToys_Userx001` / `TailspinToysFeedback_Userx001` and `containerx001`. ## Prerequisites -- **Fabric APIs for service principals** must be enabled in the tenant admin portal - (the platform runs as a service principal). Without it, gateway/workspace creation - fails. -- SQL is accessed with an Entra token as the deploying principal, which - `shared-deploy-lab.ps1` sets as the SQL MI Entra admin (plus Directory Readers on the - MI identity). No shared SQL password is distributed. +- The platform runs as a service principal. A Fabric administrator must enable both + **Service principals can call Fabric public APIs** and **Service principals can create + workspaces, connections, and deployment pipelines** in the Fabric Admin portal, with + the platform service principal included in an allowed security group. The second + setting is separate and disabled by default for new tenants. + Instead of clicking through the portal, a Fabric/Global administrator can run + `enable-fabric-sp-tenant-settings.ps1 -SecurityGroupObjectId ` + (signed in as themselves, not the deployment service principal) to enable both + settings for the security group that contains the platform service principal. +- The automation accesses SQL with a stable administrator password scoped to `rg-shared`; + both hooks derive the same value without printing or distributing it. The shared hook + also configures the first lab user as the SQL MI Entra admin and grants Directory Readers + to the MI identity for attendee login creation. +- Fabric mirroring uses the shared SQL login `demouser` / `Demo@pass1234567`; both hooks + ensure the login exists and has `db_owner` access to restored lab databases. - The scripts read `databasebackup/*.bak` and `csvdata/*.csv` and reuse - `infra/modules`, all local to this folder, so **mount this `labautomation` folder** for local testing. + `shared.bicep` and `infra/modules`, all local to this folder, so **mount this + `labautomation` folder** for local testing. ## Local testing @@ -68,5 +82,28 @@ Then run the two hooks in the real order. Pass more than one object id to -AllowedEntraUserIds (Get-AzADUser -SignedIn).Id ``` +For local testing with regular tenant accounts instead of framework-created +`labuser-####` accounts, set `TTYD_LOCAL_TEST_USER_INDEX` before each attendee +run. This keeps the hook parameters framework-compatible while producing stable +local names such as `TailspinToys_Userx001` and `containerx001`: + +```powershell +$env:TTYD_LOCAL_TEST_USER_INDEX = '1' +./deploy-lab.ps1 ` + -DeploymentType resourcegroup ` + -SubscriptionId (Get-AzContext).Subscription.Id ` + -ResourceGroupName "rg-lab-local-test-x001" ` + -PreferredLocation "swedencentral" ` + -AllowedEntraUserIds '' + +$env:TTYD_LOCAL_TEST_USER_INDEX = '2' +./deploy-lab.ps1 ` + -DeploymentType resourcegroup ` + -SubscriptionId (Get-AzContext).Subscription.Id ` + -ResourceGroupName "rg-lab-local-test-x002" ` + -PreferredLocation "swedencentral" ` + -AllowedEntraUserIds '' +``` + > Provisioning a SQL Managed Instance takes hours; `shared-deploy-lab.ps1` submits it > asynchronously and refreshes credentials while it polls. diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/deploy-lab.ps1 b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/deploy-lab.ps1 index 5b47ec5bd..cc7f7ee27 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/deploy-lab.ps1 +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/deploy-lab.ps1 @@ -31,14 +31,19 @@ param( Set-StrictMode -Version Latest $ErrorActionPreference = 'Stop' +# Invoke-Sqlcmd emits progress records that the runner's child-job receiver reports as errors. +$ProgressPreference = 'SilentlyContinue' +$script:LabCurrentStep = 'initialization' $SharedResourceGroup = 'rg-shared' $SqlAdminLogin = 'sqlmiadmin' +$DemoSqlLogin = 'demouser' +$DemoSqlPassword = 'Demo@pass1234567' $FabricApi = 'https://api.fabric.microsoft.com/v1' $TailspinToysBak = 'tailspintoys_before_launch.bak' $TailspinToysFeedbackBak = 'tailspintoysfeedback_before_launch.bak' -# Shared MI admin password: derive at subscription scope so it matches the shared hook's value. -$sqlPassword = New-MhhStablePassword -Purpose 'sql-admin' -Length 24 -ResourceGroupName '' +# Shared MI admin password: derive with the shared resource group scope in both hooks. +$sqlPassword = New-MhhStablePassword -Purpose 'sql-admin' -Length 24 -ResourceGroupName $SharedResourceGroup # ───────────────────────────────────────────── # Helpers @@ -53,6 +58,31 @@ function Update-MhhTokenQuiet { Update-MhhToken | Out-String | Write-Verbose } +function Write-LabTrace { + param( + [Parameter(Mandatory = $true)][string]$Message + ) + Write-Host "[lab][trace] $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') $Message" +} + +function Start-LabStep { + param( + [Parameter(Mandatory = $true)][string]$Name + ) + $script:LabCurrentStep = $Name + Write-LabTrace "STEP: $Name" +} + +function Get-LabUserNumber { + param( + [string]$ShortName, + [string]$UserPrincipalName + ) + if ($ShortName -match '(?i)(?:labuser|user)[-_]?0*(\d+)$') { return [int]$Matches[1] } + if ($UserPrincipalName -match '(?i)(?:labuser|user)[-_]?0*(\d+)') { return [int]$Matches[1] } + return $null +} + function Invoke-MiSql { param( [Parameter(Mandatory = $true)][string]$Server, @@ -60,12 +90,93 @@ function Invoke-MiSql { [string]$Query, [int]$QueryTimeout = 0 ) - # SQL authentication with the MI admin login: the platform cannot set an Entra admin on the MI. + # The per-user hook is not the configured Entra admin, so use the shared MI's SQL admin credential. $cred = [pscredential]::new($SqlAdminLogin, (ConvertTo-SecureString $sqlPassword -AsPlainText -Force)) + Write-LabTrace "SQL $Database on $Server using inline query. Timeout=$QueryTimeout." Invoke-Sqlcmd -ServerInstance $Server -Database $Database -Credential $cred ` -Query $Query -ConnectionTimeout 30 -QueryTimeout $QueryTimeout -ErrorAction Stop } +function New-LabRoleAssignmentIfMissing { + param( + [Parameter(Mandatory = $true)][string]$ObjectId, + [Parameter(Mandatory = $true)][string]$PrincipalType, + [Parameter(Mandatory = $true)][string]$Role, + [Parameter(Mandatory = $true)][string]$Scope + ) + $existing = az role assignment list --assignee $ObjectId --role $Role --scope $Scope --query '[0].id' -o tsv 2>$null + if (-not [string]::IsNullOrWhiteSpace($existing)) { return $false } + + az role assignment create --assignee-object-id $ObjectId --assignee-principal-type $PrincipalType ` + --role $Role --scope $Scope 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Failed to grant '$Role' on '$Scope' to '$ObjectId'." } + return $true +} + +function Ensure-DemoSqlLogin { + param( + [Parameter(Mandatory = $true)][string]$Server + ) + $login = $DemoSqlLogin.Replace(']', ']]') + $password = $DemoSqlPassword.Replace("'", "''") + Invoke-MiSql -Server $Server -Database 'master' -QueryTimeout 60 -Query @" +IF NOT EXISTS (SELECT 1 FROM sys.sql_logins WHERE name = N'$DemoSqlLogin') + CREATE LOGIN [$login] WITH PASSWORD = N'$password', CHECK_POLICY = OFF; +ELSE + ALTER LOGIN [$login] WITH PASSWORD = N'$password', CHECK_POLICY = OFF; +"@ | Out-Null +} + +function Grant-DemoSqlLoginDatabaseAccess { + param( + [Parameter(Mandatory = $true)][string]$Server, + [Parameter(Mandatory = $true)][string]$Database + ) + $login = $DemoSqlLogin.Replace(']', ']]') + Invoke-MiSql -Server $Server -Database $Database -QueryTimeout 60 -Query @" +IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = N'$DemoSqlLogin') + CREATE USER [$login] FROM LOGIN [$login]; +IF NOT EXISTS ( + SELECT 1 + FROM sys.database_role_members drm + INNER JOIN sys.database_principals roles ON roles.principal_id = drm.role_principal_id + INNER JOIN sys.database_principals members ON members.principal_id = drm.member_principal_id + WHERE roles.name = N'db_owner' AND members.name = N'$DemoSqlLogin' +) + ALTER ROLE [db_owner] ADD MEMBER [$login]; +"@ | Out-Null +} + +function Set-LabRestoreCredential { + # The credential is an instance-level object on the SHARED SQL MI, so every attendee's + # deployment targets the same name. DROP fails with "cannot drop ... because it is used + # by an active restore" while another attendee is restoring, so refresh via ALTER instead. + param( + [Parameter(Mandatory = $true)][string]$Server, + [Parameter(Mandatory = $true)][string]$CredentialName, + [Parameter(Mandatory = $true)][string]$SasToken + ) + $name = $CredentialName.Replace(']', ']]') + $secret = $SasToken.Replace("'", "''") + for ($attempt = 1; $attempt -le 5; $attempt++) { + try { + Invoke-MiSql -Server $Server -Database 'master' -QueryTimeout 60 -Query @" +IF EXISTS (SELECT 1 FROM sys.credentials WHERE name = N'$CredentialName') + ALTER CREDENTIAL [$name] WITH IDENTITY = 'Shared Access Signature', SECRET = '$secret'; +ELSE + CREATE CREDENTIAL [$name] WITH IDENTITY = 'Shared Access Signature', SECRET = '$secret'; +"@ | Out-Null + return + } + catch { + # Another attendee's run may have created the credential between the check and the CREATE. + if ($attempt -eq 5) { throw } + Write-LabTrace "Restore credential update attempt $attempt failed ($($_.Exception.Message)). Retrying." + Start-Sleep -Seconds (5 * $attempt) + } + } +} + function Invoke-FabricApi { param( [Parameter(Mandatory = $true)][ValidateSet('GET', 'POST', 'DELETE')][string]$Method, @@ -73,84 +184,159 @@ function Invoke-FabricApi { [object]$Body ) $url = "$FabricApi/$($Path.TrimStart('/'))" + Write-LabTrace "Fabric API $Method $Path" $azArgs = @('rest', '--method', $Method, '--url', $url, '--resource', 'https://api.fabric.microsoft.com') if ($Body) { $json = ($Body | ConvertTo-Json -Depth 10 -Compress) + Write-LabTrace "Fabric API $Method $Path includes body properties: $(($Body.Keys | Sort-Object) -join ', ')" $azArgs += @('--headers', 'Content-Type=application/json', '--body', $json) } $raw = az @azArgs 2>&1 if ($LASTEXITCODE -ne 0) { throw "Fabric API $Method $Path failed: $raw" } if ([string]::IsNullOrWhiteSpace($raw)) { return $null } + Write-LabTrace "Fabric API $Method $Path succeeded." return ($raw | ConvertFrom-Json) } # ───────────────────────────────────────────── # 0. Resolve the attendee + shared SQL MI # ───────────────────────────────────────────── +Start-LabStep "Validate deployment inputs" +Write-LabTrace "Starting attendee deployment. SubscriptionId=$SubscriptionId; ResourceGroupName=$ResourceGroupName; PreferredLocations=$($PreferredLocation -join ', '); AllowedEntraUserIds=$($AllowedEntraUserIds.Count)." if (-not $AllowedEntraUserIds -or $AllowedEntraUserIds.Count -eq 0) { throw "No AllowedEntraUserIds supplied." } +Start-LabStep "Resolve attendee" $attendeeId = $AllowedEntraUserIds[0] +Write-LabTrace "Resolving lab user '$attendeeId'." $user = Get-MhhLabUser -UserId $attendeeId -$short = ($user.ShortName -replace '[^a-zA-Z0-9]', '').ToLower() +$rawShortName = [string]$user.ShortName +$short = ($rawShortName -replace '[^a-zA-Z0-9]', '').ToLower() if ([string]::IsNullOrWhiteSpace($short)) { $short = 'u' + (Get-MhhStableHash -Value $attendeeId -Length 12) } $upn = $user.UserPrincipalName -$sqlDb = "TailspinToys_$short" -$feedbackDb = "TailspinToysFeedback_$short" +$labUserNumber = Get-LabUserNumber -ShortName $rawShortName -UserPrincipalName $upn +if (-not $labUserNumber) { + $localTestUserIndex = if ($env:TTYD_LOCAL_TEST_USER_INDEX) { [int]$env:TTYD_LOCAL_TEST_USER_INDEX } else { 1 } + if ($localTestUserIndex -lt 1) { throw "TTYD_LOCAL_TEST_USER_INDEX must be greater than or equal to 1." } + $labUserSuffix = 'x{0:D3}' -f $localTestUserIndex + Write-Warning "[lab] Could not derive a labuser number from ShortName '$rawShortName' or UPN '$upn'; using local-test fallback User$labUserSuffix." +} +else { + $labUserSuffix = '{0:D4}' -f [int]$labUserNumber +} +$labUserPostfix = "User$labUserSuffix" +$userContainer = "container$labUserSuffix" + +$sqlDb = "TailspinToys_$labUserPostfix" +$feedbackDb = "TailspinToysFeedback_$labUserPostfix" +Write-LabTrace "Resolved attendee '$upn'; salesDatabase=$sqlDb; feedbackDatabase=$feedbackDb; userDataContainer=$userContainer." +Start-LabStep "Resolve shared SQL Managed Instance" $mi = @(az sql mi list -g $SharedResourceGroup -o json | ConvertFrom-Json) if (-not $mi -or $mi.Count -eq 0) { throw "No shared SQL Managed Instance found in '$SharedResourceGroup'. Did the shared hook run?" } $miFqdn = $mi[0].fullyQualifiedDomainName $publicFqdn = $miFqdn -replace '^([^.]+)\.', '$1.public.' $server = "$publicFqdn,3342" +Write-LabTrace "Shared SQL MI resolved: name=$($mi[0].name); server=$server." + +Start-LabStep "Ensure shared demo SQL login" +Ensure-DemoSqlLogin -Server $server +Write-Host "[lab] Ensured SQL login '$DemoSqlLogin' for Fabric mirroring." # ───────────────────────────────────────────── -# 1. Per-attendee ARM resources (CSV storage) into the attendee's RG +# 1. Upload the attendee CSV into the shared user-data storage account # ───────────────────────────────────────────── -Update-MhhTokenQuiet -$rgResult = Invoke-MhhDeploymentWithRegionFallback ` - -PreferredLocations $PreferredLocation ` - -ResourceGroupName $ResourceGroupName ` - -RgOwnerEntraObjectIds $AllowedEntraUserIds ` - -TemplateFile (Join-Path $PSScriptRoot 'main.bicep') ` - -TemplateParameterObject @{ attendeeObjectId = $attendeeId } ` - -DeploymentNamePrefix 'lab' +Start-LabStep "Resolve deploying principal" +$azContext = Get-AzContext +$deployingAccount = $azContext.Account.Id +$deployingPrincipalId = $null +$deployingPrincipalType = 'User' +if ($deployingAccount -as [guid]) { + $deployingPrincipalType = 'ServicePrincipal' + $deployingPrincipalId = (Get-AzADServicePrincipal -ApplicationId $deployingAccount -ErrorAction SilentlyContinue).Id + if (-not $deployingPrincipalId) { $deployingPrincipalId = az ad sp show --id $deployingAccount --query id -o tsv 2>$null } +} +else { + $deployingPrincipalId = (Get-AzADUser -SignedIn -ErrorAction SilentlyContinue).Id +} +if (-not $deployingPrincipalId) { throw "Could not resolve the deploying principal object id." } -$userStorage = [string]$rgResult.Outputs['storageAccountName'] -$userContainer = [string]$rgResult.Outputs['containerName'] +Start-LabStep "Resolve shared user-data storage" +$userStorage = az storage account list -g $SharedResourceGroup --query "[?starts_with(name, 'employeedata')].name | [0]" -o tsv +if ([string]::IsNullOrWhiteSpace($userStorage)) { + Write-Warning "[lab] No 'employeedata' storage account found; falling back to legacy 'stuserdata' account for this existing test deployment." + $userStorage = az storage account list -g $SharedResourceGroup --query "[?starts_with(name, 'stuserdata')].name | [0]" -o tsv +} +if ([string]::IsNullOrWhiteSpace($userStorage)) { throw "No shared user-data storage account found in '$SharedResourceGroup'. Did the shared hook run?" } +Write-LabTrace "Shared user-data storage resolved: storageAccount=$userStorage; container=$userContainer." + +Start-LabStep "Grant blob data access on shared user-data storage" +$userStorageId = az storage account show --resource-group $SharedResourceGroup --name $userStorage --query id -o tsv +$createdUserStorageRole = New-LabRoleAssignmentIfMissing -ObjectId $deployingPrincipalId -PrincipalType $deployingPrincipalType ` + -Role 'Storage Blob Data Contributor' -Scope $userStorageId +if ($createdUserStorageRole) { + Write-Host "[lab] Granted Storage Blob Data Contributor on '$userStorage' to the deploying principal." + Write-Host "[lab] Waiting 30 seconds for shared user-data storage RBAC propagation." + Start-Sleep -Seconds 30 # RBAC assignments are eventually consistent. + Write-LabTrace "Shared user-data storage RBAC propagation wait complete." +} +else { + Write-LabTrace "Role assignment for '$userStorage' skipped (already exists)." +} +# Fabric's ADLS Gen2 connection flow needs to enumerate the storage account before the attendee selects their container. +New-LabRoleAssignmentIfMissing -ObjectId $attendeeId -PrincipalType User ` + -Role 'Storage Blob Data Reader' -Scope $userStorageId | Out-Null # Upload the employee CSV into the attendee container. +Start-LabStep "Upload attendee employee CSV" $csvPath = Join-Path $PSScriptRoot 'csvdata/employees_user_data.csv' if (-not (Test-Path $csvPath)) { throw "Employee CSV not found: $csvPath" } -$userStorageKey = az storage account keys list --resource-group $ResourceGroupName --account-name $userStorage --query "[0].value" -o tsv -az storage blob upload --account-name $userStorage --account-key $userStorageKey ` - --container-name $userContainer --name (Split-Path $csvPath -Leaf) --file $csvPath --overwrite true --only-show-errors | Out-Null +az storage blob upload --account-name $userStorage --auth-mode login ` + --container-name $userContainer --name (Split-Path $csvPath -Leaf) --file $csvPath ` + --overwrite true --only-show-errors --no-progress | Out-Null if ($LASTEXITCODE -ne 0) { throw "Failed to upload employee CSV." } # ───────────────────────────────────────────── # 2. Restore the two attendee databases in the shared SQL MI # ───────────────────────────────────────────── +Start-LabStep "Prepare attendee database restores" Update-MhhTokenQuiet -$storageAccount = az storage account list -g $SharedResourceGroup --query "[0].name" -o tsv +$storageAccount = az storage account list -g $SharedResourceGroup --query "[?starts_with(name, 'stsqlhack')].name | [0]" -o tsv +if ([string]::IsNullOrWhiteSpace($storageAccount)) { throw "No shared backup storage account found in '$SharedResourceGroup'. Did the shared hook run?" } $containerName = 'build' -$storageKey = az storage account keys list --resource-group $SharedResourceGroup --account-name $storageAccount --query "[0].value" -o tsv + +# The shared backup storage account has shared-key auth disabled; grant this run's principal +# Entra-based blob access too (it may not be the same principal that ran the shared hook). +$backupStorageId = az storage account show --resource-group $SharedResourceGroup --name $storageAccount --query id -o tsv +$createdBackupStorageRole = New-LabRoleAssignmentIfMissing -ObjectId $deployingPrincipalId -PrincipalType $deployingPrincipalType ` + -Role 'Storage Blob Data Contributor' -Scope $backupStorageId +if ($createdBackupStorageRole) { + Write-Host "[lab] Granted Storage Blob Data Contributor on '$storageAccount' to the deploying principal." + Write-Host "[lab] Waiting 30 seconds for backup storage RBAC propagation." + Start-Sleep -Seconds 30 # RBAC assignments are eventually consistent. + Write-LabTrace "Backup storage RBAC propagation wait complete." +} +else { + Write-LabTrace "Role assignment for '$storageAccount' skipped (already exists)." +} + $expiry = (Get-Date).ToUniversalTime().AddHours(4).ToString('yyyy-MM-ddTHH:mmZ') +# SQL MI RESTORE FROM URL requires an account-key-backed service SAS. +$storageAccountKey = az storage account keys list --resource-group $SharedResourceGroup --account-name $storageAccount --query '[0].value' -o tsv +if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($storageAccountKey)) { throw "Failed to read a storage account key for '$storageAccount'." } $sasToken = az storage container generate-sas --account-name $storageAccount --name $containerName ` - --account-key $storageKey --permissions rl --https-only --expiry $expiry -o tsv + --account-key $storageAccountKey --permissions rl --https-only --expiry $expiry -o tsv if ($LASTEXITCODE -ne 0) { throw "Failed to generate container SAS." } $credentialName = "https://$storageAccount.blob.core.windows.net/$containerName" -Invoke-MiSql -Server $server -Database 'master' -QueryTimeout 60 -Query @" -IF EXISTS (SELECT 1 FROM sys.credentials WHERE name = N'$credentialName') - DROP CREDENTIAL [$credentialName]; -CREATE CREDENTIAL [$credentialName] -WITH IDENTITY = 'Shared Access Signature', SECRET = '$sasToken'; -"@ | Out-Null +Start-LabStep "Create SQL restore credential" +Set-LabRestoreCredential -Server $server -CredentialName $credentialName -SasToken $sasToken $restores = @( @{ Db = $sqlDb; Bak = $TailspinToysBak }, @{ Db = $feedbackDb; Bak = $TailspinToysFeedbackBak } ) foreach ($r in $restores) { + Start-LabStep "Restore or verify attendee database $($r.Db)" $exists = (Invoke-MiSql -Server $server -Database 'master' -QueryTimeout 60 ` -Query "SELECT COUNT(*) AS C FROM sys.databases WHERE name = N'$($r.Db)'").C if ($exists -gt 0) { @@ -165,6 +351,7 @@ foreach ($r in $restores) { # ───────────────────────────────────────────── # 3. Attendee login/user (db_owner) + product in each database # ───────────────────────────────────────────── +Start-LabStep "Create attendee SQL login" Update-MhhTokenQuiet Invoke-MiSql -Server $server -Database 'master' -QueryTimeout 60 -Query @" IF NOT EXISTS (SELECT 1 FROM sys.server_principals WHERE name = N'$upn') @@ -173,15 +360,20 @@ IF NOT EXISTS (SELECT 1 FROM sys.server_principals WHERE name = N'$upn') $productSqlTemplate = Get-Content -Raw (Join-Path $PSScriptRoot 'sql/InsertProduct.sql') foreach ($db in @($sqlDb, $feedbackDb)) { + Start-LabStep "Grant attendee access to $db" Invoke-MiSql -Server $server -Database $db -QueryTimeout 60 -Query @" IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = N'$upn') CREATE USER [$upn] FROM LOGIN [$upn]; ALTER ROLE [db_owner] ADD MEMBER [$upn]; "@ | Out-Null + + Start-LabStep "Grant demo SQL login access to $db" + Grant-DemoSqlLoginDatabaseAccess -Server $server -Database $db } # Ensure the Fabric Space Ranger product exists in the sales database (needed by the # shared proc / Agent job, which read Product from each per-attendee database). +Start-LabStep "Seed attendee sales database product" $productExists = (Invoke-MiSql -Server $server -Database $sqlDb -QueryTimeout 60 ` -Query "SELECT CASE WHEN EXISTS (SELECT 1 FROM dbo.Product WHERE ProductSKU = '9000-FABRIC-RANGER') THEN 1 ELSE 0 END AS C").C if ($productExists -ne 1) { @@ -193,20 +385,41 @@ if ($productExists -ne 1) { # ───────────────────────────────────────────── # 4. Fabric workspace on the shared capacity + attendee as Member # ───────────────────────────────────────────── +Start-LabStep "Resolve Fabric capacity" Update-MhhTokenQuiet $capacityName = az resource list -g $SharedResourceGroup --resource-type 'Microsoft.Fabric/capacities' --query "[0].name" -o tsv -$capacity = (Invoke-FabricApi -Method GET -Path 'capacities').value | Where-Object { $_.displayName -eq $capacityName } | Select-Object -First 1 +$capacity = $null +for ($elapsed = 0; $elapsed -lt 120; $elapsed += 15) { + $capacity = (Invoke-FabricApi -Method GET -Path 'capacities').value | Where-Object { $_.displayName -eq $capacityName } | Select-Object -First 1 + if ($capacity -and $capacity.state -eq 'Active') { break } + Write-LabTrace "Waiting for Fabric capacity '$capacityName' to become Active (state=$($capacity.state))..." + Start-Sleep -Seconds 15 +} if (-not $capacity) { throw "Fabric capacity '$capacityName' not visible via the Fabric API." } +if ($capacity.state -ne 'Active') { throw "Fabric capacity '$capacityName' did not reach Active state (last state: $($capacity.state))." } +Write-LabTrace "Fabric capacity resolved: name=$capacityName; id=$($capacity.id); state=$($capacity.state)." +Start-LabStep "Create or resolve attendee Fabric workspace" $workspaceName = "Workspace_$short" $workspace = (Invoke-FabricApi -Method GET -Path 'workspaces').value | Where-Object { $_.displayName -eq $workspaceName } | Select-Object -First 1 if (-not $workspace) { Write-Host "[lab] Creating Fabric workspace '$workspaceName'." - $workspace = Invoke-FabricApi -Method POST -Path 'workspaces' -Body @{ - displayName = $workspaceName - capacityId = $capacity.id + try { + $workspace = Invoke-FabricApi -Method POST -Path 'workspaces' -Body @{ + displayName = $workspaceName + capacityId = $capacity.id + } + } + catch { + if ($_.Exception.Message -match '(?i)Unauthorized|not authenticated') { + $caller = (Get-AzContext).Account.Id + throw "Fabric denied workspace creation for '$caller'. In the Fabric Admin portal, enable the Developer setting 'Service principals can create workspaces, connections, and deployment pipelines' for a security group containing this service principal." + } + throw } } +Write-LabTrace "Fabric workspace resolved: name=$workspaceName; id=$($workspace.id)." +Start-LabStep "Grant attendee Fabric workspace Member role" try { Invoke-FabricApi -Method POST -Path "workspaces/$($workspace.id)/roleAssignments" -Body @{ principal = @{ id = $attendeeId; type = 'User' } @@ -220,6 +433,7 @@ catch { # ───────────────────────────────────────────── # 5. Attendee credentials # ───────────────────────────────────────────── +Start-LabStep "Emit attendee credentials" @{ HackboxCredential = @{ name = 'Sales Database'; value = $sqlDb; note = 'Your TailspinToys database on the shared SQL MI' } } @{ HackboxCredential = @{ name = 'Feedback Database'; value = $feedbackDb; note = 'Your feedback database' } } @{ HackboxCredential = @{ name = 'Fabric Workspace'; value = $workspaceName; note = 'Your Fabric workspace' } } diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/storageBackups.bicep b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/storageBackups.bicep index 50a8b0ec0..7e2e24970 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/storageBackups.bicep +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/storageBackups.bicep @@ -36,6 +36,8 @@ resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { } properties: { allowBlobPublicAccess: false + // SQL MI RESTORE FROM URL requires an account-key-backed service SAS for these .bak files. + allowSharedKeyAccess: true minimumTlsVersion: 'TLS1_2' } } diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/userDataStorage.bicep b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/userDataStorage.bicep index 633eadebd..3742ab071 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/userDataStorage.bicep +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/infra/modules/userDataStorage.bicep @@ -9,29 +9,25 @@ param location string @description('Lab/environment name (used for tagging and the generated account name).') param envName string -@description('Number of per-user containers to create.') -@minValue(1) -param userCount int +@description('Explicit per-user container names, e.g. container0001 for lab users or containerx001 for local test users.') +@minLength(1) +param containerNames array -@description('Prefix for generated container names (e.g. container001).') -param containerNamePrefix string = 'container' - -@description('Optional explicit storage account name. Defaults to stuserdata.') +@description('Optional explicit storage account name. Defaults to employeedata.') param storageAccountName string = '' @description('Tags to apply to all resources.') param tags object = {} var normalizedEnv = toLower(replace(envName, '-', '')) -var generatedName = substring('stuserdata${normalizedEnv}', 0, min(length('stuserdata${normalizedEnv}'), 24)) +var generatedNameBase = 'employeedata${uniqueString(subscription().subscriptionId, resourceGroup().id, normalizedEnv)}' +var generatedName = substring(generatedNameBase, 0, min(length(generatedNameBase), 24)) var effectiveName = empty(storageAccountName) ? generatedName : storageAccountName var mergedTags = union(tags, { environment: envName }) -var containerNames = [for i in range(0, userCount): '${containerNamePrefix}${padLeft(string(i + 1), 3, '0')}'] - resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { name: effectiveName location: location @@ -42,6 +38,7 @@ resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { } properties: { allowBlobPublicAccess: false + allowSharedKeyAccess: false minimumTlsVersion: 'TLS1_2' isHnsEnabled: true } @@ -63,4 +60,5 @@ resource containers 'Microsoft.Storage/storageAccounts/blobServices/containers@2 ] output storageAccountName string = storageAccount.name +output storageAccountDfsEndpoint string = storageAccount.properties.primaryEndpoints.dfs output containerNames array = containerNames diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/lab-defaults.json b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/lab-defaults.json index a09f1c396..27e46d050 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/lab-defaults.json +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/lab-defaults.json @@ -1,6 +1,6 @@ { "$schema": "https://raw.githubusercontent.com/microsoft/MicroHack/refs/heads/main/lab-defaults-schema.json", - "groups": [ "" ], + "groups": [ "M365-E5-Users" ], "deploymentType": "resourcegroup", "labsPerSubscription": 42, "preferredLocation": "swedencentral, westeurope, northeurope", diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared-deploy-lab.ps1 b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared-deploy-lab.ps1 index 520a2b547..d6bffe7ff 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared-deploy-lab.ps1 +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared-deploy-lab.ps1 @@ -25,29 +25,32 @@ param( Set-StrictMode -Version Latest $ErrorActionPreference = 'Stop' +# Invoke-Sqlcmd/Az cmdlets emit Write-Progress records that the runner's child-job receiver can't +# deserialize cleanly (surfaces as "NotSpecified: (:String) [], RemoteException"), which flips the +# runner's own success verdict even though the script itself completes without error. +$ProgressPreference = 'SilentlyContinue' $script:SharedCurrentStep = 'initialization' -trap { - Write-Warning "[shared] FAILED during step '$script:SharedCurrentStep'." - Write-Warning "[shared] Exception: $($_.Exception.Message)" - if ($_.InvocationInfo -and $_.InvocationInfo.PositionMessage) { - Write-Warning "[shared] Location: $($_.InvocationInfo.PositionMessage)" - } - throw -} +# Commented out to test whether trap's rethrow is what makes the runner see this as failed. +# trap { +# Write-Warning "[shared] FAILED during step '$script:SharedCurrentStep'." +# Write-Warning "[shared] Exception: $($_.Exception.Message)" +# if ($_.InvocationInfo -and $_.InvocationInfo.PositionMessage) { +# Write-Warning "[shared] Location: $($_.InvocationInfo.PositionMessage)" +# } +# throw +# } # ───────────────────────────────────────────── # Constants # ───────────────────────────────────────────── $SharedResourceGroup = 'rg-shared' $SqlAdminLogin = 'sqlmiadmin' +$DemoSqlLogin = 'demouser' +$DemoSqlPassword = 'Demo@pass1234567' $FabricApi = 'https://api.fabric.microsoft.com/v1' -$GraphApi = 'https://graph.microsoft.com/v1.0' -# Power BI Free. Assigning it to a user is what makes the tenant known to Fabric; without it -# Microsoft.Fabric/capacities fails with "Tenant ... wasn't recognized by Microsoft Fabric". -$FabricLicenseSkuPartNumber = 'POWER_BI_STANDARD' -$FabricLicenseSkuId = 'a403ebcc-fae0-4ca2-8c8c-7a907fd6c235' -# Only needs to be a country Power BI is offered in; it gates license assignment, nothing else. -$FabricUsageLocation = 'DE' +# Fabric requires a licensed user in the tenant before it recognises it; the deploying principal cannot +# grant that license (no Graph write permission), so it comes from the 'M365-E5-Users' group request in +# lab-defaults.json instead. We only poll for it here. # Microsoft.PowerPlatform is required for the Fabric VNet data gateway's subnet delegation. $RequiredProviders = @('Microsoft.Sql', 'Microsoft.Fabric', 'Microsoft.Storage', 'Microsoft.Web', 'Microsoft.Network', 'Microsoft.PowerPlatform') @@ -89,6 +92,16 @@ function Start-SharedStep { Write-SharedTrace "STEP: $Name" } +function Get-LabUserNumber { + param( + [string]$ShortName, + [string]$UserPrincipalName + ) + if ($ShortName -match '(?i)(?:labuser|user)[-_]?0*(\d+)$') { return [int]$Matches[1] } + if ($UserPrincipalName -match '(?i)(?:labuser|user)[-_]?0*(\d+)') { return [int]$Matches[1] } + return $null +} + function Invoke-MiSql { param( [Parameter(Mandatory = $true)][string]$Server, @@ -97,7 +110,7 @@ function Invoke-MiSql { [string]$InputFile, [int]$QueryTimeout = 0 ) - # SQL authentication with the MI admin login: the platform cannot set an Entra admin on the MI. + # Use SQL authentication while provisioning; the Entra admin is configured later in this hook. $cred = [pscredential]::new($SqlAdminLogin, (ConvertTo-SecureString $sqlPassword -AsPlainText -Force)) $splat = @{ ServerInstance = $Server @@ -113,32 +126,43 @@ function Invoke-MiSql { Invoke-Sqlcmd @splat } -function Invoke-GraphApi { +function Ensure-DemoSqlLogin { param( - [Parameter(Mandatory = $true)][ValidateSet('GET', 'POST', 'PATCH')][string]$Method, - [Parameter(Mandatory = $true)][string]$Path, - [object]$Body + [Parameter(Mandatory = $true)][string]$Server ) - $url = "$GraphApi/$($Path.TrimStart('/'))" - Write-SharedTrace "Graph $Method $Path" - $azArgs = @('rest', '--method', $Method, '--url', $url, '--resource', 'https://graph.microsoft.com') - if ($Body) { - Write-SharedTrace "Graph $Method $Path includes body properties: $(($Body.Keys | Sort-Object) -join ', ')" - $azArgs += @('--headers', 'Content-Type=application/json', '--body', ($Body | ConvertTo-Json -Depth 10 -Compress)) - } - $raw = az @azArgs 2>&1 - if ($LASTEXITCODE -ne 0) { - Write-Warning "[shared] Graph $Method $Path failed with exit code $LASTEXITCODE. Raw response: $raw" - throw "Graph $Method $Path failed: $raw" - } - Write-SharedTrace "Graph $Method $Path succeeded." - if ([string]::IsNullOrWhiteSpace($raw)) { return $null } - return ($raw | ConvertFrom-Json) + $login = $DemoSqlLogin.Replace(']', ']]') + $password = $DemoSqlPassword.Replace("'", "''") + Invoke-MiSql -Server $Server -Database 'master' -QueryTimeout 60 -Query @" +IF NOT EXISTS (SELECT 1 FROM sys.sql_logins WHERE name = N'$DemoSqlLogin') + CREATE LOGIN [$login] WITH PASSWORD = N'$password', CHECK_POLICY = OFF; +ELSE + ALTER LOGIN [$login] WITH PASSWORD = N'$password', CHECK_POLICY = OFF; +"@ | Out-Null +} + +function Grant-DemoSqlLoginDatabaseAccess { + param( + [Parameter(Mandatory = $true)][string]$Server, + [Parameter(Mandatory = $true)][string]$Database + ) + $login = $DemoSqlLogin.Replace(']', ']]') + Invoke-MiSql -Server $Server -Database $Database -QueryTimeout 60 -Query @" +IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = N'$DemoSqlLogin') + CREATE USER [$login] FROM LOGIN [$login]; +IF NOT EXISTS ( + SELECT 1 + FROM sys.database_role_members drm + INNER JOIN sys.database_principals roles ON roles.principal_id = drm.role_principal_id + INNER JOIN sys.database_principals members ON members.principal_id = drm.member_principal_id + WHERE roles.name = N'db_owner' AND members.name = N'$DemoSqlLogin' +) + ALTER ROLE [db_owner] ADD MEMBER [$login]; +"@ | Out-Null } function Invoke-FabricApi { param( - [Parameter(Mandatory = $true)][ValidateSet('GET', 'POST', 'DELETE')][string]$Method, + [Parameter(Mandatory = $true)][ValidateSet('GET', 'POST', 'PATCH', 'DELETE')][string]$Method, [Parameter(Mandatory = $true)][string]$Path, [object]$Body ) @@ -176,8 +200,11 @@ foreach ($rp in $RequiredProviders) { } # ───────────────────────────────────────────── -# 1. Shared resource group + Owner for every attendee in this subscription +# 1. Shared resource group (created once; attendees get no RBAC here) # ───────────────────────────────────────────── +# Attendees never touch rg-shared directly: the SQL MI is reached via a SQL login (deploy-lab.ps1) +# and Fabric via a per-attendee workspace Member role, not Azure RBAC on this resource group. Granting +# RBAC here would let any attendee modify/delete resources shared by every lab in the subscription. if (-not (Get-AzResourceGroup -Name $SharedResourceGroup -ErrorAction SilentlyContinue)) { Start-SharedStep "Create shared resource group" New-AzResourceGroup -Name $SharedResourceGroup -Location $location -Tag @{ SecurityControl = 'Ignore' } | Out-Null @@ -185,13 +212,6 @@ if (-not (Get-AzResourceGroup -Name $SharedResourceGroup -ErrorAction SilentlyCo } $rgId = (Get-AzResourceGroup -Name $SharedResourceGroup).ResourceId Write-SharedTrace "Shared resource group id: $rgId" -Start-SharedStep "Ensure attendee Owner assignments" -foreach ($uid in ($AllowedEntraUserIds | Where-Object { $_ })) { - Write-SharedTrace "Ensuring Owner role on shared resource group for attendee object id '$uid'." - if (-not (Get-AzRoleAssignment -ObjectId $uid -Scope $rgId -RoleDefinitionName 'Owner' -ErrorAction SilentlyContinue)) { - New-AzRoleAssignment -ObjectId $uid -Scope $rgId -RoleDefinitionName 'Owner' -ErrorAction SilentlyContinue | Out-Null - } -} # ───────────────────────────────────────────── # 2. Resolve principals and bootstrap the Fabric tenant @@ -217,7 +237,14 @@ if (-not $spObjectId) { if (-not $spObjectId) { throw "Could not resolve the deploying principal object id." } Write-SharedTrace "Deploying principal object id resolved: $spObjectId" -# Resolve every attendee once; reused for Fabric licensing, capacity admins and the SQL MI Entra admin. +Start-SharedStep "Resolve default lab group" +$labGroup = Get-MhhDefaultLabGroup +if (-not $labGroup -or -not $labGroup.ObjectId) { throw "Get-MhhDefaultLabGroup did not return a group with an ObjectId." } +Write-SharedTrace "Default lab group resolved: $($labGroup.DisplayName) ($($labGroup.ObjectId))." + +# Resolve every attendee once; reused below for the SQL MI Entra admin, the Fabric capacity admin +# list (per-user UPNs) and the shared user-data containers. Fabric gateway admin access is still +# granted to the whole group (gateway role assignments accept a group principal). Start-SharedStep "Resolve lab users" $labUsers = [System.Collections.Generic.List[object]]::new() foreach ($uid in ($AllowedEntraUserIds | Where-Object { $_ })) { @@ -233,50 +260,35 @@ foreach ($uid in ($AllowedEntraUserIds | Where-Object { $_ })) { Id = $uid UserPrincipalName = $memberUpn ShortName = $mhhUser.ShortName + LabUserNumber = Get-LabUserNumber -ShortName $mhhUser.ShortName -UserPrincipalName $memberUpn + LabUserSuffix = $null }) } if ($labUsers.Count -eq 0) { throw "Could not resolve any lab user from AllowedEntraUserIds." } Write-SharedTrace "Resolved $($labUsers.Count) lab users: $((@($labUsers | ForEach-Object { $_.UserPrincipalName }) -join ', '))" +foreach ($labUser in @($labUsers | Where-Object { $_.LabUserNumber })) { + $labUser.LabUserSuffix = '{0:D4}' -f [int]$labUser.LabUserNumber +} +$nextFallbackUserIndex = 1 +foreach ($labUser in @($labUsers | Where-Object { -not $_.LabUserNumber })) { + $labUser.LabUserSuffix = 'x{0:D3}' -f $nextFallbackUserIndex + Write-Warning "[shared] Could not derive a labuser number for '$($labUser.UserPrincipalName)' ($($labUser.ShortName)); using local-test fallback User$($labUser.LabUserSuffix)." + $nextFallbackUserIndex++ +} +$userDataContainerNames = @($labUsers | + Sort-Object LabUserSuffix | + ForEach-Object { "container$($_.LabUserSuffix)" } | + Select-Object -Unique) +Write-SharedTrace "Preparing shared user-data storage containers: $($userDataContainerNames -join ', ')." + $firstLabUser = $labUsers | Where-Object { $_.ShortName -match '(?i)labuser-[0-9]{4}' } | Sort-Object ShortName | Select-Object -First 1 if (-not $firstLabUser) { $firstLabUser = $labUsers | Sort-Object ShortName | Select-Object -First 1 } -Write-SharedTrace "Fabric bootstrap user: $($firstLabUser.UserPrincipalName) ($($firstLabUser.Id))." - -# A service principal cannot sign a tenant up for Fabric; only a licensed user can. Every attendee is -# licensed anyway (they need it to open Fabric), and the first assignment provisions the tenant. -Start-SharedStep "Assume Fabric license SKU" -Write-SharedTrace "Skipping Graph subscribedSkus lookup; using known SKU '$FabricLicenseSkuPartNumber' with skuId '$FabricLicenseSkuId'." +Write-SharedTrace "SQL MI Entra admin candidate: $($firstLabUser.UserPrincipalName) ($($firstLabUser.Id))." -foreach ($labUser in $labUsers) { - Start-SharedStep "Ensure Fabric license for $($labUser.UserPrincipalName)" - $isBootstrapUser = $labUser.Id -eq $firstLabUser.Id - try { - Write-SharedTrace "Checking Fabric license for $($labUser.UserPrincipalName) ($($labUser.Id))." - $graphUser = Invoke-GraphApi -Method GET -Path "users/$($labUser.Id)?`$select=id,usageLocation,assignedLicenses" - if ($graphUser.assignedLicenses.skuId -contains $FabricLicenseSkuId) { - Write-Host "[shared] $($labUser.UserPrincipalName) already holds $FabricLicenseSkuPartNumber." - continue - } - # assignLicense rejects users without a usage location. - if (-not $graphUser.usageLocation) { - Write-SharedTrace "Setting usageLocation '$FabricUsageLocation' for $($labUser.UserPrincipalName)." - Invoke-GraphApi -Method PATCH -Path "users/$($labUser.Id)" -Body @{ usageLocation = $FabricUsageLocation } | Out-Null - } - Write-SharedTrace "Assigning SKU '$FabricLicenseSkuPartNumber' to $($labUser.UserPrincipalName)." - Invoke-GraphApi -Method POST -Path "users/$($labUser.Id)/assignLicense" -Body @{ - addLicenses = @(@{ skuId = $FabricLicenseSkuId; disabledPlans = @() }) - removeLicenses = @() - } | Out-Null - Write-Host "[shared] Assigned $FabricLicenseSkuPartNumber to $($labUser.UserPrincipalName)." - } - catch { - if ($isBootstrapUser) { throw "Could not license '$($labUser.UserPrincipalName)' for Fabric, so the tenant cannot be provisioned: $($_.Exception.Message)" } - Write-Warning "[shared] Fabric license for $($labUser.UserPrincipalName) failed: $($_.Exception.Message)" - } -} - -# Tenant provisioning is asynchronous. Poll until the Fabric API answers rather than letting the -# capacity deployment fail; on timeout continue anyway so ARM produces the authoritative error. +# The 'M365-E5-Users' group request in lab-defaults.json licenses every attendee for Fabric; we cannot +# assign it ourselves here. Poll (read-only) until the tenant is recognised rather than letting the +# capacity deployment fail outright while that group-based provisioning is still catching up. $fabricTenantReady = $false Start-SharedStep "Wait for Fabric tenant provisioning" for ($elapsed = 0; $elapsed -lt 600; $elapsed += 30) { @@ -295,7 +307,9 @@ if (-not $fabricTenantReady) { Write-Warning "[shared] Fabric did not confirm the tenant within 10 minutes. Continuing; the capacity deployment will report the real cause." } -# Fabric capacity admin members: users must be UPNs (object IDs are rejected); a service principal uses its object ID. +# Fabric capacity administrators must be existing users or service principals; groups are rejected +# ("All provided principals must be existing, user or service principals"). So enumerate every attendee's +# UPN individually instead of adding the shared lab group's ObjectId. $fabricMemberList = [System.Collections.Generic.List[string]]::new() if ($account -as [guid]) { $fabricMemberList.Add($spObjectId) # deploying service principal @@ -305,7 +319,7 @@ else { } foreach ($labUser in $labUsers) { $fabricMemberList.Add($labUser.UserPrincipalName) } $fabricAdminMembers = @($fabricMemberList | Select-Object -Unique) -$sqlPassword = New-MhhStablePassword -Purpose 'sql-admin' -Length 24 +$sqlPassword = New-MhhStablePassword -Purpose 'sql-admin' -Length 24 -ResourceGroupName $SharedResourceGroup Write-SharedTrace "Prepared $($fabricAdminMembers.Count) Fabric capacity admin members. SQL admin password generated but not printed." # ───────────────────────────────────────────── @@ -328,6 +342,7 @@ New-AzResourceGroupDeployment ` # Plain string, not SecureString: -AsJob cannot serialize a SecureString across the job boundary. Bicep param stays @secure(). sqlPassword = $sqlPassword fabricAdminMembers = $fabricAdminMembers + userDataContainerNames = $userDataContainerNames sqlMiNetworkingExists = $sqlMiNetworkingExists } ` -AsJob | Out-Null @@ -360,12 +375,18 @@ $vnetName = $out.vnetName.Value $fabricSubnet = $out.fabricSubnetName.Value $storageAccount = $out.backupStorageAccountName.Value $containerName = $out.backupContainerName.Value +$userDataStorage = $out.userDataStorageAccountName.Value +$userDataStorageDfsEndpoint = $out.userDataStorageDfsEndpoint.Value $webshopHost = $out.webshopDefaultHostname.Value # Public endpoint FQDN: insert 'public.' after the instance short name; port 3342. $publicFqdn = $miFqdn -replace '^([^.]+)\.', '$1.public.' $server = "$publicFqdn,3342" -Write-SharedTrace "Shared outputs: miName=$miName; capacityName=$capacityName; vnetName=$vnetName; fabricSubnet=$fabricSubnet; storageAccount=$storageAccount; containerName=$containerName; webshopHost=$webshopHost." +Write-SharedTrace "Shared outputs: miName=$miName; capacityName=$capacityName; vnetName=$vnetName; fabricSubnet=$fabricSubnet; backupStorage=$storageAccount; backupContainer=$containerName; userDataStorage=$userDataStorage; webshopHost=$webshopHost." + +Start-SharedStep "Ensure shared demo SQL login" +Ensure-DemoSqlLogin -Server $server +Write-Host "[shared] Ensured SQL login '$DemoSqlLogin' for Fabric mirroring." # ───────────────────────────────────────────── # 4. Entra: Directory Readers for the MI identity (needed for external-provider logins) @@ -386,7 +407,12 @@ Write-Host "[shared] SQL MI identity Directory Readers: $($drResult.status -join # Deliberately not done in Bicep: ARM cannot resolve the principal in the lab tenant. # Set-AzSqlInstanceActiveDirectoryAdministrator is unreliable here too, so PUT the # administrator sub-resource directly. Directory Readers must be granted first. -Start-Sleep -Seconds 30 +# Entra directory-role assignments are eventually consistent; a short wait risks +# ServicePrincipalLookupInAadFailedIdentityForbidden when the PUT below looks up the AAD admin. +Start-SharedStep "Wait for Directory Readers propagation" +Write-Host "[shared] Waiting 5 minutes for the SQL MI Directory Readers assignment to propagate before setting the Entra admin." +Start-Sleep -Seconds 300 +Write-SharedTrace "Directory Readers propagation wait complete." Update-MhhTokenQuiet Start-SharedStep "Configure SQL MI Entra admin" @@ -426,34 +452,52 @@ Write-Host "[shared] SQL MI Entra admin is $($sqlEntraAdmin.UserPrincipalName)." # 5. Upload the .bak files and restore the demo databases # ───────────────────────────────────────────── Update-MhhTokenQuiet -Start-SharedStep "Read shared backup storage key" -$storageKey = az storage account keys list --resource-group $SharedResourceGroup --account-name $storageAccount --query "[0].value" -o tsv -if ($LASTEXITCODE -ne 0) { throw "Failed to read storage key for '$storageAccount'." } -Write-SharedTrace "Storage key retrieved for '$storageAccount'. Key value is not printed." +# The backup storage account has shared-key auth disabled; grant the deploying principal Entra-based +# blob access instead (covers both blob upload and generating a user-delegation SAS below). +Start-SharedStep "Grant deploying principal blob data access on backup storage" +$backupStorageId = az storage account show --resource-group $SharedResourceGroup --name $storageAccount --query id -o tsv +$backupPrincipalType = if ($account -as [guid]) { 'ServicePrincipal' } else { 'User' } +az role assignment create --assignee-object-id $spObjectId --assignee-principal-type $backupPrincipalType ` + --role 'Storage Blob Data Contributor' --scope $backupStorageId 2>$null | Out-Null +if ($LASTEXITCODE -eq 0) { + Write-Host "[shared] Granted Storage Blob Data Contributor on '$storageAccount' to the deploying principal." + Write-Host "[shared] Waiting 30 seconds for backup storage RBAC propagation." + Start-Sleep -Seconds 30 # RBAC assignments are eventually consistent. + Write-SharedTrace "Backup storage RBAC propagation wait complete." +} +else { + Write-SharedTrace "Role assignment for '$storageAccount' skipped (already exists)." +} foreach ($bak in @($TailspinToysBak, $TailspinToysFeedbackBak)) { Start-SharedStep "Upload backup $bak" $localBak = Join-Path $PSScriptRoot "databasebackup/$bak" if (-not (Test-Path $localBak)) { throw "Backup file not found: $localBak" } Write-Host "[shared] Uploading $bak." - az storage blob upload --account-name $storageAccount --account-key $storageKey ` - --container-name $containerName --name $bak --file $localBak --overwrite true --only-show-errors | Out-Null + # --no-progress: the CLI's own upload progress bar goes to stderr regardless of $ProgressPreference, + # which the runner's child-job receiver mis-relays as a "RemoteException" (false-positive failure). + az storage blob upload --account-name $storageAccount --auth-mode login ` + --container-name $containerName --name $bak --file $localBak --overwrite true --only-show-errors --no-progress 2>$null | Out-Null if ($LASTEXITCODE -ne 0) { throw "Failed to upload $bak." } } $expiry = (Get-Date).ToUniversalTime().AddHours(4).ToString('yyyy-MM-ddTHH:mmZ') Start-SharedStep "Generate backup container SAS" +# SQL MI RESTORE FROM URL requires an account-key-backed service SAS. +$storageAccountKey = az storage account keys list --resource-group $SharedResourceGroup --account-name $storageAccount --query '[0].value' -o tsv +if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($storageAccountKey)) { throw "Failed to read a storage account key for '$storageAccount'." } $sasToken = az storage container generate-sas --account-name $storageAccount --name $containerName ` - --account-key $storageKey --permissions rl --https-only --expiry $expiry -o tsv + --account-key $storageAccountKey --permissions rl --https-only --expiry $expiry -o tsv if ($LASTEXITCODE -ne 0) { throw "Failed to generate container SAS." } $credentialName = "https://$storageAccount.blob.core.windows.net/$containerName" Start-SharedStep "Create SQL restore credential" +# ALTER, not DROP+CREATE: DROP fails while any restore on this instance is still using the credential. Invoke-MiSql -Server $server -Database 'master' -QueryTimeout 60 -Query @" IF EXISTS (SELECT 1 FROM sys.credentials WHERE name = N'$credentialName') - DROP CREDENTIAL [$credentialName]; -CREATE CREDENTIAL [$credentialName] -WITH IDENTITY = 'Shared Access Signature', SECRET = '$sasToken'; + ALTER CREDENTIAL [$credentialName] WITH IDENTITY = 'Shared Access Signature', SECRET = '$sasToken'; +ELSE + CREATE CREDENTIAL [$credentialName] WITH IDENTITY = 'Shared Access Signature', SECRET = '$sasToken'; "@ | Out-Null foreach ($db in $DemoDatabases.Keys) { @@ -469,6 +513,11 @@ foreach ($db in $DemoDatabases.Keys) { Invoke-MiSql -Server $server -Database 'master' -Query "RESTORE DATABASE [$db] FROM URL = N'$url';" | Out-Null } +foreach ($db in $DemoDatabases.Keys) { + Start-SharedStep "Grant demo SQL login access to $db" + Grant-DemoSqlLoginDatabaseAccess -Server $server -Database $db +} + # ───────────────────────────────────────────── # 6. Seed Demo_Final: product, stored procedure, Agent job # ───────────────────────────────────────────── @@ -494,13 +543,23 @@ if ($jobExists -ne 1) { } # ───────────────────────────────────────────── -# 7. Fabric VNet data gateway + ConnectionCreator for every attendee +# 7. Fabric VNet data gateway + Admin role for every attendee # ───────────────────────────────────────────── Update-MhhTokenQuiet -Start-SharedStep "Resolve Fabric capacity" -$capacity = (Invoke-FabricApi -Method GET -Path 'capacities').value | Where-Object { $_.displayName -eq $capacityName } | Select-Object -First 1 +# ARM reporting the capacity deployment as Succeeded doesn't mean the Fabric control plane has finished +# attaching it yet; creating the gateway against a not-yet-Active capacity leaves it permanently unable +# to refresh, even though both later report healthy. Wait for the Fabric-side state explicitly. +Start-SharedStep "Wait for Fabric capacity to become Active" +$capacity = $null +for ($elapsed = 0; $elapsed -lt 300; $elapsed += 15) { + $capacity = (Invoke-FabricApi -Method GET -Path 'capacities').value | Where-Object { $_.displayName -eq $capacityName } | Select-Object -First 1 + if ($capacity -and $capacity.state -eq 'Active') { break } + Write-Host "[shared] Waiting for Fabric capacity '$capacityName' to become Active (state=$($capacity.state); ${elapsed}s elapsed)..." + Start-Sleep -Seconds 15 +} if (-not $capacity) { throw "Fabric capacity '$capacityName' not visible via the Fabric API (check tenant setting 'Service principals can use Fabric APIs')." } -Write-SharedTrace "Fabric capacity id: $($capacity.id)" +if ($capacity.state -ne 'Active') { throw "Fabric capacity '$capacityName' did not reach Active state within 5 minutes (last state: $($capacity.state))." } +Write-SharedTrace "Fabric capacity id: $($capacity.id); state confirmed Active." $gatewayName = "fabric-gateway-shared" Start-SharedStep "Resolve Fabric VNet gateway" @@ -523,27 +582,49 @@ if (-not $gateway) { } } Write-SharedTrace "Fabric gateway id: $($gateway.id)" -foreach ($uid in ($AllowedEntraUserIds | Where-Object { $_ })) { - try { - Start-SharedStep "Grant Fabric gateway ConnectionCreator to $uid" - Invoke-FabricApi -Method POST -Path "gateways/$($gateway.id)/roleAssignments" -Body @{ - principal = @{ id = $uid; type = 'User' } - role = 'ConnectionCreator' - } | Out-Null +# The lab group gets Admin (not just ConnectionCreator) so every attendee can restart the gateway +# themselves when it goes to sleep or fails during the hack. +$gatewayAssignments = @() +try { $gatewayAssignments = @((Invoke-FabricApi -Method GET -Path "gateways/$($gateway.id)/roleAssignments").value) } +catch { Write-Warning "[shared] Could not list existing gateway role assignments: $($_.Exception.Message)" } + +try { + $existing = $gatewayAssignments | Where-Object { $_.principal.id -eq $labGroup.ObjectId } | Select-Object -First 1 + if ($existing -and $existing.role -eq 'Admin') { + Write-SharedTrace "Gateway Admin already assigned to lab group $($labGroup.DisplayName)." } - catch { - Write-Warning "[shared] Gateway role assignment for $uid skipped: $($_.Exception.Message)" + else { + Start-SharedStep "Grant Fabric gateway Admin to lab group $($labGroup.DisplayName)" + if ($existing) { + # Re-run over a lab that was provisioned with ConnectionCreator: upgrade in place, since a + # second POST for the same principal is rejected. + Invoke-FabricApi -Method PATCH -Path "gateways/$($gateway.id)/roleAssignments/$($existing.id)" -Body @{ + role = 'Admin' + } | Out-Null + } + else { + Invoke-FabricApi -Method POST -Path "gateways/$($gateway.id)/roleAssignments" -Body @{ + principal = @{ id = $labGroup.ObjectId; type = 'Group' } + role = 'Admin' + } | Out-Null + } } } +catch { + Write-Warning "[shared] Gateway role assignment for lab group $($labGroup.DisplayName) skipped: $($_.Exception.Message)" +} # ───────────────────────────────────────────── # 8. Shared credentials for every attendee's dashboard # ───────────────────────────────────────────── -@{ HackboxCredential = @{ name = 'SQL MI Public Endpoint'; value = $server; note = 'SSMS: connect with the SQL admin login below' } } -@{ HackboxCredential = @{ name = 'SQL Admin Login'; value = $SqlAdminLogin; note = 'Shared SQL Managed Instance admin' } } -@{ HackboxCredential = @{ name = 'SQL Admin Password'; value = $sqlPassword; note = 'Shared SQL Managed Instance admin password' } } +@{ HackboxCredential = @{ name = 'SQL MI Endpoint'; value = $miFqdn; note = 'Fabric mirroring Server field: Fabric reaches this over the VNet gateway, so use the internal FQDN, not the public one' } } +@{ HackboxCredential = @{ name = 'SQL Login'; value = $DemoSqlLogin; note = 'Shared SQL Managed Instance demo login' } } +@{ HackboxCredential = @{ name = 'SQL Password'; value = $DemoSqlPassword; note = 'Shared SQL Managed Instance demo login password' } } +@{ HackboxCredential = @{ name = 'User Data Storage DFS Endpoint'; value = $userDataStorageDfsEndpoint; note = 'ADLS Gen2 endpoint for the shared employee CSV storage account' } } @{ HackboxCredential = @{ name = 'Webshop URL'; value = "https://$webshopHost"; note = 'Shared Tailspin Toys webshop' } } @{ HackboxCredential = @{ name = 'Fabric Capacity'; value = $capacityName; note = 'Shared Fabric F32 capacity' } } @{ HackboxCredential = @{ name = 'Shared Resource Group'; value = $SharedResourceGroup; note = 'Shared by all labs in your subscription' } } Write-Host "[shared] Shared deployment complete." +# Guard against a stale non-zero $LASTEXITCODE from an earlier, already-handled native command call. +exit 0 diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared.bicep b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared.bicep index d207f51eb..41990d873 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared.bicep +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/shared.bicep @@ -7,8 +7,8 @@ // the first lab user; ARM cannot resolve the principal in the lab tenant) // - demo database restores, stored proc, product, Agent job // - Fabric VNet gateway + gateway role assignments -// Per-attendee resources (user databases, Fabric workspaces) are created by -// deploy-lab.ps1. +// Per-attendee work (user databases, Fabric workspaces, SQL access, CSV upload) +// is performed by deploy-lab.ps1. No ARM resources are deployed per attendee. @description('Azure region for the shared resources.') param location string @@ -32,6 +32,9 @@ param fabricAdminMembers array @description('Base database the shared webshop connects to.') param webshopSqlDatabase string = 'TailspinToys_Demo_Final' +@description('Exact per-user CSV container names to create.') +param userDataContainerNames array + @description('When true, the SQL MI subnet NSG and route table already exist and are referenced instead of redeployed (avoids ConflictWithNetworkIntentPolicy on shared hook re-runs).') param sqlMiNetworkingExists bool = false @@ -86,6 +89,16 @@ module storageBackups 'infra/modules/storageBackups.bicep' = { } } +module userDataStorage 'infra/modules/userDataStorage.bicep' = { + name: 'shared-user-data' + params: { + location: location + envName: envName + containerNames: userDataContainerNames + tags: commonTags + } +} + module fabricCapacity 'infra/modules/fabricCapacity.bicep' = { name: 'shared-fabric' params: { @@ -105,5 +118,8 @@ output fabricSubnetName string = vnet.outputs.fabricSubnetName output fabricCapacityName string = fabricCapacity.outputs.capacityName output backupStorageAccountName string = storageBackups.outputs.storageAccountName output backupContainerName string = storageBackups.outputs.containerName +output userDataStorageAccountName string = userDataStorage.outputs.storageAccountName +output userDataStorageDfsEndpoint string = userDataStorage.outputs.storageAccountDfsEndpoint +output userDataContainerNames array = userDataStorage.outputs.containerNames output webshopName string = appService.outputs.webAppName output webshopDefaultHostname string = appService.outputs.defaultHostname diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/Jobs.sql b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/Jobs.sql index 5acb7613c..e642da156 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/Jobs.sql +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/Jobs.sql @@ -3,7 +3,7 @@ GO /****** Job: [2 Fabric Space Ranger Workload] ******/ -- Rewritten for the MicroHack framework: both job steps now discover the per-attendee --- databases by name pattern instead of iterating a fixed TailspinToys_User001..100 range. +-- databases by name pattern instead of iterating a fixed TailspinToys_User0001..0100 range. BEGIN TRANSACTION DECLARE @ReturnCode INT SELECT @ReturnCode = 0 diff --git a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/StoredProcedure.sql b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/StoredProcedure.sql index e29f29a2b..2538ab2ec 100644 --- a/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/StoredProcedure.sql +++ b/03-Azure/01-02 Data/03-Talk_to_your_data/labautomation/sql/StoredProcedure.sql @@ -1,5 +1,5 @@ -- usp_PurchaseSpaceRanger: shared purchase entry point called by the webshop. --- Rewritten for the MicroHack framework: instead of a hardcoded TailspinToys_User001..100 +-- Rewritten for the MicroHack framework: instead of a hardcoded TailspinToys_User0001..0100 -- loop, it discovers every per-attendee database by name pattern and fans a purchase -- out to Demo_Final plus each attendee database that currently exists. USE [TailspinToys_Demo_Final]