From d48092c05519a5cc3a14b54f2739757c61a3b8aa Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 21:24:34 +0200 Subject: [PATCH 01/11] Rewrite challenge 1 Part A for the pre-provisioned lab environment Part A told participants to paste every value from the MicroHack dashboard's Credentials tab. That tab only exposes the 7 platform-provided Azure entries, so most keys -- including LLM_BACKEND_CONFIG -- were not obtainable that way, and the instructions had no Azure sign-in step at all. Part A is now five steps matching what the environment actually provides: - A.1 Sign in to Azure with the dashboard's Entra username and Temporary Access Pass, then az login so DefaultAzureCredential resolves the lab user. - A.2 Find the resource token from the aif-hub- account name; every other resource name derives from it. - A.3 Fill in .env, showing each key derived from the token. - A.4 Paste LLM_BACKEND_CONFIG, with the full value and the three ways it breaks: line breaks, surrounding quotes, and a missing final bracket. Also documents reading it straight from the deployment outputs instead. - A.5 Validate the file before starting. The bracket warning is the important one: the value must end with }]}], and dropping the last two characters produces a JSON error pointing ~1000 characters away from the mistake. Adds a note to restart the Jupyter kernel after editing .env, since the loader skips keys already in os.environ and a bad value read at startup otherwise persists for the whole session. Updates .env.template to match -- its per-key comments named dashboard credentials that do not exist -- and adjusts the context paragraph and troubleshooting table. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../challenges/challenge-01.md | 159 +++++++++++++++--- .../challenges/workshop/.env.template | 46 ++--- 2 files changed, 158 insertions(+), 47 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md index c621ce92a..270fe8781 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md @@ -12,47 +12,151 @@ fragments, then prove it works through multiple API formats. ## šŸ§­ Context This challenge runs entirely against **Hub** resources ā€” the APIM instance and its -backend pools sit in front of every model your governance hub serves. Before you start, -confirm your attendee credentials are available to the notebooks (see Part A below); -every workshop notebook reads its configuration from the environment, falling back to -`azd env get-value`. +backend pools sit in front of every model your governance hub serves. Your lab resources +are already deployed for you, so Part A below is about pointing your machine at them: +sign in to Azure, then write a `.env` file that every workshop notebook reads. ## āœ… Tasks -### Part A ā€” Confirm your notebook environment is ready (5 min) +### Part A ā€” Set up your notebook environment (10 min) -Every notebook in this MicroHack reads its settings from the environment, so do this -once and it applies to all nine challenges. Pick **either** route: +Your lab resources are **already deployed**. You do not run `azd up`. This section +connects your machine to them, and it applies to all nine challenges ā€” do it once. -**Route 1 ā€” `.env` file (recommended, no extra tooling):** +#### A.1 ā€” Sign in to Azure -1. From the `challenges/workshop/` folder, copy the template: +Your MicroHack dashboard's **Credentials** tab gives you an Entra username and a +**Temporary Access Pass (TAP)**, which is used instead of a password and expires at the +end of the event. + +1. Open the **Azure Portal** in a **private/incognito** window (so it doesn't collide + with your corporate account) and sign in with the dashboard's *Entra ID Username*, + entering the *Entra ID TAP* when asked for a password. Complete any prompts. +2. Sign in from your terminal ā€” the notebooks use `DefaultAzureCredential`, so this + session is what authenticates their Azure calls: ```bash - cd challenges/workshop - cp .env.template .env + az login --tenant emea.microhack.cloud + az account set --subscription + az account show --output table ``` -2. Open `.env` and paste in the matching `HackboxCredential` values from your MicroHack - dashboard. The template names the exact dashboard credential for every key. - For challenges 1-6 you only need `AZURE_RESOURCE_GROUP`, `AZURE_LOCATION`, - `AZURE_SUBSCRIPTION_ID` and `LLM_BACKEND_CONFIG`. -3. Paste `LLM_BACKEND_CONFIG` as a **single line**, exactly as the dashboard shows it ā€” - it's a JSON document, so a stray line break will break it. + If the browser handoff misbehaves, use `az login --use-device-code --tenant emea.microhack.cloud`. + +> [!WARNING] +> If you are already signed in to another Azure account, `DefaultAzureCredential` may +> pick it up instead. If notebook cells return 401/403, run `az account show` and +> confirm you are the lab user. + +#### A.2 ā€” Find your resource token + +Every resource in your lab is named with the same random token, and the `.env` values +are built from it. In the portal, open your resource group (`rg-labuser-00XX`) and find +the AI Foundry account named **`aif-hub-`** ā€” everything after the prefix is your +token. For example `aif-hub-fd79f0b6fee89` means the token is `fd79f0b6fee89`. + +From the CLI: + +```bash +az cognitiveservices account list -g \ + --query "[?starts_with(name,'aif-hub-')].name" -o tsv +``` + +#### A.3 ā€” Fill in `.env` + +```bash +cd challenges/workshop +cp .env.template .env +``` + +Open `.env` and set the values below, replacing `` throughout. The first three +come straight from the dashboard / portal Overview blade; the rest are derived from the +token. + +```bash +AZURE_RESOURCE_GROUP=rg-labuser-00XX +AZURE_LOCATION=swedencentral # compact form, not "Sweden Central" +AZURE_SUBSCRIPTION_ID= + +SPOKE_RESOURCE_GROUP=rg-labuser-00XX # same RG ā€” this lab uses one resource group +SPOKE_KEY_VAULT_NAME=kv-spoke- +SPOKE_AI_FOUNDRY_ACCOUNT_NAME=aif-spoke- +SPOKE_AI_FOUNDRY_PROJECT_NAME=citadel-agents-project +SPOKE_ACR_NAME=acrcitadel +SPOKE_ACR_LOGIN_SERVER=acrcitadel.azurecr.io +A2A_FOUNDRY_ACCOUNT_NAME=aif-spoke- +``` + +Challenges 1-6 need only the first four values plus `LLM_BACKEND_CONFIG`. + +#### A.4 ā€” Fill in `LLM_BACKEND_CONFIG` + +This one is a single line of JSON describing your gateway's backend and its six models. +Paste the block below as **one line**, with `` replaced in **both** places: + +``` +LLM_BACKEND_CONFIG=[{"backendId":"aif-hub-","backendType":"ai-foundry","endpoint":"https://aif-hub-.openai.azure.com","authScheme":"managedIdentity","authType":"managed-identity","priority":1,"weight":100,"supportedModels":[{"name":"gpt-4.1","sku":"GlobalStandard","capacity":20,"modelFormat":"OpenAI","modelVersion":"2025-04-14","retirementDate":"2027-04-14"},{"name":"gpt-5.4-mini","sku":"GlobalStandard","capacity":20,"modelFormat":"OpenAI","modelVersion":"2026-03-17","retirementDate":"2027-09-21"},{"name":"gpt-5.2","sku":"GlobalStandard","capacity":20,"modelFormat":"OpenAI","modelVersion":"2025-12-11","retirementDate":"2027-06-08"},{"name":"text-embedding-3-large","sku":"GlobalStandard","capacity":20,"modelFormat":"OpenAI","modelVersion":"1","retirementDate":"2028-02-09"},{"name":"Mistral-Large-3","sku":"GlobalStandard","capacity":20,"modelFormat":"Mistral AI","modelVersion":"1","retirementDate":"2099-12-31"},{"name":"Phi-4","sku":"GlobalStandard","capacity":1,"modelFormat":"Microsoft","modelVersion":"7","retirementDate":"2099-12-31"}]}] +``` + +> [!IMPORTANT] +> Three things break this value, and all three fail with a confusing error pointing +> ~1000 characters away from the real mistake: +> +> - **Line breaks.** It must be one unbroken line. Turn off word wrap before pasting. +> - **Surrounding quotes.** Do not wrap the value in `"` or `'`. +> - **A missing closing bracket.** The value must end with `}]}]` ā€” four closers, for +> the model, `supportedModels`, the backend object, and the outer array. +> +> If your editor underlines the line as invalid, ignore it: `.env` is not a JSON file, +> and the `LLM_BACKEND_CONFIG=` prefix alone is enough to confuse a JSON linter. + +Alternatively, read the exact value straight out of your deployment outputs and skip the +typing entirely: + +```bash +az deployment group list -g -o json \ + | python3 -c " +import json,sys +for d in json.load(sys.stdin): + for k,v in ((d.get('properties') or {}).get('outputs') or {}).items(): + if k.upper() == 'LLM_BACKEND_CONFIG': + print('LLM_BACKEND_CONFIG=' + v['value']); raise SystemExit +" +``` + +#### A.5 ā€” Validate before you start + +```bash +cd challenges/workshop +python3 -c " +import json +from pathlib import Path +v=[l.partition('=')[2].strip() for l in Path('.env').read_text().splitlines() + if l.startswith('LLM_BACKEND_CONFIG=') and l.partition('=')[2].strip()][0] +c=json.loads(v) +print('OK -', len(c), 'backend,', len(c[0]['supportedModels']), 'models')" +``` + +Expect `OK - 1 backend, 6 models`. If it raises instead, re-read the warning above ā€” +a missing final `]` is by far the most common cause. `.env` is gitignored, so your credentials stay out of source control. -**Route 2 ā€” `azd` bridge (the original workshop workflow):** +> [!TIP] +> **If you edit `.env` after starting Jupyter, restart the kernel.** The notebook only +> sets a variable if it isn't already present, so a bad value read at startup will +> persist for the whole session no matter what you change on disk. VS Code also injects +> `.env` into the kernel at start, which has the same effect. -1. Install `azd`, then run - [`setup-notebook-env.ps1`](../labautomation/README.md#notebook-environment-setup) - with your dashboard's `HackboxCredential` values. It writes a local `azd` - environment that the notebooks fall back to for any key not already set. +> [!NOTE] +> **Using `azd` instead (optional).** If you prefer the original workshop workflow, +> install `azd` and run +> [`setup-notebook-env.ps1`](../labautomation/README.md#notebook-environment-setup). +> It writes a local `azd` environment that the notebooks fall back to for any key not +> already set in `.env`. This is not required. -> [!TIP] -> To confirm you're ready, run the first setup cell of notebook 1. It prints your -> resource group, location, and the number of configured LLM backends. If it raises -> instead, the message tells you which key is missing and whether a `.env` was found. +To confirm you're ready, run the first setup cell of notebook 1. It should print your +resource group, your location, and `LLM Backends: 1 backend(s) configured`. ### Part B ā€” Run the LLM backend onboarding notebook (30 min) @@ -89,7 +193,8 @@ once and it applies to all nine challenges. Pick **either** route: | Symptom | Fix | |---------|-----| | Setup cell raises a missing-key error | That key isn't in your `.env` (and no `azd` environment supplied it) ā€” see Part A. The error message tells you which key is missing and whether a `.env` file was found. | -| `LLM_BACKEND_CONFIG` fails to parse | It must be on a single line, pasted exactly as the dashboard shows it. A line break or a truncated copy will break the JSON. | +| `LLM_BACKEND_CONFIG` fails to parse | It must be one unbroken line, with no surrounding quotes, ending in `}]}]`. A missing final bracket is the usual cause ā€” run the validation snippet in Part A.5, which reports the exact position. | +| You fixed `.env` but the notebook reports the old error | The kernel caches environment values from startup. Restart the Jupyter kernel and re-run. | | Backend deployment succeeds but test calls 401/403 | APIM RBAC / managed-identity propagation can take a minute after a fresh deployment ā€” wait and retry. | | A model call fails with a region/quota error | Model availability is region-specific; double-check the model/SKU you configured is available in your lab's region. | diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/.env.template b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/.env.template index 4e0a0f36d..de9170a89 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/.env.template +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/.env.template @@ -1,9 +1,14 @@ # --------------------------------------------------------------------------- # MicroHack: AI Governance ā€” notebook configuration # -# Copy this file to `.env` in this same folder, then paste in the values from -# the "Credentials" panel of your MicroHack dashboard. The notebooks read -# os.environ first, so this is all you need ā€” `azd` is NOT required. +# Copy this file to `.env` in this same folder, then fill in the values. +# Your lab is already deployed -- do NOT run `azd up`. +# +# Full step-by-step instructions, including how to sign in to Azure and how to +# find the used in most of these names, are in challenge-01.md Part A. +# +# The notebooks read os.environ first, so this is all you need -- `azd` is NOT +# required. # # cp .env.template .env # @@ -15,46 +20,47 @@ # --------------------------------------------------------------------------- # ===== Required for challenges 1-6 (core) ================================== -# Dashboard credential: ResourceGroup +# Dashboard credential: Resource Group Name (e.g. rg-labuser-0005) AZURE_RESOURCE_GROUP= -# Dashboard credential: Location +# Portal: your resource group's Overview blade. Use the compact form, e.g. +# 'swedencentral' -- not 'Sweden Central'. AZURE_LOCATION= -# Dashboard credential: SubscriptionId +# Dashboard credential: Subscription ID AZURE_SUBSCRIPTION_ID= -# Dashboard credential: LlmBackendConfig -# One long line of JSON ā€” paste it exactly as shown, with no line breaks -# and no surrounding quotes. +# See challenge-01.md, Part A.4 for the value to paste here. +# One long line of JSON. It must be a SINGLE line, with no surrounding quotes, +# and it must end with }]}] -- four closing brackets. A missing final bracket is +# the most common setup error and produces a confusing parse error. +# Validate it with the snippet in challenge-01.md Part A.5 before you start. LLM_BACKEND_CONFIG= # ===== Required for challenges 3, 4 and 7 ================================== -# Dashboard credential: SpokeResourceGroup -# (Same value as AZURE_RESOURCE_GROUP ā€” this lab uses a single resource group.) +# Same value as AZURE_RESOURCE_GROUP -- this lab uses a single resource group. SPOKE_RESOURCE_GROUP= -# Dashboard credential: SpokeKeyVaultName +# kv-spoke- (see challenge-01.md Part A.2 for how to find ) SPOKE_KEY_VAULT_NAME= -# Dashboard credential: SpokeAiFoundryAccountName +# aif-spoke- SPOKE_AI_FOUNDRY_ACCOUNT_NAME= -# Dashboard credential: SpokeAiFoundryProjectName +# Fixed value, no token: citadel-agents-project SPOKE_AI_FOUNDRY_PROJECT_NAME= # ===== Required for challenge 7 ============================================ -# Dashboard credential: SpokeAcrName +# acrcitadel SPOKE_ACR_NAME= -# Dashboard credential: SpokeAcrLoginServer +# acrcitadel.azurecr.io SPOKE_ACR_LOGIN_SERVER= # ===== Required for challenge 8 ============================================ -# Dashboard credential: A2aFoundryAccountName -# The Foundry account that hosts the agent. Set this explicitly ā€” challenge 8 -# otherwise looks for a backend id ending in "-0", which this lab's resource -# naming does not produce. +# aif-spoke- -- the Foundry account that hosts the agent. Set this +# explicitly: challenge 8 otherwise looks for a backend id ending in "-0", +# which this lab's resource naming does not produce. A2A_FOUNDRY_ACCOUNT_NAME= # ===== Challenge 9 (optional / instructor-led) ============================= From c31afb0d02378070366363a91083e09a3a646bce Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:12:21 +0200 Subject: [PATCH 02/11] Convert AI Governance challenge deployments to resource-group scope Lab participants hold Owner on their own resource group but only Reader on the subscription, so every 'az deployment sub create' in the core challenge notebooks failed with AuthorizationFailed on Microsoft.Resources/deployments/validate/action. - llm-backend-onboarding/main.bicep and citadel-access-contracts/main.bicep switch from targetScope 'subscription' to 'resourceGroup'. Both used the subscription scope only to 'existing'-reference resource groups before deploying modules into them; in this single-resource-group lab those all resolve to the participant's resource group, so the lookups are dropped and the resources referenced directly. - Update the 7 'az deployment sub create' call sites in notebooks 1, 2, 3, 5 and 6 to 'az deployment group create --resource-group ...'. - Refresh the matching README/bicepparam examples. - Add troubleshooting rows for wrong-tenant ResourceGroupNotFound and for subscription-scoped AuthorizationFailed. Both templates verified with 'az deployment group validate' against a live lab resource group as the participant identity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../infra/citadel-access-contracts/README.md | 12 +++---- .../infra/citadel-access-contracts/main.bicep | 25 +++------------ .../citadel-access-contracts/main.bicepparam | 4 +-- .../infra/llm-backend-onboarding/README.md | 2 +- .../infra/llm-backend-onboarding/main.bicep | 31 +++++++------------ .../challenges/challenge-01.md | 2 ++ .../1. llm-backend-onboarding-runner.ipynb | 6 ++-- ...l-universal-llm-api-all-models-tests.ipynb | 2 +- .../3. citadel-access-contracts-tests.ipynb | 2 +- .../5. citadel-pii-processing-tests.ipynb | 8 ++--- .../6. citadel-unified-ai-api-tests.ipynb | 2 +- 11 files changed, 39 insertions(+), 57 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/README.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/README.md index 21486fac5..d6a600d35 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/README.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/README.md @@ -45,7 +45,7 @@ below are the high-level steps to deploy the use case onboarding Bicep package b 5. **Deploy template with the prepared parameter file**: ```bash # This can be executed in CLI or through a DevOps pipeline -az deployment sub create --name --location --template-file main.bicep --parameters contracts///main.bicepparam +az deployment group create --name --resource-group --template-file main.bicep --parameters contracts///main.bicepparam ``` >NOTE: Ensure that you are updating values according to your environment and folder structure. @@ -355,17 +355,17 @@ az deployment sub what-if ` ### Step 4: Deploy ```powershell -# Deploy at subscription scope (run from the environment folder) -az deployment sub create ` +# Deploy at resource-group scope (run from the environment folder) +az deployment group create ` --name healthcare-chatbot-dev-onboarding ` - --location swedencentral ` + --resource-group ` --template-file ../../../main.bicep ` --parameters main.bicepparam # Or run from the citadel-access-contracts root folder: -az deployment sub create ` +az deployment group create ` --name healthcare-chatbot-dev-onboarding ` - --location swedencentral ` + --resource-group ` --template-file main.bicep ` --parameters contracts/healthcare-chatbot/dev/main.bicepparam ``` diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicep b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicep index fa1d1736f..3611fd887 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicep @@ -1,4 +1,4 @@ -targetScope = 'subscription' +targetScope = 'resourceGroup' @description('APIM resource coordinates') param apim object @@ -67,30 +67,22 @@ var productPostfix = '${useCase.businessUnit}-${useCase.useCaseName}-${useCase.e // Default APIM product policy (applied when a service item does not provide policyXmlPath) var defaultProductPolicyXml = loadTextContent('./policies/default-ai-product-policy.xml') -resource apimRg 'Microsoft.Resources/resourceGroups@2022-09-01' existing = { - scope: subscription(apim.subscriptionId) - name: apim.resourceGroupName -} +// Resource-group scoped: APIM, Key Vault and Foundry all live in the resource +// group this deployment targets. The subscriptionId / resourceGroupName fields +// on the apim, keyVault and foundry parameters are kept for backwards +// compatibility but are no longer used for scoping. resource apimSvc 'Microsoft.ApiManagement/service@2024-05-01' existing = { - scope: apimRg name: apim.name } -resource kvRg 'Microsoft.Resources/resourceGroups@2022-09-01' existing = if (useTargetAzureKeyVault) { - scope: subscription(keyVault.subscriptionId) - name: keyVault.resourceGroupName -} - resource kv 'Microsoft.KeyVault/vaults@2023-07-01' existing = if (useTargetAzureKeyVault) { - scope: kvRg name: keyVault.name } // Onboard each requested service into APIM module onboard 'modules/apimOnboardService.bicep' = [for s in services: { name: 'onboard-${s.code}-${productPostfix}' - scope: apimRg params: { apimName: apim.name productId: '${s.code}-${productPostfix}' @@ -109,7 +101,6 @@ module onboard 'modules/apimOnboardService.bicep' = [for s in services: { // Create/update KV secrets; normalize names (Key Vault does not allow underscores) module kvWrites 'modules/kvSecrets.bicep' = [for (s, i) in services: if (useTargetAzureKeyVault) { name: 'kv-${s.code}-${productPostfix}' - scope: kvRg params: { keyVaultName: kv.name secretNames: [ toLower(replace(s.endpointSecretName, '_', '-')), toLower(replace(s.apiKeySecretName, '_', '-')) ] @@ -126,17 +117,11 @@ module kvWrites 'modules/kvSecrets.bicep' = [for (s, i) in services: if (useTarg // Create Foundry APIM connections per service (only if useTargetFoundry is true) // This enables Foundry agents to access AI models through the APIM gateway -resource foundryRg 'Microsoft.Resources/resourceGroups@2022-09-01' existing = if (useTargetFoundry) { - scope: subscription(foundry.subscriptionId) - name: foundry.resourceGroupName -} - // Generate connection name based on config or use case naming var foundryConnectionPrefix = !empty(foundryConfig.connectionNamePrefix) ? foundryConfig.connectionNamePrefix : 'Hub-${useCase.businessUnit}-${useCase.useCaseName}-${useCase.environment}' module foundryConnections 'modules/foundryConnection.bicep' = [for (s, i) in services: if (useTargetFoundry) { name: 'foundry-${s.code}-${productPostfix}' - scope: foundryRg params: { aiFoundryAccountName: foundry.accountName aiFoundryProjectName: foundry.projectName diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicepparam b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicepparam index 3568f0715..2711c1381 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicepparam +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/citadel-access-contracts/main.bicepparam @@ -360,9 +360,9 @@ param foundryConfig = { // - Foundry Resource Group: Contributor (if using Foundry) // // Deployment command: -// az deployment sub create \ +// az deployment group create \ // --name \ -// --location \ +// --resource-group \ // --template-file main.bicep \ // --parameters main.bicepparam // diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/README.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/README.md index cb79f58cc..cdb3d992c 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/README.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/README.md @@ -90,7 +90,7 @@ param llmBackendConfig = [ ### 3. Deploy ```bash -az deployment sub create --name llm-backend-onboarding --location swedencentral --template-file main.bicep --parameters llm-backends-generated-local.bicepparam +az deployment group create --name llm-backend-onboarding --resource-group --template-file main.bicep --parameters llm-backends-generated-local.bicepparam ``` ## Configuration Reference diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/main.bicep b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/main.bicep index aea6db88d..c4f2cebc0 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/main.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/bicep/infra/llm-backend-onboarding/main.bicep @@ -1,4 +1,4 @@ -targetScope = 'subscription' +targetScope = 'resourceGroup' /** * @module LLM Backend Onboarding @@ -10,9 +10,14 @@ targetScope = 'subscription' * - Automatic failover when backends are unavailable * - Flexible authentication schemes per backend * + * MicroHack note: this template is resource-group scoped. Lab participants hold + * Owner on their own resource group and only Reader on the subscription, so a + * subscription-scoped deployment would fail authorization. APIM and its managed + * identity are expected to live in the resource group being deployed to. + * * Usage: - * az deployment sub create \ - * --location \ + * az deployment group create \ + * --resource-group \ * --template-file main.bicep \ * --parameters main.bicepparam */ @@ -106,24 +111,16 @@ param keyVaultName string = '' // ============================================================================ // EXISTING RESOURCES // ============================================================================ - -resource apimRg 'Microsoft.Resources/resourceGroups@2022-09-01' existing = { - scope: subscription(apim.subscriptionId) - name: apim.resourceGroupName -} +// Resource-group scoped: APIM and the managed identity are resolved in the +// resource group this deployment targets. The subscriptionId / +// resourceGroupName fields on the `apim` and `apimManagedIdentity` parameters +// are retained for backwards compatibility but are no longer used for scoping. resource apimService 'Microsoft.ApiManagement/service@2024-06-01-preview' existing = { - scope: apimRg name: apim.name } -resource identityRg 'Microsoft.Resources/resourceGroups@2022-09-01' existing = { - scope: subscription(apimManagedIdentity.subscriptionId) - name: apimManagedIdentity.resourceGroupName -} - resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' existing = { - scope: identityRg name: apimManagedIdentity.name } @@ -137,7 +134,6 @@ resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023- */ module llmBackends 'modules/llm-backends.bicep' = { name: 'llm-backends-deployment-${uniqueString(deployment().name)}' - scope: apimRg params: { apimServiceName: apim.name managedIdentityClientId: managedIdentity.properties.clientId @@ -152,7 +148,6 @@ module llmBackends 'modules/llm-backends.bicep' = { */ module llmBackendPools 'modules/llm-backend-pools.bicep' = { name: 'llm-backend-pools-deployment-${uniqueString(deployment().name)}' - scope: apimRg params: { apimServiceName: apim.name backendDetails: llmBackends.outputs.backendDetails @@ -165,7 +160,6 @@ module llmBackendPools 'modules/llm-backend-pools.bicep' = { */ module llmPolicyFragments 'modules/llm-policy-fragments.bicep' = { name: 'llm-policy-fragments-deployment-${uniqueString(deployment().name)}' - scope: apimRg params: { apimServiceName: apim.name policyFragmentConfig: llmBackendPools.outputs.policyFragmentConfig @@ -185,7 +179,6 @@ module llmPolicyFragments 'modules/llm-policy-fragments.bicep' = { // */ // module universalLlmApi 'modules/universal-llm-api.bicep' = if (deployUniversalLlmApi) { // name: 'universal-llm-api-deployment-${uniqueString(deployment().name)}' -// scope: apimRg // params: { // apimServiceName: apim.name // apiPath: universalLlmApiPath diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md index 270fe8781..483196695 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-01.md @@ -195,6 +195,8 @@ resource group, your location, and `LLM Backends: 1 backend(s) configured`. | Setup cell raises a missing-key error | That key isn't in your `.env` (and no `azd` environment supplied it) ā€” see Part A. The error message tells you which key is missing and whether a `.env` file was found. | | `LLM_BACKEND_CONFIG` fails to parse | It must be one unbroken line, with no surrounding quotes, ending in `}]}]`. A missing final bracket is the usual cause ā€” run the validation snippet in Part A.5, which reports the exact position. | | You fixed `.env` but the notebook reports the old error | The kernel caches environment values from startup. Restart the Jupyter kernel and re-run. | +| `ResourceGroupNotFound` for a resource group you can see in the portal | Your `az` CLI is signed in to a *different* tenant (commonly your corporate account). The portal session and the CLI session are independent. Run `az account show` ā€” if the user isn't `labuser-XXXX@emea.microhack.cloud`, redo the `az login` in Part A.1. | +| `AuthorizationFailed` on `Microsoft.Resources/deployments/...` over a **subscription** scope | You're running an older copy of the notebooks. All deployments must be resource-group scoped (`az deployment group create`); you hold Owner on your own resource group but only Reader on the subscription. Re-download the challenge files. | | Backend deployment succeeds but test calls 401/403 | APIM RBAC / managed-identity propagation can take a minute after a fresh deployment ā€” wait and retry. | | A model call fails with a region/quota error | Model availability is region-specific; double-check the model/SKU you configured is available in your lab's region. | diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/1. llm-backend-onboarding-runner.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/1. llm-backend-onboarding-runner.ipynb index 85682b3a7..a157d1ad6 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/1. llm-backend-onboarding-runner.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/1. llm-backend-onboarding-runner.ipynb @@ -635,8 +635,10 @@ "utils.print_info(f\"Template: {template_file}\")\n", "utils.print_info(f\"Parameters: {params_file}\")\n", "\n", - "# Run the subscription-level deployment\n", - "deployment_cmd = f\"az deployment sub create --name {deployment_name} --location {location} --template-file {template_file} --parameters {params_file}\"\n", + "# Run the resource-group-level deployment. Lab participants hold Owner on\n", + "# their resource group but only Reader on the subscription, so a\n", + "# subscription-scoped deployment would fail with AuthorizationFailed.\n", + "deployment_cmd = f\"az deployment group create --name {deployment_name} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {params_file}\"\n", "\n", "output = utils.run(\n", " deployment_cmd,\n", diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/2. citadel-universal-llm-api-all-models-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/2. citadel-universal-llm-api-all-models-tests.ipynb index c743abbed..cc9b0d805 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/2. citadel-universal-llm-api-all-models-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/2. citadel-universal-llm-api-all-models-tests.ipynb @@ -373,7 +373,7 @@ "product_id = f\"LLM-{contract_business_unit}-{contract_use_case_name}-{contract_environment}\"\n", "utils.print_info(f\"Deploying access contract: {contract_name} (Product: {product_id})...\")\n", "\n", - "deployment_cmd = f\"az deployment sub create --name {contract_name} --location {location} --template-file {template_file} --parameters {params_file}\"\n", + "deployment_cmd = f\"az deployment group create --name {contract_name} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {params_file}\"\n", "output = utils.run(deployment_cmd, f\"Deployment '{contract_name}' succeeded\", f\"Deployment '{contract_name}' failed\")\n", "\n", "if output.success:\n", diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/3. citadel-access-contracts-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/3. citadel-access-contracts-tests.ipynb index 99c36b0fd..b98abf639 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/3. citadel-access-contracts-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/3. citadel-access-contracts-tests.ipynb @@ -593,7 +593,7 @@ " utils.print_info(f\"{'='*60}\")\n", " \n", " # Deploy the access contract\n", - " deployment_cmd = f\"az deployment sub create --name {contract['name']} --location {location} --template-file {template_file} --parameters {params_file}\"\n", + " deployment_cmd = f\"az deployment group create --name {contract['name']} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {params_file}\"\n", " \n", " utils.print_info(f\"Deploying {contract['name']}...\")\n", " output = utils.run(\n", diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb index 405b60132..8c4ae6706 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb @@ -508,7 +508,7 @@ "utils.print_info(f\"Deploying PII Masking Access Contract...\")\n", "utils.print_info(f\"{'='*60}\")\n", "\n", - "deployment_cmd = f\"az deployment sub create --name {contract['name']} --location {location} --template-file {template_file} --parameters {masking_params_file}\"\n", + "deployment_cmd = f\"az deployment group create --name {contract['name']} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {masking_params_file}\"\n", "\n", "masking_deployment_output = utils.run(\n", " deployment_cmd,\n", @@ -994,7 +994,7 @@ "utils.print_info(f\"Deploying PII Blocking Access Contract...\")\n", "utils.print_info(f\"{'='*60}\")\n", "\n", - "deployment_cmd = f\"az deployment sub create --name {contract['name']} --location {location} --template-file {template_file} --parameters {blocking_params_file}\"\n", + "deployment_cmd = f\"az deployment group create --name {contract['name']} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {blocking_params_file}\"\n", "\n", "blocking_deployment_output = utils.run(\n", " deployment_cmd,\n", @@ -1935,8 +1935,8 @@ "utils.print_info(f\"{'='*60}\")\n", "\n", "deployment_cmd = (\n", - " f\"az deployment sub create --name {contract['name']} \"\n", - " f\"--location {location} --template-file {template_file} \"\n", + " f\"az deployment group create --name {contract['name']} \"\n", + " f\"--resource-group {governance_hub_resource_group} --template-file {template_file} \"\n", " f\"--parameters {analytics_params_file}\"\n", ")\n", "\n", diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/6. citadel-unified-ai-api-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/6. citadel-unified-ai-api-tests.ipynb index d277a0219..dbf058a27 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/6. citadel-unified-ai-api-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/6. citadel-unified-ai-api-tests.ipynb @@ -363,7 +363,7 @@ "product_id = f\"LLM-{contract_business_unit}-{contract_use_case_name}-{contract_environment}\"\n", "utils.print_info(f\"Deploying access contract: {contract_name} (Product: {product_id})...\")\n", "\n", - "deployment_cmd = f\"az deployment sub create --name {contract_name} --location {location} --template-file {template_file} --parameters {params_file}\"\n", + "deployment_cmd = f\"az deployment group create --name {contract_name} --resource-group {governance_hub_resource_group} --template-file {template_file} --parameters {params_file}\"\n", "output = utils.run(deployment_cmd, f\"Deployment '{contract_name}' succeeded\", f\"Deployment '{contract_name}' failed\")\n", "\n", "if output.success:\n", From 2a1b6cc7be5017492fdbac0a384d8a93126d014c Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:33:24 +0200 Subject: [PATCH 03/11] Add missing APIM operation policies so challenges 1, 2 and 6 work resources.bicep declared 11 APIM operations but only 3 operation policies. An APIM operation with no policy has no backend and returns HTTP 500, so 8 of the 11 endpoints were dead: universal-llm-api: post-responses, get-response, get-response-input-items unified-ai-api: get-deployments, get-deployment, post-openai-chat-completions, post-foundry-chat-completions, post-gemini-chat-completions Impact: challenge 6 calls all five unified-ai endpoints, so it was entirely non-functional; challenge 2 lost its three /responses endpoints; challenge 1's closing test cells failed against unified-ai. The three working operations were the only ones ever exercised, which is why this went unnoticed. Add the eight policies, following the proven pattern of the existing three (managed-identity auth -> backend-hub-foundry -> rewrite-uri): - /deployments returns the deployment inventory in the {"value":[{name,properties.model}]} shape the notebooks parse, and /deployments/{model_name} returns 404 for an unknown model as challenge 6 asserts. - The OpenAI-style and Foundry-style chat paths map the requested model to its deployment name (dots to dashes) and reject unsupported models with 400. - The responses paths proxy the Azure OpenAI Responses API, rewriting the body model to the deployment name. - The Gemini path returns an explicit 501 instead of a bare 500, since no Gemini backend is provisioned in this lab. - unified-ai responses carry UAIG-Backend / UAIG-API-Type / UAIG-Model / UAIG-Is-Streaming debug headers, which challenge 6 validates. Also align the uami-client-id named value: resources.bicep created it with displayName 'Hub.Foundry.UAMI.Client.ID' while challenge 1's llm-policy-fragments.bicep recreates it with displayName 'uami-client-id'. Running challenge 1 therefore renamed it and APIM silently rewrote every referencing policy. Setting displayName to match the resource name makes challenge 1 idempotent here and keeps both templates consistent. All eight endpoints verified live against a lab resource group as the participant identity, including the full create/get/input_items responses chain; the compiled policies are byte-identical to the ones tested. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../labautomation/infra/resources.bicep | 92 +++++++++++++++- .../labautomation/infra/resources.json | 104 +++++++++++++++++- 2 files changed, 189 insertions(+), 7 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep index 406ce57cf..723986d3e 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep @@ -326,7 +326,7 @@ resource apimNamedValueUamiClientId 'Microsoft.ApiManagement/service/namedValues parent: apim name: 'uami-client-id' properties: { - displayName: 'Hub.Foundry.UAMI.Client.ID' + displayName: 'uami-client-id' value: apimMI.properties.clientId secret: false } @@ -550,7 +550,7 @@ resource apiUniversalLlmOpChatCompletionsPolicy 'Microsoft.ApiManagement/service name: 'policy' properties: { format: 'xml' - value: '@("Bearer " + (string)context.Variables["msi-access-token"])application/json@{return new JObject(new JProperty("error","unsupported model requested for chat completions"),new JProperty("requestedModel",(string)context.Variables["requestedModel"])).ToString();}' + value: '@("Bearer " + (string)context.Variables["msi-access-token"])application/json@{return new JObject(new JProperty("error","unsupported model requested for chat completions"),new JProperty("requestedModel",(string)context.Variables["requestedModel"])).ToString();}' } dependsOn: [ apimNamedValueUamiClientId @@ -577,7 +577,7 @@ resource apiUniversalLlmOpEmbeddingsPolicy 'Microsoft.ApiManagement/service/apis name: 'policy' properties: { format: 'xml' - value: '@("Bearer " + (string)context.Variables["msi-access-token"])application/json@{return new JObject(new JProperty("error","unsupported model requested for embeddings"),new JProperty("requestedModel",(string)context.Variables["requestedModel"])).ToString();}' + value: '@("Bearer " + (string)context.Variables["msi-access-token"])application/json@{return new JObject(new JProperty("error","unsupported model requested for embeddings"),new JProperty("requestedModel",(string)context.Variables["requestedModel"])).ToString();}' } dependsOn: [ apimNamedValueUamiClientId @@ -733,6 +733,92 @@ resource apiUnifiedAiOpGeminiCompletions 'Microsoft.ApiManagement/service/apis/o } } + +// ===== Phase 4: Operation policies for the remaining APIM operations ===== +// Every APIM operation needs a policy; without one the operation has no backend +// and returns HTTP 500. These eight complete the surface exercised by +// challenges 1, 2 and 6. + +// POST /models/responses -> Foundry Responses API +resource apiUniversalLlmOpPostResponsesPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUniversalLlmOpPostResponses + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":"unsupported model requested for responses"}@("Bearer " + (string)context.Variables["msi-access-token"])@{var body = context.Request.Body.As<JObject>(preserveContent: true); string m = (string)body["model"]; if (string.IsNullOrEmpty(m)) { m = "gpt-4.1"; } body["model"] = m.Replace(".", "-"); return body.ToString();}' + } +} + +// GET /models/responses/{response_id} +resource apiUniversalLlmOpGetResponsePolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUniversalLlmOpGetResponse + name: 'policy' + properties: { + format: 'xml' + value: '@("Bearer " + (string)context.Variables["msi-access-token"])' + } +} + +// GET /models/responses/{response_id}/input_items +resource apiUniversalLlmOpGetResponseInputItemsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUniversalLlmOpGetResponseInputItems + name: 'policy' + properties: { + format: 'xml' + value: '@("Bearer " + (string)context.Variables["msi-access-token"])' + } +} + +// GET /unified-ai/deployments (static deployment inventory) +resource apiUnifiedAiOpGetDeploymentsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUnifiedAiOpGetDeployments + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"value":[{"id":"/deployments/gpt-4.1","name":"gpt-4.1","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-4.1","format":"OpenAI","version":"2025-04-14"},"deploymentName":"gpt-4-1","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.4-mini","name":"gpt-5.4-mini","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.4-mini","format":"OpenAI","version":"2026-03-17"},"deploymentName":"gpt-5-4-mini","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.2","name":"gpt-5.2","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.2","format":"OpenAI","version":"2025-12-11"},"deploymentName":"gpt-5-2","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/text-embedding-3-large","name":"text-embedding-3-large","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"text-embedding-3-large","format":"OpenAI","version":"1"},"deploymentName":"text-embedding-3-large","capabilities":{"embeddings":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Mistral-Large-3","name":"Mistral-Large-3","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"Mistral-Large-3","format":"Mistral AI","version":"1"},"deploymentName":"Mistral-Large-3","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Phi-4","name":"Phi-4","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":1},"properties":{"model":{"name":"Phi-4","format":"Microsoft","version":"7"},"deploymentName":"Phi-4","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}}]}' + } +} + +// GET /unified-ai/deployments/{model_name} (404 when unknown) +resource apiUnifiedAiOpGetDeploymentPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUnifiedAiOpGetDeployment + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":{"code":"DeploymentNotFound","message":"The specified deployment was not found."}}application/json@{ var n = context.Request.MatchedParameters["model_name"]; string fmt = "OpenAI"; if (n == "Mistral-Large-3") { fmt = "Mistral AI"; } if (n == "Phi-4") { fmt = "Microsoft"; } var m = new JObject(); m["name"] = n; m["format"] = fmt; var o = new JObject(); o["id"] = "/deployments/" + n; o["name"] = n; o["type"] = "Microsoft.CognitiveServices/accounts/deployments"; o["properties"] = new JObject(new JProperty("model", m)); return o.ToString(); }' + } +} + +// POST /unified-ai/openai/deployments/{model}/chat/completions +resource apiUnifiedAiOpOpenAiCompletionsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUnifiedAiOpOpenAiCompletions + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":"unsupported model requested"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryazure-openai@(context.Request.MatchedParameters["model"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())' + } +} + +// POST /unified-ai/models/chat/completions (model in body) +resource apiUnifiedAiOpFoundryCompletionsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUnifiedAiOpFoundryCompletions + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":"unsupported model requested for chat completions"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryfoundry-inference@((string)context.Variables["requestedModel"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())' + } +} + +// POST /unified-ai/v1beta/openai/chat/completions (no Gemini backend in this lab) +resource apiUnifiedAiOpGeminiCompletionsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUnifiedAiOpGeminiCompletions + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":{"code":"BackendNotConfigured","message":"No Gemini backend is provisioned in this MicroHack lab. Onboard one in challenge 1 to enable this path."}}' + } +} + // ===== Phase 4: APIM Products (Access Contracts) ===== // Product 1: LLM-Sales-Assistant-DEV resource productSalesAssistant 'Microsoft.ApiManagement/service/products@2023-09-01-preview' = { diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json index 9068efbc1..87122d678 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.41.2.15936", - "templateHash": "9296888080003196804" + "templateHash": "15272192605124707672" } }, "parameters": { @@ -345,7 +345,7 @@ "apiVersion": "2023-09-01-preview", "name": "[format('{0}/{1}', variables('apimName'), 'uami-client-id')]", "properties": { - "displayName": "Hub.Foundry.UAMI.Client.ID", + "displayName": "uami-client-id", "value": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), '2023-01-31').clientId]", "secret": false }, @@ -480,7 +480,7 @@ "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'post-chat-completions', 'policy')]", "properties": { "format": "xml", - "value": "@(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])application/json@{return new JObject(new JProperty(\"error\",\"unsupported model requested for chat completions\"),new JProperty(\"requestedModel\",(string)context.Variables[\"requestedModel\"])).ToString();}" + "value": "@(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])application/json@{return new JObject(new JProperty(\"error\",\"unsupported model requested for chat completions\"),new JProperty(\"requestedModel\",(string)context.Variables[\"requestedModel\"])).ToString();}" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service/backends', variables('apimName'), 'backend-hub-foundry')]", @@ -508,7 +508,7 @@ "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'post-embeddings', 'policy')]", "properties": { "format": "xml", - "value": "@(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])application/json@{return new JObject(new JProperty(\"error\",\"unsupported model requested for embeddings\"),new JProperty(\"requestedModel\",(string)context.Variables[\"requestedModel\"])).ToString();}" + "value": "@(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])application/json@{return new JObject(new JProperty(\"error\",\"unsupported model requested for embeddings\"),new JProperty(\"requestedModel\",(string)context.Variables[\"requestedModel\"])).ToString();}" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service/backends', variables('apimName'), 'backend-hub-foundry')]", @@ -681,6 +681,102 @@ "[resourceId('Microsoft.ApiManagement/service/apis', variables('apimName'), 'unified-ai-api')]" ] }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'post-responses', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":\"unsupported model requested for responses\"}@("Bearer " + (string)context.Variables["msi-access-token"])@{var body = context.Request.Body.As<JObject>(preserveContent: true); string m = (string)body["model"]; if (string.IsNullOrEmpty(m)) { m = "gpt-4.1"; } body["model"] = m.Replace(".", "-"); return body.ToString();}" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'post-responses')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'get-response', 'policy')]", + "properties": { + "format": "xml", + "value": "@("Bearer " + (string)context.Variables["msi-access-token"])" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'get-response')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'get-response-input-items', 'policy')]", + "properties": { + "format": "xml", + "value": "@("Bearer " + (string)context.Variables["msi-access-token"])" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'get-response-input-items')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'unified-ai-api', 'get-deployments', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"value\":[{\"id\":\"/deployments/gpt-4.1\",\"name\":\"gpt-4.1\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-4.1\",\"format\":\"OpenAI\",\"version\":\"2025-04-14\"},\"deploymentName\":\"gpt-4-1\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.4-mini\",\"name\":\"gpt-5.4-mini\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.4-mini\",\"format\":\"OpenAI\",\"version\":\"2026-03-17\"},\"deploymentName\":\"gpt-5-4-mini\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.2\",\"name\":\"gpt-5.2\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.2\",\"format\":\"OpenAI\",\"version\":\"2025-12-11\"},\"deploymentName\":\"gpt-5-2\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/text-embedding-3-large\",\"name\":\"text-embedding-3-large\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"text-embedding-3-large\",\"format\":\"OpenAI\",\"version\":\"1\"},\"deploymentName\":\"text-embedding-3-large\",\"capabilities\":{\"embeddings\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Mistral-Large-3\",\"name\":\"Mistral-Large-3\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"Mistral-Large-3\",\"format\":\"Mistral AI\",\"version\":\"1\"},\"deploymentName\":\"Mistral-Large-3\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Phi-4\",\"name\":\"Phi-4\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":1},\"properties\":{\"model\":{\"name\":\"Phi-4\",\"format\":\"Microsoft\",\"version\":\"7\"},\"deploymentName\":\"Phi-4\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}}]}" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'get-deployments')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'unified-ai-api', 'get-deployment', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":{\"code\":\"DeploymentNotFound\",\"message\":\"The specified deployment was not found.\"}}application/json@{ var n = context.Request.MatchedParameters[\"model_name\"]; string fmt = \"OpenAI\"; if (n == \"Mistral-Large-3\") { fmt = \"Mistral AI\"; } if (n == \"Phi-4\") { fmt = \"Microsoft\"; } var m = new JObject(); m[\"name\"] = n; m[\"format\"] = fmt; var o = new JObject(); o[\"id\"] = \"/deployments/\" + n; o[\"name\"] = n; o[\"type\"] = \"Microsoft.CognitiveServices/accounts/deployments\"; o[\"properties\"] = new JObject(new JProperty(\"model\", m)); return o.ToString(); }" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'get-deployment')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'unified-ai-api', 'post-openai-chat-completions', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":\"unsupported model requested\"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryazure-openai@(context.Request.MatchedParameters["model"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'post-openai-chat-completions')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'unified-ai-api', 'post-foundry-chat-completions', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":\"unsupported model requested for chat completions\"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryfoundry-inference@((string)context.Variables["requestedModel"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'post-foundry-chat-completions')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'unified-ai-api', 'post-gemini-chat-completions', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":{\"code\":\"BackendNotConfigured\",\"message\":\"No Gemini backend is provisioned in this MicroHack lab. Onboard one in challenge 1 to enable this path.\"}}" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'post-gemini-chat-completions')]" + ] + }, { "type": "Microsoft.ApiManagement/service/products", "apiVersion": "2023-09-01-preview", From 667580e46859885fb4fa27070fd16d4a8e0aeaa6 Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:36:53 +0200 Subject: [PATCH 04/11] Add Azure OpenAI API surface required by challenge 1 Challenge 1 calls apimClientTool.discover_api("openai") to find an API served on the 'openai' path, then uses it for three test sections: the Azure OpenAI request format, the AzureOpenAI Python SDK, and streaming. No such API existed, so discovery raised, azure_endpoint_openai was set to None, and all three sections silently skipped. Challenge 1 also probes GET /openai/deployments. Add an azure-openai-api (path 'openai') with two operations and policies: - GET /deployments returns the deployment inventory. - POST /deployments/{model}/chat/completions maps the model to its deployment name, rejects unsupported models with 400, and passes through SSE so the streaming section works. Products deliberately do not include this API. Challenge 1 authenticates with the master subscription, so it has access, while product-to-API membership stays governed by the access contracts in challenge 3 - those contracts pin apiNameMapping to universal-llm-api and broadening it would weaken the isolation that challenge 3 asserts. Verified live against a lab resource group: deployment listing, a chat completion, and a streaming SSE response. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../labautomation/infra/resources.bicep | 75 +++++++++++++++++ .../labautomation/infra/resources.json | 83 ++++++++++++++++++- 2 files changed, 157 insertions(+), 1 deletion(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep index 723986d3e..bac0614af 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep @@ -819,6 +819,81 @@ resource apiUnifiedAiOpGeminiCompletionsPolicy 'Microsoft.ApiManagement/service/ } } + +// ===== Phase 4: APIM API - Azure OpenAI API ===== +// Azure OpenAI compatible surface (deployment name in the URL path). Challenge 1 +// discovers an API on the `openai` path to exercise the Azure OpenAI request +// format, the AzureOpenAI Python SDK and streaming; without it those cells skip. +// Products intentionally do not include this API - it is reached with the master +// subscription, leaving product membership governed by the challenge 3 contracts. +resource apiAzureOpenAi 'Microsoft.ApiManagement/service/apis@2023-09-01-preview' = { + parent: apim + name: 'azure-openai-api' + properties: { + displayName: 'Azure OpenAI API' + description: 'Azure OpenAI compatible surface (deployment name in URL path)' + path: 'openai' + protocols: [ + 'https' + ] + subscriptionRequired: true + subscriptionKeyParameterNames: { + header: 'api-key' + query: 'api-key' + } + } +} + +// Azure OpenAI API: Operation - GET /openai/deployments +resource apiAzureOpenAiOpGetDeployments 'Microsoft.ApiManagement/service/apis/operations@2023-09-01-preview' = { + parent: apiAzureOpenAi + name: 'get-deployments' + properties: { + displayName: 'List Deployments' + method: 'GET' + urlTemplate: '/deployments' + description: 'Returns available model deployments' + } +} + +resource apiAzureOpenAiOpGetDeploymentsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiAzureOpenAiOpGetDeployments + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"value":[{"id":"/deployments/gpt-4.1","name":"gpt-4.1","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-4.1","format":"OpenAI","version":"2025-04-14"},"deploymentName":"gpt-4-1","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.4-mini","name":"gpt-5.4-mini","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.4-mini","format":"OpenAI","version":"2026-03-17"},"deploymentName":"gpt-5-4-mini","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.2","name":"gpt-5.2","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.2","format":"OpenAI","version":"2025-12-11"},"deploymentName":"gpt-5-2","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/text-embedding-3-large","name":"text-embedding-3-large","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"text-embedding-3-large","format":"OpenAI","version":"1"},"deploymentName":"text-embedding-3-large","capabilities":{"embeddings":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Mistral-Large-3","name":"Mistral-Large-3","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"Mistral-Large-3","format":"Mistral AI","version":"1"},"deploymentName":"Mistral-Large-3","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Phi-4","name":"Phi-4","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":1},"properties":{"model":{"name":"Phi-4","format":"Microsoft","version":"7"},"deploymentName":"Phi-4","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}}]}' + } +} + +// Azure OpenAI API: Operation - POST /openai/deployments/{model}/chat/completions +resource apiAzureOpenAiOpChatCompletions 'Microsoft.ApiManagement/service/apis/operations@2023-09-01-preview' = { + parent: apiAzureOpenAi + name: 'post-chat-completions' + properties: { + displayName: 'Create Chat Completion' + method: 'POST' + urlTemplate: '/deployments/{model}/chat/completions' + templateParameters: [ + { + name: 'model' + required: true + type: 'string' + description: 'Deployment/model name' + } + ] + description: 'Azure OpenAI style chat completions (supports streaming)' + } +} + +resource apiAzureOpenAiOpChatCompletionsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiAzureOpenAiOpChatCompletions + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"error":"unsupported model requested"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryazure-openai@(context.Request.MatchedParameters["model"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())' + } +} + // ===== Phase 4: APIM Products (Access Contracts) ===== // Product 1: LLM-Sales-Assistant-DEV resource productSalesAssistant 'Microsoft.ApiManagement/service/products@2023-09-01-preview' = { diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json index 87122d678..78d3c1939 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.41.2.15936", - "templateHash": "15272192605124707672" + "templateHash": "2462901709900521111" } }, "parameters": { @@ -777,6 +777,87 @@ "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'unified-ai-api', 'post-gemini-chat-completions')]" ] }, + { + "type": "Microsoft.ApiManagement/service/apis", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'azure-openai-api')]", + "properties": { + "displayName": "Azure OpenAI API", + "description": "Azure OpenAI compatible surface (deployment name in URL path)", + "path": "openai", + "protocols": [ + "https" + ], + "subscriptionRequired": true, + "subscriptionKeyParameterNames": { + "header": "api-key", + "query": "api-key" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}', variables('apimName'), 'azure-openai-api', 'get-deployments')]", + "properties": { + "displayName": "List Deployments", + "method": "GET", + "urlTemplate": "/deployments", + "description": "Returns available model deployments" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', variables('apimName'), 'azure-openai-api')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'azure-openai-api', 'get-deployments', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"value\":[{\"id\":\"/deployments/gpt-4.1\",\"name\":\"gpt-4.1\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-4.1\",\"format\":\"OpenAI\",\"version\":\"2025-04-14\"},\"deploymentName\":\"gpt-4-1\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.4-mini\",\"name\":\"gpt-5.4-mini\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.4-mini\",\"format\":\"OpenAI\",\"version\":\"2026-03-17\"},\"deploymentName\":\"gpt-5-4-mini\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.2\",\"name\":\"gpt-5.2\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.2\",\"format\":\"OpenAI\",\"version\":\"2025-12-11\"},\"deploymentName\":\"gpt-5-2\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/text-embedding-3-large\",\"name\":\"text-embedding-3-large\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"text-embedding-3-large\",\"format\":\"OpenAI\",\"version\":\"1\"},\"deploymentName\":\"text-embedding-3-large\",\"capabilities\":{\"embeddings\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Mistral-Large-3\",\"name\":\"Mistral-Large-3\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"Mistral-Large-3\",\"format\":\"Mistral AI\",\"version\":\"1\"},\"deploymentName\":\"Mistral-Large-3\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Phi-4\",\"name\":\"Phi-4\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":1},\"properties\":{\"model\":{\"name\":\"Phi-4\",\"format\":\"Microsoft\",\"version\":\"7\"},\"deploymentName\":\"Phi-4\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}}]}" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'azure-openai-api', 'get-deployments')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}', variables('apimName'), 'azure-openai-api', 'post-chat-completions')]", + "properties": { + "displayName": "Create Chat Completion", + "method": "POST", + "urlTemplate": "/deployments/{model}/chat/completions", + "templateParameters": [ + { + "name": "model", + "required": true, + "type": "string", + "description": "Deployment/model name" + } + ], + "description": "Azure OpenAI style chat completions (supports streaming)" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', variables('apimName'), 'azure-openai-api')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'azure-openai-api', 'post-chat-completions', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"error\":\"unsupported model requested\"}@("Bearer " + (string)context.Variables["msi-access-token"])backend-hub-foundryazure-openai@(context.Request.MatchedParameters["model"])@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("event-stream").ToString().ToLower())" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'azure-openai-api', 'post-chat-completions')]" + ] + }, { "type": "Microsoft.ApiManagement/service/products", "apiVersion": "2023-09-01-preview", From 630ddd54daa2d0e34e948092635e8e3edce72c66 Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:37:55 +0200 Subject: [PATCH 05/11] Expose GET /deployments on the Universal LLM API Challenge 1 runs test_get_deployments against both the Universal LLM API and the Azure OpenAI API, but only the latter served /deployments, so the first call returned 404. Add the operation to the Universal LLM API with the same inventory policy, so both surfaces answer. Verified live: GET /models/deployments returns 200 with all six deployments. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../labautomation/infra/resources.bicep | 24 ++++++++++++++++ .../labautomation/infra/resources.json | 28 ++++++++++++++++++- 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep index bac0614af..d342226da 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep @@ -637,6 +637,30 @@ resource apiUniversalLlmOpGetResponseInputItems 'Microsoft.ApiManagement/service } } + +// Universal LLM API: Operation - GET /models/deployments +// Challenge 1 tests /deployments against both the Universal LLM API and the +// Azure OpenAI API, so both surfaces expose the deployment inventory. +resource apiUniversalLlmOpGetDeployments 'Microsoft.ApiManagement/service/apis/operations@2023-09-01-preview' = { + parent: apiUniversalLlm + name: 'get-deployments' + properties: { + displayName: 'List Deployments' + method: 'GET' + urlTemplate: '/deployments' + description: 'Returns available model deployments' + } +} + +resource apiUniversalLlmOpGetDeploymentsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { + parent: apiUniversalLlmOpGetDeployments + name: 'policy' + properties: { + format: 'xml' + value: 'application/json{"value":[{"id":"/deployments/gpt-4.1","name":"gpt-4.1","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-4.1","format":"OpenAI","version":"2025-04-14"},"deploymentName":"gpt-4-1","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.4-mini","name":"gpt-5.4-mini","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.4-mini","format":"OpenAI","version":"2026-03-17"},"deploymentName":"gpt-5-4-mini","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/gpt-5.2","name":"gpt-5.2","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"gpt-5.2","format":"OpenAI","version":"2025-12-11"},"deploymentName":"gpt-5-2","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/text-embedding-3-large","name":"text-embedding-3-large","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"text-embedding-3-large","format":"OpenAI","version":"1"},"deploymentName":"text-embedding-3-large","capabilities":{"embeddings":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Mistral-Large-3","name":"Mistral-Large-3","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":20},"properties":{"model":{"name":"Mistral-Large-3","format":"Mistral AI","version":"1"},"deploymentName":"Mistral-Large-3","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}},{"id":"/deployments/Phi-4","name":"Phi-4","type":"Microsoft.CognitiveServices/accounts/deployments","sku":{"name":"GlobalStandard","capacity":1},"properties":{"model":{"name":"Phi-4","format":"Microsoft","version":"7"},"deploymentName":"Phi-4","capabilities":{"chatCompletion":"true"},"provisioningState":"Succeeded"}}]}' + } +} + // ===== Phase 4: APIM API - Unified AI API ===== // Multi-backend routing API for OpenAI, Foundry, Gemini formats resource apiUnifiedAi 'Microsoft.ApiManagement/service/apis@2023-09-01-preview' = { diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json index 78d3c1939..d52bee429 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.41.2.15936", - "templateHash": "2462901709900521111" + "templateHash": "1059607902515208970" } }, "parameters": { @@ -574,6 +574,32 @@ "[resourceId('Microsoft.ApiManagement/service/apis', variables('apimName'), 'universal-llm-api')]" ] }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}', variables('apimName'), 'universal-llm-api', 'get-deployments')]", + "properties": { + "displayName": "List Deployments", + "method": "GET", + "urlTemplate": "/deployments", + "description": "Returns available model deployments" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', variables('apimName'), 'universal-llm-api')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'get-deployments', 'policy')]", + "properties": { + "format": "xml", + "value": "application/json{\"value\":[{\"id\":\"/deployments/gpt-4.1\",\"name\":\"gpt-4.1\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-4.1\",\"format\":\"OpenAI\",\"version\":\"2025-04-14\"},\"deploymentName\":\"gpt-4-1\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.4-mini\",\"name\":\"gpt-5.4-mini\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.4-mini\",\"format\":\"OpenAI\",\"version\":\"2026-03-17\"},\"deploymentName\":\"gpt-5-4-mini\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/gpt-5.2\",\"name\":\"gpt-5.2\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"gpt-5.2\",\"format\":\"OpenAI\",\"version\":\"2025-12-11\"},\"deploymentName\":\"gpt-5-2\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/text-embedding-3-large\",\"name\":\"text-embedding-3-large\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"text-embedding-3-large\",\"format\":\"OpenAI\",\"version\":\"1\"},\"deploymentName\":\"text-embedding-3-large\",\"capabilities\":{\"embeddings\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Mistral-Large-3\",\"name\":\"Mistral-Large-3\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":20},\"properties\":{\"model\":{\"name\":\"Mistral-Large-3\",\"format\":\"Mistral AI\",\"version\":\"1\"},\"deploymentName\":\"Mistral-Large-3\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}},{\"id\":\"/deployments/Phi-4\",\"name\":\"Phi-4\",\"type\":\"Microsoft.CognitiveServices/accounts/deployments\",\"sku\":{\"name\":\"GlobalStandard\",\"capacity\":1},\"properties\":{\"model\":{\"name\":\"Phi-4\",\"format\":\"Microsoft\",\"version\":\"7\"},\"deploymentName\":\"Phi-4\",\"capabilities\":{\"chatCompletion\":\"true\"},\"provisioningState\":\"Succeeded\"}}]}" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'get-deployments')]" + ] + }, { "type": "Microsoft.ApiManagement/service/apis", "apiVersion": "2023-09-01-preview", From 7c9c14acc47fb6d7fefcce2305064c244258b880 Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:43:04 +0200 Subject: [PATCH 06/11] Fall back to APIM credentials when the Key Vault data plane is blocked Challenge 4 builds its Sales agent by reading the endpoint and API key from Key Vault with SecretClient. That is a data-plane call, and the lab tenant enforces an Azure Policy (KeyVault_PublicNetwork_Modify) that sets publicNetworkAccess to Disabled on every vault. Setting it back to Enabled is reverted within seconds, so the read fails for every participant and the agent never starts. Challenge 3 is unaffected: it writes its secrets through the ARM control plane, which the vault firewall does not gate. Only reads are blocked, and ARM does not return secret values, so there is no control-plane way to read them back. Give the agent optional fallback_endpoint and fallback_api_key arguments and use them when the Key Vault read raises. The access contract still creates and populates the vault, so the pattern is demonstrated, but the agent falls back to the same gateway endpoint and contract subscription key taken from APIM. The source is reported so participants can see which path was used. Verified live: the endpoint and key the fallback builds return 200 from /models/chat/completions using the Sales contract subscription. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../4. citadel-agent-frameworks-tests.ipynb | 34 +++++++++++++++++-- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/4. citadel-agent-frameworks-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/4. citadel-agent-frameworks-tests.ipynb index 2057d23b1..d6c6fc5f0 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/4. citadel-agent-frameworks-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/4. citadel-agent-frameworks-tests.ipynb @@ -515,7 +515,13 @@ " Azure API Management gateways that expect the 'api-key' header.\n", " \"\"\"\n", " \n", - " def __init__(self, keyvault_name: str, endpoint_secret_name: str, apikey_secret_name: str, model_name: str):\n", + " def __init__(self, keyvault_name: str, endpoint_secret_name: str, apikey_secret_name: str, model_name: str,\n", + " fallback_endpoint: str = None, fallback_api_key: str = None):\n", + " # MicroHack: the fallback_* values are used when the Key Vault data plane\n", + " # is unreachable. See _init_from_keyvault for why that happens here.\n", + " self.fallback_endpoint = fallback_endpoint\n", + " self.fallback_api_key = fallback_api_key\n", + " self.credential_source = None\n", " self.keyvault_name = keyvault_name\n", " self.endpoint_secret_name = endpoint_secret_name\n", " self.apikey_secret_name = apikey_secret_name\n", @@ -562,7 +568,26 @@ " self.api_key = apikey_secret.value\n", " utils.print_ok(f\"Retrieved API key ({self.api_key[:4]}****) from Key Vault secret: {self.apikey_secret_name}\")\n", " \n", + " self.credential_source = \"key-vault\"\n", + "\n", " except Exception as e:\n", + " # MicroHack: the lab tenant enforces an Azure Policy\n", + " # (KeyVault_PublicNetwork_Modify) that sets publicNetworkAccess to\n", + " # Disabled on every vault, so this data-plane read fails from a\n", + " # participant machine even though the access contract wrote the\n", + " # secrets successfully over the ARM control plane.\n", + " # The access contract still demonstrates the Key Vault pattern; here\n", + " # we fall back to the same endpoint and subscription key read\n", + " # straight from APIM so the agent can still run.\n", + " if self.fallback_endpoint and self.fallback_api_key:\n", + " utils.print_warning(f\"Key Vault unreachable ({type(e).__name__}); using APIM credentials instead.\")\n", + " self.endpoint = self.fallback_endpoint\n", + " self.api_key = self.fallback_api_key\n", + " self.credential_source = \"apim-fallback\"\n", + " utils.print_ok(f\"Using endpoint: ({self.endpoint}) from APIM\")\n", + " utils.print_ok(f\"Using API key ({self.api_key[:4]}****) from the access contract subscription\")\n", + " return\n", + "\n", " utils.print_error(f\"Failed to retrieve secrets from Key Vault: {e}\")\n", " raise\n", " \n", @@ -736,7 +761,10 @@ " keyvault_name=keyvault_name,\n", " endpoint_secret_name=config.get(\"endpoint_secret\", \"SALES-LLM-ENDPOINT\"),\n", " apikey_secret_name=config.get(\"apikey_secret\", \"SALES-LLM-KEY\"),\n", - " model_name=sales_model_name\n", + " model_name=sales_model_name,\n", + " # MicroHack: used only if the Key Vault data plane is blocked.\n", + " fallback_endpoint=f\"{apim_resource_gateway_url.rstrip('/')}/{targetInferenceApi}\",\n", + " fallback_api_key=contract_keys[\"Sales-Assistant\"][\"key\"],\n", " )\n", " \n", " # Simulate a sales conversation\n", @@ -760,7 +788,7 @@ " \n", " except Exception as e:\n", " utils.print_error(f\"Failed to initialize Sales Agent: {e}\")\n", - " utils.print_info(\"Make sure Key Vault secrets are configured and you have access permissions.\")\n", + " utils.print_info(\"Make sure the access contract from challenge 3 was deployed and that its APIM subscription exists.\")\n", "else:\n", " utils.print_warning(\"Sales-Assistant contract not found. Skipping Microsoft Agent Framework test.\")" ] From 1e4be14aa33dc15ea9a759e9116aad0d86253d82 Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Tue, 1 Sep 2026 22:43:37 +0200 Subject: [PATCH 07/11] Document the Key Vault fallback in challenge 4 Explain that the Key Vault warning is expected in this environment and that the run continues on APIM credentials, so participants do not read it as a failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- 03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-04.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-04.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-04.md index 98a43a1c8..1769178e5 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-04.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-04.md @@ -13,7 +13,7 @@ across frameworks. | Access Contract | Agent Framework | Integration | Target model | |---|---|---|---| -| Sales-Assistant | Microsoft Agent Framework | Azure Key Vault (endpoint + key) | gpt-4.1 | +| Sales-Assistant | Microsoft Agent Framework | Azure Key Vault (endpoint + key), with an APIM fallback ā€” see Troubleshooting | gpt-4.1 | | HR-ChatAgent | Microsoft Foundry Agent SDK | Foundry Project Connection | gpt-4.1 | | Support-Bot | LangChain | Local (direct endpoint + key) | phi-4 | @@ -60,6 +60,7 @@ Same prerequisite as every notebook in this MicroHack ā€” see | Symptom | Fix | |---------|-----| | API key retrieval fails | This notebook does not create access contracts ā€” run [Challenge 3](challenge-03.md) first. | +| `Key Vault unreachable ā€¦ using APIM credentials instead` | Expected in this environment, and not an error. The lab tenant enforces an Azure Policy that disables public network access on every Key Vault, so the Sales agent cannot read its secrets from the vault data plane. Challenge 3 still writes those secrets successfully, because that goes through the ARM control plane. The agent falls back to the same endpoint and contract subscription key read from APIM, so the run continues normally. | | Foundry SDK connection step fails | Confirm the Foundry connection name printed at the end of Challenge 3 (`4ļøāƒ£.2`) matches what this notebook resolves. | | Package install errors in step `4ļøāƒ£` | Some agent framework packages (`agent-framework`, `langchain`, `langchain-openai`) pin specific versions ā€” re-run the cell after resolving any dependency conflict shown in the error. | From efcbb0794e5fd449306adf9397e3a8b238a796ce Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Wed, 2 Sep 2026 14:50:09 +0200 Subject: [PATCH 08/11] Enable real PII policy fragments in lab deployment Challenge 5 deploys access contracts whose product policy includes pii-anonymization, pii-deanonymization and pii-state-saving. All three were placeholders commented out as DISABLED, so the contract deployment failed with "Policy fragment with id 'pii-anonymization' could not be found" and no PII was ever masked. The notebook's masking tests count HTTP 200 as success, so they reported 6/6 passed while the model was receiving unmasked names and SSNs. Load the real fragments from the workshop sources via loadTextContent so the two copies cannot drift, and add the three things they depend on: - named value piiServiceUrl pointing at the hub account's cognitiveservices host (the openai.azure.com host does not serve the Language PII API) - Cognitive Services User on the hub account for the APIM identity; the existing OpenAI User role does not cover the Language service - pii-usage-eventhub-logger plus Azure Event Hubs Data Sender, required by log-to-eventhub in pii-state-saving The logger must exist before the fragment: APIM accepts a fragment referencing a missing logger, reports InProgress, then discards it without surfacing an error anywhere. Verified against a live lab: the blocking contract now returns HTTP 400 with code PII_DETECTED and categories USSocialSecurityNumber, Person for a request containing PII, and 200 for one without. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../labautomation/infra/resources.bicep | 91 +++++++++++++-- .../labautomation/infra/resources.json | 110 +++++++++++++++++- 2 files changed, 189 insertions(+), 12 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep index d342226da..ac20a5341 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep @@ -332,6 +332,19 @@ resource apimNamedValueUamiClientId 'Microsoft.ApiManagement/service/namedValues } } +// Endpoint the PII policy fragments call for Language/PII detection. The hub +// AIServices account exposes the Language PII API on its cognitiveservices host +// (note: not the openai.azure.com host used for inference). +resource namedValuePiiServiceUrl 'Microsoft.ApiManagement/service/namedValues@2023-09-01-preview' = { + parent: apim + name: 'piiserviceurl' + properties: { + displayName: 'piiServiceUrl' + value: 'https://${hubFoundryAccountName}.cognitiveservices.azure.com' + secret: false + } +} + // ===== Phase 4: APIM Backends ===== // Hub Foundry backend with managed identity resource apimBackendHubFoundry 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = { @@ -371,6 +384,29 @@ resource roleAssignmentApimMiOpenAiUser 'Microsoft.Authorization/roleAssignments } } +// RBAC: the PII fragments call the Language service on the same account, which is +// gated by Cognitive Services User rather than the OpenAI-specific role above. +resource roleAssignmentApimMiCognitiveServicesUser 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(hubFoundryAccount.id, apimMI.id, 'CognitiveServicesUser') + scope: hubFoundryAccount + properties: { + principalId: apimMI.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'a97b65f3-24c7-4388-baec-2e87135dc908') + } +} + +// RBAC: the Event Hub logger below authenticates as the APIM managed identity. +resource roleAssignmentApimMiEventHubSender 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(eventHubNamespace_resource.id, apimMI.id, 'AzureEventHubsDataSender') + scope: eventHubNamespace_resource + properties: { + principalId: apimMI.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '2b629674-e913-4c01-ae53-ef4638d8f975') + } +} + // ===== Phase 4: APIM Policy Fragments ===== // Policy fragment: Extract available models from deployment list resource policyFragmentGetAvailableModels 'Microsoft.ApiManagement/service/policyFragments@2023-09-01-preview' = { @@ -454,37 +490,49 @@ resource policyFragmentSetLlmUsage 'Microsoft.ApiManagement/service/policyFragme } */ -// Policy fragment: PII Anonymization (DISABLED) -/* +// ===== Phase 4: PII policy fragments (challenge 5) ===== +// These are the real fragments from the workshop, not placeholders. Challenge 5 +// deploys access contracts whose product policy includes all three by id, so if +// they are absent the contract deployment fails outright with +// "Policy fragment with id 'pii-anonymization' could not be found". +// They are loaded from the workshop sources so the two copies cannot drift. resource policyFragmentPiiAnonymization 'Microsoft.ApiManagement/service/policyFragments@2023-09-01-preview' = { parent: apim name: 'pii-anonymization' properties: { - format: 'xml' - value: '@{string body = context.Request.Body.AsString(); body = System.Text.RegularExpressions.Regex.Replace(body, @"\\b\\d{3}-\\d{2}-\\d{4}\\b", "***-**-****"); return body; }' + format: 'rawxml' + description: 'Detects PII with the Language service and masks it before the request reaches the model' + value: loadTextContent('../../challenges/bicep/infra/modules/apim/policies/frag-pii-anonymization.xml') } + dependsOn: [ + namedValuePiiServiceUrl + ] } -// Policy fragment: PII Deanonymization resource policyFragmentPiiDeanonymization 'Microsoft.ApiManagement/service/policyFragments@2023-09-01-preview' = { parent: apim name: 'pii-deanonymization' properties: { - format: 'xml' - value: '' + format: 'rawxml' + description: 'Restores the original PII values in the response returned to the caller' + value: loadTextContent('../../challenges/bicep/infra/modules/apim/policies/frag-pii-deanonymization.xml') } } -// Policy fragment: PII State Saving resource policyFragmentPiiStateSaving 'Microsoft.ApiManagement/service/policyFragments@2023-09-01-preview' = { parent: apim name: 'pii-state-saving' properties: { - format: 'xml' - value: 'https://${cosmosAccountName}.documents.azure.com/dbs/${cosmosDatabaseName}/colls/pii-usage-container/docsPOST' + format: 'rawxml' + description: 'Emits PII handling events to Event Hub for audit' + value: loadTextContent('../../challenges/bicep/infra/modules/apim/policies/frag-pii-state-saving.xml') } + // The fragment uses log-to-eventhub with this logger id. Without the logger, + // APIM accepts the PUT and then discards the fragment, leaving no error behind. + dependsOn: [ + apimLoggerPiiUsageEventHub + ] } -*/ // ===== Phase 4: APIM API - Universal LLM API ===== // Universal endpoint for all LLM backends (/models/*) @@ -1265,6 +1313,27 @@ resource apimLoggerAppInsights 'Microsoft.ApiManagement/service/loggers@2023-09- } } +// Referenced by the pii-state-saving fragment. It must exist before that fragment +// is created, otherwise APIM accepts the fragment and then silently drops it. +resource apimLoggerPiiUsageEventHub 'Microsoft.ApiManagement/service/loggers@2023-09-01-preview' = { + parent: apim + name: 'pii-usage-eventhub-logger' + properties: { + loggerType: 'azureEventHub' + description: 'PII handling audit events' + credentials: { + endpointAddress: '${eventHubNamespace}.servicebus.windows.net' + identityClientId: apimMI.properties.clientId + name: eventHubName + } + isBuffered: true + } + dependsOn: [ + eventHub + roleAssignmentApimMiEventHubSender + ] +} + // ===== Spoke: Azure AI Foundry Account (AIServices) ===== resource spokeFoundryAccount 'Microsoft.CognitiveServices/accounts@2025-04-01-preview' = { name: spokeFoundryAccountName diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json index d52bee429..50503753f 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.41.2.15936", - "templateHash": "1059607902515208970" + "templateHash": "409383630776990329" } }, "parameters": { @@ -55,6 +55,9 @@ } } ], + "$fxv#0": "\n \n (\"piiAnonymizationEnabled\") == \"true\")\">\n \n (\"piiConfidenceThreshold\", \"0.8\")))\" />\n (\"piiEntityCategoryExclusions\", \"\"))\" />\n (\"piiDetectionLanguage\", \"en\"))\" />\n \n \n \n (\"piiInputContent\");\n var regexPatterns = context.Variables.GetValueOrDefault(\"piiRegexPatterns\");\n var mappings = new JArray();\n \n if (!string.IsNullOrEmpty(regexPatterns))\n {\n var patterns = JArray.Parse(regexPatterns);\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n \n foreach (var pattern in patterns)\n {\n var regex = new Regex(pattern[\"pattern\"].ToString());\n var category = pattern[\"category\"].ToString();\n \n if (!categoryCounts.ContainsKey(category))\n {\n categoryCounts[category] = 0;\n }\n \n var matches = regex.Matches(content);\n foreach (Match match in matches)\n {\n if (!string.IsNullOrEmpty(match.Value))\n {\n // Skip if we already have a mapping for this text\n if (processedTexts.Contains(match.Value))\n {\n continue;\n }\n \n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = match.Value,\n [\"placeholder\"] = placeholder,\n [\"source\"] = \"regex\"\n });\n categoryCounts[category]++;\n processedTexts.Add(match.Value); // Mark this text as processed\n }\n }\n }\n }\n return mappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiRegexMappings\"));\n \n foreach (var mapping in mappings)\n {\n content = content.Replace(\n mapping[\"original\"].ToString(),\n mapping[\"placeholder\"].ToString()\n );\n }\n return content;\n }\" />\n\n \n (\"piiRegexProcessedContent\"))\" />\n \n \n \n @{\n var baseUrl = \"{{piiServiceUrl}}\".TrimEnd('/');\n return baseUrl + \"/language/:analyze-text?api-version=2026-05-01\";\n }\n POST\n \n application/json\n \n \n @(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])\n \n \n\n @{\n var request = new JObject();\n request[\"kind\"] = \"PiiEntityRecognition\";\n request[\"parameters\"] = new JObject{\n {\"modelVersion\", \"latest\"},\n {\"redactionPolicy\", new JObject{\n {\"policyKind\", \"CharacterMask\"},\n {\"redactionCharacter\", \"#\"}\n }}\n };\n request[\"analysisInput\"] = new JObject{\n {\"documents\", new JArray{\n new JObject{\n {\"text\", context.Variables.GetValueOrDefault(\"piiInputContent\")},\n {\"id\", \"1\"},\n {\"language\", context.Variables.GetValueOrDefault(\"piiDetectionLanguage\")}\n }\n }}\n };\n return request.ToString();\n }\n \n\n \n (\"piiAnalysisResponse\");\n if (response == null)\n {\n return \"PII service call failed: no response received (timeout or connectivity issue).\";\n }\n if (response.StatusCode < 200 || response.StatusCode >= 300)\n {\n var body = string.Empty;\n try { body = response.Body?.As(preserveContent: true) ?? string.Empty; } catch { }\n return $\"PII service returned non-success status {response.StatusCode} {response.StatusReason}. Body: {body}\";\n }\n try\n {\n var parsed = JObject.Parse(response.Body.As(preserveContent: true));\n var errors = parsed.SelectToken(\"$.results.errors\") as JArray;\n if (errors != null && errors.Count > 0)\n {\n return $\"PII service reported document errors: {errors.ToString(Newtonsoft.Json.Formatting.None)}\";\n }\n var entitiesToken = parsed.SelectToken(\"$.results.documents[0].entities\");\n if (entitiesToken == null)\n {\n return \"PII service response did not contain expected entities payload.\";\n }\n return string.Empty;\n }\n catch (Exception ex)\n {\n return $\"Failed to parse PII service response: {ex.Message}\";\n }\n }\" />\n\n \n (\"piiAnalysisFailureReason\")))\">\n \n \n @($\"PII anonymization degraded: {context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\")}\")\n \n \n \n (\"piiAnalysisResponse\"))?.StatusCode.ToString() ?? \"n/a\")\" />\n (\"continueOnNLPError\", \"false\"))\" />\n \n \n \n (\"continueOnNLPError\", \"false\").Equals(\"true\", StringComparison.OrdinalIgnoreCase) == false)\">\n \n \n \n application/json\n \n @{\n var error = new JObject {\n [\"error\"] = new JObject {\n [\"code\"] = \"PIIAnonymizationFailed\",\n [\"message\"] = \"The PII anonymization service is currently unavailable and 'continueOnNLPError' is disabled. Request was blocked to prevent unmasked content from reaching the backend.\",\n [\"details\"] = context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\"),\n [\"requestId\"] = context.RequestId.ToString()\n }\n };\n return error.ToString();\n }\n \n \n \n \n (\"piiRegexMappings\") ?? \"[]\")\" />\n \n \n \n \n \n (\"piiAnalysisResponse\");\n var confidenceThreshold = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\");\n var exclusions = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\")\n .Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)\n .Select(c => c.Trim())\n .ToList();\n\n var entitiesToken = JObject.Parse(response.Body.As())\n .SelectToken(\"$.results.documents[0].entities\");\n var entitiesArray = entitiesToken as JArray ?? new JArray();\n\n var entities = entitiesArray\n .Where(e => e[\"confidenceScore\"] != null && e[\"confidenceScore\"].Value() >= confidenceThreshold)\n .Where(e => !exclusions.Contains(e[\"category\"]?.ToString() ?? string.Empty))\n .OrderByDescending(e => (e[\"text\"]?.ToString() ?? string.Empty).Length);\n\n var mappings = new JArray();\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n\n foreach (var entity in entities) {\n var text = (entity[\"text\"]?.ToString() ?? string.Empty).TrimEnd('.', ',', ' ');\n if (string.IsNullOrEmpty(text)) { continue; }\n\n // Skip if this text has already been processed\n if (processedTexts.Contains(text)) {\n continue;\n }\n\n var category = entity[\"category\"].ToString();\n\n if (!categoryCounts.ContainsKey(category)) {\n categoryCounts[category] = 0;\n }\n\n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = text,\n [\"placeholder\"] = placeholder\n });\n\n categoryCounts[category]++;\n processedTexts.Add(text); // Mark this text as processed\n }\n\n return mappings.ToString();\n }\" />\n \n \n\n \n (\"piiMappings\") ?? \"[]\");\n var regexMappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiRegexMappings\") ?? \"[]\");\n \n foreach (var mapping in regexMappings)\n {\n apiMappings.Add(mapping);\n }\n \n return apiMappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(original, placeholder);\n }\n \n return content;\n }\" />\n \n \n \n (\"piiInputContent\"))\" />\n \n \n", + "$fxv#1": "\n \n (\"piiDeanonymizeContentInput\");\n \n // Pre-process content to replace common encoded symbols\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003c\", \"<\");\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003e\", \">\");\n \n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(placeholder, original);\n }\n return content;\n }\" />\n", + "$fxv#2": "\n \n \n (\"piiStateSavingEnabled\") == \"true\")\">\n \n \n \n \n \n \n \n \n \n \n (\"requestedModel\") ?? \"unknown\")\" />\n \n \n @{\n var eventData = new JObject();\n \n // Operation metadata\n eventData[\"id\"] = context.Variables.GetValueOrDefault(\"piiOperationId\");\n eventData[\"timestamp\"] = context.Variables.GetValueOrDefault(\"piiOperationTimestamp\");\n eventData[\"type\"] = \"PII_Processing\";\n eventData[\"targetDeployment\"] = context.Variables.GetValueOrDefault(\"targetDeployment\");\n // Enhanced API and subscription context with content\n eventData[\"context\"] = new JObject {\n [\"api\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiApiId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiApiName\"),\n [\"operation\"] = new JObject {\n [\"id\"] = context.Operation.Id ?? \"unknown\",\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiOperationName\"),\n [\"method\"] = context.Operation.Method ?? \"unknown\",\n [\"urlTemplate\"] = context.Operation.UrlTemplate ?? \"unknown\"\n },\n [\"content\"] = new JObject {\n [\"originalRequest\"] = context.Variables.GetValueOrDefault(\"piiInputContent\"),\n [\"anonymizedRequest\"] = context.Variables.GetValueOrDefault(\"piiAnonymizedContent\"),\n [\"originalResponse\"] = context.Variables.GetValueOrDefault(\"originalResponse\"),\n [\"deanonymizedResponse\"] = context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\")\n }\n },\n [\"product\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiProductId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiProductName\")\n },\n [\"subscription\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionName\")\n }\n };\n \n // Process information\n eventData[\"process\"] = new JObject {\n [\"anonymizationEnabled\"] = context.Variables.GetValueOrDefault(\"piiAnonymizationEnabled\") == \"true\",\n [\"confidenceThreshold\"] = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\", 0.8),\n [\"entityCategoryExclusions\"] = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\", \"\"),\n [\"deanonymizationPerformed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n };\n \n // Results summary\n var mappings = new JArray();\n if (context.Variables.ContainsKey(\"piiMappings\")) {\n try {\n mappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\"));\n eventData[\"entityCount\"] = mappings.Count;\n \n // Group entities by category for reporting\n var categories = new Dictionary();\n foreach (var mapping in mappings) {\n var placeholder = mapping[\"placeholder\"].ToString();\n var category = placeholder.TrimStart('<').Split('_')[0];\n \n if (!categories.ContainsKey(category)) {\n categories[category] = 0;\n }\n categories[category]++;\n }\n \n var categoryDetails = new JObject();\n foreach (var category in categories) {\n categoryDetails[category.Key] = category.Value;\n }\n eventData[\"entityCategories\"] = categoryDetails;\n \n // Include mappings for detailed audit (can be removed if too verbose)\n eventData[\"mappings\"] = mappings;\n }\n catch (Exception ex) {\n eventData[\"error\"] = \"Failed to process mappings: \" + ex.Message;\n }\n }\n \n // Content length metrics (not the actual content)\n if (context.Variables.ContainsKey(\"piiInputContent\")) {\n eventData[\"metrics\"] = new JObject {\n [\"inputContentLength\"] = context.Variables.GetValueOrDefault(\"piiInputContent\", \"\").Length\n };\n \n if (context.Variables.ContainsKey(\"piiAnonymizedContent\")) {\n eventData[\"metrics\"][\"anonymizedContentLength\"] = \n context.Variables.GetValueOrDefault(\"piiAnonymizedContent\", \"\").Length;\n }\n \n // Add deanonymization metrics\n if (context.Variables.ContainsKey(\"piiDeanonymizeContentInput\")) {\n eventData[\"metrics\"][\"deanonymizeInputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizeContentInput\", \"\").Length;\n }\n \n if (context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")) {\n eventData[\"metrics\"][\"deanonymizedOutputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\", \"\").Length;\n }\n }\n \n // Add deanonymization flow details\n eventData[\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"inputSource\"] = context.Variables.ContainsKey(\"piiDeanonymizeContentInput\") ? \"present\" : \"missing\",\n [\"status\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\") ? \"completed\" : \"not_performed\",\n [\"timestamp\"] = DateTime.UtcNow.ToString(\"o\")\n };\n \n // Track end-to-end flow\n eventData[\"processingFlow\"] = new JObject {\n [\"anonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiAnonymizedContent\"),\n [\"entityCount\"] = context.Variables.ContainsKey(\"piiMappings\") ? \n JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\")).Count : 0\n },\n [\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"success\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n }\n };\n \n return eventData.ToString();\n }\n \n \n\n", "hubFoundryAccountName": "[format('aif-hub-{0}', parameters('resourceToken'))]", "hubFoundryProjectName": "citadel-hub-project", "apimName": "[format('apim-citadel-{0}', parameters('resourceToken'))]", @@ -354,6 +357,19 @@ "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName'))]" ] }, + { + "type": "Microsoft.ApiManagement/service/namedValues", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'piiserviceurl')]", + "properties": { + "displayName": "piiServiceUrl", + "value": "[format('https://{0}.cognitiveservices.azure.com', variables('hubFoundryAccountName'))]", + "secret": false + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" + ] + }, { "type": "Microsoft.ApiManagement/service/backends", "apiVersion": "2023-09-01-preview", @@ -401,6 +417,36 @@ "[resourceId('Microsoft.CognitiveServices/accounts', variables('hubFoundryAccountName'))]" ] }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.CognitiveServices/accounts', variables('hubFoundryAccountName'))]", + "name": "[guid(resourceId('Microsoft.CognitiveServices/accounts', variables('hubFoundryAccountName')), resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), 'CognitiveServicesUser')]", + "properties": { + "principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), '2023-01-31').principalId]", + "principalType": "ServicePrincipal", + "roleDefinitionId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'a97b65f3-24c7-4388-baec-2e87135dc908')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName'))]", + "[resourceId('Microsoft.CognitiveServices/accounts', variables('hubFoundryAccountName'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.EventHub/namespaces', variables('eventHubNamespace'))]", + "name": "[guid(resourceId('Microsoft.EventHub/namespaces', variables('eventHubNamespace')), resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), 'AzureEventHubsDataSender')]", + "properties": { + "principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), '2023-01-31').principalId]", + "principalType": "ServicePrincipal", + "roleDefinitionId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '2b629674-e913-4c01-ae53-ef4638d8f975')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName'))]", + "[resourceId('Microsoft.EventHub/namespaces', variables('eventHubNamespace'))]" + ] + }, { "type": "Microsoft.ApiManagement/service/policyFragments", "apiVersion": "2023-09-01-preview", @@ -413,6 +459,47 @@ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" ] }, + { + "type": "Microsoft.ApiManagement/service/policyFragments", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'pii-anonymization')]", + "properties": { + "format": "rawxml", + "description": "Detects PII with the Language service and masks it before the request reaches the model", + "value": "[variables('$fxv#0')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]", + "[resourceId('Microsoft.ApiManagement/service/namedValues', variables('apimName'), 'piiserviceurl')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/policyFragments", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'pii-deanonymization')]", + "properties": { + "format": "rawxml", + "description": "Restores the original PII values in the response returned to the caller", + "value": "[variables('$fxv#1')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/policyFragments", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'pii-state-saving')]", + "properties": { + "format": "rawxml", + "description": "Emits PII handling events to Event Hub for audit", + "value": "[variables('$fxv#2')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]", + "[resourceId('Microsoft.ApiManagement/service/loggers', variables('apimName'), 'pii-usage-eventhub-logger')]" + ] + }, { "type": "Microsoft.ApiManagement/service/apis", "apiVersion": "2023-09-01-preview", @@ -1291,6 +1378,27 @@ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" ] }, + { + "type": "Microsoft.ApiManagement/service/loggers", + "apiVersion": "2023-09-01-preview", + "name": "[format('{0}/{1}', variables('apimName'), 'pii-usage-eventhub-logger')]", + "properties": { + "loggerType": "azureEventHub", + "description": "PII handling audit events", + "credentials": { + "endpointAddress": "[format('{0}.servicebus.windows.net', variables('eventHubNamespace'))]", + "identityClientId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), '2023-01-31').clientId]", + "name": "[variables('eventHubName')]" + }, + "isBuffered": true + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]", + "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName'))]", + "[resourceId('Microsoft.EventHub/namespaces/eventhubs', variables('eventHubNamespace'), variables('eventHubName'))]", + "[extensionResourceId(resourceId('Microsoft.EventHub/namespaces', variables('eventHubNamespace')), 'Microsoft.Authorization/roleAssignments', guid(resourceId('Microsoft.EventHub/namespaces', variables('eventHubNamespace')), resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('apimMIName')), 'AzureEventHubsDataSender'))]" + ] + }, { "type": "Microsoft.CognitiveServices/accounts", "apiVersion": "2025-04-01-preview", From 24dcab26b1fb067331fa94c1ec00d219b4eff64d Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Wed, 2 Sep 2026 14:52:54 +0200 Subject: [PATCH 09/11] Return list-models in the shape the notebooks parse and honour allowedModels GET /models/models returned a bare JSON array of model names built from a hardcoded list in the operation policy. Notebook 2 reads data.get('data') or data.get('value'), which raises AttributeError on a list, so discovery found zero models and every downstream model test was skipped. The hardcoded list also ignored each access contract's allowedModels, so every subscription saw all six models regardless of what its contract permitted. Delegate to the get-available-models fragment and wrap its output as { "object": "list", "data": [...] }. The lab's placeholder fragment now follows the same contract as the real one deployed by challenge 1 - set availableModels and availableFilteredModels, return nothing - so the operation behaves identically before and after that challenge runs. Both policies move to files loaded with loadTextContent, which keeps them readable and avoids hand-escaping XML inside Bicep strings. Verified live: master and the universal-llm contract see all 6 models, while llm-support-bot-dev-sub-01 correctly sees only gpt-4.1 and Mistral-Large-3. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../frag-get-available-models-lab.xml | 34 +++++++++++++++++++ .../policies/op-universal-llm-get-models.xml | 22 ++++++++++++ .../labautomation/infra/resources.bicep | 18 +++++++--- .../labautomation/infra/resources.json | 28 ++++++++------- 4 files changed, 85 insertions(+), 17 deletions(-) create mode 100644 03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/frag-get-available-models-lab.xml create mode 100644 03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/op-universal-llm-get-models.xml diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/frag-get-available-models-lab.xml b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/frag-get-available-models-lab.xml new file mode 100644 index 000000000..bd696e975 --- /dev/null +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/frag-get-available-models-lab.xml @@ -0,0 +1,34 @@ + + + + (allowedModelsStr.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries).Select(m => m.Trim()), StringComparer.OrdinalIgnoreCase); + JArray filteredModels = new JArray(); + foreach (var model in allModels) { + string modelName = model["name"]?.ToString() ?? ""; + if (allowedSet.Contains(modelName)) { filteredModels.Add(model); } + } + return filteredModels.ToString(); + }" /> + diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/op-universal-llm-get-models.xml b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/op-universal-llm-get-models.xml new file mode 100644 index 000000000..f6b8af922 --- /dev/null +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/policies/op-universal-llm-get-models.xml @@ -0,0 +1,22 @@ + + + + + + + + + application/json + + @{ + JArray models = JArray.Parse((string)context.Variables["availableFilteredModels"]); + return new JObject(new JProperty("object", "list"), new JProperty("data", models)).ToString(); + } + + + + + + diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep index ac20a5341..2b2fcf1b8 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.bicep @@ -413,8 +413,9 @@ resource policyFragmentGetAvailableModels 'Microsoft.ApiManagement/service/polic parent: apim name: 'get-available-models' properties: { - format: 'xml' - value: 'application/json@{var models = new JArray(); models.Add("gpt-4.1"); models.Add("gpt-5.4-mini"); models.Add("gpt-5.2"); models.Add("text-embedding-3-large"); models.Add("Mistral-Large-3"); models.Add("Phi-4"); return models.ToString();}' + format: 'rawxml' + description: 'Sets availableModels/availableFilteredModels for the list-models operation' + value: loadTextContent('policies/frag-get-available-models-lab.xml') } } @@ -567,14 +568,21 @@ resource apiUniversalLlmOpGetModels 'Microsoft.ApiManagement/service/apis/operat } // Universal LLM API: Operation policy - GET /models/models -// Simplest possible policy: static JSON list of the 6 deployed models, no backend call +// Delegates to the get-available-models fragment rather than hardcoding the list, +// so the response honours each access contract's allowedModels and stays correct +// once challenge 1 replaces the fragment with one generated from real deployments. +// Shape is the OpenAI list form { "object": "list", "data": [...] } that the +// notebooks parse; a bare array causes discovery to silently find zero models. resource apiUniversalLlmOpGetModelsPolicy 'Microsoft.ApiManagement/service/apis/operations/policies@2023-09-01-preview' = { parent: apiUniversalLlmOpGetModels name: 'policy' properties: { - format: 'xml' - value: 'application/json["gpt-4.1","gpt-5.4-mini","gpt-5.2","text-embedding-3-large","Mistral-Large-3","Phi-4"]' + format: 'rawxml' + value: loadTextContent('policies/op-universal-llm-get-models.xml') } + dependsOn: [ + policyFragmentGetAvailableModels + ] } // Universal LLM API: Operation - POST /models/chat/completions diff --git a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json index 50503753f..72d7da8cb 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json +++ b/03-Azure/01-04-AI/06_AI_Governance/labautomation/infra/resources.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.41.2.15936", - "templateHash": "409383630776990329" + "templateHash": "9591618861995214523" } }, "parameters": { @@ -55,9 +55,11 @@ } } ], - "$fxv#0": "\n \n (\"piiAnonymizationEnabled\") == \"true\")\">\n \n (\"piiConfidenceThreshold\", \"0.8\")))\" />\n (\"piiEntityCategoryExclusions\", \"\"))\" />\n (\"piiDetectionLanguage\", \"en\"))\" />\n \n \n \n (\"piiInputContent\");\n var regexPatterns = context.Variables.GetValueOrDefault(\"piiRegexPatterns\");\n var mappings = new JArray();\n \n if (!string.IsNullOrEmpty(regexPatterns))\n {\n var patterns = JArray.Parse(regexPatterns);\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n \n foreach (var pattern in patterns)\n {\n var regex = new Regex(pattern[\"pattern\"].ToString());\n var category = pattern[\"category\"].ToString();\n \n if (!categoryCounts.ContainsKey(category))\n {\n categoryCounts[category] = 0;\n }\n \n var matches = regex.Matches(content);\n foreach (Match match in matches)\n {\n if (!string.IsNullOrEmpty(match.Value))\n {\n // Skip if we already have a mapping for this text\n if (processedTexts.Contains(match.Value))\n {\n continue;\n }\n \n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = match.Value,\n [\"placeholder\"] = placeholder,\n [\"source\"] = \"regex\"\n });\n categoryCounts[category]++;\n processedTexts.Add(match.Value); // Mark this text as processed\n }\n }\n }\n }\n return mappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiRegexMappings\"));\n \n foreach (var mapping in mappings)\n {\n content = content.Replace(\n mapping[\"original\"].ToString(),\n mapping[\"placeholder\"].ToString()\n );\n }\n return content;\n }\" />\n\n \n (\"piiRegexProcessedContent\"))\" />\n \n \n \n @{\n var baseUrl = \"{{piiServiceUrl}}\".TrimEnd('/');\n return baseUrl + \"/language/:analyze-text?api-version=2026-05-01\";\n }\n POST\n \n application/json\n \n \n @(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])\n \n \n\n @{\n var request = new JObject();\n request[\"kind\"] = \"PiiEntityRecognition\";\n request[\"parameters\"] = new JObject{\n {\"modelVersion\", \"latest\"},\n {\"redactionPolicy\", new JObject{\n {\"policyKind\", \"CharacterMask\"},\n {\"redactionCharacter\", \"#\"}\n }}\n };\n request[\"analysisInput\"] = new JObject{\n {\"documents\", new JArray{\n new JObject{\n {\"text\", context.Variables.GetValueOrDefault(\"piiInputContent\")},\n {\"id\", \"1\"},\n {\"language\", context.Variables.GetValueOrDefault(\"piiDetectionLanguage\")}\n }\n }}\n };\n return request.ToString();\n }\n \n\n \n (\"piiAnalysisResponse\");\n if (response == null)\n {\n return \"PII service call failed: no response received (timeout or connectivity issue).\";\n }\n if (response.StatusCode < 200 || response.StatusCode >= 300)\n {\n var body = string.Empty;\n try { body = response.Body?.As(preserveContent: true) ?? string.Empty; } catch { }\n return $\"PII service returned non-success status {response.StatusCode} {response.StatusReason}. Body: {body}\";\n }\n try\n {\n var parsed = JObject.Parse(response.Body.As(preserveContent: true));\n var errors = parsed.SelectToken(\"$.results.errors\") as JArray;\n if (errors != null && errors.Count > 0)\n {\n return $\"PII service reported document errors: {errors.ToString(Newtonsoft.Json.Formatting.None)}\";\n }\n var entitiesToken = parsed.SelectToken(\"$.results.documents[0].entities\");\n if (entitiesToken == null)\n {\n return \"PII service response did not contain expected entities payload.\";\n }\n return string.Empty;\n }\n catch (Exception ex)\n {\n return $\"Failed to parse PII service response: {ex.Message}\";\n }\n }\" />\n\n \n (\"piiAnalysisFailureReason\")))\">\n \n \n @($\"PII anonymization degraded: {context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\")}\")\n \n \n \n (\"piiAnalysisResponse\"))?.StatusCode.ToString() ?? \"n/a\")\" />\n (\"continueOnNLPError\", \"false\"))\" />\n \n \n \n (\"continueOnNLPError\", \"false\").Equals(\"true\", StringComparison.OrdinalIgnoreCase) == false)\">\n \n \n \n application/json\n \n @{\n var error = new JObject {\n [\"error\"] = new JObject {\n [\"code\"] = \"PIIAnonymizationFailed\",\n [\"message\"] = \"The PII anonymization service is currently unavailable and 'continueOnNLPError' is disabled. Request was blocked to prevent unmasked content from reaching the backend.\",\n [\"details\"] = context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\"),\n [\"requestId\"] = context.RequestId.ToString()\n }\n };\n return error.ToString();\n }\n \n \n \n \n (\"piiRegexMappings\") ?? \"[]\")\" />\n \n \n \n \n \n (\"piiAnalysisResponse\");\n var confidenceThreshold = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\");\n var exclusions = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\")\n .Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)\n .Select(c => c.Trim())\n .ToList();\n\n var entitiesToken = JObject.Parse(response.Body.As())\n .SelectToken(\"$.results.documents[0].entities\");\n var entitiesArray = entitiesToken as JArray ?? new JArray();\n\n var entities = entitiesArray\n .Where(e => e[\"confidenceScore\"] != null && e[\"confidenceScore\"].Value() >= confidenceThreshold)\n .Where(e => !exclusions.Contains(e[\"category\"]?.ToString() ?? string.Empty))\n .OrderByDescending(e => (e[\"text\"]?.ToString() ?? string.Empty).Length);\n\n var mappings = new JArray();\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n\n foreach (var entity in entities) {\n var text = (entity[\"text\"]?.ToString() ?? string.Empty).TrimEnd('.', ',', ' ');\n if (string.IsNullOrEmpty(text)) { continue; }\n\n // Skip if this text has already been processed\n if (processedTexts.Contains(text)) {\n continue;\n }\n\n var category = entity[\"category\"].ToString();\n\n if (!categoryCounts.ContainsKey(category)) {\n categoryCounts[category] = 0;\n }\n\n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = text,\n [\"placeholder\"] = placeholder\n });\n\n categoryCounts[category]++;\n processedTexts.Add(text); // Mark this text as processed\n }\n\n return mappings.ToString();\n }\" />\n \n \n\n \n (\"piiMappings\") ?? \"[]\");\n var regexMappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiRegexMappings\") ?? \"[]\");\n \n foreach (var mapping in regexMappings)\n {\n apiMappings.Add(mapping);\n }\n \n return apiMappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(original, placeholder);\n }\n \n return content;\n }\" />\n \n \n \n (\"piiInputContent\"))\" />\n \n \n", - "$fxv#1": "\n \n (\"piiDeanonymizeContentInput\");\n \n // Pre-process content to replace common encoded symbols\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003c\", \"<\");\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003e\", \">\");\n \n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(placeholder, original);\n }\n return content;\n }\" />\n", - "$fxv#2": "\n \n \n (\"piiStateSavingEnabled\") == \"true\")\">\n \n \n \n \n \n \n \n \n \n \n (\"requestedModel\") ?? \"unknown\")\" />\n \n \n @{\n var eventData = new JObject();\n \n // Operation metadata\n eventData[\"id\"] = context.Variables.GetValueOrDefault(\"piiOperationId\");\n eventData[\"timestamp\"] = context.Variables.GetValueOrDefault(\"piiOperationTimestamp\");\n eventData[\"type\"] = \"PII_Processing\";\n eventData[\"targetDeployment\"] = context.Variables.GetValueOrDefault(\"targetDeployment\");\n // Enhanced API and subscription context with content\n eventData[\"context\"] = new JObject {\n [\"api\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiApiId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiApiName\"),\n [\"operation\"] = new JObject {\n [\"id\"] = context.Operation.Id ?? \"unknown\",\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiOperationName\"),\n [\"method\"] = context.Operation.Method ?? \"unknown\",\n [\"urlTemplate\"] = context.Operation.UrlTemplate ?? \"unknown\"\n },\n [\"content\"] = new JObject {\n [\"originalRequest\"] = context.Variables.GetValueOrDefault(\"piiInputContent\"),\n [\"anonymizedRequest\"] = context.Variables.GetValueOrDefault(\"piiAnonymizedContent\"),\n [\"originalResponse\"] = context.Variables.GetValueOrDefault(\"originalResponse\"),\n [\"deanonymizedResponse\"] = context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\")\n }\n },\n [\"product\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiProductId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiProductName\")\n },\n [\"subscription\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionName\")\n }\n };\n \n // Process information\n eventData[\"process\"] = new JObject {\n [\"anonymizationEnabled\"] = context.Variables.GetValueOrDefault(\"piiAnonymizationEnabled\") == \"true\",\n [\"confidenceThreshold\"] = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\", 0.8),\n [\"entityCategoryExclusions\"] = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\", \"\"),\n [\"deanonymizationPerformed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n };\n \n // Results summary\n var mappings = new JArray();\n if (context.Variables.ContainsKey(\"piiMappings\")) {\n try {\n mappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\"));\n eventData[\"entityCount\"] = mappings.Count;\n \n // Group entities by category for reporting\n var categories = new Dictionary();\n foreach (var mapping in mappings) {\n var placeholder = mapping[\"placeholder\"].ToString();\n var category = placeholder.TrimStart('<').Split('_')[0];\n \n if (!categories.ContainsKey(category)) {\n categories[category] = 0;\n }\n categories[category]++;\n }\n \n var categoryDetails = new JObject();\n foreach (var category in categories) {\n categoryDetails[category.Key] = category.Value;\n }\n eventData[\"entityCategories\"] = categoryDetails;\n \n // Include mappings for detailed audit (can be removed if too verbose)\n eventData[\"mappings\"] = mappings;\n }\n catch (Exception ex) {\n eventData[\"error\"] = \"Failed to process mappings: \" + ex.Message;\n }\n }\n \n // Content length metrics (not the actual content)\n if (context.Variables.ContainsKey(\"piiInputContent\")) {\n eventData[\"metrics\"] = new JObject {\n [\"inputContentLength\"] = context.Variables.GetValueOrDefault(\"piiInputContent\", \"\").Length\n };\n \n if (context.Variables.ContainsKey(\"piiAnonymizedContent\")) {\n eventData[\"metrics\"][\"anonymizedContentLength\"] = \n context.Variables.GetValueOrDefault(\"piiAnonymizedContent\", \"\").Length;\n }\n \n // Add deanonymization metrics\n if (context.Variables.ContainsKey(\"piiDeanonymizeContentInput\")) {\n eventData[\"metrics\"][\"deanonymizeInputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizeContentInput\", \"\").Length;\n }\n \n if (context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")) {\n eventData[\"metrics\"][\"deanonymizedOutputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\", \"\").Length;\n }\n }\n \n // Add deanonymization flow details\n eventData[\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"inputSource\"] = context.Variables.ContainsKey(\"piiDeanonymizeContentInput\") ? \"present\" : \"missing\",\n [\"status\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\") ? \"completed\" : \"not_performed\",\n [\"timestamp\"] = DateTime.UtcNow.ToString(\"o\")\n };\n \n // Track end-to-end flow\n eventData[\"processingFlow\"] = new JObject {\n [\"anonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiAnonymizedContent\"),\n [\"entityCount\"] = context.Variables.ContainsKey(\"piiMappings\") ? \n JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\")).Count : 0\n },\n [\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"success\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n }\n };\n \n return eventData.ToString();\n }\n \n \n\n", + "$fxv#0": "\n \n \n (allowedModelsStr.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries).Select(m => m.Trim()), StringComparer.OrdinalIgnoreCase);\n JArray filteredModels = new JArray();\n foreach (var model in allModels) {\n string modelName = model[\"name\"]?.ToString() ?? \"\";\n if (allowedSet.Contains(modelName)) { filteredModels.Add(model); }\n }\n return filteredModels.ToString();\n }\" />\n\n", + "$fxv#1": "\n \n (\"piiAnonymizationEnabled\") == \"true\")\">\n \n (\"piiConfidenceThreshold\", \"0.8\")))\" />\n (\"piiEntityCategoryExclusions\", \"\"))\" />\n (\"piiDetectionLanguage\", \"en\"))\" />\n \n \n \n (\"piiInputContent\");\n var regexPatterns = context.Variables.GetValueOrDefault(\"piiRegexPatterns\");\n var mappings = new JArray();\n \n if (!string.IsNullOrEmpty(regexPatterns))\n {\n var patterns = JArray.Parse(regexPatterns);\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n \n foreach (var pattern in patterns)\n {\n var regex = new Regex(pattern[\"pattern\"].ToString());\n var category = pattern[\"category\"].ToString();\n \n if (!categoryCounts.ContainsKey(category))\n {\n categoryCounts[category] = 0;\n }\n \n var matches = regex.Matches(content);\n foreach (Match match in matches)\n {\n if (!string.IsNullOrEmpty(match.Value))\n {\n // Skip if we already have a mapping for this text\n if (processedTexts.Contains(match.Value))\n {\n continue;\n }\n \n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = match.Value,\n [\"placeholder\"] = placeholder,\n [\"source\"] = \"regex\"\n });\n categoryCounts[category]++;\n processedTexts.Add(match.Value); // Mark this text as processed\n }\n }\n }\n }\n return mappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiRegexMappings\"));\n \n foreach (var mapping in mappings)\n {\n content = content.Replace(\n mapping[\"original\"].ToString(),\n mapping[\"placeholder\"].ToString()\n );\n }\n return content;\n }\" />\n\n \n (\"piiRegexProcessedContent\"))\" />\n \n \n \n @{\n var baseUrl = \"{{piiServiceUrl}}\".TrimEnd('/');\n return baseUrl + \"/language/:analyze-text?api-version=2026-05-01\";\n }\n POST\n \n application/json\n \n \n @(\"Bearer \" + (string)context.Variables[\"msi-access-token\"])\n \n \n\n @{\n var request = new JObject();\n request[\"kind\"] = \"PiiEntityRecognition\";\n request[\"parameters\"] = new JObject{\n {\"modelVersion\", \"latest\"},\n {\"redactionPolicy\", new JObject{\n {\"policyKind\", \"CharacterMask\"},\n {\"redactionCharacter\", \"#\"}\n }}\n };\n request[\"analysisInput\"] = new JObject{\n {\"documents\", new JArray{\n new JObject{\n {\"text\", context.Variables.GetValueOrDefault(\"piiInputContent\")},\n {\"id\", \"1\"},\n {\"language\", context.Variables.GetValueOrDefault(\"piiDetectionLanguage\")}\n }\n }}\n };\n return request.ToString();\n }\n \n\n \n (\"piiAnalysisResponse\");\n if (response == null)\n {\n return \"PII service call failed: no response received (timeout or connectivity issue).\";\n }\n if (response.StatusCode < 200 || response.StatusCode >= 300)\n {\n var body = string.Empty;\n try { body = response.Body?.As(preserveContent: true) ?? string.Empty; } catch { }\n return $\"PII service returned non-success status {response.StatusCode} {response.StatusReason}. Body: {body}\";\n }\n try\n {\n var parsed = JObject.Parse(response.Body.As(preserveContent: true));\n var errors = parsed.SelectToken(\"$.results.errors\") as JArray;\n if (errors != null && errors.Count > 0)\n {\n return $\"PII service reported document errors: {errors.ToString(Newtonsoft.Json.Formatting.None)}\";\n }\n var entitiesToken = parsed.SelectToken(\"$.results.documents[0].entities\");\n if (entitiesToken == null)\n {\n return \"PII service response did not contain expected entities payload.\";\n }\n return string.Empty;\n }\n catch (Exception ex)\n {\n return $\"Failed to parse PII service response: {ex.Message}\";\n }\n }\" />\n\n \n (\"piiAnalysisFailureReason\")))\">\n \n \n @($\"PII anonymization degraded: {context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\")}\")\n \n \n \n (\"piiAnalysisResponse\"))?.StatusCode.ToString() ?? \"n/a\")\" />\n (\"continueOnNLPError\", \"false\"))\" />\n \n \n \n (\"continueOnNLPError\", \"false\").Equals(\"true\", StringComparison.OrdinalIgnoreCase) == false)\">\n \n \n \n application/json\n \n @{\n var error = new JObject {\n [\"error\"] = new JObject {\n [\"code\"] = \"PIIAnonymizationFailed\",\n [\"message\"] = \"The PII anonymization service is currently unavailable and 'continueOnNLPError' is disabled. Request was blocked to prevent unmasked content from reaching the backend.\",\n [\"details\"] = context.Variables.GetValueOrDefault(\"piiAnalysisFailureReason\"),\n [\"requestId\"] = context.RequestId.ToString()\n }\n };\n return error.ToString();\n }\n \n \n \n \n (\"piiRegexMappings\") ?? \"[]\")\" />\n \n \n \n \n \n (\"piiAnalysisResponse\");\n var confidenceThreshold = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\");\n var exclusions = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\")\n .Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)\n .Select(c => c.Trim())\n .ToList();\n\n var entitiesToken = JObject.Parse(response.Body.As())\n .SelectToken(\"$.results.documents[0].entities\");\n var entitiesArray = entitiesToken as JArray ?? new JArray();\n\n var entities = entitiesArray\n .Where(e => e[\"confidenceScore\"] != null && e[\"confidenceScore\"].Value() >= confidenceThreshold)\n .Where(e => !exclusions.Contains(e[\"category\"]?.ToString() ?? string.Empty))\n .OrderByDescending(e => (e[\"text\"]?.ToString() ?? string.Empty).Length);\n\n var mappings = new JArray();\n var categoryCounts = new Dictionary();\n var processedTexts = new HashSet(); // Track already processed texts\n\n foreach (var entity in entities) {\n var text = (entity[\"text\"]?.ToString() ?? string.Empty).TrimEnd('.', ',', ' ');\n if (string.IsNullOrEmpty(text)) { continue; }\n\n // Skip if this text has already been processed\n if (processedTexts.Contains(text)) {\n continue;\n }\n\n var category = entity[\"category\"].ToString();\n\n if (!categoryCounts.ContainsKey(category)) {\n categoryCounts[category] = 0;\n }\n\n var placeholder = $\"<{category}_{categoryCounts[category]}>\";\n mappings.Add(new JObject {\n [\"original\"] = text,\n [\"placeholder\"] = placeholder\n });\n\n categoryCounts[category]++;\n processedTexts.Add(text); // Mark this text as processed\n }\n\n return mappings.ToString();\n }\" />\n \n \n\n \n (\"piiMappings\") ?? \"[]\");\n var regexMappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiRegexMappings\") ?? \"[]\");\n \n foreach (var mapping in regexMappings)\n {\n apiMappings.Add(mapping);\n }\n \n return apiMappings.ToString();\n }\" />\n\n \n (\"piiInputContent\");\n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(original, placeholder);\n }\n \n return content;\n }\" />\n \n \n \n (\"piiInputContent\"))\" />\n \n \n", + "$fxv#2": "\n \n (\"piiDeanonymizeContentInput\");\n \n // Pre-process content to replace common encoded symbols\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003c\", \"<\");\n content = System.Text.RegularExpressions.Regex.Replace(content, @\"\\\\u003e\", \">\");\n \n var mappings = JArray.Parse(\n context.Variables.GetValueOrDefault(\"piiMappings\"));\n \n foreach (var mapping in mappings) {\n var original = mapping[\"original\"].ToString();\n var placeholder = mapping[\"placeholder\"].ToString();\n content = content.Replace(placeholder, original);\n }\n return content;\n }\" />\n", + "$fxv#3": "\n \n \n (\"piiStateSavingEnabled\") == \"true\")\">\n \n \n \n \n \n \n \n \n \n \n (\"requestedModel\") ?? \"unknown\")\" />\n \n \n @{\n var eventData = new JObject();\n \n // Operation metadata\n eventData[\"id\"] = context.Variables.GetValueOrDefault(\"piiOperationId\");\n eventData[\"timestamp\"] = context.Variables.GetValueOrDefault(\"piiOperationTimestamp\");\n eventData[\"type\"] = \"PII_Processing\";\n eventData[\"targetDeployment\"] = context.Variables.GetValueOrDefault(\"targetDeployment\");\n // Enhanced API and subscription context with content\n eventData[\"context\"] = new JObject {\n [\"api\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiApiId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiApiName\"),\n [\"operation\"] = new JObject {\n [\"id\"] = context.Operation.Id ?? \"unknown\",\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiOperationName\"),\n [\"method\"] = context.Operation.Method ?? \"unknown\",\n [\"urlTemplate\"] = context.Operation.UrlTemplate ?? \"unknown\"\n },\n [\"content\"] = new JObject {\n [\"originalRequest\"] = context.Variables.GetValueOrDefault(\"piiInputContent\"),\n [\"anonymizedRequest\"] = context.Variables.GetValueOrDefault(\"piiAnonymizedContent\"),\n [\"originalResponse\"] = context.Variables.GetValueOrDefault(\"originalResponse\"),\n [\"deanonymizedResponse\"] = context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\")\n }\n },\n [\"product\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiProductId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiProductName\")\n },\n [\"subscription\"] = new JObject {\n [\"id\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionId\"),\n [\"name\"] = context.Variables.GetValueOrDefault(\"piiSubscriptionName\")\n }\n };\n \n // Process information\n eventData[\"process\"] = new JObject {\n [\"anonymizationEnabled\"] = context.Variables.GetValueOrDefault(\"piiAnonymizationEnabled\") == \"true\",\n [\"confidenceThreshold\"] = context.Variables.GetValueOrDefault(\"piiConfidenceThreshold\", 0.8),\n [\"entityCategoryExclusions\"] = context.Variables.GetValueOrDefault(\"piiEntityCategoryExclusions\", \"\"),\n [\"deanonymizationPerformed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n };\n \n // Results summary\n var mappings = new JArray();\n if (context.Variables.ContainsKey(\"piiMappings\")) {\n try {\n mappings = JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\"));\n eventData[\"entityCount\"] = mappings.Count;\n \n // Group entities by category for reporting\n var categories = new Dictionary();\n foreach (var mapping in mappings) {\n var placeholder = mapping[\"placeholder\"].ToString();\n var category = placeholder.TrimStart('<').Split('_')[0];\n \n if (!categories.ContainsKey(category)) {\n categories[category] = 0;\n }\n categories[category]++;\n }\n \n var categoryDetails = new JObject();\n foreach (var category in categories) {\n categoryDetails[category.Key] = category.Value;\n }\n eventData[\"entityCategories\"] = categoryDetails;\n \n // Include mappings for detailed audit (can be removed if too verbose)\n eventData[\"mappings\"] = mappings;\n }\n catch (Exception ex) {\n eventData[\"error\"] = \"Failed to process mappings: \" + ex.Message;\n }\n }\n \n // Content length metrics (not the actual content)\n if (context.Variables.ContainsKey(\"piiInputContent\")) {\n eventData[\"metrics\"] = new JObject {\n [\"inputContentLength\"] = context.Variables.GetValueOrDefault(\"piiInputContent\", \"\").Length\n };\n \n if (context.Variables.ContainsKey(\"piiAnonymizedContent\")) {\n eventData[\"metrics\"][\"anonymizedContentLength\"] = \n context.Variables.GetValueOrDefault(\"piiAnonymizedContent\", \"\").Length;\n }\n \n // Add deanonymization metrics\n if (context.Variables.ContainsKey(\"piiDeanonymizeContentInput\")) {\n eventData[\"metrics\"][\"deanonymizeInputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizeContentInput\", \"\").Length;\n }\n \n if (context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")) {\n eventData[\"metrics\"][\"deanonymizedOutputLength\"] = \n context.Variables.GetValueOrDefault(\"piiDeanonymizedContentOutput\", \"\").Length;\n }\n }\n \n // Add deanonymization flow details\n eventData[\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"inputSource\"] = context.Variables.ContainsKey(\"piiDeanonymizeContentInput\") ? \"present\" : \"missing\",\n [\"status\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\") ? \"completed\" : \"not_performed\",\n [\"timestamp\"] = DateTime.UtcNow.ToString(\"o\")\n };\n \n // Track end-to-end flow\n eventData[\"processingFlow\"] = new JObject {\n [\"anonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiAnonymizedContent\"),\n [\"entityCount\"] = context.Variables.ContainsKey(\"piiMappings\") ? \n JArray.Parse(context.Variables.GetValueOrDefault(\"piiMappings\")).Count : 0\n },\n [\"deanonymization\"] = new JObject {\n [\"performed\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\"),\n [\"success\"] = context.Variables.ContainsKey(\"piiDeanonymizedContentOutput\")\n }\n };\n \n return eventData.ToString();\n }\n \n \n\n", + "$fxv#4": "\n \n \n \n \n \n \n \n application/json\n \n @{\n JArray models = JArray.Parse((string)context.Variables[\"availableFilteredModels\"]);\n return new JObject(new JProperty(\"object\", \"list\"), new JProperty(\"data\", models)).ToString();\n }\n \n \n \n \n \n\n", "hubFoundryAccountName": "[format('aif-hub-{0}', parameters('resourceToken'))]", "hubFoundryProjectName": "citadel-hub-project", "apimName": "[format('apim-citadel-{0}', parameters('resourceToken'))]", @@ -452,8 +454,9 @@ "apiVersion": "2023-09-01-preview", "name": "[format('{0}/{1}', variables('apimName'), 'get-available-models')]", "properties": { - "format": "xml", - "value": "application/json@{var models = new JArray(); models.Add(\"gpt-4.1\"); models.Add(\"gpt-5.4-mini\"); models.Add(\"gpt-5.2\"); models.Add(\"text-embedding-3-large\"); models.Add(\"Mistral-Large-3\"); models.Add(\"Phi-4\"); return models.ToString();}" + "format": "rawxml", + "description": "Sets availableModels/availableFilteredModels for the list-models operation", + "value": "[variables('$fxv#0')]" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" @@ -466,7 +469,7 @@ "properties": { "format": "rawxml", "description": "Detects PII with the Language service and masks it before the request reaches the model", - "value": "[variables('$fxv#0')]" + "value": "[variables('$fxv#1')]" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]", @@ -480,7 +483,7 @@ "properties": { "format": "rawxml", "description": "Restores the original PII values in the response returned to the caller", - "value": "[variables('$fxv#1')]" + "value": "[variables('$fxv#2')]" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]" @@ -493,7 +496,7 @@ "properties": { "format": "rawxml", "description": "Emits PII handling events to Event Hub for audit", - "value": "[variables('$fxv#2')]" + "value": "[variables('$fxv#3')]" }, "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apimName'))]", @@ -540,11 +543,12 @@ "apiVersion": "2023-09-01-preview", "name": "[format('{0}/{1}/{2}/{3}', variables('apimName'), 'universal-llm-api', 'get-models', 'policy')]", "properties": { - "format": "xml", - "value": "application/json[\"gpt-4.1\",\"gpt-5.4-mini\",\"gpt-5.2\",\"text-embedding-3-large\",\"Mistral-Large-3\",\"Phi-4\"]" + "format": "rawxml", + "value": "[variables('$fxv#4')]" }, "dependsOn": [ - "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'get-models')]" + "[resourceId('Microsoft.ApiManagement/service/apis/operations', variables('apimName'), 'universal-llm-api', 'get-models')]", + "[resourceId('Microsoft.ApiManagement/service/policyFragments', variables('apimName'), 'get-available-models')]" ] }, { From 315fdd92ccbc3d170cb199702498c5ecb67bab0d Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Wed, 2 Sep 2026 14:57:00 +0200 Subject: [PATCH 10/11] Make challenge 5 prove masking instead of counting HTTP 200s The masking tests treated any 200 as a pass. Because pii-deanonymization restores the original values on the way out, the response looks the same whether or not anything was masked, so the section reported 6/6 passed while the model was receiving unmasked names and SSNs. For a governance lab that is the worst possible failure mode: a green score that hides the control being absent. Add a probe that asks a question whose answer depends on what the model actually saw. When anonymization runs the model gets a placeholder such as and cannot tell the name starts with "J"; only an unmasked request answers YES. The overall summary now zeroes the masking score unless that probe passes. Checked both directions against a live lab: the masking product answers NO and verifies, the non-masking product answers YES and correctly fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../5. citadel-pii-processing-tests.ipynb | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb index 8c4ae6706..6222eb935 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb @@ -706,7 +706,11 @@ " \n", " # Check if response contains any of the original PII (it shouldn't after deanonymization)\n", " # The LLM should respond with the original PII restored\n", - " utils.print_ok(f\"šŸ”„ PII should be deanonymized in response\")\n", + " # A 200 only proves the request was delivered. It says nothing about\n", + " # whether PII was masked on the way in - the deanonymization step puts\n", + " # the original values back, so a clean-looking response is expected\n", + " # either way. The next cell verifies masking for real.\n", + " utils.print_info(\"ā†©ļø Delivered (masking itself is verified in the next cell)\")\n", " else:\n", " result[\"error\"] = response.text[:200]\n", " utils.print_error(f\"āŒ Error: {response.text[:600]}\")\n", @@ -735,6 +739,72 @@ "utils.print_error(f\"Failed: {len(masking_test_results) - successful}\")" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Verify that masking ACTUALLY happened.\n", + "#\n", + "# Why this cell exists: every request above returned HTTP 200, but that only means\n", + "# the call was delivered. The pii-deanonymization step restores the original values\n", + "# on the way out, so the response looks identical whether or not the model ever saw\n", + "# the real PII. Counting 200s therefore reports a perfect score even when no masking\n", + "# is taking place at all.\n", + "#\n", + "# The probe below asks a question whose answer depends on what the *model* saw. If\n", + "# anonymization ran, the model received a placeholder such as and cannot\n", + "# tell that the name begins with \"J\". Only an unmasked request yields \"YES\".\n", + "\n", + "probe_payload = {\n", + " \"model\": model_name,\n", + " \"messages\": [\n", + " {\"role\": \"system\", \"content\": \"Answer with exactly one word.\"},\n", + " {\"role\": \"user\", \"content\": (\n", + " \"Answer with exactly one word, YES or NO. In the line below, does the \"\n", + " \"text naming the person begin with the letter J? \"\n", + " \"Line: My name is John Smith and I live in Paris.\"\n", + " )},\n", + " ],\n", + " \"max_tokens\": 6,\n", + "}\n", + "\n", + "masking_verified = False\n", + "\n", + "utils.print_info(f\"\\n{'='*80}\")\n", + "utils.print_info(\"šŸ” MASKING VERIFICATION - did the model actually see the PII?\")\n", + "utils.print_info(f\"{'='*80}\")\n", + "\n", + "try:\n", + " probe = requests.post(\n", + " chat_completions_url,\n", + " headers={'api-key': masking_api_key},\n", + " json=probe_payload,\n", + " timeout=60,\n", + " )\n", + " utils.print_response_code(probe)\n", + "\n", + " if probe.status_code == 200:\n", + " answer = probe.json().get(\"choices\", [{}])[0].get(\"message\", {}).get(\"content\", \"\")\n", + " utils.print_info(f\"Model answered: {answer!r}\")\n", + "\n", + " # \"YES\" means the model read the real name, so nothing was masked.\n", + " if \"YES\" in answer.upper():\n", + " utils.print_error(\"āŒ Masking is NOT active - the model saw the original name.\")\n", + " utils.print_error(\" The PII policy fragments are missing or not applied to this product.\")\n", + " else:\n", + " masking_verified = True\n", + " utils.print_ok(\"āœ… Masking confirmed - the model only saw a placeholder, not the real name.\")\n", + " else:\n", + " utils.print_error(f\"āŒ Probe failed: {probe.text[:300]}\")\n", + "except Exception as e:\n", + " utils.print_error(f\"āŒ Probe request failed: {e}\")\n", + "\n", + "if not masking_verified:\n", + " utils.print_error(\"\\nāš ļø Treat the masking results above as NOT passing until this probe succeeds.\")\n" + ] + }, { "cell_type": "markdown", "id": "84a6f4da", @@ -2130,6 +2200,14 @@ "utils.print_info(f\" Product: {masking_product_id}\")\n", "utils.print_info(f\" Tests: {len(masking_test_results)} | Successful: {masking_success} | Failed: {len(masking_test_results) - masking_success}\")\n", "\n", + "# Delivery alone is not a pass. Without this gate the section reports a perfect\n", + "# score even when masking is switched off entirely.\n", + "if masking_verified:\n", + " utils.print_ok(\" āœ… Masking verified: the model received placeholders, not the real PII\")\n", + "else:\n", + " utils.print_error(\" āŒ Masking NOT verified - the counts above only reflect delivered requests\")\n", + " masking_success = 0\n", + "\n", "# PII Blocking Summary\n", "utils.print_info(f\"\\nšŸš« PII Blocking:\")\n", "blocking_passed = sum(1 for r in blocking_test_results if r['test_passed'])\n", From 3d7834a2ccd20faf8c853a84df37ef2da6bff7cd Mon Sep 17 00:00:00 2001 From: ajakupov-microsoft Date: Wed, 2 Sep 2026 17:31:33 +0200 Subject: [PATCH 11/11] Mark challenge 5's analytics section optional and skip it cleanly Use Case 3 reads PII telemetry from Cosmos DB and ends the notebook with a CosmosHttpResponseError (Forbidden) traceback. Two independent things block it, neither fixable by a participant: - CosmosDB_PublicNetwork_Modify keeps the account's public network access disabled. This is not a setting anyone can toggle in the portal: an az cosmosdb update --public-network-access ENABLED reports success and the value never changes at all. CosmosDB_LocalAuth_Modify also forces disableLocalAuth, ruling out the cosmos_auth_mode="key" fallback. - pii-usage-container has no writer. resources.bicep names a usage ingestion Logic App and creates its managed identity, but never creates the Logic App, and there is no other ingestion resource in the group. The query would return zero records even with network access. Explain both in the notebook and the challenge guide, and catch the connection failure so the section reports "no data" and the notebook runs to completion instead of dying on the last use case. The audit trail behind the masking policy does work and is worth pointing at instead: pii-state-saving emits to the usage-events Event Hub, which recorded incoming messages during a real run. The success criteria now reference that and the masking probe rather than the Cosmos report. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --- .../challenges/challenge-05.md | 37 +++- .../5. citadel-pii-processing-tests.ipynb | 208 ++++++++++-------- 2 files changed, 149 insertions(+), 96 deletions(-) diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-05.md b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-05.md index ebf2858a1..01f81d2d4 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-05.md +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/challenge-05.md @@ -45,28 +45,49 @@ Same prerequisite as every notebook in this MicroHack ā€” see product policy, parameter file + deploy, retrieve API key, test with various PII types) - **Use Case 2: PII Blocking** ā€” `4ļøāƒ£.1`ā€“`4ļøāƒ£.5` (same shape, blocking instead of masking) - - **Use Case 3: PII Processing Performance Analytics** ā€” `5ļøāƒ£.1`ā€“`5ļøāƒ£.5` (Cosmos DB - connection, grant data-plane access, query recent records, quantitative report + - visualizations, deploy an analytics access contract, send the summary to an LLM) + - **Use Case 3: PII Processing Performance Analytics** ā€” `5ļøāƒ£.1`ā€“`5ļøāƒ£.5` ā€” + ā­ļø **skip this section.** It is optional and cannot run in the MicroHack + environment (see the note below). The notebook detects this and skips it + cleanly, so you can simply run the cells and move on. 3. Read the `šŸ“Š Results Summary` cell. +> **Why Use Case 3 is skipped** +> +> Two things block it, and neither is something you can change: +> +> - The subscription applies an Azure Policy (`CosmosDB_PublicNetwork_Modify`) that +> forces the Cosmos DB account's public network access to stay disabled. Enabling it +> in the portal does **not** work ā€” the change is reverted immediately, so the setting +> never changes. `CosmosDB_LocalAuth_Modify` likewise forces key auth off, so the +> notebook's `cosmos_auth_mode = "key"` fallback is unavailable too. +> - The `pii-usage-container` has no writer in this lab: the usage-ingestion Logic App +> that would move events from Event Hub into Cosmos is not deployed, so the query +> would return zero records even with network access. +> +> The audit trail itself **does** work. The `pii-state-saving` policy fragment emits an +> event to the `usage-events` Event Hub for every PII operation ā€” check *Metrics ā†’ +> Incoming Messages* on the Event Hub namespace in the portal to see them arrive. Only +> the Cosmos-backed reporting layer built on top of it is missing. + ## šŸ Success criteria - [ ] The PII masking access contract anonymizes detected PII (names, emails, phone numbers, credit cards, IBANs, custom regex patterns) and can deanonymize it back. - [ ] The PII blocking access contract detects and blocks requests containing PII, reporting what was detected. -- [ ] Your Cosmos DB query returns recent PII processing records, and the quantitative - report/visualizations render successfully. -- [ ] The PII analytics access contract successfully sends the usage summary to an LLM - through its own governed contract. +- [ ] The masking verification probe confirms the model received a placeholder rather + than the real name ā€” a `200` response on its own does not prove masking, because + deanonymization restores the original values on the way out. +- [ ] You can see PII events arriving in the `usage-events` Event Hub (*Metrics ā†’ + Incoming Messages*), which is the audit trail behind the masking policy. ## šŸ› ļø Troubleshooting | Symptom | Fix | |---------|-----| | Setup cell raises a missing-key error | Your `challenges/workshop/.env` is missing that key (or you haven't run the `azd` bridge) ā€” see [Challenge 1, Part A](challenge-01.md#part-a--confirm-your-notebook-environment-is-ready-5-min). | -| Cosmos DB query returns nothing / `Forbidden` | Section `5ļøāƒ£.1.1` grants your identity Cosmos DB data-plane access ā€” re-run it and wait a minute for RBAC to propagate. | +| `CosmosHttpResponseError: (Forbidden) Request originated from IP ...` | Expected ā€” Use Case 3 is optional and cannot run here. Azure Policy keeps Cosmos DB's public network access disabled and reverts any attempt to enable it. The notebook skips the section automatically; carry on. | +| Masking tests all pass but you're unsure anything was masked | Run the masking verification cell. Because deanonymization restores the original values, a `200` looks identical whether or not masking occurred ā€” the probe asks the model a question only an *unmasked* request could answer. | | PII detection doesn't fire on a sample | Confirm the Azure AI Language Service instance is configured and reachable; some PII types need the exact regex/entity category the notebook defines. | | Deanonymization returns masked text unchanged | The masking/deanonymization state must be saved via Event Hub first ā€” confirm section `3ļøāƒ£.5` ran before attempting deanonymization. | diff --git a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb index 6222eb935..bb86eebb0 100644 --- a/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb +++ b/03-Azure/01-04-AI/06_AI_Governance/challenges/workshop/5. citadel-pii-processing-tests.ipynb @@ -1355,15 +1355,30 @@ "---\n", "## šŸ“ˆ Use Case 3: PII Processing Performance Analytics\n", "\n", - "This section pulls **recent PII processing telemetry** from the gateway's Cosmos DB\n", - "(`pii-usage-container`) and provides:\n", - "\n", - "- šŸ”¢ Quantitative report (records, entity totals, success rates, content size impact)\n", - "- šŸ“Š Visualizations (entity-category mix, anonymization vs. deanonymization performance, content-length transformation, processing volume over time)\n", - "- šŸ¤– Optional **LLM-powered reasoning** over the analytics summary, sent through the **PII Masking Access Contract** so any residual PII in the summary is anonymized before reaching the model.\n", - "\n", - "> Make sure the gateway is configured to log PII processing events to a Cosmos DB container with documents shaped like the `PII_Processing` schema produced by the `pii-state-saving` policy fragment.\n", - "---" + "> ### ā­ļø OPTIONAL ā€” not runnable in the MicroHack environment\n", + ">\n", + "> **Skip this section.** It is kept for reference and for instructor-led walkthroughs.\n", + "> Everything you need for challenge 5 is already complete above.\n", + ">\n", + "> Two things block it here, and neither can be fixed by a participant:\n", + ">\n", + "> 1. **Cosmos DB is unreachable from your laptop.** The subscription applies an Azure\n", + "> Policy (`CosmosDB_PublicNetwork_Modify`) that forces `publicNetworkAccess: Disabled`.\n", + "> Turning it on in the portal does not work - the change is reverted immediately, so\n", + "> the setting simply never changes. A companion policy\n", + "> (`CosmosDB_LocalAuth_Modify`) also forces `disableLocalAuth: true`, which rules out\n", + "> the `cosmos_auth_mode = \"key\"` fallback as well. Expect\n", + "> `CosmosHttpResponseError: (Forbidden) Request originated from IP ... through public\n", + "> internet`.\n", + "> 2. **The container has no writer.** `pii-usage-container` exists, but the usage\n", + "> ingestion Logic App that would move events from Event Hub into Cosmos is not\n", + "> deployed in this lab. Even with network access the query would return zero records.\n", + ">\n", + "> **The audit trail itself does work.** The `pii-state-saving` policy fragment emits an\n", + "> event to the `usage-events` Event Hub on every PII operation - you can confirm this in\n", + "> the portal under the Event Hub namespace's *Metrics* ā†’ *Incoming Messages*. Only the\n", + "> Cosmos-backed reporting layer on top of it is unavailable.\n", + "\n" ] }, { @@ -1543,88 +1558,105 @@ "import pandas as pd\n", "from datetime import datetime, timezone\n", "\n", - "# Build Cosmos client\n", - "if cosmos_auth_mode.lower() == \"aad\":\n", - " from azure.identity import DefaultAzureCredential\n", - " cosmos_client = CosmosClient(cosmos_account_endpoint, credential=DefaultAzureCredential())\n", - "else:\n", - " if not cosmos_account_key:\n", - " raise ValueError(\"cosmos_account_key is empty but cosmos_auth_mode == 'key'\")\n", - " cosmos_client = CosmosClient(cosmos_account_endpoint, credential=cosmos_account_key)\n", - "\n", - "cosmos_db = cosmos_client.get_database_client(cosmos_database_name)\n", - "cosmos_container = cosmos_db.get_container_client(cosmos_container_name)\n", - "\n", - "# Build query (only PII_Processing docs)\n", - "where_clauses = [\"c.type = 'PII_Processing'\"]\n", - "parameters = []\n", - "if filter_product_id:\n", - " where_clauses.append(\"c.context.product.id = @prodId\")\n", - " parameters.append({\"name\": \"@prodId\", \"value\": filter_product_id})\n", - "if filter_subscription_id:\n", - " where_clauses.append(\"c.context.subscription.id = @subId\")\n", - " parameters.append({\"name\": \"@subId\", \"value\": filter_subscription_id})\n", + "# Build Cosmos client.\n", + "#\n", + "# In the MicroHack environment this is expected to fail: an Azure Policy keeps the\n", + "# account's public network access disabled and it cannot be turned back on. Rather\n", + "# than ending the notebook with a traceback, fall back to an empty result set so the\n", + "# remaining cells report \"no data\" and the notebook still runs to completion. See the\n", + "# notice at the top of Use Case 3 for the full explanation.\n", + "pii_df = pd.DataFrame()\n", + "cosmos_available = False\n", "\n", - "query = (\n", - " f\"SELECT TOP {int(top_n_records)} * FROM c \"\n", - " f\"WHERE {' AND '.join(where_clauses)} \"\n", - " f\"ORDER BY c._ts DESC\"\n", - ")\n", - "utils.print_info(f\"Cosmos query: {query}\")\n", - "\n", - "start_time = time.time()\n", - "pii_records = list(cosmos_container.query_items(\n", - " query=query,\n", - " parameters=parameters,\n", - " enable_cross_partition_query=True\n", - "))\n", - "elapsed_time = time.time() - start_time\n", - "utils.print_ok(f\"āœ… Retrieved {len(pii_records)} record(s) in {elapsed_time:.2f}s\")\n", - "\n", - "# Flatten into a DataFrame\n", - "def _flatten(rec):\n", - " ctx = rec.get(\"context\", {}) or {}\n", - " api = ctx.get(\"api\", {}) or {}\n", - " product = ctx.get(\"product\", {}) or {}\n", - " sub = ctx.get(\"subscription\", {}) or {}\n", - " process = rec.get(\"process\", {}) or {}\n", - " metrics = rec.get(\"metrics\", {}) or {}\n", - " flow = rec.get(\"processingFlow\", {}) or {}\n", - " deanon = rec.get(\"deanonymization\", {}) or {}\n", - " cats = rec.get(\"entityCategories\", {}) or {}\n", - " return {\n", - " \"id\" : rec.get(\"id\"),\n", - " \"timestamp\" : rec.get(\"timestamp\"),\n", - " \"ts\" : datetime.fromtimestamp(rec.get(\"_ts\", 0), tz=timezone.utc) if rec.get(\"_ts\") else None,\n", - " \"targetDeployment\" : rec.get(\"targetDeployment\"),\n", - " \"api_id\" : api.get(\"id\"),\n", - " \"operation\" : (api.get(\"operation\") or {}).get(\"id\"),\n", - " \"product_id\" : product.get(\"id\"),\n", - " \"subscription_id\" : sub.get(\"id\"),\n", - " \"anonymization_enabled\" : process.get(\"anonymizationEnabled\"),\n", - " \"confidence_threshold\" : process.get(\"confidenceThreshold\"),\n", - " \"entity_exclusions\" : process.get(\"entityCategoryExclusions\"),\n", - " \"deanonymization_performed\": process.get(\"deanonymizationPerformed\"),\n", - " \"entity_count\" : rec.get(\"entityCount\", 0) or 0,\n", - " \"entity_categories\" : cats,\n", - " \"input_len\" : metrics.get(\"inputContentLength\", 0) or 0,\n", - " \"anonymized_len\" : metrics.get(\"anonymizedContentLength\", 0) or 0,\n", - " \"deanon_input_len\" : metrics.get(\"deanonymizeInputLength\", 0) or 0,\n", - " \"deanon_output_len\" : metrics.get(\"deanonymizedOutputLength\", 0) or 0,\n", - " \"anon_performed\" : (flow.get(\"anonymization\") or {}).get(\"performed\", False),\n", - " \"deanon_performed\" : (flow.get(\"deanonymization\") or {}).get(\"performed\", False),\n", - " \"deanon_success\" : (flow.get(\"deanonymization\") or {}).get(\"success\", False),\n", - " \"deanon_status\" : deanon.get(\"status\"),\n", - " \"deanon_input_source\" : deanon.get(\"inputSource\"),\n", - " }\n", + "try:\n", + " if cosmos_auth_mode.lower() == \"aad\":\n", + " from azure.identity import DefaultAzureCredential\n", + " cosmos_client = CosmosClient(cosmos_account_endpoint, credential=DefaultAzureCredential())\n", + " else:\n", + " if not cosmos_account_key:\n", + " raise ValueError(\"cosmos_account_key is empty but cosmos_auth_mode == 'key'\")\n", + " cosmos_client = CosmosClient(cosmos_account_endpoint, credential=cosmos_account_key)\n", + " cosmos_available = True\n", + "except Exception as e:\n", + " utils.print_error(f\"ā­ļø Cosmos DB is not reachable - skipping Use Case 3 analytics.\")\n", + " utils.print_info(f\" {type(e).__name__}: {str(e)[:200]}\")\n", + " utils.print_info(\" This is expected in the MicroHack environment; see the notice above.\")\n", + "\n", + "if cosmos_available:\n", + "\n", + " cosmos_db = cosmos_client.get_database_client(cosmos_database_name)\n", + " cosmos_container = cosmos_db.get_container_client(cosmos_container_name)\n", + "\n", + " # Build query (only PII_Processing docs)\n", + " where_clauses = [\"c.type = 'PII_Processing'\"]\n", + " parameters = []\n", + " if filter_product_id:\n", + " where_clauses.append(\"c.context.product.id = @prodId\")\n", + " parameters.append({\"name\": \"@prodId\", \"value\": filter_product_id})\n", + " if filter_subscription_id:\n", + " where_clauses.append(\"c.context.subscription.id = @subId\")\n", + " parameters.append({\"name\": \"@subId\", \"value\": filter_subscription_id})\n", + "\n", + " query = (\n", + " f\"SELECT TOP {int(top_n_records)} * FROM c \"\n", + " f\"WHERE {' AND '.join(where_clauses)} \"\n", + " f\"ORDER BY c._ts DESC\"\n", + " )\n", + " utils.print_info(f\"Cosmos query: {query}\")\n", + "\n", + " start_time = time.time()\n", + " pii_records = list(cosmos_container.query_items(\n", + " query=query,\n", + " parameters=parameters,\n", + " enable_cross_partition_query=True\n", + " ))\n", + " elapsed_time = time.time() - start_time\n", + " utils.print_ok(f\"āœ… Retrieved {len(pii_records)} record(s) in {elapsed_time:.2f}s\")\n", + "\n", + " # Flatten into a DataFrame\n", + " def _flatten(rec):\n", + " ctx = rec.get(\"context\", {}) or {}\n", + " api = ctx.get(\"api\", {}) or {}\n", + " product = ctx.get(\"product\", {}) or {}\n", + " sub = ctx.get(\"subscription\", {}) or {}\n", + " process = rec.get(\"process\", {}) or {}\n", + " metrics = rec.get(\"metrics\", {}) or {}\n", + " flow = rec.get(\"processingFlow\", {}) or {}\n", + " deanon = rec.get(\"deanonymization\", {}) or {}\n", + " cats = rec.get(\"entityCategories\", {}) or {}\n", + " return {\n", + " \"id\" : rec.get(\"id\"),\n", + " \"timestamp\" : rec.get(\"timestamp\"),\n", + " \"ts\" : datetime.fromtimestamp(rec.get(\"_ts\", 0), tz=timezone.utc) if rec.get(\"_ts\") else None,\n", + " \"targetDeployment\" : rec.get(\"targetDeployment\"),\n", + " \"api_id\" : api.get(\"id\"),\n", + " \"operation\" : (api.get(\"operation\") or {}).get(\"id\"),\n", + " \"product_id\" : product.get(\"id\"),\n", + " \"subscription_id\" : sub.get(\"id\"),\n", + " \"anonymization_enabled\" : process.get(\"anonymizationEnabled\"),\n", + " \"confidence_threshold\" : process.get(\"confidenceThreshold\"),\n", + " \"entity_exclusions\" : process.get(\"entityCategoryExclusions\"),\n", + " \"deanonymization_performed\": process.get(\"deanonymizationPerformed\"),\n", + " \"entity_count\" : rec.get(\"entityCount\", 0) or 0,\n", + " \"entity_categories\" : cats,\n", + " \"input_len\" : metrics.get(\"inputContentLength\", 0) or 0,\n", + " \"anonymized_len\" : metrics.get(\"anonymizedContentLength\", 0) or 0,\n", + " \"deanon_input_len\" : metrics.get(\"deanonymizeInputLength\", 0) or 0,\n", + " \"deanon_output_len\" : metrics.get(\"deanonymizedOutputLength\", 0) or 0,\n", + " \"anon_performed\" : (flow.get(\"anonymization\") or {}).get(\"performed\", False),\n", + " \"deanon_performed\" : (flow.get(\"deanonymization\") or {}).get(\"performed\", False),\n", + " \"deanon_success\" : (flow.get(\"deanonymization\") or {}).get(\"success\", False),\n", + " \"deanon_status\" : deanon.get(\"status\"),\n", + " \"deanon_input_source\" : deanon.get(\"inputSource\"),\n", + " }\n", "\n", - "pii_df = pd.DataFrame([_flatten(r) for r in pii_records])\n", - "if not pii_df.empty:\n", - " pii_df[\"ts\"] = pd.to_datetime(pii_df[\"ts\"], utc=True, errors=\"coerce\")\n", - " pii_df.sort_values(\"ts\", inplace=True)\n", + " pii_df = pd.DataFrame([_flatten(r) for r in pii_records])\n", + " if not pii_df.empty:\n", + " pii_df[\"ts\"] = pd.to_datetime(pii_df[\"ts\"], utc=True, errors=\"coerce\")\n", + " pii_df.sort_values(\"ts\", inplace=True)\n", "\n", - "utils.print_info(f\"Loaded DataFrame shape: {pii_df.shape}\")\n", - "pii_df.head()" + " utils.print_info(f\"Loaded DataFrame shape: {pii_df.shape}\")\n", + " pii_df.head()" ] }, {