Repository navigation
Rewrite challenge 1 Part A for the pre-provisioned lab environment - #470
Merged
Marco Weber (qxsch) merged 11 commits intoSep 3, 2026
Conversation
Part A told participants to paste every value from the MicroHack dashboard's Credentials tab. That tab only exposes the 7 platform-provided Azure entries, so most keys -- including LLM_BACKEND_CONFIG -- were not obtainable that way, and the instructions had no Azure sign-in step at all. Part A is now five steps matching what the environment actually provides: - A.1 Sign in to Azure with the dashboard's Entra username and Temporary Access Pass, then az login so DefaultAzureCredential resolves the lab user. - A.2 Find the resource token from the aif-hub-<TOKEN> account name; every other resource name derives from it. - A.3 Fill in .env, showing each key derived from the token. - A.4 Paste LLM_BACKEND_CONFIG, with the full value and the three ways it breaks: line breaks, surrounding quotes, and a missing final bracket. Also documents reading it straight from the deployment outputs instead. - A.5 Validate the file before starting. The bracket warning is the important one: the value must end with }]}], and dropping the last two characters produces a JSON error pointing ~1000 characters away from the mistake. Adds a note to restart the Jupyter kernel after editing .env, since the loader skips keys already in os.environ and a bad value read at startup otherwise persists for the whole session. Updates .env.template to match -- its per-key comments named dashboard credentials that do not exist -- and adjusts the context paragraph and troubleshooting table. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Lab participants hold Owner on their own resource group but only Reader on the subscription, so every 'az deployment sub create' in the core challenge notebooks failed with AuthorizationFailed on Microsoft.Resources/deployments/validate/action. - llm-backend-onboarding/main.bicep and citadel-access-contracts/main.bicep switch from targetScope 'subscription' to 'resourceGroup'. Both used the subscription scope only to 'existing'-reference resource groups before deploying modules into them; in this single-resource-group lab those all resolve to the participant's resource group, so the lookups are dropped and the resources referenced directly. - Update the 7 'az deployment sub create' call sites in notebooks 1, 2, 3, 5 and 6 to 'az deployment group create --resource-group ...'. - Refresh the matching README/bicepparam examples. - Add troubleshooting rows for wrong-tenant ResourceGroupNotFound and for subscription-scoped AuthorizationFailed. Both templates verified with 'az deployment group validate' against a live lab resource group as the participant identity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
resources.bicep declared 11 APIM operations but only 3 operation policies.
An APIM operation with no policy has no backend and returns HTTP 500, so 8
of the 11 endpoints were dead:
universal-llm-api: post-responses, get-response, get-response-input-items
unified-ai-api: get-deployments, get-deployment,
post-openai-chat-completions,
post-foundry-chat-completions,
post-gemini-chat-completions
Impact: challenge 6 calls all five unified-ai endpoints, so it was entirely
non-functional; challenge 2 lost its three /responses endpoints; challenge 1's
closing test cells failed against unified-ai. The three working operations
were the only ones ever exercised, which is why this went unnoticed.
Add the eight policies, following the proven pattern of the existing three
(managed-identity auth -> backend-hub-foundry -> rewrite-uri):
- /deployments returns the deployment inventory in the
{"value":[{name,properties.model}]} shape the notebooks parse, and
/deployments/{model_name} returns 404 for an unknown model as challenge 6
asserts.
- The OpenAI-style and Foundry-style chat paths map the requested model to its
deployment name (dots to dashes) and reject unsupported models with 400.
- The responses paths proxy the Azure OpenAI Responses API, rewriting the body
model to the deployment name.
- The Gemini path returns an explicit 501 instead of a bare 500, since no
Gemini backend is provisioned in this lab.
- unified-ai responses carry UAIG-Backend / UAIG-API-Type / UAIG-Model /
UAIG-Is-Streaming debug headers, which challenge 6 validates.
Also align the uami-client-id named value: resources.bicep created it with
displayName 'Hub.Foundry.UAMI.Client.ID' while challenge 1's
llm-policy-fragments.bicep recreates it with displayName 'uami-client-id'.
Running challenge 1 therefore renamed it and APIM silently rewrote every
referencing policy. Setting displayName to match the resource name makes
challenge 1 idempotent here and keeps both templates consistent.
All eight endpoints verified live against a lab resource group as the
participant identity, including the full create/get/input_items responses
chain; the compiled policies are byte-identical to the ones tested.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Challenge 1 calls apimClientTool.discover_api("openai") to find an API served
on the 'openai' path, then uses it for three test sections: the Azure OpenAI
request format, the AzureOpenAI Python SDK, and streaming. No such API existed,
so discovery raised, azure_endpoint_openai was set to None, and all three
sections silently skipped. Challenge 1 also probes GET /openai/deployments.
Add an azure-openai-api (path 'openai') with two operations and policies:
- GET /deployments returns the deployment inventory.
- POST /deployments/{model}/chat/completions maps the model to its deployment
name, rejects unsupported models with 400, and passes through SSE so the
streaming section works.
Products deliberately do not include this API. Challenge 1 authenticates with
the master subscription, so it has access, while product-to-API membership
stays governed by the access contracts in challenge 3 - those contracts pin
apiNameMapping to universal-llm-api and broadening it would weaken the
isolation that challenge 3 asserts.
Verified live against a lab resource group: deployment listing, a chat
completion, and a streaming SSE response.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Challenge 1 runs test_get_deployments against both the Universal LLM API and the Azure OpenAI API, but only the latter served /deployments, so the first call returned 404. Add the operation to the Universal LLM API with the same inventory policy, so both surfaces answer. Verified live: GET /models/deployments returns 200 with all six deployments. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Challenge 4 builds its Sales agent by reading the endpoint and API key from Key Vault with SecretClient. That is a data-plane call, and the lab tenant enforces an Azure Policy (KeyVault_PublicNetwork_Modify) that sets publicNetworkAccess to Disabled on every vault. Setting it back to Enabled is reverted within seconds, so the read fails for every participant and the agent never starts. Challenge 3 is unaffected: it writes its secrets through the ARM control plane, which the vault firewall does not gate. Only reads are blocked, and ARM does not return secret values, so there is no control-plane way to read them back. Give the agent optional fallback_endpoint and fallback_api_key arguments and use them when the Key Vault read raises. The access contract still creates and populates the vault, so the pattern is demonstrated, but the agent falls back to the same gateway endpoint and contract subscription key taken from APIM. The source is reported so participants can see which path was used. Verified live: the endpoint and key the fallback builds return 200 from /models/chat/completions using the Sales contract subscription. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Explain that the Key Vault warning is expected in this environment and that the run continues on APIM credentials, so participants do not read it as a failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Challenge 5 deploys access contracts whose product policy includes pii-anonymization, pii-deanonymization and pii-state-saving. All three were placeholders commented out as DISABLED, so the contract deployment failed with "Policy fragment with id 'pii-anonymization' could not be found" and no PII was ever masked. The notebook's masking tests count HTTP 200 as success, so they reported 6/6 passed while the model was receiving unmasked names and SSNs. Load the real fragments from the workshop sources via loadTextContent so the two copies cannot drift, and add the three things they depend on: - named value piiServiceUrl pointing at the hub account's cognitiveservices host (the openai.azure.com host does not serve the Language PII API) - Cognitive Services User on the hub account for the APIM identity; the existing OpenAI User role does not cover the Language service - pii-usage-eventhub-logger plus Azure Event Hubs Data Sender, required by log-to-eventhub in pii-state-saving The logger must exist before the fragment: APIM accepts a fragment referencing a missing logger, reports InProgress, then discards it without surfacing an error anywhere. Verified against a live lab: the blocking contract now returns HTTP 400 with code PII_DETECTED and categories USSocialSecurityNumber, Person for a request containing PII, and 200 for one without. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
…dModels
GET /models/models returned a bare JSON array of model names built from a
hardcoded list in the operation policy. Notebook 2 reads
data.get('data') or data.get('value'), which raises AttributeError on a
list, so discovery found zero models and every downstream model test was
skipped. The hardcoded list also ignored each access contract's
allowedModels, so every subscription saw all six models regardless of
what its contract permitted.
Delegate to the get-available-models fragment and wrap its output as
{ "object": "list", "data": [...] }. The lab's placeholder fragment now
follows the same contract as the real one deployed by challenge 1 - set
availableModels and availableFilteredModels, return nothing - so the
operation behaves identically before and after that challenge runs.
Both policies move to files loaded with loadTextContent, which keeps them
readable and avoids hand-escaping XML inside Bicep strings.
Verified live: master and the universal-llm contract see all 6 models,
while llm-support-bot-dev-sub-01 correctly sees only gpt-4.1 and
Mistral-Large-3.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
The masking tests treated any 200 as a pass. Because pii-deanonymization restores the original values on the way out, the response looks the same whether or not anything was masked, so the section reported 6/6 passed while the model was receiving unmasked names and SSNs. For a governance lab that is the worst possible failure mode: a green score that hides the control being absent. Add a probe that asks a question whose answer depends on what the model actually saw. When anonymization runs the model gets a placeholder such as <Person_1> and cannot tell the name starts with "J"; only an unmasked request answers YES. The overall summary now zeroes the masking score unless that probe passes. Checked both directions against a live lab: the masking product answers NO and verifies, the non-masking product answers YES and correctly fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Use Case 3 reads PII telemetry from Cosmos DB and ends the notebook with a CosmosHttpResponseError (Forbidden) traceback. Two independent things block it, neither fixable by a participant: - CosmosDB_PublicNetwork_Modify keeps the account's public network access disabled. This is not a setting anyone can toggle in the portal: an az cosmosdb update --public-network-access ENABLED reports success and the value never changes at all. CosmosDB_LocalAuth_Modify also forces disableLocalAuth, ruling out the cosmos_auth_mode="key" fallback. - pii-usage-container has no writer. resources.bicep names a usage ingestion Logic App and creates its managed identity, but never creates the Logic App, and there is no other ingestion resource in the group. The query would return zero records even with network access. Explain both in the notebook and the challenge guide, and catch the connection failure so the section reports "no data" and the notebook runs to completion instead of dying on the last use case. The audit trail behind the masking policy does work and is worth pointing at instead: pii-state-saving emits to the usage-events Event Hub, which recorded incoming messages during a real run. The success criteria now reference that and the masking probe rather than the Cosmos report. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part A told participants to paste every value from the MicroHack dashboard's Credentials tab. That tab only exposes the 7 platform-provided Azure entries, so most keys -- including LLM_BACKEND_CONFIG -- were not obtainable that way, and the instructions had no Azure sign-in step at all.
Part A is now five steps matching what the environment actually provides:
The bracket warning is the important one: the value must end with }]}], and dropping the last two characters produces a JSON error pointing ~1000 characters away from the mistake. Adds a note to restart the Jupyter kernel after editing .env, since the loader skips keys already in os.environ and a bad value read at startup otherwise persists for the whole session.
Updates .env.template to match -- its per-key comments named dashboard credentials that do not exist -- and adjusts the context paragraph and troubleshooting table.