Repository navigation
Agentic Governance Micro Hack - #573
Merged
Merged
Conversation
…use error The region-fallback loop assumed an empty resource group. When a lab is re-deployed into an RG that already holds a previous run, the first attempt pins every resource name to that region, so each subsequent candidate fails with "InvalidResourceLocation: mi-apim-<token> already exists in location <first region>". The retries can never succeed, and the final error reports only the last region's symptom, hiding the real preflight failure. Observed in the Sept-10 event: labuser-0005's RG still contained the previous event's stack, so a CognitiveServices preflight error in swedencentral was reported as "failed in all candidate regions: swedencentral, westeurope, norwayeast" with the actual cause discarded. - Pin to the region the RG already uses when it is non-empty, preferring one the platform asked for. 'global' resources are ignored, since they are not deployable regions; an RG holding only global resources still gets the full fallback list. - Carry the first failure into the final throw, as later regions report downstream symptoms rather than the root cause. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
resourceToken is derived from subscription id + resource group name, and the platform reuses both when an event is torn down and a new one is deployed over it. Foundry accounts and Key Vaults are soft deleted rather than removed, and a soft-deleted resource keeps its name reserved -- for Foundry, also its globally unique customSubDomainName. Re-deploying therefore fails during preflight with an opaque "Microsoft.CognitiveServices/accounts (2025-04-01-preview) reported preflight validation errors" before any resource is touched. This took out all three labs on the Sept-10 re-run (event 004). The same two labs had deployed cleanly an hour earlier (event 003) when their resource groups were still empty, which is why it read as an environment problem rather than a template one. Purge any soft-deleted Foundry account or Key Vault whose name ends in this lab's token before deploying. Matching on the token means labs sharing a subscription are never touched, and a purge failure stays non-fatal so the deployment still surfaces the underlying preflight error. Also corrects the README's "re-runs skip existing resources" claim, which held only for resources that are not soft deleted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
Re-deploying an event into a reused resource group failed for every participant
(Sept-10 event 004). Root cause, reproduced and confirmed locally:
An `AIServices` account with `allowProjectManagement` is backed by a hidden Azure ML
(`AmlRp`) workspace. Deleting the account soft-deletes that workspace for ~14 days.
Purging the Cognitive Services account does NOT purge it, and Azure exposes no API to
list or purge it (`deletedWorkspaces` 404s on every api-version; `forceToPurge` DELETEs
are no-ops). Recreating the account therefore always lands in `provisioningState:
Failed` with "Soft-deleted workspace exists", which in turn poisons the resource group
permanently. Since `resourceToken` is derived from subscription id + RG name and the
platform reuses both, every re-run was guaranteed to fail.
Proven by isolation test: purge + recreate under the same name failed three times;
a bare account under a fresh name succeeded.
- Foundry accounts now take a `foundryNameSuffix`. The script reuses the existing
accounts while they are healthy, and otherwise deploys under a fresh name, recording
the suffix as the `citadelFoundrySuffix` resource-group tag so re-runs stay stable.
- Recycle Foundry accounts stuck in `Failed` state, which ARM otherwise refuses to
update ('AccountIsNotSucceeded').
- Wait for purges to propagate before deploying; purging is asynchronous, and
recreating a name too early was itself creating the Failed accounts.
- Fix the Key Vault purge verb (POST .../purge; DELETE returns 405) and add the
missing APIM purge scope ('ServiceAlreadyExistsInSoftDeletedState').
- Retry Azure REST calls: a transient TLS error silently skipped the rest of the purge
pass, and per-item isolation stops one failure skipping the others.
- Retry a deployment in place up to 3 times for genuinely transient states
(APIM 'ServiceLocked', AML soft-delete lag), re-running cleanup before each attempt.
- Document the extra Foundry suffix in challenge-01 (attendees now read the two account
names directly instead of building them from the token) and in the lab README.
Verified end-to-end: full local deployment completes with 6 model deployments, both
Foundry projects and all RBAC grants succeeding.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Citadel adaptation from BE workshop (tested during the hackathon in Brussels)