Refine migration monitoring and modernization challenges

03-Azure/01-03-Infrastructure/06_Migration_Secure_AI_Ready/Readme.md

@@ -60,10 +60,10 @@ After completing this Hack, you will be able to:
 * Build an assessment and business case for a datacenter transformation.
 * Plan and execute a right-sized Hyper-V VM migration to Azure.
 * Apply security practices to migrated workloads.
-* Enable Azure Monitor Agent, DCR-based telemetry, VM insights, and actionable alerting.
-* Investigate a deterministic IIS or Apache incident with Azure Copilot Observability Agent or a KQL fallback.
-* Restore and verify service using evidence from both the VM and Azure Monitor.
-* Inspect a selected IIS or Apache workload and manually replatform its static content to Windows App Service with ZIP deployment.
+* Enable OpenTelemetry VM metrics, DCR-based Windows event collection, and service-specific alerting.
+* Investigate deterministic IIS and Apache incidents with Azure Copilot Observability Agent or the manual/KQL fallback.
+* Restore and verify both services using evidence from the VMs and Azure Monitor.
+* Select an IIS or Apache workload and manually replatform its static content to the appropriate Windows or Linux App Service target with ZIP deployment.
 * Explain why Azure Storage static website hosting or Azure Static Web Apps can be a better production target for a static workload.
 
 <a id="hack-challenges"></a>
@@ -79,7 +79,7 @@ Complete these prerequisites before the session:
 * A Microsoft Entra user with Contributor or Owner permissions on the Azure subscription.
 * A client network and browser that support the Azure portal, Azure Bastion, and Azure Cloud Shell.
 
-Challenge 7 includes a preflight for Azure Copilot tenant access, RBAC, network access, and supported regions. Azure Copilot is not mandatory because the challenge includes a full KQL/VM insights fallback. No Azure OpenAI resource, model deployment, or model quota is required anywhere in this Hack.
+Challenge 7 assumes Hack participants have Azure Copilot access, suitable RBAC, network access, and a supported region. It also includes a concise manual/KQL fallback if agent output is unavailable. No Azure OpenAI resource, model deployment, or model quota is required anywhere in this Hack.
 
 ## Challenges
 
@@ -89,8 +89,8 @@ Challenge 7 includes a preflight for Azure Copilot tenant access, RBAC, network
 * [Challenge 4 - Assess VMs for the migration](challenges/challenge-04.md)
 * [Challenge 5 - Migrate Hyper-V virtual machines to Azure](challenges/challenge-05.md)
 * [Optional Challenge 6 - Secure on Azure](challenges/challenge-06.md)
-* [Challenge 7 - Operate the migrated workload with intelligent observability](challenges/challenge-07.md)
-* [Challenge 8 - Replatform the selected migrated web workload to Azure App Service](challenges/challenge-08.md)
+* [Challenge 7 - Operate the migrated workloads with intelligent observability](challenges/challenge-07.md)
+* [Challenge 8 - Replatform a migrated web workload to Azure App Service](challenges/challenge-08.md)
 
 ## Solutions - Spoiler warning
 
@@ -100,8 +100,8 @@ Challenge 7 includes a preflight for Azure Copilot tenant access, RBAC, network
 * [Solution 4 - Assess VMs for the migration](./walkthrough/challenge-04/solution-04.md)
 * [Solution 5 - Migrate Hyper-V virtual machines to Azure](./walkthrough/challenge-05/solution-05.md)
 * [Optional Solution 6 - Secure on Azure](./walkthrough/challenge-06/solution-06.md)
-* [Solution 7 - Operate the migrated workload with intelligent observability](./walkthrough/challenge-07/solution-07.md)
-* [Solution 8 - Replatform the selected migrated web workload to Azure App Service](./walkthrough/challenge-08/solution-08.md)
+* [Solution 7 - Operate the migrated workloads with intelligent observability](./walkthrough/challenge-07/solution-07.md)
+* [Solution 8 - Replatform a migrated web workload to Azure App Service](./walkthrough/challenge-08/solution-08.md)
 
 ## Contributors
 

03-Azure/01-03-Infrastructure/06_Migration_Secure_AI_Ready/challenges/challenge-05.md

@@ -15,16 +15,17 @@ The goal of this exercise is to ...
 * Configure and register the Hyper-V replication provider and Recovery Services agent on the Hyper-V host.
 * Select the Windows and Ubuntu VMs and enable replication.
 * Review the compute, network, and disk target settings.
-* Run and clean up test migrations.
-* Prepare destination load balancing and cutover.
+* Run, validate, and clean up test migrations for both VMs.
 * Perform the final migration with a planned shutdown.
-* Validate the migrated workloads and complete the migration to stop replication.
+* Directly validate both migrated web workloads in Azure.
+* Complete both migrations to stop replication.
 
 ## Success criteria
 
-* The Windows and Ubuntu web VMs are successfully migrated to and running in Azure.
-* Both web workloads are accessible through the dedicated public load balancer.
-* Test-migration resources are cleaned up, and the final migrations are completed.
+* The Windows/IIS and Ubuntu/Apache VMs are successfully replicated and test-migrated.
+* Both test-migrated web workloads return HTTP success with the correct hostname, platform, and web-server details.
+* Test-migration resources are cleaned up.
+* Both web VMs are successfully migrated, directly validated in Azure, and completed in Azure Migrate.
 
 ## Learning resources
 * [Migrate Hyper-V VMs to Azure](https://learn.microsoft.com/en-us/azure/migrate/tutorial-migrate-hyper-v?view=migrate)

03-Azure/01-03-Infrastructure/06_Migration_Secure_AI_Ready/challenges/challenge-07.md

@@ -1,89 +1,87 @@
-# Challenge 7 - Operate the migrated workload with intelligent observability
+# Challenge 7 - Operate the migrated workloads with intelligent observability
 
 [Previous Challenge](challenge-06.md) - **[Home](../Readme.md)** - [Next Challenge](challenge-08.md)
 
-Duration: 40 minutes
+Duration: 70 minutes
 
 ## Goal
 
-Operate one migrated web workload by combining Azure Monitor telemetry, an actionable alert, and an AI-assisted investigation. You will create a safe service-level incident, determine its scope, restore service, and prove recovery.
+Operate both migrated web workloads through a closed-loop incident exercise. Enable modern OpenTelemetry monitoring, create an alert from the authoritative signal for each workload, induce controlled service failures, investigate the fired alerts with Azure Copilot Observability Agent, restore service under human control, and prove recovery.
 
-## Select one track
+## Workloads and authoritative signals
 
-At the start of this challenge, select one track. Follow the same VM and track through Challenge 8. Teams do not need to complete both.
+| Workload | Service | Authoritative alert signal |
+| --- | --- | --- |
+| Windows Server / IIS | `W3SVC` | Windows System event ID 7036 in Log Analytics |
+| Ubuntu Linux / Apache | `apache2` | Absence of all matching `process.uptime` OpenTelemetry series |
 
-| Track | Migrated workload | Service | Incident telemetry |
-| --- | --- | --- | --- |
-| A | Windows Server / IIS | `W3SVC` | Windows System event ID 7036 in `Event` |
-| B | Ubuntu Linux / Apache | `apache2` | Tagged `daemon.warning` record in `Syslog` |
+The Linux alert monitors every process whose `"process.executable.name"` is `apache2`. It becomes true only when all matching Apache worker and parent process series are absent for the configured window. The Windows alert intentionally uses the Service Control Manager event instead of `w3wp.exe`; IIS application-pool idle shutdown can stop a worker process while the web service remains healthy.
 
-## Prerequisites and preflight
+## Prerequisites and access assumptions
 
-* Challenge 5 is complete and the selected migrated web VM in `destination-rg` is running.
-* The selected IIS or Apache site responds locally and through its existing endpoint.
-* You can create or reuse a Log Analytics workspace, install VM extensions, create data collection rules (DCRs), create alert rules, and create or reuse an action group.
-* Use a Log Analytics workspace in the same region as the selected VM for this lab.
-* Confirm that the workspace region is listed in the current [Azure Copilot Observability Agent regions](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/observability-agent-overview#regions).
-* In the Azure portal, confirm that you can open Azure Copilot and that **Observability Agent** appears on the Log Analytics workspace **Logs** page.
-* If tenant access is restricted, an administrator must grant the **Copilot for Azure User** role. Your account must also be able to read the VM and its monitoring data; **Monitoring Reader** and **Log Analytics Reader** are suitable least-privilege starting points.
-* The client network must allow WebSocket connections to `https://directline.botframework.com`.
+* Challenge 5 is complete and both migrated web VMs in `destination-rg` are running.
+* You can connect to both VMs through Azure Bastion and each site responds locally.
+* Before opening **Infrastructure monitoring**, you can create or reuse one user-assigned managed identity for both migrated VMs. Using the destination resource group and the same region is recommended for this lab.
+* You can grant that identity **Monitoring Reader** on both monitored VMs or their containing destination resource group, and on the Azure Monitor workspace when the alert uses workspace scope.
+* This user-assigned identity authorizes OpenTelemetry query-based and recommended alerts. It is separate from the VM's system-assigned identity used by Azure Monitor Agent; the portal doesn't automatically create the user-assigned alert identity.
+* Azure Copilot access, a supported region, and the required telemetry permissions are available to Hack participants. No Azure OpenAI resource, model deployment, or model quota is required.
+* The client network allows WebSocket connections to `https://directline.botframework.com`.
 
 > [!IMPORTANT]
-> Azure Copilot availability is tenant-, user-, region-, and policy-dependent. No Azure OpenAI resource, model deployment, or model quota is required. If the preflight fails, complete the clearly labeled KQL and VM insights fallback. The monitoring, incident, remediation, and recovery outcomes remain the same.
+> Azure Copilot Observability Agent, deep investigations, Azure Monitor issues, and query-based metric alerts include preview experiences whose portal labels and availability can vary by region and tenant. A deep investigation can consume Azure Agent Credits. Follow your instructor's lab access and cost guidance.
 
 > [!NOTE]
-> On-demand chat and investigation do not require an Azure Copilot Observability Agent resource. Creating that preview resource is outside the required path.
+> On-demand chat and deep investigations use the signed-in user's identity and require no `Microsoft.Monitor/observabilityAgents` resource. Do not provision an autonomous Observability Agent resource for this VM-only Hack. The portal preview scopes that resource to Application Insights, and virtual-machine scope isn't a supported new configuration.
 
-## Actions
-
-### Common actions
-
-* Create or reuse a Log Analytics workspace.
-* Enable enhanced monitoring/VM insights on the selected VM with Azure Monitor Agent.
-* Verify heartbeat and performance telemetry.
-* Create or reuse an action group and create an actionable log search alert.
-* Use Azure Copilot Observability Agent chat, or the KQL/VM insights fallback, to distinguish a service outage from a VM outage.
-* Restore the selected service and verify local, external, and telemetry recovery.
-* Decide which lab monitoring resources should remain after the Hack.
+## Monitoring models
 
-### Track A - Windows Server / IIS
+| | OpenTelemetry metrics-based | Classic logs-based |
+| --- | --- | --- |
+| Store | Azure Monitor workspace | Log Analytics workspace |
+| Query language | PromQL | KQL |
+| Typical latency | Near real-time | 1-3 minutes |
+| Cost model | Default metrics are free; added metrics can incur cost | Log ingestion and retention charges |
+| Strength | Consistent Windows/Linux schema and low-latency metrics | Multi-VM views and same-workspace metric/log correlation |
+| Limitation | Separate queries are required to correlate with logs; built-in multi-VM views are limited | Platform-specific counters and higher ingestion cost |
 
-* Associate a Windows event DCR that collects System event ID 7036.
-* Alert on `W3SVC` stopping, safely stop IIS, and prove the local outage.
-* Investigate `Event`, `Heartbeat`, and `InsightsMetrics`, then restore `W3SVC`.
+Enable metrics-based OpenTelemetry monitoring for both VMs. Use classic log collection only on Windows where the authoritative `W3SVC` service event must be delivered to Log Analytics.
 
-### Track B - Ubuntu Linux / Apache
-
-* Associate a Linux Syslog DCR that collects the `daemon` facility at `Warning` or higher.
-* Stop `apache2` and emit an explicit tagged Syslog event that records the controlled lab action.
-* Alert on the exact tag/message, investigate `Syslog`, `Heartbeat`, and `InsightsMetrics`, then restore `apache2`.
+## Actions
 
-> [!NOTE]
-> The explicit Linux Syslog event makes lab ingestion deterministic without relying only on distribution-dependent systemd journal forwarding.
+* Warm and validate both web pages before changing either service.
+* Create one reusable user-assigned alert identity before opening Infrastructure monitoring, grant its required **Monitoring Reader** access, and select that existing identity in the portal's **Identity** field for both VMs.
+* Let Infrastructure monitoring select or create the regional default Azure Monitor workspace and generated OpenTelemetry DCR; do not pre-create a custom workspace.
+* Enable OpenTelemetry metrics, per-process metrics, built-in Grafana dashboards, and recommended infrastructure alerts.
+* Confirm query-based and recommended alerts use the pre-created identity and that it retains the required monitoring read access.
+* Locate the Linux VM's generated `MSVMOtel-<region>-<name>` DCR and ensure that `process.uptime` is collected.
+* Reuse the portal-selected compatible Log Analytics workspace for Windows events and associate a focused Event DCR.
+* Create an Apache PromQL absence alert and an IIS KQL scheduled-query alert.
+* Stop Apache and IIS in a controlled manner, prove local HTTP failure, and wait for both alerts.
+* Start a deep investigation from each fired alert, evaluate evidence and ruled-out hypotheses, and save useful context as an Azure Monitor issue when available.
+* Restore both services manually, validate both pages and telemetry recovery, confirm alert resolution, and request a post-incident summary and runbook improvements.
 
 ## Success criteria
 
-* `AzureMonitorWindowsAgent` or `AzureMonitorLinuxAgent` is healthy and the VM has the intended DCR associations.
-* `Heartbeat`, `InsightsMetrics`, and the selected track's `Event` or `Syslog` record are visible in the Log Analytics workspace.
-* An enabled alert rule monitors the selected service-stop event and has an action group.
-* The incident is proven by local HTTP failure, an inactive service, and the selected track's ingested event.
-* The investigation shows whether the VM remained online and produces a safe service-level remediation.
-* `W3SVC` or `apache2` is running again and the site responds successfully.
-
-> [!NOTE]
-> Log ingestion and alert evaluation are asynchronous. The walkthrough uses the ingested event query, not a notification arrival time, as the deterministic success gate. Alert notification delivery is still reviewed as an operational outcome.
-
-## Optional stretch - autonomous operations (preview)
-
-Only if your instructor has prepared the required resources and permissions, explore an [Azure Copilot Observability Agent resource](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/observability-agent-resource) for autonomous alert correlation, issue creation, and custom instructions. The resource, autonomous operations, Azure Monitor issues, and custom-instruction experience are preview capabilities. They are not required for this challenge.
+* Both VMs send default OpenTelemetry guest metrics to the portal-selected Azure Monitor workspace through their intended generated DCR associations.
+* Linux `process.uptime` metrics include one or more `apache2` process series before the incident.
+* Windows System event ID 7036 records for `W3SVC` are visible in the portal-selected Log Analytics workspace.
+* The Apache PromQL alert and IIS KQL alert are enabled, use the intended signals, and fire after the controlled failures.
+* An inactive service and local HTTP failure prove user-visible impact on each VM; process or service telemetry alone isn't treated as a synthetic availability test.
+* The AI-assisted or manual investigation distinguishes service failure from VM/platform failure and produces evidence-backed remediation with rollback and prevention guidance.
+* A human restores `W3SVC` and `apache2`; both sites return HTTP `200`, service/process telemetry returns, and alerts resolve.
+* Both recovered workloads are available for the path selection at the start of Challenge 8.
 
 ## Learning resources
 
-* [Azure Copilot Observability Agent overview](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/observability-agent-overview)
-* [Chat with observability data](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/observability-agent-chat)
-* [Manage access to Azure Copilot](https://learn.microsoft.com/en-us/azure/copilot/manage-access)
-* [Enable VM monitoring in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/vm/vm-enable-monitoring)
-* [Install and manage Azure Monitor Agent](https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-manage)
-* [Collect Windows events with Azure Monitor Agent](https://learn.microsoft.com/en-us/azure/azure-monitor/vm/data-collection-windows-events)
-* [Collect Syslog events with Azure Monitor Agent](https://learn.microsoft.com/en-us/azure/azure-monitor/vm/data-collection-syslog)
-* [Create Azure Monitor log search alert rules](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-create-log-alert-rule)
+* [Metrics experience for virtual machines](https://learn.microsoft.com/azure/azure-monitor/vm/metrics-opentelemetry-guest)
+* [Enable enhanced monitoring for an Azure VM](https://learn.microsoft.com/azure/azure-monitor/vm/tutorial-enable-monitoring)
+* [Customize OpenTelemetry metrics for VMs](https://learn.microsoft.com/azure/azure-monitor/vm/metrics-opentelemetry-guest-modify)
+* [OpenTelemetry guest OS metrics reference](https://learn.microsoft.com/azure/azure-monitor/vm/metrics-guest-reference)
+* [PromQL for system and guest OS metrics](https://learn.microsoft.com/azure/azure-monitor/metrics/prometheus-system-metrics-best-practices)
+* [Query-based metric alerts](https://learn.microsoft.com/azure/azure-monitor/alerts/alerts-query-based-metric-alerts-overview)
+* [Create a user-assigned managed identity in the Azure portal](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/manage-user-assigned-managed-identities-azure-portal)
+* [Assign Azure roles using the Azure portal](https://learn.microsoft.com/azure/role-based-access-control/role-assignments-portal)
+* [Collect Windows events with Azure Monitor Agent](https://learn.microsoft.com/azure/azure-monitor/vm/data-collection-windows-events)
+* [Create Azure Monitor log search alert rules](https://learn.microsoft.com/azure/azure-monitor/alerts/alerts-create-log-alert-rule)
+* [Azure Copilot Observability Agent overview](https://learn.microsoft.com/azure/azure-monitor/aiops/observability-agent-overview)
+* [Deep investigations with Observability Agent](https://learn.microsoft.com/azure/azure-monitor/aiops/observability-agent-deep-investigations)