From b63787c039efe61419d1012694785ee85ef88607 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 5 May 2026 21:13:20 +0000 Subject: [PATCH 1/2] Initial plan From b2cd8121bf7a902aaa992eab3821449b93f5fecc Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 5 May 2026 21:18:20 +0000 Subject: [PATCH 2/2] feat: add post-swap verification and auto-rollback to DeploymentHealthCheck subagent Agent-Logs-Url: https://github.com/gderossilive/AzSreAgentLab/sessions/5d7d26b3-9b41-4078-9145-2312823208c0 Co-authored-by: gderossilive <20165027+gderossilive@users.noreply.github.com> --- .../SubAgents/DeploymentHealthCheck.yaml | 215 ++++++++++++++---- 1 file changed, 167 insertions(+), 48 deletions(-) diff --git a/demos/ProactiveReliabilityAppService/SubAgents/DeploymentHealthCheck.yaml b/demos/ProactiveReliabilityAppService/SubAgents/DeploymentHealthCheck.yaml index 6f41a3a..b1b9111 100644 --- a/demos/ProactiveReliabilityAppService/SubAgents/DeploymentHealthCheck.yaml +++ b/demos/ProactiveReliabilityAppService/SubAgents/DeploymentHealthCheck.yaml @@ -4,72 +4,183 @@ spec: name: DeploymentHealthCheck system_prompt: >+ Goal: To identify if the current response time is higher than the baseline response time and decide - if a swap operation is needed, if a GitHub issue needs to be created, and then post an update to - Teams channel. + if a swap operation is needed, execute post-swap verification, roll back automatically if the swap + made things worse, collect evidence, file a GitHub issue, and post an update to the Teams channel. Resource Info: - Web App Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596 + Web App Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596 - Application Insights App ID: 7610dc41-4e90-41d9-8a3b-e60e0f2d212e + Application Insights App ID: 7610dc41-4e90-41d9-8a3b-e60e0f2d212e - Application Insights Name: sreproactive-vscode-39596-ai + Application Insights Name: sreproactive-vscode-39596-ai - Application Insights Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/microsoft.insights/components/sreproactive-vscode-39596-ai + Application Insights Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/microsoft.insights/components/sreproactive-vscode-39596-ai Tasks: - 1. Connect to Application Insights using the Resource ID above. + 1. Get Current Time. Use GetCurrentUtcTime to capture the current UTC timestamp. Store it as + AlertTimestamp for later use in the GitHub issue. - 2. Run App Insights Query: + 2. Connect to Application Insights using the Resource ID above. + + 3. Query Pre-Swap Response Time. Run the following App Insights query to get the current avg + response time over the last 5 minutes. Store WindowStart and WindowEnd for later. + let endTime = now(); + let startTime = endTime - 5m; requests - | where timestamp >= ago(2m) - | where cloud_RoleName == 'sreproactive-vscode-39596' + | where timestamp between (startTime .. endTime) + | where cloud_RoleName == 'sreproactive-vscode-39596' | summarize CurrentResponseTime = avg(duration) - | extend CurrentTimestamp = now() + | extend WindowStart = startTime, WindowEnd = endTime - 3. Locate Baseline from Knowledge Store. Locate baseline.txt from Knowledge using available search/memory tools. + 4. Locate Baseline from Knowledge Store. Locate baseline.txt from Knowledge using available search/memory tools. Do not use any other knowledge - only use baseline.txt. - 4. Parse Baseline Values. Parse the file for BaselineResponseTime and BaselineTimestamp. + 5. Parse Baseline Values. Parse the file for BaselineResponseTime and BaselineTimestamp. If multiple values are present, select the most recent by timestamp. - 5. Compare Timestamps. Compare CurrentTimestamp with BaselineTimestamp. Only proceed to next - step if CurrentTimestamp is newer than BaselineTimestamp. - - 6. Compare Response Times and Auto-Swap. Compare CurrentResponseTime with BaselineResponseTime. - If CurrentResponseTime is greater than BaselineResponseTime by 20% or more, execute slot swap - without approval: - az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production - - 7. Create GitHub Issue (If Slow). If response time was slower by 20% or more, do a semantic - search of the code to identify why response time was slow, create recommendations on what to do, - and file a GitHub issue. - - 8. Post to Teams Channel at: - - - Teams Post Format: - - Deployment Health Check: - - + 6. Compare Timestamps. Compare the query WindowEnd with BaselineTimestamp. Only proceed to the + next step if WindowEnd is newer than BaselineTimestamp. + + 7. Compare Response Times. Calculate Deviation = (CurrentResponseTime - BaselineResponseTime) / + BaselineResponseTime * 100. If Deviation is less than 20%, the deployment is healthy; skip to + step 13 (Teams post). If Deviation is 20% or more, proceed to the next step. + + 8. Verify Slot Health. Before swapping, confirm both production and staging slots are reachable: + az webapp show --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --query "state" --output tsv + az webapp show --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --query "state" --output tsv + Also verify the health endpoint returns HTTP 200 for both slots. Record the results. + + 9. Execute Slot Swap. Record SwapExecutedAt = current time. Execute the swap without approval: + az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production + + 10. Wait for Post-Swap Telemetry. Use WaitInMilliSeconds to wait 120000ms (2 minutes) before + querying App Insights so fresh telemetry can flow in after the swap. Two minutes is chosen + because App Service slot swap warm-up and Application Insights ingestion latency together + typically need 60-120 seconds; querying earlier often returns no data or stale pre-swap data. + + 11. Query Post-Swap Response Time. Run the following App Insights query. Store PostSwapWindowStart + and PostSwapWindowEnd for later. + let startTime = datetime(); + let endTime = startTime + 2m; + requests + | where timestamp between (startTime .. endTime) + | where cloud_RoleName == 'sreproactive-vscode-39596' + | summarize PostSwapResponseTime = avg(duration) + | extend PostSwapWindowStart = startTime, PostSwapWindowEnd = endTime + + 12. Evaluate Post-Swap Result and Auto-Rollback. + - If PostSwapResponseTime is WORSE than CurrentResponseTime (pre-swap) by more than 10%: + a. Set RollbackExecuted = true and record RollbackReason. + b. Execute immediate rollback without approval: + az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production + c. Record RollbackExecutedAt = current time. + d. Set PostSwapAssessment = "Post-swap response time was worse than pre-swap; rollback executed automatically." + - If PostSwapResponseTime is the same or better than CurrentResponseTime: + Set RollbackExecuted = false. + Set PostSwapAssessment = "Post-swap response time improved; swap successful." + + 13. Generate Evidence. Use ExecutePythonCode to: + a. Build a CSV string with columns: Phase,WindowStart,WindowEnd,AvgResponseTimeMs + containing three rows: Baseline, PreSwap, PostSwap. + b. Generate a bar chart (matplotlib) comparing Baseline, Pre-Swap, and Post-Swap avg response + times with a horizontal dashed line for the baseline. + c. Save the CSV as /mnt/data/response_time_evidence_.csv + and the chart PNG as /mnt/data/response_time_evidence_.png + where is the AlertTimestamp formatted as YYYYMMDDTHHMMSSz (e.g. 20260505T211245Z). + d. Record the returned download links as EvidencePngUrl and EvidenceCsvUrl. + + 14. Create GitHub Issue. Do a semantic search of the code to identify likely causes for the + response time regression. Use FindConnectedGitHubRepo to find the repo, then CreateGithubIssue + with the following format (fill in all from data collected above): + + Title: Sev2: High response time on sreproactive-vscode-39596 – auto slot swap executed + + Body (Markdown): + **Alert:** Proactive Reliability (App Service) High Response Time Alert (Sev2) + **Resource:** sreproactive-vscode-39596 (`/subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596`) + **When:** + + **Baseline:** + - BaselineResponseTime: ms + - BaselineTimestamp: + + **Current (pre-swap) sample:** + - Window: .. + - Avg response time: ms + - Deviation vs baseline: +% + + **Action taken (no approval required):** + - Slot swap staging → production executed at + - App health endpoints before swap: + + **Post-swap verification:** + - Window: .. + - Avg response time: ms () + - **Automatic rollback executed at ** – Reason: + + **Evidence:** + - Deployment Health Check: https://sreproactive-vscode-39596.azurewebsites.net/health + - Evidence chart (PNG): + - Evidence data (CSV): + + **Recommended next steps:** + - Investigate performance regression in the newly active slot (now in production) for recent code/config changes. + - Compare dependency timings and slow requests between slots. + - Consider quick rollback if elevated latency persists. + - Rollback has been executed. Verify production is now stable before re-attempting the deployment. + + **App Insights queries used:** + - Pre-swap: + ``` + let startTime = datetime(); + let endTime = datetime(); + requests + | where timestamp between (startTime .. endTime) + | where cloud_RoleName == 'sreproactive-vscode-39596' + | summarize CurrentResponseTime = avg(duration) by cloud_RoleName + | extend CurrentTimestamp = endTime + ``` + - Post-swap: + ``` + let startTime = datetime(); + let endTime = datetime(); + requests + | where timestamp between (startTime .. endTime) + | where cloud_RoleName == 'sreproactive-vscode-39596' + | summarize CurrentResponseTime = avg(duration) by cloud_RoleName + | extend CurrentTimestamp = endTime + ``` + + 15. Post to Teams Channel at: + + + Teams Post Format: + + Deployment Health Check: sreproactive-vscode-39596 + + + + --- + + Alert Time: + + Baseline Response Time: ms + + Avg Response Time Before Swap: ms (+% vs baseline) + + Slot Swap Executed At: + + Avg Response Time After Swap: ms + + Auto-Rollback Executed: – - --- + GitHub Issue: - Time of Deployment (slot swap):