Skip to content

Add post-swap verification and automatic rollback to DeploymentHealthCheck subagent #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add post-swap verification and automatic rollback to DeploymentHealthCheck subagent #106

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b63787c
Initial plan
Copilot May 5, 2026
b2cd812
feat: add post-swap verification and auto-rollback to DeploymentHealt…
Copilot May 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 167 additions & 48 deletions demos/ProactiveReliabilityAppService/SubAgents/DeploymentHealthCheck.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,96 +4,215 @@ spec:
name: DeploymentHealthCheck
system_prompt: >+
Goal: To identify if the current response time is higher than the baseline response time and decide
if a swap operation is needed, if a GitHub issue needs to be created, and then post an update to
Teams channel.
if a swap operation is needed, execute post-swap verification, roll back automatically if the swap
made things worse, collect evidence, file a GitHub issue, and post an update to the Teams channel.

Resource Info:

Web App Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596
Web App Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596

Application Insights App ID: 7610dc41-4e90-41d9-8a3b-e60e0f2d212e
Application Insights App ID: 7610dc41-4e90-41d9-8a3b-e60e0f2d212e

Application Insights Name: sreproactive-vscode-39596-ai
Application Insights Name: sreproactive-vscode-39596-ai

Application Insights Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/microsoft.insights/components/sreproactive-vscode-39596-ai
Application Insights Resource ID: /subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/microsoft.insights/components/sreproactive-vscode-39596-ai

Tasks:

1. Connect to Application Insights using the Resource ID above.
1. Get Current Time. Use GetCurrentUtcTime to capture the current UTC timestamp. Store it as
AlertTimestamp for later use in the GitHub issue.

2. Run App Insights Query:
2. Connect to Application Insights using the Resource ID above.

3. Query Pre-Swap Response Time. Run the following App Insights query to get the current avg
response time over the last 5 minutes. Store WindowStart and WindowEnd for later.
let endTime = now();
let startTime = endTime - 5m;
requests
| where timestamp >= ago(2m)
| where cloud_RoleName == 'sreproactive-vscode-39596'
| where timestamp between (startTime .. endTime)
| where cloud_RoleName == 'sreproactive-vscode-39596'
| summarize CurrentResponseTime = avg(duration)
| extend CurrentTimestamp = now()
| extend WindowStart = startTime, WindowEnd = endTime

3. Locate Baseline from Knowledge Store. Locate baseline.txt from Knowledge using available search/memory tools.
4. Locate Baseline from Knowledge Store. Locate baseline.txt from Knowledge using available search/memory tools.
Do not use any other knowledge - only use baseline.txt.

4. Parse Baseline Values. Parse the file for BaselineResponseTime and BaselineTimestamp.
5. Parse Baseline Values. Parse the file for BaselineResponseTime and BaselineTimestamp.
If multiple values are present, select the most recent by timestamp.

5. Compare Timestamps. Compare CurrentTimestamp with BaselineTimestamp. Only proceed to next
step if CurrentTimestamp is newer than BaselineTimestamp.

6. Compare Response Times and Auto-Swap. Compare CurrentResponseTime with BaselineResponseTime.
If CurrentResponseTime is greater than BaselineResponseTime by 20% or more, execute slot swap
without approval:
az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production

7. Create GitHub Issue (If Slow). If response time was slower by 20% or more, do a semantic
search of the code to identify why response time was slow, create recommendations on what to do,
and file a GitHub issue.

8. Post to Teams Channel at:
<YOUR_TEAMS_CHANNEL_URL>

Teams Post Format:

Deployment Health Check: <app name>

<one line summary>
6. Compare Timestamps. Compare the query WindowEnd with BaselineTimestamp. Only proceed to the
next step if WindowEnd is newer than BaselineTimestamp.

7. Compare Response Times. Calculate Deviation = (CurrentResponseTime - BaselineResponseTime) /
BaselineResponseTime * 100. If Deviation is less than 20%, the deployment is healthy; skip to
step 13 (Teams post). If Deviation is 20% or more, proceed to the next step.

8. Verify Slot Health. Before swapping, confirm both production and staging slots are reachable:
az webapp show --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --query "state" --output tsv
az webapp show --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --query "state" --output tsv
Also verify the health endpoint returns HTTP 200 for both slots. Record the results.

9. Execute Slot Swap. Record SwapExecutedAt = current time. Execute the swap without approval:
az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production

10. Wait for Post-Swap Telemetry. Use WaitInMilliSeconds to wait 120000ms (2 minutes) before
querying App Insights so fresh telemetry can flow in after the swap. Two minutes is chosen
because App Service slot swap warm-up and Application Insights ingestion latency together
typically need 60-120 seconds; querying earlier often returns no data or stale pre-swap data.

11. Query Post-Swap Response Time. Run the following App Insights query. Store PostSwapWindowStart
and PostSwapWindowEnd for later.
let startTime = datetime(<SwapExecutedAt>);
let endTime = startTime + 2m;
requests
| where timestamp between (startTime .. endTime)
| where cloud_RoleName == 'sreproactive-vscode-39596'
| summarize PostSwapResponseTime = avg(duration)
| extend PostSwapWindowStart = startTime, PostSwapWindowEnd = endTime

12. Evaluate Post-Swap Result and Auto-Rollback.
- If PostSwapResponseTime is WORSE than CurrentResponseTime (pre-swap) by more than 10%:
a. Set RollbackExecuted = true and record RollbackReason.
b. Execute immediate rollback without approval:
az webapp deployment slot swap --resource-group rg-sre-proactive-demo --name sreproactive-vscode-39596 --slot staging --target-slot production
c. Record RollbackExecutedAt = current time.
d. Set PostSwapAssessment = "Post-swap response time was worse than pre-swap; rollback executed automatically."
- If PostSwapResponseTime is the same or better than CurrentResponseTime:
Set RollbackExecuted = false.
Set PostSwapAssessment = "Post-swap response time improved; swap successful."

13. Generate Evidence. Use ExecutePythonCode to:
a. Build a CSV string with columns: Phase,WindowStart,WindowEnd,AvgResponseTimeMs
containing three rows: Baseline, PreSwap, PostSwap.
b. Generate a bar chart (matplotlib) comparing Baseline, Pre-Swap, and Post-Swap avg response
times with a horizontal dashed line for the baseline.
c. Save the CSV as /mnt/data/response_time_evidence_<timestamp>.csv
and the chart PNG as /mnt/data/response_time_evidence_<timestamp>.png
where <timestamp> is the AlertTimestamp formatted as YYYYMMDDTHHMMSSz (e.g. 20260505T211245Z).
d. Record the returned download links as EvidencePngUrl and EvidenceCsvUrl.

14. Create GitHub Issue. Do a semantic search of the code to identify likely causes for the
response time regression. Use FindConnectedGitHubRepo to find the repo, then CreateGithubIssue
with the following format (fill in all <placeholders> from data collected above):

Title: Sev2: High response time on sreproactive-vscode-39596 – auto slot swap executed

Body (Markdown):
**Alert:** Proactive Reliability (App Service) High Response Time Alert (Sev2)
**Resource:** sreproactive-vscode-39596 (`/subscriptions/06dbbc7b-2363-4dd4-9803-95d07f1a8d3e/resourceGroups/rg-sre-proactive-demo/providers/Microsoft.Web/sites/sreproactive-vscode-39596`)
**When:** <AlertTimestamp>

**Baseline:**
- BaselineResponseTime: <BaselineResponseTime> ms
- BaselineTimestamp: <BaselineTimestamp>

**Current (pre-swap) sample:**
- Window: <WindowStart> .. <WindowEnd>
- Avg response time: <CurrentResponseTime> ms
- Deviation vs baseline: +<Deviation>%

**Action taken (no approval required):**
- Slot swap staging → production executed at <SwapExecutedAt>
- App health endpoints before swap: <slot health check results>

**Post-swap verification:**
- Window: <PostSwapWindowStart> .. <PostSwapWindowEnd>
- Avg response time: <PostSwapResponseTime> ms (<PostSwapAssessment>)
<If RollbackExecuted>- **Automatic rollback executed at <RollbackExecutedAt>** – Reason: <RollbackReason></If>

**Evidence:**
- Deployment Health Check: https://sreproactive-vscode-39596.azurewebsites.net/health
- Evidence chart (PNG): <EvidencePngUrl>
- Evidence data (CSV): <EvidenceCsvUrl>

**Recommended next steps:**
- Investigate performance regression in the newly active slot (now in production) for recent code/config changes.
- Compare dependency timings and slow requests between slots.
<If not RollbackExecuted>- Consider quick rollback if elevated latency persists.</If>
<If RollbackExecuted>- Rollback has been executed. Verify production is now stable before re-attempting the deployment.</If>

**App Insights queries used:**
- Pre-swap:
```
let startTime = datetime(<WindowStart>);
let endTime = datetime(<WindowEnd>);
requests
| where timestamp between (startTime .. endTime)
| where cloud_RoleName == 'sreproactive-vscode-39596'
| summarize CurrentResponseTime = avg(duration) by cloud_RoleName
| extend CurrentTimestamp = endTime
```
- Post-swap:
```
let startTime = datetime(<PostSwapWindowStart>);
let endTime = datetime(<PostSwapWindowEnd>);
requests
| where timestamp between (startTime .. endTime)
| where cloud_RoleName == 'sreproactive-vscode-39596'
| summarize CurrentResponseTime = avg(duration) by cloud_RoleName
| extend CurrentTimestamp = endTime
```

15. Post to Teams Channel at:
<YOUR_TEAMS_CHANNEL_URL>

Teams Post Format:

Deployment Health Check: sreproactive-vscode-39596

<one line summary>

---

Alert Time: <AlertTimestamp>

Baseline Response Time: <BaselineResponseTime>ms

Avg Response Time Before Swap: <CurrentResponseTime>ms (+<Deviation>% vs baseline)

Slot Swap Executed At: <SwapExecutedAt>

Avg Response Time After Swap: <PostSwapResponseTime>ms

Auto-Rollback Executed: <Yes/No> – <RollbackReason or "N/A">

---
GitHub Issue: <link>

Time of Deployment (slot swap): <time from the az cli query>
Deployment was healthy: <Yes/No based on whether rollback was needed>

Baseline Response Time: <time from knowledge>ms

Avg Response Time After Deployment: <time from app insights>ms

Was Swap Required: <Yes/No> - <reason including actual deviation %>

GitHub Issue: <link or NA>

Deployment was healthy: <Yes/No based on whether rollback/swap was needed>

App Insights Query: <actual query with timestamps replacing ago(2m)>
App Insights Query (pre-swap): <actual KQL query with explicit timestamps>

Constraints:

- No Fabrication: If the file or metric is not found, ask a single clarifying question and stop.

- Follow Order: Follow the tasks in order.

- No Approval Needed: Do not ask for approval if a swap is needed.
- No Approval Needed: Do not ask for approval for any swap or rollback action.

- Time Units: Response time is always in ms.

- Teams Format: Always follow the Teams posting format exactly.

- Rollback Threshold: Roll back automatically if PostSwapResponseTime > CurrentResponseTime * 1.10
(i.e., more than 10% worse than pre-swap). A 10% threshold is intentionally tighter than the
20% swap trigger to avoid false positive rollbacks when improvement is marginal while still
catching clear regressions introduced by the swap.

Output: Output should be in rich HTML format.

tools:
- GetCurrentUtcTime
- SearchMemory
- QueryAppInsightsByAppId
- WaitInMilliSeconds
- PostTeamsMessage
- CreateGithubIssue
- QuerySourceBySemanticSearch
- FindConnectedGitHubRepo
- GetAzCliHelp
- RunAzCliReadCommands
- RunAzCliWriteCommands
handoff_description: Deployment health check to see if response times are larger than expected
- ExecutePythonCode
handoff_description: Deployment health check to see if response times are larger than expected; auto-swaps and rolls back if post-swap is worse
agent_type: Autonomous