[Incident] False Positive Slow Response Alert — 4th Recurrence (2026-07-01)

[DarrenJohns]

Incident Report: False Positive Slow Response Alert — 4th Recurrence

• Incident ID: 4f8e940c-d7ae-4d3b-96c4-96541452f000
• Service: Azure Container Apps — ca-grubify-45ne7zvrqsjxa (rg: rg-sre-lab)
• Subscription: 573c01a0-ef63-4535-a78e-0bc7f79c87c9
• FQDN: ca-grubify-45ne7zvrqsjxa.graycoast-6fc58ae6.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-45ne7zvrqsjxa--0000013 (100% traffic, created 2026-06-16)

Summary

The alert-slow-response-sre-lab alert fired at 18:40:46 UTC on 2026-07-01. Investigation confirmed this is a false positive — the 4th recurrence of the same pattern (previous: issues #3, #4, #5). One ARM AlertsManagement GET call timed out at ~99,999ms at 18:38:02Z, inflating the App Insights requests/duration average above the 3000ms threshold. The Grubify API itself was responding in ~85ms and is fully healthy.

Impact

• No user-facing impact. The Grubify API was healthy and responsive throughout.
• Operational noise: SRE agent time spent investigating a known false positive pattern.

Timeline (UTC)

• ~18:34: ARM AlertsManagement GET request initiated
• ~18:38:02: ARM GET request timed out at 99,998.9ms after ~100s
• ~18:38–18:40: ARM timeout inflated the 5-minute rolling average of requests/duration above 3000ms
• ~18:40:46: Alert alert-slow-response-sre-lab fired (Sev3)
• ~18:42: SRE Agent investigation started; confirmed false positive within minutes

Evidence

App Insights Request Analysis

Check	Result
Grubify app requests (cloud_RoleName has "grubify" OR name has "/api/")	ZERO_ROWS_RETURNED — no application telemetry
Requests with duration > 3000ms (last 1h)	1 result: ARM AlertsManagement GET at 18:38:02Z — 99,998.9ms
Total App Insights requests (last 30m)	141 rows — all ARM management.azure.com calls

ARM API Timeout (trigger event)

Timestamp: 2026-07-01T18:38:02.9375933Z
Name: GET https://management.azure.com/.../Microsoft.AlertsManagement/alerts?...
Duration: 99,998.9161 ms
ResultCode: unknown
Success: True

Live Endpoint Verification

curl https://ca-grubify-45ne7zvrqsjxa.graycoast-6fc58ae6.eastus2.azurecontainerapps.io/api/restaurants
→ HTTP 200 in 0.085s (healthy)

Metrics Snapshot (Azure Monitor, 18:12–18:42 UTC)

Metric	Average	Max	Status
CpuUsageNanoCores	~296K (~0.3 vCPU)	~504K	✅ Healthy (well under 800M threshold)
WorkingSetBytes	~57MB	~57MB	✅ Healthy (well under 400MB threshold)

Container App Configuration

Setting	Value
CPU	0.5 cores
Memory	1Gi
Image	acrcagrubify45ne7zvrqsjxa.azurecr.io/grubify-api:latest
SIMULATE_SLOW	Not present
APPLICATIONINSIGHTS_CONNECTION_STRING	Present (but SDK not integrated in code)

Root Cause

The .NET Grubify API is missing the Application Insights SDK (Microsoft.ApplicationInsights.AspNetCore NuGet package and builder.Services.AddApplicationInsightsTelemetry() in Program.cs). The APPLICATIONINSIGHTS_CONNECTION_STRING env var is set, but without the SDK, no application-level requests telemetry flows to App Insights. Only ARM management API calls are recorded. One ARM AlertsManagement GET timed out at ~100s, pushing the requests/duration average above the 3000ms alert threshold — a false positive.

This is the 4th occurrence of this exact pattern:

• 1st: Issue [Incident] False Positive Slow Response Alert — Missing App Insights Instrumentation #3 (2026-06-11)
• 2nd: Issue [Incident] Recurring False Positive Slow Response Alert — App Insights SDK Missing from Application Code #4 (2026-06-18)
• 3rd: Issue [Incident] False Positive Slow Response Alert — 3rd Recurrence (2026-06-25) #5 (2026-06-25)
• 4th: This issue (2026-07-01)

Remediation

• No immediate remediation needed — the API is healthy and unaffected.

Permanent Fix Options (pick one or both):

1. Integrate App Insights SDK (recommended):

   • Add Microsoft.ApplicationInsights.AspNetCore to GrubifyApi.csproj
   • Add builder.Services.AddApplicationInsightsTelemetry() to Program.cs
   • This enables real application telemetry, making the alert meaningful

2. Add dimension filter to alert rule:

   • Filter requests/duration metric to only include requests where cloud/roleName contains "grubify" or name starts with "/api/"
   • This excludes ARM management API calls from the alert evaluation

Action Items

#	Action	Priority
1	Add Microsoft.ApplicationInsights.AspNetCore NuGet package to GrubifyApi.csproj	High
2	Add builder.Services.AddApplicationInsightsTelemetry() to Program.cs	High
3	Redeploy with App Insights SDK integrated	High
4	Add dimension filter to alert-slow-response-sre-lab to exclude ARM calls	Medium
5	Consider suppressing this alert until SDK is integrated to reduce noise	Low

References

• Container App: /subscriptions/573c01a0-ef63-4535-a78e-0bc7f79c87c9/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerapps/ca-grubify-45ne7zvrqsjxa
• App Insights: /subscriptions/573c01a0-ef63-4535-a78e-0bc7f79c87c9/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-45ne7zvrqsjxa
• Log Analytics Workspace ID: 24fa3c3f-c6cb-4be9-ba43-5f09562e869d
• Alert: View in Azure Portal
• Prior incidents: [Incident] False Positive Slow Response Alert — Missing App Insights Instrumentation #3, [Incident] Recurring False Positive Slow Response Alert — App Insights SDK Missing from Application Code #4, [Incident] False Positive Slow Response Alert — 3rd Recurrence (2026-06-25) #5

---

This issue was created by sre-agent-45ne7zvrqsjxa--fcd9117a
Tracked by the SRE agent here

[DarrenJohns]

🤖 Grubify SRE Agent Bot

5th recurrence of this false positive alert recorded on 2026-07-15T02:52:47Z.

Same root cause: Three ARM AlertsManagement GET calls timed out (100,000ms, 99,998ms, 46,179ms), inflating requests/duration average above 3000ms. Zero Grubify application telemetry in App Insights. API healthy (HTTP 200, 93ms).

Tracked in issue #7.

⚠️ This is now the 5th occurrence. The permanent fix (integrating the App Insights SDK or adding a dimension filter to the alert rule) should be prioritized to eliminate recurring alert noise.