Incident: HTTP 5xx due to OutOfMemoryException in Cart API — 3rd Recurrence (Grubify Container App)

[wilkinshum]

Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API (3rd Recurrence)

• Incident ID: 2c068682-d580-4a6c-9a32-b6379bc3f000
• Service: Azure Container Apps — ca-grubify-jkazytu5kl5py (rg: rg-sre-lab)
• Subscription: f627598e-05c5-4093-8667-5730c4026ea3
• FQDN: ca-grubify-jkazytu5kl5py.salmonstone-16ca5c6b.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-jkazytu5kl5py--0000002 (100% traffic)

Summary

The Grubify backend API experienced sustained System.OutOfMemoryException crashes caused by CartController.AddItemToCart (line 30), which allocates a 10 MB byte[] per request and appends it to an unbounded static List<byte[]> RequestDataCache that is never cleaned up. With the container limited to 1 Gi memory, a traffic spike of ~76 req/min at 13:29 UTC exhausted the .NET managed heap, triggering cascading HTTP 500 errors. This is the third occurrence of this exact issue — the code fix identified in issues #305 and #306 has still not been deployed.

Impact

• User-facing: All API endpoints returned HTTP 5xx errors during the OOM window — cart operations, order placement, and restaurant browsing were unavailable
• Duration: OOM burst from ~13:25 to ~13:31 UTC (approximately 6 minutes)
• Blast radius: 100% of users affected during burst (single replica, no failover)
• Recurrence: Third occurrence — same root cause as Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #305 (2026-06-21) and Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #306 (2026-06-21). The code fix has never been deployed.

Timeline (UTC)

• ~13:25: First OOM exceptions detected — 106 exceptions in the 13:25–13:30 bin
• ~13:28: Traffic spike begins — 33 req/min, escalating to 76 req/min at 13:29
• ~13:29–13:30: Peak OOM activity — sustained System.OutOfMemoryException across 8+ concurrent connections
• ~13:30: Traffic drops to 66 req/min as requests fail, 22 additional OOM exceptions
• ~13:31: Alert alert-http-5xx-sre-lab fires (Sev3); traffic drops to 35 req/min
• ~13:32: Traffic drops to 0 — app effectively unresponsive
• ~13:35: SRE Agent investigation started, container restart initiated

Evidence

Console logs (active revision)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNMFRTTUKQRP", Request id "0HNMFRTTUKQRP:00000020": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request) in /app/Controllers/CartController.cs:line 30

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNMFRTTUKQRT", Request id "0HNMFRTTUKQRT:00000013": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNMFRTTUKQRU", Request id "0HNMFRTTUKQRU:00000016": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

OOM Exception Distribution (5-min bins)

Time (UTC)	OOM Exceptions
13:25–13:30	106
13:30–13:35	22
Total	128

Traffic and Metrics (per minute)

Time (UTC)	Requests/min	CPU %	Memory %
13:27	0	0	5
13:28	33	1.25	5
13:29	76	2.0	5.5
13:30	66	0.75	4
13:31	35	0	4
13:32	0	0	4

Metrics snapshot (Azure Monitor)

• Requests: 0 → 33 → 76 → 66 → 35 → 0 req/min (spike at 13:28–13:31)
• CPU avg: peaked at 2% — not CPU-bound
• Memory avg: ~5% container-level — misleading (Azure Monitor measures container RSS, not .NET managed heap)
• Container config: 0.5 CPU, 1 Gi memory, minReplicas=1, maxReplicas=5
• RestartCount: Container restarted automatically after OOM
• App Insights error rate: 0% — telemetry gap — OOM crashes the request pipeline before App Insights records the failure; errors only visible in ContainerAppConsoleLogs_CL

Root cause code

// CartController.cs — lines 14 and 30-31
private static readonly List<byte[]> RequestDataCache = new();  // line 14 — unbounded, never cleared

var requestData = new byte[10 * 1024 * 1024]; // line 30 — 10MB per request
RequestDataCache.Add(requestData);             // line 31 — appended with no cleanup
// TODO: Implement cache cleanup mechanism in future sprint  // line 33

Root Cause

Memory leak in CartController.AddItemToCart — each POST /api/cart/{userId}/items call allocates a 10 MB byte[] and appends it to a static, unbounded List<byte[]> RequestDataCache (lines 14 and 30-31 in CartController.cs) that is never cleaned up. With the container limited to 1 Gi memory, approximately 100 add-to-cart requests exhaust the .NET managed heap, causing System.OutOfMemoryException and cascading HTTP 500 errors until the container restarts. This clears the static list temporarily, but the leak recurs with the next traffic spike.

Remediation

• Code: Remove the RequestDataCache static list and the 10 MB allocation entirely from CartController.cs. If analytics data is needed, use a bounded buffer (e.g., ConcurrentQueue with max capacity) or external store (Application Insights custom events, Event Hub).
• Defensive: Add payload size limits and rate limiting on the /api/cart/{userId}/items endpoint. Add request body validation to reject oversized payloads.
• Platform: Increase container memory from 1 Gi to 2 Gi as a safety margin. Set minReplicas to at least 2 to eliminate single-replica exposure. Add memory-based autoscale rule.
• Observability: Add an alert for System.OutOfMemoryException in ContainerAppConsoleLogs_CL (since App Insights does not capture OOM-caused 5xx). Instrument .NET managed heap metrics (GC.GetTotalMemory or dotnet-counters) to detect heap exhaustion before container-level memory thresholds are reached.

Action Items

#	Action	Priority
1	Remove RequestDataCache and 10 MB allocation from CartController.cs	Critical
2	Deploy the fix to production (this has been pending since #305)	Critical
3	Increase container memory to 2 Gi	High
4	Set minReplicas=2 to eliminate single-replica exposure	High
5	Add memory-based autoscale rule	Medium
6	Add Log Analytics alert for OOM exceptions	Medium
7	Instrument .NET managed heap metrics	Low
8	Add rate limiting on cart endpoints	Low

References

• Container App: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-jkazytu5kl5py
• Log Analytics Workspace ID: 7ba6ec0e-ad5c-4d6f-8c2b-9e94d7d3a59b
• Log Analytics ARM ID: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.OperationalInsights/workspaces/law-jkazytu5kl5py
• App Insights: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-jkazytu5kl5py
• Prior incidents: #305, #306
• Alert: alert-http-5xx-sre-lab (Sev3), fired 2026-06-22T13:31:26Z
• Source file: GrubifyApi/Controllers/CartController.cs lines 14, 30-31

---

This issue was created by sre-agent-jkazytu5kl5py--70975bf6
Tracked by the SRE agent here