Incident: Recurring HTTP 5xx due to OutOfMemoryException in CartController (Grubify Container App)

[duygusilivri]

Incident Report: Recurring HTTP 5xx due to OutOfMemoryException in CartController

• Incident ID: d383b539-5512-4506-aefe-c057b49ef000
• Service: Azure Container Apps — ca-grubify-vv36phphsnxnq (rg: rg-sre-lab)
• Subscription: 537ed198-06da-473b-b72c-b75168240ed2
• FQDN: ca-grubify-vv36phphsnxnq.livelyforest-59606736.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-vv36phphsnxnq--0000002 (100% traffic)

Summary

Azure Monitor alert alert-http-5xx-sre-lab fired at 13:02:25 UTC on 2026-08-04 for the Grubify Container App. Investigation shows 200+ System.OutOfMemoryException errors flooding from the CartController.AddItemToCart endpoint between 12:59–13:03 UTC. This is the same unresolved root cause as the incident on 2026-08-03 (see #319) — the unbounded RequestDataCache memory leak in CartController.cs was never fixed.

Impact

• All Cart API operations (POST /api/cart/{userId}/items) returning HTTP 500 during the incident window
• 200+ failed requests across multiple Kestrel connections in ~5 minutes
• Users unable to add items to cart, blocking the core ordering flow
• Other API endpoints (restaurants, food items, orders) may also be degraded due to memory pressure in the shared process

Timeline (UTC)

• ~12:59: First System.OutOfMemoryException errors appear in logs (36 OOM errors/min)
• ~13:00: Error rate escalates — 45 OOM exceptions/min, 26 HTTP requests/min recorded
• ~13:01: Peak OOM rate — 48 exceptions/min, 42 requests/min
• ~13:02: Alert alert-http-5xx-sre-lab fires at 13:02:25 UTC; 46 OOM exceptions/min
• ~13:03: Errors continue — 25 OOM exceptions/min, 46 requests/min

Evidence

Console logs (active revision ca-grubify-vv36phphsnxnq--0000002)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNNHL2PFK5BF", Request id "0HNNHL2PFK5BF:0000001B": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNNHL2PFK5BC", Request id "0HNNHL2PFK5BC:0000001E": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNNHL2PFK5B5", Request id "0HNNHL2PFK5B5:00000015": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Errors seen on 7+ unique Kestrel connections (0HNNHL2PFK5BF, BC, B5, AM, B6, B9, BB, B3) — indicating broad impact across all incoming connections.

OOM Exception Rate (1-min bins)

Time (UTC)	OOM Exceptions/min
12:59	36
13:00	45
13:01	48
13:02	46
13:03	25
Total	200

HTTP Request Volume (Azure Monitor, 1-min bins)

Time (UTC)	Requests/min
12:55–12:58	0
12:59	1
13:00	26
13:01	42
13:02	48
13:03	46
13:04	41

Metrics snapshot (Azure Monitor)

• Requests: Ramped from 0 to 48/min at peak (13:02 UTC)
• RestartCount: 0 — no container-level restarts; OOM caught at application level before cgroup kill
• MemoryPercentage: Container-level memory was low (~3-5%) — the .NET managed heap exhausted before hitting cgroup limit
• UsageNanoCores: CPU was normal (0-2%) — this is a pure memory issue, not CPU

Root Cause

Unbounded static memory leak in CartController.AddItemToCart (file: GrubifyApi/Controllers/CartController.cs, lines 30-31).

Every POST /api/cart/{userId}/items request allocates a new byte[10 * 1024 * 1024] (10 MB) buffer and appends it to static List<byte[]> RequestDataCache, which is never cleaned up. The TODO on line 33 ("Implement cache cleanup mechanism in future sprint") was never implemented. Under moderate load (~50 req/min), the managed heap exhausts within seconds, causing System.OutOfMemoryException on all subsequent requests.

This is a recurring incident — the identical root cause was identified on 2026-08-03 (issue #319) but the code fix was not applied.

// Lines 30-31 — the faulting code:
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);

Remediation

• Code (HIGH): Remove the RequestDataCache static field and the 10MB allocation entirely from CartController.AddItemToCart. If analytics tracking is needed, use a proper bounded buffer, streaming approach, or external analytics service instead of in-memory byte arrays.
• Defensive: Add request payload size validation and rate limiting on the cart endpoint to prevent memory pressure from high request volumes.
• Platform: Consider increasing container memory from 1Gi to 2Gi as a safety margin, though this does not fix the root cause (the leak is unbounded and will exhaust any amount of memory).
• Observability: Add a dedicated alert for System.OutOfMemoryException in container logs to catch this class of issue earlier.

Action Items

#	Action	Priority
1	Remove RequestDataCache and 10MB allocation from CartController.cs lines 14, 30-35	Critical
2	Restart the container app revision to clear accumulated memory	High
3	Add unit test to verify AddItemToCart does not allocate unbounded memory	Medium
4	Add OOM-specific log alert on ContainerAppConsoleLogs_CL	Medium
5	Implement proper analytics tracking via Application Insights instead of in-memory buffers	Medium
6	Consider memory-based autoscale rule as defense-in-depth	Low

References

• Container App: /subscriptions/537ed198-06da-473b-b72c-b75168240ed2/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-vv36phphsnxnq
• Log Analytics Workspace ID: 1c9a5ca5-8c2a-48f7-b5e8-c29f76be52ba
• App Insights: /subscriptions/537ed198-06da-473b-b72c-b75168240ed2/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-vv36phphsnxnq
• Previous incident: Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #319 (2026-08-03, identical root cause)
• Alert: alert-http-5xx-sre-lab — /subscriptions/537ed198-06da-473b-b72c-b75168240ed2/resourcegroups/rg-sre-lab/providers/Microsoft.Insights/metricAlerts/alert-http-5xx-sre-lab
• Source code: GrubifyApi/Controllers/CartController.cs lines 14, 30-35

---

This issue was created by sre-agent-vv36phphsnxnq--3b062ec3
Tracked by the SRE agent here