Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App)

[DarrenJohns]

Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API

• Incident ID: 960c0703-619c-4194-8d4d-de92222ff000
• Service: Azure Container Apps — ca-grubify-45ne7zvrqsjxa (rg: rg-sre-lab)
• Subscription: 573c01a0-ef63-4535-a78e-0bc7f79c87c9
• FQDN: ca-grubify-45ne7zvrqsjxa.graycoast-6fc58ae6.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-45ne7zvrqsjxa--0000002 (100% traffic)

Summary

The Grubify API Container App experienced a sustained System.OutOfMemoryException failure starting at ~04:04 UTC on 2026-06-10. Every incoming request to the POST /api/cart/{userId}/items endpoint (and eventually all endpoints) failed with OOM errors thrown by ASP.NET Core Kestrel. The root cause is a memory leak in CartController.cs — a static List<byte[]> (RequestDataCache) allocates 10MB per POST request with no eviction, exhausting the container's 1Gi memory limit after sustained cart API traffic.

Impact

• All API requests began failing with HTTP 500 errors once memory was exhausted (~04:04 UTC)
• Cart operations (add to cart) were the primary failure endpoint, but OOM affected all endpoints served by the process
• ~200 OOM exceptions logged over a ~4 minute window (04:04–04:08 UTC)
• 6 concurrent connections were affected simultaneously
• The HTTP 5xx alert (alert-http-5xx-sre-lab) fired at 04:07:19 UTC

Timeline (UTC)

• ~04:02: Traffic burst observed (20 requests/min vs baseline 6/min)
• ~04:04: First System.OutOfMemoryException errors appear in console logs (33 OOM errors/min)
• ~04:05: OOM errors escalate to 51/min — memory fully exhausted
• ~04:06: Peak OOM rate: 56 errors/min — all incoming requests failing
• ~04:07:19: Azure Monitor alert alert-http-5xx-sre-lab fires (Sev3)
• ~04:08:05: Last OOM error logged before container becomes unresponsive
• ~04:10:50: SRE Agent restarts container app revision; app restarts on port 8080
• ~04:11: Container healthy — 1/1 replicas running, no new OOM errors

Evidence

Console logs (active revision)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNM5RGI8USD0", Request id "0HNM5RGI8USD0:00000016": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request)
            in /app/Controllers/CartController.cs:line 30

26 distinct OOM failures across 6 concurrent connections in ~28 seconds (04:07:38–04:08:05 UTC).

OOM Error Rate (from ContainerAppConsoleLogs_CL)

Time (UTC)	OOM Errors/min
04:04	33
04:05	51
04:06	56
04:07	55
04:08	5

Request Metrics (Azure Monitor — Requests metric)

Time (UTC)	Total Requests
03:50–04:01	~6/min (baseline)
04:02	20 (burst)
04:05	17
04:06	54
04:07	53
04:08	55
04:09	24

Metrics snapshot (Azure Monitor)

• MemoryPercentage: 4% avg (Azure Monitor reports container-level, not .NET managed heap — does not reflect the internal OOM pressure)
• CpuPercentage: 0% (OOM is memory-driven, not CPU)
• RestartCount: 0 (no automated restart before agent intervention)
• Replicas: 1/1

KQL Queries Used

OOM error timeline:

ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "ca-grubify-45ne7zvrqsjxa"
| where Log_s has "OutOfMemoryException"
| summarize OOMCount = count() by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

Failed requests (App Insights):

requests
| where timestamp > ago(30m)
| where success == false
| summarize failure_count = count() by name, resultCode
| order by failure_count desc

Root Cause

The CartController.AddItemToCart method (line 30 of Controllers/CartController.cs) allocates a 10MB byte array per POST request into a static List<byte[]> called RequestDataCache. This list is never cleared or bounded. Under sustained cart API traffic, memory consumption grows linearly at 10MB per request until the container's 1Gi limit is exhausted, causing System.OutOfMemoryException on all subsequent requests.

Offending code:

private static readonly List<byte[]> RequestDataCache = new();

// In AddItemToCart:
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);
// TODO: Implement cache cleanup mechanism in future sprint

Remediation

• Immediate (done): Restarted the container app revision ca-grubify-45ne7zvrqsjxa--0000002 to clear the in-memory leaked cache. Service restored at ~04:11 UTC.
• Code: Remove the RequestDataCache static list and the 10MB allocation from CartController.AddItemToCart. If analytics caching is needed, implement a bounded cache (e.g., MemoryCache with size limits and expiration) or offload to an external store.
• Defensive: Add request payload size validation and rate limiting on the /api/cart/{userId}/items endpoint to prevent rapid-fire memory exhaustion.
• Platform: Consider increasing container memory limit from 1Gi to 2Gi as a safety margin, and add a memory-percentage-based autoscale rule.
• Observability: Add an Azure Monitor alert for MemoryPercentage > 80% to catch memory pressure before OOM occurs.

Action Items

#	Action	Priority
1	Remove RequestDataCache and 10MB allocation from CartController.AddItemToCart	High
2	Implement bounded MemoryCache with TTL if analytics caching is still needed	Medium
3	Add rate limiting on POST /api/cart/{userId}/items	Medium
4	Add memory-percentage alert threshold (>80%)	Medium
5	Consider memory-based autoscale rule for the Container App	Low
6	Add integration test to validate cart endpoint under sustained load	Low

References

• Container App: /subscriptions/573c01a0-ef63-4535-a78e-0bc7f79c87c9/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerapps/ca-grubify-45ne7zvrqsjxa
• Log Analytics Workspace ID: 24fa3c3f-c6cb-4be9-ba43-5f09562e869d
• App Insights: /subscriptions/573c01a0-ef63-4535-a78e-0bc7f79c87c9/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-45ne7zvrqsjxa
• Alert: /subscriptions/573c01a0-ef63-4535-a78e-0bc7f79c87c9/resourcegroups/rg-sre-lab/providers/microsoft.app/containerapps/ca-grubify-45ne7zvrqsjxa/providers/Microsoft.AlertsManagement/alerts/960c0703-619c-4194-8d4d-de92222ff000
• Source code: GrubifyApi/Controllers/CartController.cs line 14 (RequestDataCache) and line 30 (allocation)

---

This issue was created by sre-agent-45ne7zvrqsjxa--fcd9117a
Tracked by the SRE agent here