Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App)

[wilkinshum]

Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API

• Incident ID: a5d26f57-c78c-46d3-b90b-78ef9cacf000
• Service: Azure Container Apps — ca-grubify-jkazytu5kl5py (rg: rg-sre-lab)
• Subscription: f627598e-05c5-4093-8667-5730c4026ea3
• FQDN: ca-grubify-jkazytu5kl5py.salmonstone-16ca5c6b.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-jkazytu5kl5py--0000002 (100% traffic)

Summary

The Grubify API began returning HTTP 500 errors due to sustained System.OutOfMemoryException exceptions originating from CartController.AddItemToCart(). The root cause is a memory leak in CartController.cs — every call to POST /api/cart/{userId}/items allocates a 10 MB byte array into a static RequestDataCache list that is never cleaned up. Under concurrent traffic (~65–69 req/min), this rapidly exhausted the container's 1 Gi memory limit, causing all AddItemToCart requests to fail with OOM errors.

Impact

• API errors (HTTP 500) on the POST /api/cart/{userId}/items endpoint during the spike window (~21:08–21:11 UTC)
• 20+ failed requests across 8 concurrent connections in a ~22-second burst
• All cart add-to-cart operations failed for affected users during the incident window
• Other API endpoints (restaurants, food items, orders) were likely degraded due to shared process memory pressure

Timeline (UTC)

• ~19:43: Revision ca-grubify-jkazytu5kl5py--0000002 deployed successfully; image pulled in 2.03s, traffic cut over to 100%
• ~21:08: Traffic begins hitting the app (~8 req/min), including AddItemToCart calls that start accumulating 10 MB allocations in RequestDataCache
• ~21:10: Traffic spikes to ~65 req/min; memory pressure builds rapidly as each cart request adds 10 MB to the unbounded static list
• ~21:11:15Z: First System.OutOfMemoryException logged at CartController.cs:line 30
• ~21:11:29Z: Azure Monitor alert alert-http-5xx-sre-lab fires (Sev3)
• ~21:11:37Z: OOM burst subsides after ~22 seconds; 20+ requests failed across 8 connection IDs

Evidence

Console logs (active revision)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNMFPLA451UT", Request id "0HNMFPLA451UT:00000021": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request) in /app/Controllers/CartController.cs:line 30
         at lambda_method8(Closure, Object, Object[])
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(...)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](...)

Affected connections (sample):

Connection ID	Failed Request IDs
0HNMFPLA451TV	00000017, 00000018
0HNMFPLA451UT	0000001F, 00000020, 00000021, 00000022
0HNMFPLA451UU	0000001B–0000001F
0HNMFPLA451UV	00000014–00000016
0HNMFPLA451V0	0000001E, 0000001F
0HNMFPLA451V2	00000017–0000001A
0HNMFPLA451V5	00000013–00000015
0HNMFPLA451VA	00000012–00000014

Buggy code path (CartController.cs:line 28–31)

// Cache for performance optimization - stores request data for analytics
private static readonly List<byte[]> RequestDataCache = new();

[HttpPost("{userId}/items")]
public ActionResult<Cart> AddItemToCart(string userId, [FromBody] AddCartItemRequest request)
{
    // Store request data for analytics and performance monitoring
    var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics  ← BUG
    RequestDataCache.Add(requestData);                                                    ← NEVER CLEANED

    // TODO: Implement cache cleanup mechanism in future sprint
    ...
}

Each invocation of AddItemToCart allocates a 10 MB byte array and appends it to a static, unbounded list (RequestDataCache). With the container limited to 1 Gi memory, only ~100 requests are needed to exhaust the heap. The // TODO comment confirms the cleanup was deferred intentionally and never implemented.

Metrics snapshot (Azure Monitor, 30-min window)

Metric	Value	Interpretation
Requests (peak)	69 req/min at 21:11	Traffic spike coincides with OOM burst
CPU avg	0%	Not a CPU issue
Memory avg	4%	Azure Monitor container-level avg does not capture in-process .NET managed heap spikes
RestartCount	0	No container restarts observed in this window
Replicas	1	Single replica; no scale-out rules configured
Container memory limit	1 Gi	Insufficient headroom given the 10 MB/request leak

Traffic and Response Time

Metrics chart showing request count, CPU, and memory over the incident window is attached below. Note that Azure Monitor's container-level memory average (4%) masks the in-process .NET heap exhaustion — the OOM occurs within the managed heap before the container-level metric spikes.

Root Cause

Memory leak in CartController.AddItemToCart() (CartController.cs:line 30). A static List<byte[]> named RequestDataCache accumulates a 10 MB byte array on every POST /api/cart/{userId}/items request and is never cleared. Under concurrent traffic, the .NET managed heap exhausts the 1 Gi container memory limit, throwing System.OutOfMemoryException on all subsequent cart requests. The cleanup was marked as a TODO but never implemented.

Remediation

• Code: Remove the RequestDataCache static list and the 10 MB allocation entirely — this "analytics cache" serves no functional purpose and is the direct cause of the OOM. If analytics buffering is genuinely needed, use a bounded, size-limited buffer (e.g., circular buffer with max entries) or offload to an external store (Application Insights, Event Hub).
• Defensive: Add request payload size validation and rate limiting on the AddItemToCart endpoint to prevent abuse even after the leak is fixed. Implement health check probes (liveness/readiness) that monitor managed heap usage.
• Platform: Increase container memory from 1 Gi to 2 Gi as a safety margin. Add a memory-based autoscale rule. Set minReplicas to at least 2 for redundancy.
• Observability: Add an Azure Monitor alert for System.OutOfMemoryException in console logs. Add .NET GC.GetTotalMemory() as a custom metric to Application Insights to catch managed heap growth before it triggers OOM.

Action Items

#	Action	Priority
1	Remove RequestDataCache and the 10 MB allocation from CartController.AddItemToCart()	High
2	Add bounded analytics buffering or remove the analytics cache entirely	High
3	Add liveness/readiness probes with memory checks	Medium
4	Increase container memory limit to 2 Gi	Medium
5	Add memory-based autoscale rule and set minReplicas >= 2	Medium
6	Add Application Insights custom metric for managed heap size	Low
7	Add Azure Monitor alert for OOM exceptions in console logs	Low

References

• Container App: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-jkazytu5kl5py
• Log Analytics Workspace ID: 7ba6ec0e-ad5c-4d6f-8c2b-9e94d7d3a59b
• Log Analytics Workspace: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.OperationalInsights/workspaces/law-jkazytu5kl5py
• App Insights: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-jkazytu5kl5py
• Alert: /subscriptions/f627598e-05c5-4093-8667-5730c4026ea3/resourcegroups/rg-sre-lab/providers/microsoft.app/containerapps/ca-grubify-jkazytu5kl5py/providers/Microsoft.AlertsManagement/alerts/a5d26f57-c78c-46d3-b90b-78ef9cacf000
• Source file: GrubifyApi/Controllers/CartController.cs

---

This issue was created by sre-agent-jkazytu5kl5py--70975bf6
Tracked by the SRE agent here