Incident: HTTP 5xx due to OutOfMemory in Cart API (Grubify Container App)

[TraderShan]

Incident Report: OutOfMemoryException in CartController

• Incident ID: bc554079-129f-4e06-8b16-85e13846f000
• Service: Azure Container Apps — ca-grubify-yrhvssbcb5n2y (rg: rg-sre-lab)
• Subscription: 71aedd71-54de-4647-bbc4-aa3d8502f313
• Active revision: ca-grubify-yrhvssbcb5n2y--0000002 (100% traffic)

Summary

The Grubify API started returning HTTP 500 errors due to System.OutOfMemoryException caused by an unbounded memory leak in CartController.AddItemToCart. Every call to POST /api/cart/{userId}/items allocates a 10MB byte array to a static RequestDataCache list that is never cleaned up, exhausting the container's 1Gi memory limit.

Impact

• API errors (5xx) on all endpoints once memory was exhausted — not just the Cart endpoint
• All cart, order, and menu operations failed for users during the OOM window (~01:22–01:25 UTC)

Timeline (UTC)

• ~01:21:41 — Cache size reached 500MB (50 entries × 10MB); continued climbing rapidly
• ~01:22:12 — Cache reached 750MB (75 entries); first OOM errors appeared
• ~01:22–01:23 — Massive OOM error storm (172 error log lines in one minute)
• ~01:25:48 — Alert alert-http-5xx-sre-lab fired (Sev3)
• ~01:28 — SRE Agent restarted revision --0000002, clearing in-memory cache
• ~01:29 — Service recovered, no new errors

Evidence

Console logs (active revision)

Analytics cache: Added request data. Total entries: 75
Cache size: 750MB
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNOTIARC8JNO", Request id "0HNOTIARC8JNO:0000001C": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request) in /app/Controllers/CartController.cs:line 30

Error volume per minute

Time (UTC)	Error log lines
01:21	6
01:22	172
01:23	78

Root Cause

CartController.cs line 30 allocates a 10MB byte array on every AddItemToCart call and appends it to a static List<byte[]> RequestDataCache that is never cleared. With the container limited to 1Gi memory, ~75 cart-add requests are enough to exhaust all available memory and trigger OutOfMemoryException on every subsequent request.

// Line 30-31 — the leak
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);

Remediation

• Code (P0): Remove the RequestDataCache allocation entirely — it provides no real analytics value and is the direct cause of OOM. If request analytics are needed, use a bounded queue or external telemetry (App Insights).
• Defensive: Add payload size limits and rate limiting on the cart endpoint.
• Platform: Consider increasing container memory limit above 1Gi as a safety net.
• Observability: Instrument App Insights SDK for proper distributed tracing and memory tracking.

Action Items

#	Action	Priority
1	Remove RequestDataCache static list and 10MB allocation from CartController.cs	High
2	Add memory-based autoscale rule or increase memory limit	Medium
3	Instrument App Insights SDK for memory and request telemetry	Medium
4	Add a health check that reports memory usage	Low

References

• Container App: /subscriptions/71aedd71-54de-4647-bbc4-aa3d8502f313/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-yrhvssbcb5n2y
• Log Analytics Workspace ID: bc1ceff3-3f34-4b3e-91ae-bd9f99735169
• Alert: alert-http-5xx-sre-lab

---

This issue was created by sre-agent-yrhvssbcb5n2y--e066baca
Tracked by the SRE agent here