Incident: HTTP 5xx due to System.OutOfMemoryException in CartController.AddItemToCart (Grubify Container App)

[simonbMS]

Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API

• Incident ID: b44f4931-8e17-4570-9fc7-226adf85f000
• Service: Azure Container Apps — ca-grubify-5t5utmo2yorzk (rg: rg-sre-lab)
• Subscription: 0dd96c3f-318b-4e1e-9ee9-29ebe70a993e
• FQDN: ca-grubify-5t5utmo2yorzk.politeforest-9260c20c.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-5t5utmo2yorzk--0000002 (100% traffic)

Summary

Azure Monitor alert alert-http-5xx-sre-lab fired at 2026-06-25T10:35:18Z due to sustained HTTP 5xx errors on the Grubify Container App. Investigation revealed 25+ System.OutOfMemoryException errors in CartController.AddItemToCart (CartController.cs:line 30) within a 21-second window (10:33:23Z–10:33:44Z), affecting 7+ concurrent connections. The root cause is an unbounded in-memory cart storage with no eviction policy — under a traffic burst of 62–75 requests/minute, the cart data structure grew without limits, exhausting application-level memory and producing HTTP 500 responses on every subsequent request to the Cart API.

Impact

• API errors (5xx) on the /api/cart/{userId}/items POST endpoint during the 10:33–10:34 UTC window
• 25+ failed requests across 7+ concurrent connections in 21 seconds
• All cart operations (add item to cart) were failing for affected users
• Other endpoints (OrdersController, WeatherForecast) remained operational — impact was isolated to the Cart API

Timeline (UTC)

• ~10:12: Application revision ca-grubify-5t5utmo2yorzk--0000002 started and initialized successfully (image pulled in 2.49s)
• ~10:25: Initial low traffic begins (~1 req/min)
• ~10:26: Small traffic spike (11 req/min)
• ~10:32: Major traffic burst begins (62 req/min)
• ~10:33:23: First System.OutOfMemoryException in CartController.AddItemToCart at CartController.cs:line 30
• ~10:33:44: OOM exceptions continue — 25+ occurrences across 7+ connections
• ~10:34: Traffic remains elevated (69 req/min), OOM errors persist
• ~10:35:18: Azure Monitor alert alert-http-5xx-sre-lab fires (Sev3)
• ~10:35: Traffic drops to 1 req/min (likely client-side failures/timeouts)

Evidence

Console logs (active revision)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNMIK9QKRCA3", Request id "0HNMIK9QKRCA3:0000000E": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request) in /app/Controllers/CartController.cs:line 30
         at lambda_method8(Closure, Object, Object[])
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(...)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Affected connections: 0HNMIK9QKRCA0, 0HNMIK9QKRCA2, 0HNMIK9QKRCA3, 0HNMIK9QKRCA5, 0HNMIK9QKRCA6, 0HNMIK9QKRCA9, 0HNMIK9QKRC6E

Metrics snapshot (Azure Monitor)

Time (UTC)	Requests/min	CPU %	Memory %
10:25	1	0.25	2.0
10:26	11	2.75	2.75
10:31	1	0.0	3.0
10:32	62	2.5	3.75
10:33	75	1.0	4.0
10:34	69	1.25	4.0
10:35	1	0.0	4.0

• Requests: Spiked from ~1/min to 62–75/min at 10:32–10:34
• CPU avg: Low (max 2.75%) — not a CPU-bound issue
• Memory %: Gradually rising (2% → 4%) at container level, but application-level OOM at line 30 indicates unbounded in-process allocation
• RestartCount: 0 (process survived, but requests failed with 500)
• Container resources: 0.5 CPU, 1Gi memory

Root Cause

The CartController.AddItemToCart method at CartController.cs:line 30 allocates cart items into an unbounded in-memory data structure (likely a Dictionary or List) with no eviction policy, no size limits, and no TTL. Under sustained traffic to POST /api/cart/{userId}/items (~62–75 req/min), this collection grew without limits until the .NET runtime threw System.OutOfMemoryException. Because the allocation is in the shared process memory space, the OOM cascaded to all concurrent connections handling cart requests, producing HTTP 500 errors across 7+ simultaneous connections within seconds.

This is a known failure mode documented in the Grubify architecture: the /api/cart/{userId}/items endpoint is identified as a "memory leak trigger" — cart items stored in memory with no eviction cause OOM under load.

Remediation

• Code: Implement bounded cart storage — use a size-limited concurrent dictionary with LRU eviction or TTL-based expiry. Alternatively, move cart state to a persistent store (Redis, database) instead of in-process memory.
• Defensive: Add payload size validation on AddCartItemRequest, enforce per-user cart item limits (e.g., max 100 items), and implement rate limiting on the cart POST endpoint to prevent burst-driven OOM.
• Platform: Consider increasing container memory from 1Gi to 2Gi as a short-term mitigation. Add memory-based autoscaling rules to scale out before OOM threshold. Configure health probes to detect degraded state earlier.
• Observability: Add an alert for System.OutOfMemoryException in application logs. Add a custom metric for in-memory cart size to detect unbounded growth before OOM. Monitor memory working set at the application level, not just container level.

Action Items

#	Action	Priority
1	Replace unbounded in-memory cart with bounded storage (LRU/TTL cache or external store like Redis)	High
2	Add per-user cart item limits and payload size validation in CartController.AddItemToCart	High
3	Increase container memory allocation from 1Gi to 2Gi as interim mitigation	Medium
4	Add rate limiting on POST /api/cart/{userId}/items endpoint	Medium
5	Add Azure Monitor alert for OOM exceptions in application logs	Medium
6	Add custom metric tracking in-memory cart collection size	Low
7	Configure memory-based autoscale rule for the container app	Low

References

• Container App: /subscriptions/0dd96c3f-318b-4e1e-9ee9-29ebe70a993e/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-5t5utmo2yorzk
• Log Analytics Workspace ID: cd26bbbc-9e28-487e-a492-923edfc64c49
• App Insights: /subscriptions/0dd96c3f-318b-4e1e-9ee9-29ebe70a993e/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-5t5utmo2yorzk
• Container Environment: /subscriptions/0dd96c3f-318b-4e1e-9ee9-29ebe70a993e/resourceGroups/rg-sre-lab/providers/Microsoft.App/managedEnvironments/cae-5t5utmo2yorzk
• Alert Rule: /subscriptions/0dd96c3f-318b-4e1e-9ee9-29ebe70a993e/resourcegroups/rg-sre-lab/providers/Microsoft.Insights/metricAlerts/alert-http-5xx-sre-lab
• Azure Portal Alert: View Alert
• SRE Agent Thread: Open thread

---

This issue was created by sre-agent-5t5utmo2yorzk--3279a27e
Tracked by the SRE agent here

[simonbMS]

Test