Incident: HTTP 5xx during deployment — Datadog sidecar crash-loop & latent memory leak (2026-09-28)

[dajwalia-ms]

Incident Report: HTTP 5xx during deployment — Datadog agent sidecar crash-loop

• Incident ID: fe518660-7b59-4bac-a225-6a1a0c72f000
• Service: Azure Container Apps — ca-grubify-ijid2sq6n3hr6 (rg: rg-sre-lab02)
• Subscription: a9096d06-8ee3-4532-9b34-aa3bb3570226
• FQDN: ca-grubify-ijid2sq6n3hr6.victoriousmeadow-21ce84b0.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-ijid2sq6n3hr6--0000006 (100% traffic)

Summary

HTTP 5xx errors were triggered during a deployment that created new revisions (0000005 → 0000006) with a misconfigured datadog/agent:7 sidecar container. The Datadog agent immediately entered a crash-loop (ContainerBackOff) due to missing DD_HOSTNAME environment variable, causing transient 5xx responses during the revision transition. The Grubify API container itself started successfully and is serving HTTP 200.

Impact

• Transient 5xx errors during ~3-minute deployment window (15:19–15:23 UTC)
• 21 requests observed in the affected window (11 + 10 in two 5-min buckets)
• Datadog agent sidecar is still crash-looping (wasting resources, no APM telemetry)
• Latent risk: The deployed image grubify-api:dd-instrumented likely still contains the CartController memory leak (10MB/request unbounded allocation — see [Incident] Recurring memory leak in CartController.AddItemToCart causing HTTP 5xx #318)

Timeline (UTC)

• ~15:19:29: First containerApps/write operation (caller: 4ace62cf-...) — created revision 0000005
• ~15:20:37: Second containerApps/write operation — created revision 0000006
• ~15:20:42: Revision --0000006 created (image: grubify-api:dd-instrumented + datadog/agent:7 sidecar)
• ~15:20:51: Grubify API container started successfully on port 8080
• ~15:20:53: Traffic weight set to 100% for revision 0000006; old revision 0000005 decommissioned
• ~15:21:00: Datadog agent sidecar begins crash-looping (hostname resolution failure)
• ~15:23:12: First ContainerBackOff warning — "Persistent Failure to start container"
• ~15:23:29: Alert alert-http-5xx-sre-lab02 fired (Sev3)
• ~15:23:55: Alert auto-resolved
• ~15:24:36: Second ContainerBackOff event (sidecar still crash-looping)

Evidence

System logs (revision 0000006)

ContainerBackOff | Persistent Failure to start container (count: 2)

Datadog agent errors (sidecar)

Error: Error while getting hostname, exiting: unable to reliably determine the host name.
You can define one in the agent config file or in your hosts file

Error: couldn't get hostname from core agent at "/opt/datadog-agent/bin/agent/agent": exit status 255.
Set DD_HOSTNAME or hostname in the agent config

Metrics snapshot (Azure Monitor)

• Requests (1m bins, 15:10–15:24): 0,0,0,0,0,0,0,11,0,0,0,10,0,0,0
• RestartCount (5m bins, 15:00–15:20): max 1, 0, 1, 1, 1 (sidecar restarts)
• App container: Running, HTTP 200 verified on /api/restaurants (473ms) and /api/fooditems (200)

Current CartController.cs (repo main branch)

The memory leak code is still present in the source:

private static readonly List<byte[]> RequestDataCache = new();
// ...
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer
RequestDataCache.Add(requestData);

Root Cause

Primary: A deployment at 15:19–15:20 UTC added a datadog/agent:7 sidecar container without configuring the required DD_HOSTNAME environment variable. The sidecar immediately crash-looped, and during the revision transition window, incoming requests received 5xx responses before the new revision's main container was fully ready.

Secondary (latent): The deployed grubify-api:dd-instrumented image and the source repo (main branch) still contain the CartController memory leak (static unbounded List<byte[]> with 10MB allocations per POST request). This will cause OutOfMemoryException and HTTP 5xx under sustained POST /api/cart traffic. See #318.

Remediation

• Sidecar fix: Set DD_HOSTNAME environment variable on the datadog-agent container (e.g., via az containerapp update with container env vars, or use DD_HOSTNAME set to the container app name)
• Memory leak fix: Remove RequestDataCache and 10MB allocation from CartController.AddItemToCart — code fix still pending from [Incident] Recurring memory leak in CartController.AddItemToCart causing HTTP 5xx #318
• Deployment: Rebuild and push a fixed container image to ACR

Action Items

#	Action	Priority
1	Set DD_HOSTNAME env var on datadog-agent sidecar to stop crash-loop	High
2	Remove memory leak from CartController.cs and deploy fixed image (#318)	High
3	Add deployment validation checks to prevent misconfigured sidecars	Medium
4	Add pre-deployment smoke tests before shifting traffic to new revisions	Medium

References

• Container App: /subscriptions/a9096d06-8ee3-4532-9b34-aa3bb3570226/resourceGroups/rg-sre-lab02/providers/Microsoft.App/containerApps/ca-grubify-ijid2sq6n3hr6
• Log Analytics Workspace ID: e00dbb2d-5906-4228-97c0-3cf6538e8aa7
• Alert: Portal Link
• Prior memory leak incident: [Incident] Recurring memory leak in CartController.AddItemToCart causing HTTP 5xx #318
• SRE Agent thread: Open thread

[dajwalia-ms]

Correction: root cause in this issue is wrong

The original report blamed the Datadog sidecar crash loop. The evidence doesn't support that.

Actual root cause: revision --0000005 (created 15:19:36 UTC) set API_VERSION=v2. In OrdersController.ProcessPayment, the v2 path uses the wrong payment gateway URL (https://payment-gateway-staging.internal.com/v1/process) and always returns Connection to payment gateway timed out. So every POST /api/orders failed.

Evidence

• Console logs at 15:20:26 on --0000005 show 10 PlaceOrder called - Version: v2 entries, each followed by Payment processing failed in v2: Connection to payment gateway timed out.
• The Requests metric split by status code shows exactly 10 5xx (15:21 bucket), and no other 5xx between 15:10 and 15:24.
• Revision --0000006 (15:20:42, API_VERSION=v1) replaced it about 1 minute later. That was effectively a rollback by the same deploying identity (4ace62cf…). No 5xx after that.

Ruled out

• Datadog sidecar crash loop: it started around 09:01 UTC, about 6 hours before the alert, and the main container keeps serving 200s. It's a real but separate problem (still needs DD_HOSTNAME).
• BadHttpRequestException: Unexpected end of request content at 15:16:10 on --0000004: the client aborted the request body, and it shows up in the 4xx bucket (8 × 4xx at 15:17), not 5xx.

Impact: 10 failed order placements over about 1 minute.

Action items (replace the earlier list)

1. Fix GetPaymentGatewayUrlV2() in OrdersController.cs (wrong gateway URL) before rolling v2 out again. High priority.
2. Gate API_VERSION changes behind a smoke test of POST /api/orders. Medium.
3. Datadog sidecar: set DD_HOSTNAME (separate issue). Medium.
4. CartController memory leak ([Incident] Recurring memory leak in CartController.AddItemToCart causing HTTP 5xx #318) is still in the deployed image. It's a latent risk and not related to this alert.