Incident: Revision Unhealthy — Port Mismatch (TargetPort 9090 vs App Listening 8080) (Grubify Container App)

[daeungo1]

Incident Report: Revision Unhealthy — Port Mismatch Causes Crash Loop

• Incident ID: b7cdfe5a-60ce-08ab-e827-ef9a8b870004
• Service: Azure Container Apps — ca-grubify-huvqg3bjooyw6 (rg: rg-sre-lab)
• Subscription: 2cf925b6-80cb-4567-abda-5ccd3010aab5
• FQDN: ca-grubify-huvqg3bjooyw6.mangoflower-119d6108.eastus2.azurecontainerapps.io
• Active revision: ca-grubify-huvqg3bjooyw6--0000005 (100% traffic)

Summary

Revision ca-grubify-huvqg3bjooyw6--0000005 was deployed at 05:47:43 UTC on 2026-08-20 with environment variable ASPNETCORE_URLS=http://+:8080, causing the ASP.NET Core application to listen on port 8080. However, the Container App ingress targetPort remained set to 9090 (the port used by the previous revision --0000004). This mismatch caused startup probe failures ("connection refused" on port 9090), triggering a container crash loop. The app itself started cleanly each cycle — no OOM, no code errors — but was killed by the platform because the health probe could never reach port 9090.

Impact

• Total service outage from 05:47 to 05:55 UTC (~8 minutes). No requests could be served by the new revision.
• Requests arriving during 05:51–05:53 UTC (67 total) received errors or were dropped.
• The previous revision --0000004 was deactivated, so there was no fallback.

Timeline (UTC)

Time	Event
~05:46	Revision --0000004 healthy, app binding to http://+:9090, targetPort 9090 — normal
05:47:43	Revision --0000005 created with ASPNETCORE_URLS=http://+:8080 — app now listens on 8080
05:47:50	App starts cleanly on :8080, but startup probe targets :9090 → connection refused
05:49:15	Continuous Pending:PortMismatch and ReplicaUnhealthy events every ~1 second
05:49:30	Container killed: "Container grubify-api failed startup probe, will be restarted"
05:49–05:54	Crash loop — 4 restart cycles, each with image pull, clean app start on :8080, probe fail on :9090, kill
05:51:49	Alert alert-revision-unhealthy-sre-lab fired (Sev2)
05:55:00	Remediation applied: targetPort updated from 9090 → 8080 via SRE Agent
05:55:00	Revision updated in-place, container restarted with correct port mapping
05:57:00	Revision --0000005 confirmed Healthy, Running, 1/1 replicas, no PortMismatch errors

Evidence

System Logs — PortMismatch (ContainerAppSystemLogs_CL)

2026-08-20T05:50:11Z  Pending:PortMismatch  The TargetPort 9090 does not match the listening port 8080.
2026-08-20T05:49:39Z  ReplicaUnhealthy      startup probe failed: connection refused
2026-08-20T05:49:30Z  ReplicaUnhealthy      startup probe failed: connection refused
(repeated ~30 times between 05:49:28–05:50:11)

Console Logs — App binds to 8080 (ContainerAppConsoleLogs_CL)

2026-08-20T05:53:16Z  Overriding HTTP_PORTS 8080. Binding to values defined by URLS instead http://+:8080.
2026-08-20T05:49:16Z  Overriding HTTP_PORTS 8080. Binding to values defined by URLS instead http://+:8080.

Compared to previous revision --0000004 which bound to :9090:

2026-08-20T05:46:47Z  Overriding HTTP_PORTS 8080. Binding to values defined by URLS instead http://+:9090.

Container Configuration (at incident time)

• Image: acrcagrubifyhuvqg3bjooyw6.azurecr.io/grubify-api:latest
• CPU: 1 core, Memory: 2Gi
• ASPNETCORE_URLS=http://+:8080
• Ingress targetPort was 9090 — mismatch with app listening port

Request Metrics

Window	Requests	Notes
05:35–05:46	1/min	Normal baseline (idle environment)
05:47–05:50	0	No traffic served — revision unhealthy
05:51–05:53	15, 29, 23	Requests hitting unhealthy endpoint
05:54–05:57	0	Container restarting with fix

Root Cause

The deployment of revision --0000005 changed the ASPNETCORE_URLS environment variable from http://+:9090 to http://+:8080, but the Container App ingress targetPort was not updated to match. This caused a structural port mismatch: the platform startup probe attempted to reach port 9090, but the application was listening on port 8080. Every probe attempt returned "connection refused", causing the platform to kill and restart the container in a crash loop. The application code itself was healthy — the failure was purely a configuration mismatch.

Remediation

• Immediate (applied): Updated ingress targetPort from 9090 to 8080 using UpdateTargetPort API. Revision restarted in-place and recovered within ~2 minutes.
• Code: Ensure deployment pipelines validate that ASPNETCORE_URLS port and targetPort are consistent before deploying a new revision.
• Defensive: Add a pre-deployment check or CI/CD gate that compares the container listening port config against the ingress targetPort.
• Platform: Consider pinning ASPNETCORE_URLS to a fixed port (e.g., 8080) in the Dockerfile or app config, and setting targetPort=8080 in IaC as the canonical value.
• Observability: The existing alert-revision-unhealthy-sre-lab alert fired correctly (within ~4 minutes of the broken revision). Consider adding a faster probe for PortMismatch events specifically.

Action Items

#	Action	Priority
1	Pin ASPNETCORE_URLS port and targetPort to a single canonical value (8080) in IaC/Dockerfile	High
2	Add CI/CD validation gate: fail deployment if ASPNETCORE_URLS port != ingress targetPort	High
3	Investigate who/what changed ASPNETCORE_URLS from :9090 to :8080 in revision --0000005	Medium
4	Consider keeping the previous revision active as a fallback during new deployments	Medium
5	Add specific monitoring for Pending:PortMismatch events with faster evaluation window	Low

References

• Container App: /subscriptions/2cf925b6-80cb-4567-abda-5ccd3010aab5/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-huvqg3bjooyw6
• Log Analytics Workspace ID: b31cf7c6-e044-48ac-b7dc-d831aa06cdbb
• App Insights: /subscriptions/2cf925b6-80cb-4567-abda-5ccd3010aab5/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-huvqg3bjooyw6
• Alert Rule: /subscriptions/2cf925b6-80cb-4567-abda-5ccd3010aab5/resourceGroups/rg-sre-lab/providers/microsoft.insights/scheduledqueryrules/alert-revision-unhealthy-sre-lab
• Alert ID: b7cdfe5a-60ce-08ab-e827-ef9a8b870004
• Container Image: acrcagrubifyhuvqg3bjooyw6.azurecr.io/grubify-api:latest