Incident: Deployment ActivationFailed due to Port Mismatch (ca-api-deployment-compliance-dem)

[TraderShan]

Incident Report: Deployment ActivationFailed due to Port Mismatch

• Incident ID: f2e4b276-a136-ca15-8e78-42f20921f000
• Service: Azure Container Apps — ca-api-deployment-compliance-dem (rg: rg-deployment-compliance-demo)
• Subscription: 2c14ac17-ac47-4a4d-b2a4-4607601eab49
• FQDN: ca-api-deployment-compliance-dem.politehill-cd450ee4.eastus2.azurecontainerapps.io
• Active revision: ca-api-deployment-compliance-dem--d3nx0bb (100% traffic)

Summary

A deployment compliance alert fired at 23:51 UTC on 2026-08-31 after a Container App write operation (Microsoft.App/containerApps/write) was detected on ca-api-deployment-compliance-dem. Investigation reveals the sole revision is in an ActivationFailed / Unhealthy state due to a port mismatch: the ingress is configured to target port 8080, but the deployed container image (mcr.microsoft.com/azuredocs/containerapps-helloworld:latest) listens on port 80. This causes the platform's startup probe to fail continuously with "connection refused", preventing any replica from becoming ready.

Impact

• Service unavailable — The Container App has 0 healthy replicas; all inbound traffic to the FQDN returns errors.
• Continuous restart loop — The container is repeatedly started, fails startup probe, and is killed (~4-minute cycle observed).
• Deployment tags indicate pipeline origin — Tags show deployed-by: pipeline and commit-sha: initial, but the deployment was performed manually by shannichols@MngEnvMCAP776009.onmicrosoft.com via Azure CLI PUT, raising a compliance concern about deployment provenance tracking.

Timeline (UTC)

• ~23:35:04 — First Microsoft.App/containerApps/write (PUT Create) by Shan Nichols → revision --d3nx0bb created at 23:35:12
• ~23:35:40 — Container starts, logs "Listening on :80"
• ~23:35:54 — Startup probe failures begin: "connection refused" (probing port 8080, container on 80)
• ~23:39:40 — First container restart due to failed startup probe
• ~23:43:17 — Previous alert (dc920bcc-...) acknowledged by SRE Agent
• ~23:45:14 — "Deployment Progress Deadline Exceeded. 0/1 replicas ready." — revision marked ActivationFailed
• ~23:45:25 — Second PUT (Update) by Shan Nichols → no new revision provisioned, same configuration retained
• ~23:47–23:51 — Restart loop continues: start → listen on :80 → probe fail on 8080 → restart
• ~23:51:49 — Deployment compliance alert f2e4b276-a136-ca15-8e78-42f20921f000 fired

Evidence

Console logs (active revision)

2026-08-31T23:35:40Z  Listening on :80...
2026-08-31T23:39:43Z  Listening on :80...
2026-08-31T23:43:42Z  Listening on :80...
2026-08-31T23:47:43Z  Listening on :80...
2026-08-31T23:51:44Z  Listening on :80...

Container consistently starts and listens on port 80, while ingress targetPort is 8080.

System logs (startup probe failures)

2026-08-31T23:35:54Z  [Warning] startup probe failed: connection refused
2026-08-31T23:35:55Z  [Warning] startup probe failed: connection refused
2026-08-31T23:35:56Z  [Warning] startup probe failed: connection refused
...  (continuous every ~1 second)
2026-08-31T23:39:40Z  [Normal]  Container api failed startup probe, will be restarted
2026-08-31T23:43:40Z  [Normal]  Container api failed startup probe, will be restarted
2026-08-31T23:45:14Z  [Warning] Deployment Progress Deadline Exceeded. 0/1 replicas ready.
2026-08-31T23:47:41Z  [Normal]  Container api failed startup probe, will be restarted
2026-08-31T23:51:42Z  [Normal]  Container api failed startup probe, will be restarted

Deployment Timeline Chart

Activity Log (deployment operations)

Time (UTC)	Operation	Caller	Status	HTTP
23:35:04	containerApps/write	shannichols@MngEnv...	Accepted → Succeeded	PUT 201
23:45:25	containerApps/write	shannichols@MngEnv...	Accepted → Succeeded	PUT 201

Configuration Snapshot

Setting	Value	Issue
Image	mcr.microsoft.com/azuredocs/containerapps-helloworld:latest	Placeholder/demo image, not actual app
Ingress targetPort	8080	Mismatched with container port 80
Container ENV PORT	8080	Set but ignored by helloworld image
Registry auth	Password-based (acr-password secret)	Should use managed identity
Health probes	None configured	No custom liveness/readiness probes
Revision health	Unhealthy / ActivationFailed	0/1 replicas ready
Tag commit-sha	initial	Placeholder, not a real commit
Tag deployed-by	pipeline	Misleading — deployed manually via CLI
Tag pipeline-run-id	initial	Placeholder, not a real pipeline run
IP restrictions	None	Externally accessible with no restrictions
Min/Max replicas	1 / 3	—
Resources	0.5 CPU / 1Gi memory	—

Root Cause

The deployment failed because of a port mismatch: the Container App ingress is configured with targetPort: 8080, but the deployed image (mcr.microsoft.com/azuredocs/containerapps-helloworld:latest) only listens on port 80. The PORT=8080 environment variable is set but this demo image does not respect it. The platform's implicit startup probe checks the ingress target port (8080), gets "connection refused", and continuously restarts the container — ultimately exceeding the deployment progress deadline with 0/1 replicas ready.

Remediation

• Immediate fix: Change the ingress targetPort to 80 to match the container's listening port, OR deploy an application image that respects the PORT environment variable and listens on 8080.
• Code/Image: Replace the placeholder containerapps-helloworld:latest image with the actual Grubify API image from acrdeploymentcompliancedemol3ym45.azurecr.io.
• Defensive: Add explicit health probes (liveness + readiness) that target the correct port and path.
• Platform: Switch ACR authentication from password-based secrets to managed identity (identity field on the registry config).
• Observability: Ensure deployment tags (commit-sha, pipeline-run-id, deployed-by) reflect actual deployment metadata — not placeholder values — for accurate audit trail.

Action Items

#	Action	Priority
1	Fix port mismatch: set targetPort to 80 or deploy an image that listens on 8080	High
2	Replace placeholder containerapps-helloworld image with actual Grubify API image	High
3	Switch ACR registry auth from password-based to managed identity	Medium
4	Add explicit liveness and readiness health probes	Medium
5	Enforce accurate deployment tags (commit-sha, pipeline-run-id) via CI/CD pipeline	Medium
6	Add IP security restrictions or VNET integration for the ingress	Low
7	Configure Application Insights or diagnostic settings for the Container Apps environment	Low

References

• Container App: /subscriptions/2c14ac17-ac47-4a4d-b2a4-4607601eab49/resourceGroups/rg-deployment-compliance-demo/providers/Microsoft.App/containerApps/ca-api-deployment-compliance-dem
• Container Apps Environment: /subscriptions/2c14ac17-ac47-4a4d-b2a4-4607601eab49/resourceGroups/rg-deployment-compliance-demo/providers/Microsoft.App/managedEnvironments/cae-deployment-compliance-demo
• Log Analytics Workspace ID: b51d4293-b00b-4483-8738-5359895a0206
• ACR: acrdeploymentcompliancedemol3ym45.azurecr.io
• Alert ID: f2e4b276-a136-ca15-8e78-42f20921f000
• Alert Rule: /subscriptions/2c14ac17-ac47-4a4d-b2a4-4607601eab49/resourceGroups/rg-deployment-compliance-demo/providers/microsoft.insights/activityLogAlerts/alert-containerapp-deployment-deployment-compliance-demo
• App Insights: Not configured for this environment

---

This issue was created by sre-agent-zltoweask56oi--b504a391
Tracked by the SRE agent here