From 051e870cfaf23c99439ac313854ea125c6e20913 Mon Sep 17 00:00:00 2001 From: bgard68 <30295154+bgard68@users.noreply.github.com> Date: Mon, 24 Aug 2026 10:42:06 -0500 Subject: [PATCH] fix(deploy): require three consecutive ready responses before the smoke test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The readiness fix closed one race and revealed a second. webapps-deploy returns while the old process is still serving, so a single /ready 200 can come from the outgoing instance seconds before App Service restarts it. The wait passed on that stale answer and the smoke test landed mid-restart — the same 500-on-every- endpoint signature, on a build that served 200 everywhere once it settled. One ready answer proves the old instance was alive. Three in a row spanning twenty seconds cannot all be the doomed instance. Co-Authored-By: Claude Opus 5 --- .github/workflows/deploy.yml | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 85f5b1d..de6303c 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -188,19 +188,29 @@ jobs: BASE_URL: ${{ vars.API_BASE_URL }} run: | set -uo pipefail + # Two rules, each earned by a deploy that failed without them: + # + # /ready, not /live — liveness answers as soon as the process listens, before the + # app can serve, and the smoke test then measured a half-started app (500 on every + # endpoint, including readiness itself). + # + # Consecutive successes, not one — webapps-deploy returns while the OLD process is + # still serving, so a single 200 can come from the outgoing instance seconds before + # the restart. The smoke test then landed mid-restart and saw the same 500s. Three + # in a row spanning twenty seconds cannot all be the doomed instance. + streak=0 for attempt in $(seq 1 30); do - # /live, not /ready, was the original poll target, and it answers as soon as the - # process is listening — before the container has finished resolving. The smoke - # test then ran against a half-started app and reported 500 on every endpoint, - # including "Health readiness expected 200 got 500", which is the readiness probe - # saying it was not ready while the liveness probe said the app was up. That is - # the distinction the two endpoints exist for; the wait was asking the wrong one. code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 90 "$BASE_URL/api/health/ready" || true) if [[ "$code" == "200" ]]; then - echo "up after ${attempt} attempt(s)" - exit 0 + streak=$((streak + 1)) + if [[ "$streak" -ge 3 ]]; then + echo "ready: 3 consecutive 200s by attempt ${attempt}" + exit 0 + fi + else + streak=0 fi - echo "attempt ${attempt}: HTTP ${code:-none}" + echo "attempt ${attempt}: HTTP ${code:-none} (streak ${streak})" sleep 10 done echo "::error::App did not return 200 from /api/health/ready within the wait window."