From 9bb4b41e6aeecffa32172eca543cb0e81105818d Mon Sep 17 00:00:00 2001 From: Hussain Chinoy Date: Mon, 28 Sep 2026 23:41:40 +0000 Subject: [PATCH] fix(gateway-deploy): submit the build async and poll its status; log gateway v0.1.2 gcloud builds submit with --config streams logs and exits non-zero when the caller can't read the default logs bucket, even though the build succeeds (hit while deploying v0.1.2). Submit with --async and poll the build status. Deployment log: gateway revision 00061 (v0.1.2). --- .../runs/20260925-serving-speed/deployments.md | 2 ++ scripts/deploy_cloudrun_gateway.sh | 16 +++++++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/benchmarks/runs/20260925-serving-speed/deployments.md b/benchmarks/runs/20260925-serving-speed/deployments.md index d24144d..719ac5a 100644 --- a/benchmarks/runs/20260925-serving-speed/deployments.md +++ b/benchmarks/runs/20260925-serving-speed/deployments.md @@ -33,6 +33,8 @@ | 09-28 23:10 | Cloud Run `dgemma`, `dgemma-gateway`; Vertex model registry | Pruned to n and n-1: 81 revisions deleted (kept gateway `00057`, `00055`; serving `00034`, `00032`); 3 older model resources deleted (kept v0.1.0 and `504638d`) | — | — | | 09-28 22:50–23:01 | Monitoring | Health check `dgem-probe-hourly` / `-daily` (Cloud Run jobs + Scheduler, SA `dgem-probe-sa`), 4 log-based metrics, 7 `dgem:` alert policies (email) | First runs: Vertex ok 78 ms, gateway ok 192 ms, Cloud Run ok 82 ms | — | +| 09-28 23:39 | Cloud Run `dgemma-gateway` → revision `00061` | **v0.1.2** (gateway/monitoring): decision logs carry `dgem_backend_requested`, failed requests log backend `none` | All checks pass; new log fields confirmed; failover and p95 alerts now have data | Revision `00057` (v0.1.1); older revisions pruned | + Follow-ups: - ~~Rotate the Hugging Face token~~ done (version 2). - ~~Cloud Run health lacks `warmed`~~ not a bug: the earlier check ran before warmup finished; later checks show diff --git a/scripts/deploy_cloudrun_gateway.sh b/scripts/deploy_cloudrun_gateway.sh index 309816b..bd34e4a 100644 --- a/scripts/deploy_cloudrun_gateway.sh +++ b/scripts/deploy_cloudrun_gateway.sh @@ -63,11 +63,21 @@ steps: args: [build, --build-arg=DGEM_VERSION=${GW_VERSION}, --build-arg=DGEM_REVISION=$(git rev-parse --short HEAD), -t, "${IMAGE}", .] images: ["${IMAGE}"] YAML -gcloud builds submit "${TMP_CTX}" \ +# Submit asynchronously and poll: a synchronous submit exits non-zero when the caller cannot stream from the +# default logs bucket, even though the build succeeds. +BUILD_ID="$(gcloud builds submit "${TMP_CTX}" \ --project="${PROJECT}" \ --config="${TMP_CTX}/cloudbuild.yaml" \ - --suppress-logs \ - --quiet + --async --format='value(id)' --quiet)" +echo " build ${BUILD_ID}" +while true; do + BUILD_STATUS="$(gcloud builds describe "${BUILD_ID}" --project="${PROJECT}" --format='value(status)')" + case "${BUILD_STATUS}" in + SUCCESS) break ;; + QUEUED|WORKING|PENDING) sleep 10 ;; + *) echo "ERROR: build ${BUILD_ID} ended with status ${BUILD_STATUS}" >&2; exit 1 ;; + esac +done GPU_IDLE_TTL="${GPU_IDLE_TTL:-3h}" VERTEX_ENDPOINT_ID="${DGEM_VERTEX_URL:-}"