Skip to content

Wait for worker health before staging API startup - #296

Merged
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/wait-for-worker-health
Aug 15, 2026
Merged

suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/wait-for-worker-health

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • replace the staging start workflow's one-shot worker health assertion with a bounded polling gate
  • wait for exactly two RUNNING and HEALTHY worker tasks before opening API admission
  • retain a five-minute failure bound and keep the API stopped when worker health never converges
  • add a contract for the cold-start health-check startPeriod regression

Evidence

The first manual cold-start run failed safely: run 31894301138. Both revision-14 tasks entered RUNNING around 00:01:20 Asia/Shanghai, while the workflow checked health around 00:01:33 and found zero healthy tasks. The task definition has a 60-second container health-check startPeriod; both tasks subsequently became HEALTHY. The API remained at desired count 0.

The AWS CLI services-stable waiter checks deployment count and runningCount == desiredCount; it does not wait for container health. We therefore poll task health explicitly every 15 seconds after the service waiter completes.

This is the bounded cold-start correction for Ontos-AI/knowhere-api-infra#30. It does not create resources or dispatch another AWS operation.

Verification

  • uv run pytest deploy/ecs -q — 15 passed
  • uv run --group lint ruff check deploy/ecs/test_manage_staging_workflow.py deploy/ecs/render_task_definitions.py deploy/ecs/test_render_task_definitions.py
  • uv run --group typecheck pyright deploy/ecs/test_manage_staging_workflow.py
  • workflow YAML parsing and embedded Bash syntax check
  • git diff --check
  • standards review: no findings
  • spec review: no findings

@suguanYang
suguanYang merged commit b5e039c into main Aug 15, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant