Skip to content

revert(worker): rely on visibility-timeout recovery - #284

Merged
suguanYang merged 1 commit into
mainfrom
revert/wangbinqi/bounded-worker-shutdown
Aug 14, 2026
Merged

suguanYang merged 1 commit into
mainfrom
revert/wangbinqi/bounded-worker-shutdown

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • Remove the bounded gevent active-task cancellation introduced by fix(worker): bound gevent warm shutdown #282.
  • Retain the reconnect-safe gevent cancellation no-op and document why interruption recovery relies on Redis visibility recovery.
  • Retain the independent watchdog, the 4,500-second visibility contract, and the existing billing/idempotency behavior.

Staging evidence

During the approved active-job drill, job job_07feda62dfc9 was processing when ECS sent SIGTERM. The 90-second callback cancelled the greenlet and removed its workspace while a PyMuPDF subprocess was still writing. The process still missed the ECS stop deadline, exited 137, and did not restore the reservation promptly. The owner accepted the already-tested visibility-timeout fallback, with up to approximately 75 minutes of interruption delay, as the migration reliability trade-off.

Validation

  • 14 passed across worker shutdown, visibility recovery, watchdog, bootstrap, and health contracts
  • Ruff passed for changed Python files
  • Pyright passed for worker_bootstrap.py
  • Standards/spec review found no blocking findings; unrelated bootstrap extraction remains out of scope

Tracks Ontos-AI/knowhere-api-infra#22.

The bounded gevent cancellation raced with an active PyMuPDF subprocess, removed its workspace, and still missed the ECS stop deadline. Keep the independent Redis visibility watchdog as the accepted interruption fallback.
@suguanYang
suguanYang merged commit e905b9f into main Aug 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant