revert(worker): rely on visibility-timeout recovery - #284
Merged
Merged
Conversation
The bounded gevent cancellation raced with an active PyMuPDF subprocess, removed its workspace, and still missed the ECS stop deadline. Keep the independent Redis visibility watchdog as the accepted interruption fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Staging evidence
During the approved active-job drill, job
job_07feda62dfc9was processing when ECS sent SIGTERM. The 90-second callback cancelled the greenlet and removed its workspace while a PyMuPDF subprocess was still writing. The process still missed the ECS stop deadline, exited 137, and did not restore the reservation promptly. The owner accepted the already-tested visibility-timeout fallback, with up to approximately 75 minutes of interruption delay, as the migration reliability trade-off.Validation
14 passedacross worker shutdown, visibility recovery, watchdog, bootstrap, and health contractsworker_bootstrap.pyTracks Ontos-AI/knowhere-api-infra#22.