Make ship_and_run primitives safe for concurrent pod fan-out - #2
Open
protereus wants to merge 1 commit into
Open
Make ship_and_run primitives safe for concurrent pod fan-out#2protereus wants to merge 1 commit into
protereus wants to merge 1 commit into
Conversation
ship_and_run/ship_and_run_detached already guaranteed pod cleanup via finally, but could still raise past it (only CancelledError was caught) -- fine for one pod, but the moment several run concurrently via asyncio.gather, one raised exception surfaces to the caller immediately while sibling tasks keep running unmonitored in the background. Add ShipAndRunResult.error (+ raise_if_error() to opt back into today's behavior) and capture generic exceptions into it instead. Add ship_and_run_many() as a thin concurrent fan-out on top, using gather(..., return_exceptions=True) so a failing job never starves or loses a sibling's result. local_roots is required (not defaulting to None-per-job) since ship_and_run's own local_root has no such default and unconditionally uploads from it -- a None default here would have made every job silently fail with an AttributeError instead of failing loudly at the call boundary. Add PodGuard.terminate(verify=True) for an optional post-terminate status re-check, opt-in and fully backward compatible with existing zero-arg call sites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
protereus
force-pushed
the
feat/pod-fanout-safety
branch
from
July 22, 2026 15:22
690a687 to
8a21f71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1.
Changes
ShipAndRunResult.error(+raise_if_error()) —ship_and_run/ship_and_run_detachednow capture generic exceptions instead of raising past their existingfinallycleanup.ship_and_run_many()— concurrent fan-out overship_and_run, one pod per script, viagather(..., return_exceptions=True).local_rootsis required (not None-per-job default) sinceship_and_run's ownlocal_roothas no such default and unconditionally uploads from it.PodGuard.terminate(verify=True)— optional post-terminate status re-check, opt-in, defaults to today's behavior.All additive and backward-compatible — no existing caller or test needed to change.
Testing
pytest -q: 133 passed, 6 skipped (was 124/6 onmain; 9 new tests, zero regressions). New tests cover exception capture, the fan-out starvation property (a failing job doesn't block/lose a concurrent success), thelocal_rootsrequirement, andterminate's verify path.