Skip to content

Run parallel local batches with durable attempts and explicit retries - #2

Merged
matchalatteguy merged 1 commit into
mainfrom
usefulness-20261002
Oct 2, 2026
Merged

matchalatteguy merged 1 commit into
mainfrom
usefulness-20261002

Conversation

@matchalatteguy

Copy link
Copy Markdown
Owner

What changes

Turns the subprocess dispatcher into a parallel, bounded local command queue. SQLite stores individual attempts, ownership, outcomes, exit codes, log paths, retry budgets, and retry due times. Finite JSON batch manifests can be loaded idempotently and resumed after fixing a failed command.

Adds cancellation, explicit retries with bounded backoff, conservative abandoned-attempt recovery, capped stdout/stderr logs, a repository-check and repair example, schema migration, and practical installation and operations guides.

Verification

  • Actual parallel execution and atomic capacity claims.
  • Cancellation, timeout, crash recovery, and owned process-group cleanup.
  • Retry budgets, due times, attempt history, and batch identity conflicts.
  • Binary and large output without blocking or unbounded log retention.
  • Real repair-and-resume example from an installed wheel outside the checkout.
  • Hosted Python and platform checks.

Scope and limits

The dispatcher stays alive while its commands run. Retries are explicit, and a crashed dispatcher may require operator inspection and recovery. This is a local queue with no distributed leases or exactly-once guarantee. On Python builds without waitid(WNOWAIT), successful commands must not leave background children running.

@matchalatteguy
matchalatteguy merged commit f5b3578 into main Oct 2, 2026
10 checks passed
@matchalatteguy
matchalatteguy deleted the usefulness-20261002 branch October 2, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant