Skip to content

Run parallel local batches with durable attempts and explicit retries #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matchalatteguy merged 1 commit into from
Oct 2, 2026
Merged

Run parallel local batches with durable attempts and explicit retries #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ concurrency:
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
Expand All @@ -29,14 +30,42 @@ jobs:
- name: Lint and test
run: |
uv run --no-sync ruff check .
uv run --no-sync ruff format --check .
uv run --no-sync pytest
- name: Build and verify installed wheel outside the checkout
run: |
uv build --no-sources
uv build --no-sources --out-dir .agent-backlog/dist
uv venv --python "${{ matrix.python }}" .wheel-smoke
uv pip install --python .wheel-smoke/bin/python dist/*.whl
uv pip install --python .wheel-smoke/bin/python .agent-backlog/dist/*.whl
smoke_python="$PWD/.wheel-smoke/bin/python"
smoke_dir=$(mktemp -d)
cd "$smoke_dir"
"$smoke_python" -m agent_backlog_runner.demo --output demo
"$smoke_python" -m agent_backlog_runner.cli status --db demo/tasks.sqlite3 --format json
"$smoke_python" -m agent_backlog_runner.cli --version
"$smoke_python" -m agent_backlog_runner.cli demo --output maintenance
"$smoke_python" -m agent_backlog_runner.cli status --db maintenance/checks.sqlite3 --format json
macos-processes:
runs-on: macos-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.14'
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
- name: Install dependencies
run: uv sync --locked --extra dev --python 3.14
- name: Check attempts, parallel workers, recovery and process groups
run: uv run --no-sync pytest tests/test_batch_execution.py tests/test_dispatch.py
- name: Verify installed repair demo outside the checkout
run: |
uv build --no-sources --out-dir .agent-backlog/dist
uv venv --python 3.14 .wheel-smoke
uv pip install --python .wheel-smoke/bin/python .agent-backlog/dist/*.whl
smoke_python="$PWD/.wheel-smoke/bin/python"
smoke_dir=$(mktemp -d)
cd "$smoke_dir"
"$smoke_python" -m agent_backlog_runner.cli --version
"$smoke_python" -m agent_backlog_runner.cli demo --output maintenance
"$smoke_python" -m agent_backlog_runner.cli status --db maintenance/checks.sqlite3 --format json
35 changes: 35 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Changelog

## 0.3.0

- Run local commands concurrently with a real shared SQLite capacity cap. `--drain`
processes a finite queued snapshot, filling slots as workers exit.
- Add idempotent finite command manifests, explicit bounded retries with capped
exponential backoff, per-task timeouts, and durable token-owned attempt history.
- Store separate capped stdout/stderr files, drain both streams without unbounded
capture, and record observed bytes/truncation plus short event tails.
- Add `cancel`, conservative `recover`, `attempts`, `logs`, `report`, `demo`, and
`--version` CLI paths. Cancellation is handled by the owning coordinator;
recovery never signals stored PIDs and never silently retries uncertain work.
- Stop owned POSIX process groups on timeout/cancel/interruption. Keep leaders
unreaped during group signalling; document the older-macOS normal-exit fallback.
- Add an executable repository-maintenance repair/resume walkthrough and a
reusable pytest/Ruff manifest, plus targeted concurrency/crash/log/retry tests.

### Compatibility and migration

Schema 1 migrates automatically to schema 2, preserving task/event/scheduler data.
Old tasks receive three future attempts and a one-second base backoff; past
execution history cannot be reconstructed. Old running tasks retain their state
and require explicit inspection and unknown-identity recovery. Version 0.2
refuses schema 2; retain an earlier copy before upgrading if rollback is needed.

Default dispatch still selects a bounded batch and defaults to preview, but its
workers now execute concurrently. Only `--drain` waits for future retry due times.
Worker failure/cancellation now returns CLI exit `1`, blocked capacity/budget `2`,
and Ctrl-C `130`. `DispatchResult` adds completed/failed/cancelled ids; existing
`started` and `skipped_reason` fields remain. Existing string commands/placeholders
and scheduler APIs remain available. Duplicate YAML/JSON fields are now rejected.

For managed attempts, use `cancel`, `retry`, and `recover`; the CLI rejects raw
status edits that would bypass live-attempt ownership or retry budgets. Low-level
manual `mark_task`/`claim_task` bookkeeping APIs remain for compatibility.

## 0.2.0

- Claim a task and a capacity slot atomically before a subprocess starts.
Expand Down
220 changes: 99 additions & 121 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,164 +2,142 @@

[![CI](https://github.com/matchalatteguy/agent-backlog-runner/actions/workflows/ci.yml/badge.svg)](https://github.com/matchalatteguy/agent-backlog-runner/actions/workflows/ci.yml)

Keep a small local task queue replenished from a validated JSON or YAML template
catalog. Preview the next batch, store tasks and events in SQLite, then run short
worker commands and inspect what completed or failed.
A local command queue for repository checks and repeatable maintenance. Load a finite batch, run independent commands in parallel, inspect each attempt's exit code and logs, repair failures, and resume without repeating completed work. SQLite stores the queue and attempt history across CLI invocations.

**This project is a queue replenisher with a synchronous dispatcher.** Its sibling
[Local Agent Task Runtime](https://github.com/matchalatteguy/local-agent-task-runtime)
manages registered tasks in detached sessions and contained workspaces. Choose
this runner for recurring template work and short commands; choose the runtime
when independent workers need to outlive the dispatching process.
Use it when a shell loop is becoming hard to resume or debug. Templates can also replenish a recurring backlog below a chosen depth. Commands run while the dispatcher is open; detached sessions, workspace provisioning, dependency graphs, and distributed workers are outside this tool's scope.

```text
template catalog -> depth plan -> SQLite queue -> local command -> status + events
## Install and try the repair/resume walkthrough

Python 3.11+ on Linux or macOS. Version 0.3 is an alpha release; unfinished attempts require operator inspection and explicit recovery.

```bash
uv tool install 'git+https://github.com/matchalatteguy/agent-backlog-runner.git@v0.3.0'
agent-backlog --help
```

## Run a real example
From a checkout, use `uv sync --locked --extra dev` and prefix commands with `uv run`.

Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/). From a fresh checkout:
The packaged demonstration requires no test framework or external service:

```bash
uv sync --locked --extra dev
uv run python -m agent_backlog_runner.demo --output .agent-backlog/demo
agent-backlog demo --output .agent-backlog/maintenance-demo
```

The demo creates three Markdown documents, previews a queue replenishment plan,
enqueues three audits, then executes a real Python worker for each document.
Workers count headings and TODO lines, write JSON reports, and send heartbeats.
The final report is read after reopening SQLite:

```json
{
"completed_tasks": 3,
"created_tasks": 3,
"follow_up_items": 3,
"reports": [
{"document": "onboarding.md", "headings": 1, "todo_items": 1},
{"document": "release.md", "headings": 2, "todo_items": 0},
{"document": "troubleshooting.md", "headings": 1, "todo_items": 2}
]
}
```
From a checkout, run `uv run agent-backlog demo --output .agent-backlog/maintenance-demo`.

Whitespace in the printed JSON differs. Outputs are under
`.agent-backlog/demo/reports/`; task state is in
`.agent-backlog/demo/tasks.sqlite3`. Use a new `--output` directory for another
run; the demo refuses to overwrite an existing directory.
The walkthrough creates a tiny Python project, runs its unit tests, reopens the queue, then runs syntax and README-link checks. The missing usage document fails its check. It writes the document, explicitly schedules one retry, and resumes through another SQLite connection:

```bash
uv run agent-backlog status --db .agent-backlog/demo/tasks.sqlite3
uv run agent-backlog events --db .agent-backlog/demo/tasks.sqlite3 --limit 20
```text
checks: 3
initial_completed: 1
resume_failed: 1
repaired_checks: 1
attempts: 4
final_done: 3
```

## Plan and replenish your own queue
[Implementation](src/agent_backlog_runner/maintenance.py). These are real commands on illustrative files, not a claim of production use. Inspect `before-repair.md`, `after-repair.md`, and `logs/` under the output directory. The demo refuses to overwrite an existing output directory.

## Run checks on your own repository

A template describes a repeatable unit of work:
Save a trusted manifest as `checks.yaml`:

```yaml
templates:
- slug: docs-refresh
title: "Review documentation ${sequence}"
body: "Review one documentation page and record an improvement."
lane: docs
role: writer
tags: [docs]
schema_version: 1
name: repo-checks
tasks:
- id: unit-tests
argv: ['{python}', '-m', pytest, '-q']
timeout_seconds: 600
max_attempts: 3
retry_backoff_seconds: 2
- id: lint-check
argv: ['{python}', '-m', ruff, check, '.']
timeout_seconds: 120
max_attempts: 3
```

Try the bundled catalog in a fresh database:
Use your project's interpreter, with its test dependencies installed:

```bash
uv run agent-backlog templates validate --templates examples/templates/basic-backlog.yaml
uv run agent-backlog scheduler plan --db .agent-backlog/queue.sqlite3 \
--templates examples/templates/basic-backlog.yaml --min-depth 2 --target-depth 4 --max-enqueue 3 --json
uv run agent-backlog scheduler run --db .agent-backlog/queue.sqlite3 \
--templates examples/templates/basic-backlog.yaml --min-depth 2 --target-depth 4 --max-enqueue 3
agent-backlog batch load checks.yaml --root . \
--python "$PWD/.venv/bin/python" --run-id first-pass \
--db .agent-backlog/checks.sqlite3

# Preview; commands do not run until --backend subprocess is selected.
agent-backlog dispatch --db .agent-backlog/checks.sqlite3 --max-workers 2 --drain
agent-backlog dispatch --db .agent-backlog/checks.sqlite3 \
--backend subprocess --max-workers 2 --drain

agent-backlog status --db .agent-backlog/checks.sqlite3
agent-backlog attempts --db .agent-backlog/checks.sqlite3
agent-backlog report --db .agent-backlog/checks.sqlite3 \
--json-out .agent-backlog/checks.json --md-out .agent-backlog/checks.md
```

Expected: `valid templates: 3`, a preview containing three planned tasks, then
`created 3`. Active depth counts `todo` and `running`. A later replenishment run
adds nothing while depth remains at or above the minimum. Titles are checked for
duplicates, including duplicates within the same plan.
`--max-workers 2` really runs two commands concurrently. Claims and the shared capacity are reserved atomically; cancelled attempts retain their slots until stopped. Concurrent coordinators must use the same cap. Avoid putting commands that write the same files into one parallel batch.

Planning creates no tasks or events and does not advance the cursor. The CLI may
initialize the selected SQLite database on first use. Applying a plan commits
its tasks, events, and cursor together; stale plans must be regenerated.
Reloading the same manifest/root/run id is idempotent. It preserves task statuses and rejects changed command definitions. Choose a new `--run-id` for a new full check pass. The [checked-in manifest](examples/repo-checks.yaml) runs this repository's tests and lint after installing development dependencies.

## Dispatch short commands
## Repair, retry, and resume

Preview selected task ids with the default backend:
A failed command stays failed. Inspect its task id and retained stderr:

```bash
uv run agent-backlog dispatch --db .agent-backlog/queue.sqlite3 --max-workers 2
agent-backlog attempts --db .agent-backlog/checks.sqlite3 --format json
agent-backlog logs --db .agent-backlog/checks.sqlite3 TASK_ID --stream stderr
# Repair the source or configuration, then schedule failed tasks with budget remaining.
agent-backlog retry --db .agent-backlog/checks.sqlite3 --failed
agent-backlog dispatch --db .agent-backlog/checks.sqlite3 \
--backend subprocess --max-workers 2 --drain
```

This records `dispatch_preview` events and leaves tasks `todo`. Execute the
bundled worker explicitly:
Retry is explicit, bounded by `max_attempts`, and delayed by exponential backoff capped at 60 seconds. Spawn failures also consume attempts. `--drain` waits for scheduled retries in its initial queued snapshot, fills slots as workers finish, and leaves completed tasks untouched. Without `--drain`, one ready batch is selected; delayed work does not block later ready tasks.

Ctrl-C stops owned workers and records interrupted attempts as failed. Queued tasks remain ready. An abrupt dispatcher crash leaves unfinished work requiring [explicit recovery](docs/batches-and-recovery.md#interruption-and-recovery), because its side effects may already have happened. This is at-least-once execution after an operator chooses recovery and retry.

| Dispatch exit | Meaning |
|---|---|
| `0` | Selected work passed, preview completed, or no eligible work was selected. |
| `1` | A selected worker failed/cancelled, or a CLI input/operation failed. |
| `2` | Capacity or exhausted attempts prevented dispatch. |
| `130` | Ctrl-C interrupted the dispatcher. |

Use `status` to distinguish an empty queue from future scheduled work. JSON dispatch output lists started, completed, failed, and cancelled task ids.

## Recurring template work

The original queue-replenishment workflow remains available:

```bash
uv run agent-backlog dispatch --db .agent-backlog/queue.sqlite3 \
--backend subprocess --max-workers 1 --timeout 60 \
--command "bash examples/worker_echo.sh {task_id}"
agent-backlog scheduler plan --db .agent-backlog/recurring.sqlite3 \
--templates examples/templates/basic-backlog.yaml \
--min-depth 2 --target-depth 4 --max-enqueue 3 --json
agent-backlog scheduler run --db .agent-backlog/recurring.sqlite3 \
--templates examples/templates/basic-backlog.yaml \
--min-depth 2 --target-depth 4 --max-enqueue 3
```

A zero exit code marks the task `done`. Nonzero exits, timeouts, missing binaries,
and invalid workdirs mark it `failed`; later tasks in the selected batch still
run. `--max-workers` bounds the batch against current running work. Commands run
one at a time inside the CLI process, even when the limit is larger than one.
Output snippets are decoded as UTF-8 with replacement characters for invalid
bytes, so binary or differently encoded output cannot strand the batch.

Each task can have its own `--command` and `--workdir` through `enqueue`. A
command supplied to `dispatch` overrides those commands. Supported placeholders
are `{task_id}`, `{title}`, `{lane}`, `{role}`, `{priority}`, and `{workdir}`.
Shell quoting is parsed **before** substitution, preserving each placeholder as
one argument. No shell is started automatically. Treat command templates as
trusted code, especially if you invoke a shell or interpolate into program text.

Atomic claims prevent two local dispatchers from executing the same queued task.
Completion updates preserve an operator's intervening cancellation or status
change; changing a status does not itself stop an already running process.

## Python API

```python
from agent_backlog_runner import BacklogPolicy, init_store, load_template_catalog
from agent_backlog_runner.scheduler import apply_backlog_plan, plan_backlog

with init_store(".agent-backlog/api.sqlite3") as store:
catalog = load_template_catalog("examples/templates/basic-backlog.yaml")
policy = BacklogPolicy(min_queue_depth=2, target_queue_depth=4)
plan = plan_backlog(store, catalog, policy, dry_run=False)
tasks = apply_backlog_plan(store, plan)
print(len(tasks))
```
Templates create work descriptions. Supply per-task commands or a dispatch command when executing them. Plans check queue depth and duplicate titles; applying a stale plan is rejected. The scheduler cursor is shared across catalogs and filters. [Template schema](docs/template-schema.md) · [Scheduler policy](docs/scheduler-policy.md).

## Logs, processes, and trust

Each attempt retains stdout and stderr in separate files, up to 1 MiB per stream by default. Excess bytes are drained and discarded; attempts record observed byte counts and truncation flags. `--max-log-bytes` changes the per-stream cap up to 64 MiB. Full output is available only below the cap. Historical attempt files are retained until you remove or archive them yourself.

A plan marked `dry_run=True` cannot create tasks. See [CHANGELOG.md](CHANGELOG.md)
for the 0.2 API change. Every SQLite CLI command accepts `--db`; the fallback is
`AGENT_BACKLOG_DB`, then `.agent-backlog/tasks.sqlite3`.
Timeout and cancellation stop the owning dispatcher's process group on POSIX. Workers that escape their group, change privileges, or leave uninterruptible OS operations are outside that guarantee. On interpreters with `waitid(WNOWAIT)`, normal exit also cleans up remaining group members before the leader is reaped. Older macOS interpreters lack that operation: normal-exit background children cannot be safely adopted; commands must wait for their children. Windows supports queue inspection/planning but not subprocess dispatch in this release.

## Verification and boundaries
Subprocess dispatch requires the default `SIGCHLD` handler. Embedded callers must not reap its children independently; lost ownership prevents group signals and leaves unfinished work for inspection.

Commands and manifests are trusted local code, with normal access to your files, environment, and network. Workdir containment prevents accidental manifest path escapes; it is not a process sandbox. No shell is inserted automatically. Legacy command placeholders are substituted after argument splitting. SQLite, logs, and exported reports can contain command arguments, local paths, and sensitive output; keep them in a protected ignored directory.

## Development and documentation

```bash
uv run pytest
uv sync --locked --extra dev
uv run ruff check .
uv build
uv run ruff format --check .
uv run pytest
uv build --no-sources
```

CI tests Python 3.11–3.14 and executes the demo from an installed wheel outside
the checkout. Regression tests cover timeout recovery, competing claims, partial
plan rollback, duplicate titles, and stale plans.

This is an alpha library for one local machine. It has no detached worker pool,
automatic retries, process sandbox, distributed leases, or exactly-once execution.
A killed dispatcher can leave `running` tasks requiring operator recovery.
`stale` reports those tasks; it does not restart them. Timeouts terminate the
direct worker process, not an arbitrary tree of child processes. The cursor is
shared across catalogs and filters; title-based deduplication is deliberately
simple. Events retain the last 500 characters of worker output, so protect the
SQLite database and keep command output free of secrets.

[Template schema](docs/template-schema.md) · [Scheduler policy](docs/scheduler-policy.md) ·
[Dispatch](docs/dispatch-backends.md) · [Python API](docs/python-api.md) ·
[Troubleshooting](docs/troubleshooting.md)
CI tests Python 3.11–3.14 on Linux, plus process ownership and the repair/resume walkthrough on macOS 3.14. It covers overlapping workers, competing capacity claims, interruption, conservative recovery, bounded logs, and retries; installed-wheel examples run outside the checkout. [0.3 migration](CHANGELOG.md) · [Batch manifests and recovery](docs/batches-and-recovery.md) · [Dispatch](docs/dispatch-backends.md) · [Python API](docs/python-api.md) · [Troubleshooting](docs/troubleshooting.md) · [MIT license](LICENSE).
Loading
Loading