Skip to content

Stop one pipeline stalling lease reclaim, and sweep runs left with pending dependents - #169

Open
blakebauman wants to merge 4 commits into
mainfrom
fix/reclaim-stall-and-orphan-sweep
Open

blakebauman wants to merge 4 commits into
mainfrom
fix/reclaim-stall-and-orphan-sweep

Conversation

@blakebauman

Copy link
Copy Markdown
Contributor

Audit findings COR-1 and COR-3 (11 Nov), plus the retries overflows.

1. An unbounded timeout_minutes stalled reclaim instance-wide

timeout_minutes is a u32; two SQL statements cast the stored value to ::int. One pipeline with 3000000000 whose run stayed running made the timeout query raise every tick, and reclaim_once returned on that error before lease reclaim and orphan finalisation — for every project, on every replica, with /ready still ok. Writer-reachable.

  • The fix is the SQL: both queries read the snapshot through numeric and clamp to a year, so a value already stored cannot raise.
  • Compile bound stops new values: timeout_minutes ≤ 525600 (DagError::TimeoutTooLong).
  • Stage isolation limits the next such bug: every stage of a reclaim tick runs regardless of the others, and the first error is still returned naming the failed stages.
  • One rule for a stored timeout on both sides — dag::stored_timeout_minutes, used by the offer and restated by the SQL: a whole number ≥ 1 clamped to a year, anything else absent. Before, the agent and the server backstop disagreed on out-of-range values.

2. retries overflowed in three readers

Bound at compile (≤ 100); the column write clamps; requeue_outcome and the backoff use saturating arithmetic. In-range backoff values are pinned against the old expression for attempts 0–63.

3. The orphan sweep missed runs with pending dependents

A step's completion commits, then propagate_after_step runs as a separate transaction. If the process died or the pool timed out between the two, dependents stayed pending, nothing was queued or running, and the run was running forever. The sweep only selected runs with no pending steps, so it never saw them. It now selects running runs with nothing queued or running, and the caller re-checks before warning.

Upgrade note

A stored pipeline with timeout_minutes above 525600 or retries above 100 now fails to compile; the boot scan names each one, and a push or schedule against it leaves a failed run saying why. Runs already in flight keep their snapshot and are handled by the clamps.

Verified

  • make check green; cargo test --workspace green (467 tests). Each new test was broken, seen red, and restored — including source-audit tests that fail if the bare cast or a ? in reclaim_once comes back.
  • SQL executed against Postgres 17 (throwaway container, all 18 migrations applied):
    • The old run-timeout query raises value "3000000000" is out of range for type integer.
    • The new queries return without error; 3000000000, 5000000000 and 1e20 read as 525600; "abc", 1.5, 0, -5, null and absent read as absent; 5.0 reads as 5; absurd default/grace binds do not raise.
    • The old sweep predicate misses a run with one succeeded step and a pending dependent; the new one selects it and leaves a run with a queued step alone.
  • Independent review of the first three commits: nothing blocking; its follow-ups are the fourth commit.

Not verified / left out

  • No smoke was run locally; reclaim_once invoking every stage is covered by the source audit, not by execution.
  • A reclaim stage that fails every tick is still only a log line — there is no counter in /metrics or /ready for a loop that errors without dying. Left for a follow-up.
  • A hand-edited run whose needs names a step that does not exist is re-selected by the sweep every tick and holds one of its 100 slots.

🤖 Generated with Claude Code

blakebauman and others added 4 commits November 11, 2026 15:41
A timeout_minutes above 2^31-1 fits the u32 and overflowed the int cast in the
timeout sweep's make_interval, so the query raised for as long as the run stayed
running. The sweep ran before lease reclaim and orphan finalisation and they
were chained with ?, so every reclaim tick on every replica ended there.

- Reject timeout_minutes above 525600 (one year) at compile time, on the
  pipeline and on a step.
- Clamp in bigint before the cast in both sweep queries, and read a snapshot
  value that is not a non-negative whole number as absent, so a snapshot stored
  before the bound cannot raise.
- Run every reclaim stage regardless of the others, log each failure at error
  with its stage, and return the first error so the tick still reports failure.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
retries is a u32 with no upper limit and lands in an int4 column: 4294967295
was stored as -1 (a step that never retries), 2147483647 overflowed
retries + 1 inside the reclaim transaction, and 2^attempt overflowed a u64
from attempt 64 (a panic in debug, a zero backoff in release).

Reject retries above 100 at compile time, saturate the column and the
lost-lease decision, and use saturating_pow under the existing 60 s cap. The
backoff for every attempt that used to fit is unchanged and pinned.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A step's outcome commits in one transaction and propagate_after_step runs as
the next. A crash or pool timeout between them leaves the dependents pending
with nothing queued or running, and the run running forever: the agent does
not resend a completion. The reclaim loop's sweep exists for this window but
treated pending as open, so it only caught runs whose every step had finished.

Select running runs with no queued or running step instead. Pending is never
a resting state on its own (backoff, backed-out offers and unmatched labels
are all queued), so propagation always moves such a run; an exhaustive test
over a diamond pins that. Rename the store method and the scheduler stage to
say what they do, and stop warning about a run the sweep picked just before
its own propagation ran.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ies.

Review follow-ups to the three fixes before this.

- The offer and the timeout sweep disagreed on an out-of-range stored
  timeout_minutes: above u32::MAX the agent got the 60-minute default while the
  server waited a year, and a 19+ digit value was no timeout at all. Both now
  follow dag::stored_timeout_minutes: a whole number of at least 1, clamped to
  a year; anything else absent. The SQL goes through numeric, so there is no
  digit limit.
- retries_column clamps to MAX_RETRIES instead of saturating to i32::MAX, so a
  retry of an old run that asked for 4294967295 does not retry for ever.
- A run deleted between list_timed_out_runs and its cancel is done, not a
  failed stage; cancel_run_with_reason returns the typed StoreError::NotFound.
- Source audits: the two timeout queries only cast a guarded, clamped value,
  and reclaim_once contains no ? operator.
- Correct the changelog: the SQL clamp is the fix, the bound stops new values,
  and stage isolation limits the next such bug.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant