Repository navigation
Stop one pipeline stalling lease reclaim, and sweep runs left with pending dependents - #169
Open
blakebauman wants to merge 4 commits into
Open
blakebauman wants to merge 4 commits into
blakebauman wants to merge 4 commits into
Conversation
A timeout_minutes above 2^31-1 fits the u32 and overflowed the int cast in the timeout sweep's make_interval, so the query raised for as long as the run stayed running. The sweep ran before lease reclaim and orphan finalisation and they were chained with ?, so every reclaim tick on every replica ended there. - Reject timeout_minutes above 525600 (one year) at compile time, on the pipeline and on a step. - Clamp in bigint before the cast in both sweep queries, and read a snapshot value that is not a non-negative whole number as absent, so a snapshot stored before the bound cannot raise. - Run every reclaim stage regardless of the others, log each failure at error with its stage, and return the first error so the tick still reports failure. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
retries is a u32 with no upper limit and lands in an int4 column: 4294967295 was stored as -1 (a step that never retries), 2147483647 overflowed retries + 1 inside the reclaim transaction, and 2^attempt overflowed a u64 from attempt 64 (a panic in debug, a zero backoff in release). Reject retries above 100 at compile time, saturate the column and the lost-lease decision, and use saturating_pow under the existing 60 s cap. The backoff for every attempt that used to fit is unchanged and pinned. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A step's outcome commits in one transaction and propagate_after_step runs as the next. A crash or pool timeout between them leaves the dependents pending with nothing queued or running, and the run running forever: the agent does not resend a completion. The reclaim loop's sweep exists for this window but treated pending as open, so it only caught runs whose every step had finished. Select running runs with no queued or running step instead. Pending is never a resting state on its own (backoff, backed-out offers and unmatched labels are all queued), so propagation always moves such a run; an exhaustive test over a diamond pins that. Rename the store method and the scheduler stage to say what they do, and stop warning about a run the sweep picked just before its own propagation ran. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ies. Review follow-ups to the three fixes before this. - The offer and the timeout sweep disagreed on an out-of-range stored timeout_minutes: above u32::MAX the agent got the 60-minute default while the server waited a year, and a 19+ digit value was no timeout at all. Both now follow dag::stored_timeout_minutes: a whole number of at least 1, clamped to a year; anything else absent. The SQL goes through numeric, so there is no digit limit. - retries_column clamps to MAX_RETRIES instead of saturating to i32::MAX, so a retry of an old run that asked for 4294967295 does not retry for ever. - A run deleted between list_timed_out_runs and its cancel is done, not a failed stage; cancel_run_with_reason returns the typed StoreError::NotFound. - Source audits: the two timeout queries only cast a guarded, clamped value, and reclaim_once contains no ? operator. - Correct the changelog: the SQL clamp is the fix, the bound stops new values, and stage isolation limits the next such bug. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Audit findings COR-1 and COR-3 (11 Nov), plus the
retriesoverflows.1. An unbounded
timeout_minutesstalled reclaim instance-widetimeout_minutesis au32; two SQL statements cast the stored value to::int. One pipeline with3000000000whose run stayedrunningmade the timeout query raise every tick, andreclaim_oncereturned on that error before lease reclaim and orphan finalisation — for every project, on every replica, with/readystill ok. Writer-reachable.numericand clamp to a year, so a value already stored cannot raise.timeout_minutes≤ 525600 (DagError::TimeoutTooLong).dag::stored_timeout_minutes, used by the offer and restated by the SQL: a whole number ≥ 1 clamped to a year, anything else absent. Before, the agent and the server backstop disagreed on out-of-range values.2.
retriesoverflowed in three readersBound at compile (≤ 100); the column write clamps;
requeue_outcomeand the backoff use saturating arithmetic. In-range backoff values are pinned against the old expression for attempts 0–63.3. The orphan sweep missed runs with pending dependents
A step's completion commits, then
propagate_after_stepruns as a separate transaction. If the process died or the pool timed out between the two, dependents stayedpending, nothing wasqueuedorrunning, and the run wasrunningforever. The sweep only selected runs with nopendingsteps, so it never saw them. It now selectsrunningruns with nothingqueuedorrunning, and the caller re-checks before warning.Upgrade note
A stored pipeline with
timeout_minutesabove 525600 orretriesabove 100 now fails to compile; the boot scan names each one, and a push or schedule against it leaves a failed run saying why. Runs already in flight keep their snapshot and are handled by the clamps.Verified
make checkgreen;cargo test --workspacegreen (467 tests). Each new test was broken, seen red, and restored — including source-audit tests that fail if the bare cast or a?inreclaim_oncecomes back.value "3000000000" is out of range for type integer.3000000000,5000000000and1e20read as 525600;"abc",1.5,0,-5,nulland absent read as absent;5.0reads as 5; absurd default/grace binds do not raise.Not verified / left out
reclaim_onceinvoking every stage is covered by the source audit, not by execution./metricsor/readyfor a loop that errors without dying. Left for a follow-up.needsnames a step that does not exist is re-selected by the sweep every tick and holds one of its 100 slots.🤖 Generated with Claude Code