Scheduler: a single slow run stalls every other schedule and the delivery pump

[SouravRoy-ETL]

What happens

duckle serve runs its scheduler on one thread, and a scheduled run is executed inline on it. Any run that takes a while therefore stops every other schedule, and the subscription delivery pump, until it finishes.

Measured on a real server, tick interval 2s, both schedules on a 5-second interval:

fast schedule alone, 60s window                  -> fired 9 times
same window, one slow schedule beside it         -> fired 2 times
                                                    (the slow one ran once)

One run cost a five-second schedule 78% of its occurrences in the same minute. The slow pipeline here is a single ~25s DuckDB aggregate; a real extract that takes an hour stops everything for an hour.

Why

Three things combine, all in crates/duckle-runner/src/serve.rs unless noted.

One thread, one loop. spawn_scheduler spawns a single thread (:4264) whose body is loop { sleep(tick_interval); ... } (:4279-4280).

Runs are inline. fire_schedule(&state, id, cfg, &pipes) is a plain synchronous call in that loop - the cron path at :4392 and the interval path at :4439. It reaches run_scheduled (:3583), which calls execute_one (:3591) and waits for the pipeline to finish.

Admission blocks. Gates::acquire (crates/duckdb-engine/src/pools.rs:227) delegates to Gate::acquire (pools.rs:172), an untimed condvar wait:

    pub fn acquire(&self) -> Permit<'_> {
        let mut free = self.free.lock().unwrap_or_else(|p| p.into_inner());
        while *free == 0 {
            free = self.ready.wait(free).unwrap_or_else(|p| p.into_inner());
        }

There is no try-variant and no timeout, so a saturated pool holds the scheduler thread as effectively as a long run does.

What else stops

pump_deliveries(&state) is called at the top of the same loop (:4286). So a slow scheduled run also delays every materialization-triggered run (#325) - and the comment right there says why that matters: "the point of a data trigger is that the consumer runs when the data lands, not up to a minute later."

The hazard is already recognised, one line away

Freshness evaluation in the same loop was deliberately moved off the thread (:4288-4295):

                // Off the scheduler's thread, so a slow evaluation delays no
                // schedule, and guarded so two can never overlap on a workspace
                // where it takes longer than the interval.
                std::thread::spawn(move || { ... sla::check_and_alert(...) });

The same argument applies with more force to the runs themselves, which are unboundedly slower than an SLA sweep.

Why this is not #296

#296 is about policy - what should happen when occurrences overlap or are missed. This is liveness: a schedule that has no overlap at all, and is not overdue, simply does not fire on time because an unrelated pipeline is running. It is present regardless of which overlap or misfire policy is chosen.

It does interact with #296 in one direction worth noting: overlapPolicy: queue cannot be built on the current shape, because "queued" today means a blocked thread. retry::enqueue (crates/duckdb-engine/src/retry.rs:385) only stamps the run's own receipt - there is no durable queue, nothing reads a queued receipt back, and reconcile (retry.rs:417-427) deliberately turns a queued receipt into interrupted on restart.

Direction

Not prescribing one, but the shape that seems to follow: the tick thread should decide and hand off, never execute. That is the same split #296 settled on for occurrences - the scheduler decides what should run, the durable run path decides how - and it is what a durable queue would need anyway.

A bounded worker pool reading a durable queue would fix this and unblock overlapPolicy: queue and replace at the same time, since both need runs that are startable by something other than the thread that decided them.

Scope

Single server, as #296 is. No leader election, no distributed queue.

Acceptance criteria

1. a long-running scheduled pipeline does not delay an unrelated schedule;
2. it does not delay materialization deliveries;
3. a saturated resource pool does not block the scheduler thread;
4. the measurement above (fast schedule alone vs. beside a slow one) shows no material difference.

[SouravRoy-ETL]

Fixed in c04cbe8. The tick loop now hands each due schedule to its own thread instead of running it.

The measurement, with the cause isolated

I have to correct the evidence in the issue above. My original numbers were taken with the default run pool, which is one concurrent run - so they conflated the tick-loop stall with ordinary pool serialisation. The stall is real, but that measurement did not prove it on its own.

Re-run with the pool raised to 4, so pool capacity cannot be the cause:

fast schedule alone                    9 firings in 60s
inline (before)                        2
dispatched (after)                     9

Same as running alone, while the slow pipeline is going. The number in the original report was right; the isolation was not, and it would have been fair to say the pool was doing some of that work.

What changed

dispatch puts the firing on its own thread and returns. Catch-up dispatches too - a catch-up is N runs rather than one, so it is the worst case of the stall rather than an exception to it.

A thread per firing rather than a pool, because everything that bounds concurrency already exists and would be duplicated: the resource pool bounds what RUNS, the on-disk run lock stops one pipeline running twice, and a new in_flight set stops the tick handing the same schedule out again while it is going. What is left to bound is the number of schedules an operator wrote.

This is not a durable queue. overlapPolicy: queue still needs one (#296); this only makes the scheduler responsive, which is true whichever overlap policy a schedule chooses.

AC1, AC3 and AC4 are covered. 1917 tests pass.

AC2 is half done, and the other half is a real decision rather than an oversight

"A long-running scheduled pipeline does not delay materialization deliveries" - yes, that is what dispatching the schedules fixes.

The reverse is still true: a slow DELIVERY still stalls the scheduler, because pump_deliveries runs on the same tick thread and calls execute_one synchronously.

I did not fix it the same way, because the same change there is not safe. A delivery is recorded only AFTER its run (serve.rs:3788). pending() derives what is owed from (subscription x event) minus what is recorded, so a delivery handed to a thread and left unrecorded would be re-derived on the very next tick and run again - the duplicate consumer run the delivery id exists to prevent.

Recording it before spawning fixes that and costs something else: today a process that dies mid-delivery leaves the delivery unrecorded, so it is re-derived and retried. Record-then-spawn turns that into a delivery stuck pending forever, which nothing currently reclaims - deliveries retry only touches failed.

So the honest options are a claim on a delivery with an owner and a reconcile (the shape retry::reconcile already uses for runs), or leaving deliveries inline. I would rather ask than pick: is a slow consumer stalling the scheduler worth a claim/reconcile cycle on deliveries, or is it rare enough to leave until there is a durable queue that would subsume both?

[LouisDeconinck]

I would implement the delivery claim/reconcile cycle rather than leave deliveries inline until a future general durable queue.

Now that #325 is a first-class orchestration primitive, a materialization-triggered consumer can be just as long-running as a scheduled pipeline:

registry publish
    ↓
entity resolution       40 min

accounts publish
    ↓
normalization           2 h

website discovery
    ↓
enrichment              20 min

If one of those runs inline on the scheduler thread, then #329 is still operationally significant even though scheduled runs themselves have been dispatched.

The invariant I would aim for is:

scheduler tick
    -> decide / claim work
    -> return quickly

worker
    -> perform the run

for both clock occurrences and materialization deliveries.

For a delivery, I think the smallest safe lifecycle is:

pending / derivable
    ↓ atomic claim
claimed
    owner = process/worker identity
    claimed_at
    ↓
consumer run
    ↓
delivered | failed

On startup/reconciliation:

claimed delivery
+ owner no longer alive / lease expired
    -> pending again

or equivalently:

claimed -> interrupted -> eligible for retry

The exact intermediate state name is not important.

What matters is that the claim is written before handing execution to another thread, so the next scheduler tick cannot derive and launch the same (subscription, event) again.

I would reuse the existing deterministic delivery_id as the claim identity rather than introduce another key.

This also fits the existing recovery philosophy:

runs        -> claim / reconcile
slices      -> claim / reconcile
deliveries  -> claim / reconcile

A future general durable queue could certainly absorb this later, but I would not make correctness/liveness of today's materialization triggers depend on that refactor.

For me, #329 can close once neither direction holds:

slow scheduled run  -X-> scheduler
slow triggered run  -X-> scheduler

and a crash between delivery claim and completion safely makes that delivery eligible again without launching it twice concurrently.