fix(pool): prevent num_idle underflow that wedges the maintenance task

[yuhao-su]

Cherry-pick of transact-rs#4289 (unmerged upstream) onto madsim-v0.8.2, fixing the num_idle underflow that RisingWave root-caused as the source of its CI silent meta hang (risingwavelabs/risingwave#24991, full RCA: risingwavelabs/risingwave#24991 (comment)). Same bug reported upstream as transact-rs#3645 ("100% CPU in spawn_maintenance_tasks").

The bug

PoolInner::release() published a returned connection before updating the idle counter:

1. idle_conns.push(conn)
2. guard.release_permit() — this wakes a queued waiter, which can immediately pop the connection on another core
3. num_idle.fetch_add(1)

A pop_idle racing between 2 and 3 runs num_idle.fetch_sub(1) while the counter is still 0 → wraps to usize::MAX. The counter self-heals microseconds later, but the maintenance task's sweep uses it as a loop bound:

for _ in 0..pool.num_idle() {   // frozen at ~1.8e19 if it sampled the underflow
    if let Some(conn) = pool.try_acquire() { ... }   // no await on the failure path
}

The sweep then spins inside a single non-yielding poll (~forever; try_acquire consumes no coop budget so tokio cannot preempt it), pegging a worker. On a single-connection pool (RisingWave's SQLite meta store) one of the spin's add_permits handoffs assigns the only permit to a queued waiter whose wakeup lands in the spinning worker's LIFO slot — unstealable on tokio 1.49 (tokio-rs/tokio#4941) — permanently wedging the pool.

Observed in production captures: thread dumps catch the reaper mid-try_acquire minutes into total quiescence; an isolation repro observes the usize::MAX read within 11 ms of hammering.

The fix (three layers, each independently sufficient)

1. release(): increment num_idle before publishing the connection. Invariant: every successful pop is preceded by its push, which is preceded by its increment, so subs ≤ pops ≤ pushes ≤ adds at all times ⇒ the counter can provably never go below 0 (and never exceeds max_connections). The transient over-count (incremented, not yet pushed) is harmless: a racing pop_idle just finds an empty queue and returns the permit.
2. pop_idle(): saturating decrement — a future ordering regression can then only over-count, never wrap.
3. Maintenance sweep: cap iterations at max_connections, so even a corrupt counter cannot spin the non-yielding loop unboundedly.

Test

Port of the upstream stress test to the v0.8.2 Any API: 12 OS threads × 200k real try_acquire/release cycles on a max_connections = 3 pool, sampling num_idle() after every op and asserting the high-water mark never exceeds max_connections (the bound is exact: legitimate transient over-count is still ≤ total connections, so the assert cannot flake).

• Before the fix (order reverted, plain fetch_sub): fails in ~0.2 s with saw 18446744073709551615 — the exact production signature.
• With the fix: passes; full sqlx-core suite green (cargo test -p sqlx-core --features any,_rt-tokio, 30 passed).

Deviations from the upstream test: idle_timeout(None)/max_lifetime(None)/min_connections(0) so pool creation spawns no maintenance task and the test needs no async runtime; cfg-gated on a runtime feature because v0.8.2's AsyncSemaphore::new panics without one.

RisingWave will bump its pinned rev to pick this up.

[hzxa21]

Rubber stamp