chaos: scenario 09's post-failover produce can miss quorum under heavy I/O load (Overloaded, 0 follower acks)

[allamiro]

Found by the #318 closing soak — filed separately for the same reason #318 was split from #310: it is a different failure in the same scenario, and conflating them would let one fix be credited with the other's disappearance.

What is seen

One run in thirty, under deliberately extreme I/O load (a continuous cargo build/cargo clean loop churning the same disk as the suite):

range leader killed (SIGKILL) mid-flight; acknowledged floor: 500 of 3000
follower offsets: f1=600 f2=500; promoting follower 1
follower 1 took the range at epoch 2
follower 2 restarted at epoch 2 to rejoin replication
FAIL: post-failover produce never reached quorum within 30s: error: produce rejected: Overloaded quorum not reached: 0 follower ack(s), need majority of 2

Why it is NOT #318

#318's signature is a promotion that never acquires the range (holder: none for the full wait). Here the promotion succeeded — the range was held at epoch 2 and the rejoining follower restarted cleanly. What failed is the post-failover produce reaching quorum: the promoted leader reported Overloaded with zero follower acks for 30 seconds.

What is not yet known

Whether this is the 30s bound being too tight for a follower whose disk is saturated by an adversarial background load (the broker's backpressure doing exactly its job, slowly), or a rejoining follower that stalls in a way fresh traffic cannot unstick. The idle half of the same soak was 30/30, and the load half was 29/30, so load is clearly a precondition. Worth establishing first: does the follower ack shortly AFTER the deadline (bound too tight), or never (rejoin stall)?

Soak log preserved from the failing run; the Overloaded error code is the broker's own admission-control refusal, which may simply be honest under a load no deployment should run at.

Refs #318.

[allamiro]

Discriminating soak — 40 load runs at a 120s bound

The issue's first question was whether the follower acks shortly AFTER the 30s deadline (bound too tight) or never (rejoin stall). Ran the soak built for it: 40 consecutive scenario 09 runs under the same continuous rebuild load, with CHAOS_PROGRESS_TIMEOUT_SECONDS=120 so any recurrence would answer the fork.

40/40 PASS, ~20s each. No recurrence at all — the failure did not reproduce even at the original 1-in-30 rate, so the fork remains unanswered and the original failure evidently needs heavier contention than this machine produced today (the original run's load loop also had cargo clean churn on larger crates, which the corrected soak deliberately avoids).

Suggest the same posture #318 held: keep open as a watch item. The failure message already carries the broker's own Overloaded admission-control verdict and the ack count, so a CI or soak recurrence self-describes. If a future soak wants to chase it deliberately, the reproduction likely needs an I/O-heavier load generator than a build loop — e.g. fio against the same disk — plus the 120s bound to answer late-vs-never.