3.0.x: L3 l3-full fails on v3.0.0: 2fe-async-3-blocks refuses one valid share per run (#716), and 2fe-sync-r3 falls 479 offers short #779

The L3 production-window matrix (l3-full) fails on 3.0.0's tree in both runs. All the failures are in the two 400k two-frontend presets. Every other preset passed.

Collated verdicts (prism-l3-verdict-*): the tag run passed 32 of 33 planned runs, and #778's run passed 29 of 33. Every run reached its gate. Collate the suite fails in both runs only because a planned run failed its gate. It is the suite's summary job, not a separate test.

For comparison, these full runs passed every job: rc.6's bump PR #756 (37768281661) and 3.1.0-rc.1's #774 (39336365630). 3.1.0-rc.1 already contains everything 3.0.0 adds to rc.6 (#760, #770, and rustls 0.23.46).

1. throughput-400k-window-2fe-async-3-blocks: one valid share refused in steady state

The gate fails on steady_state: valid shares refused | 1 | <= 0 in four of four repeats: r1–r3 on #778 and r3 on the tag. Everything else in the gate passes: no lost or unexplained acknowledged shares, reconciliation exact, no steady-state shortfall, ACK p99 about 3 ms.

In each repeat, the frontends' logs show the refusal as #716's commit-gate class:

share commit gate refused before COMMIT error=share commit gate closed before COMMIT closure="authority-unavailable"

This is the race #716 describes: a share admitted under a tip-change lease meets a writer on an authority RwLock, and try_read fails. This preset mines three blocks in steady state, so it changes tips under load. #716 recorded the same failure on the same preset at rc.2 (job 112207024475). The 3.0.0 notes list #716's open part as a known issue. So this is not a regression in 3.0.0. It means the matrix can't pass reliably on this preset until #716 is fixed.

2. throughput-400k-window-2fe-sync-r3: 479-offer steady-state shortfall

Job 119991362518 fails on steady_state: shortfall (offers no session could take) | 479 | <= 0. No valid share was refused. The steady-state line:

phase steady_state 300.0s target=500 offered=149520 acked=149520 rate=498.4/s ack p50=1.66 ms p99=3883 ms order_waiters_max=30 settlement_waiters_max=3 shortfall=479

An ACK p99 of 3.9 s points to a stall of a few seconds. Over the whole run, the frontends logged 579 slow SELECT pg_advisory_xact_lock($1) statements and one 2.65 s pool acquire. The 7,725 share commit deadline passed before COMMIT was sent warnings belong to the ungated slow_database phase. r1 and r2 of the same preset passed. The runner was blacksmith-32vcpu-ubuntu-2404. At that time, the tag's l3-full and #778's l3-full were running at once. No issue tracks this shortfall yet.

To do:

• Re-run 2fe-sync-r3 alone, and also on 3.1.0-rc.1, to tell runner noise apart from a stall in the code.
• If it recurs, find which lock or connection wait the stalled seconds spent their time on.

What this means for 3.0.0

• v3.0.0's own L3 tag run is red, so the release has no green L3 evidence. Its only gate failure is 3.x.x: a share under a tip-change lease is refused 'commit gate closed' when the authority try_read meets a writer, which can drop a valid share's credit and fires PrismShareAppendFailures #716's single refusal.
• v3.0.0 chore: consolidate release for main #778's run is red on the same jobs. The matrix can't go green on 3.0.0 or 3.0.x until 3.x.x: a share under a tip-change lease is refused 'commit gate closed' when the authority try_read meets a writer, which can drop a valid share's credit and fires PrismShareAppendFailures #716 is fixed, or until this preset tolerates 3.x.x: a share under a tip-change lease is refused 'commit gate closed' when the authority try_read meets a writer, which can drop a valid share's credit and fires PrismShareAppendFailures #716's class.

Evidence artifacts: prism-l3-run-throughput-400k-window-2fe-* on both runs.

Related: #716, #291, #776, #778.