[elyerinfox]

Cause of the develop failures

multi-GPU scheduling failed on develop in runs 37220358649 (#52's merge) and 37226241965 (#55's merge). Neither failure was the timing assertions, and neither was card contention:

• Timings passed both times with their usual margins. Across the six develop runs since speed-aware scheduling landed: ladder 0.78-0.79x (limit 0.95), 10 s single file 1.06-1.07x (limit 1.10), archive ladders 0.66-0.71x (limit 0.85), one job per card 0.96-1.00x (limit 1.15). Run-to-run spread per plan is 1-2%.
• No contention. The x2 pod requests gpu.intel.com/i915: 2 on devbox (allocatable: exactly 2). The Intel device plugin runs with its default -shared-dev-num 1. Every other GPU runner set (safewords-runners-gpu, transcdr-intel-gpu) requests i915, and no other pod on devbox mounts /dev/dri. All of this was checked on the live cluster. In practice, when run 37229429851's four per-card jobs were queued during 37228352673's x2 job, all four ran on vulpes-zerda's A310. The one time windows overlapped (37222955807's vp9 job, 18:25-18:27), that job was also on the A310.
• The actual failure is FAIL: reuse-h264.log: no chunk was encoded on card(s) ['0']: {'1': 4}. The reuse step runs a 70 s H.264 rung as four chunks. That encode is decode-bound: the A750 finishes each chunk before the next one is decoded. So the speed gate (Speed-aware multi-GPU scheduling: decode pulled by speed, tail gate, fastest card for single-device work #52) keeps the A380 out whenever the A750 is free first, which is correct. It happened in 2 of 6 runs, and those runs were faster: 7.7 s, against 8.1-8.7 s when the A380 took a chunk. The assertion was racing the scheduler. It was not a regression.

Fix

• Reuse step: it now asserts what the step exists to check (multi_gpu_check.py reuse). Every card that encoded built exactly one session and reused it for each further chunk. No reset failed, was unsupported or was evicted. At least one session was reused. The previous step only printed these counters. The spread of chunks across cards is still asserted by the ladder and archive steps, where it is stable.
• Timings: every timing assertion now compares within the run, using the best of 3 interleaved rounds, with the plan order rotated each round (speed_aware_timings.py). It now also times the archive ladders. The single-shot 0.95x and 0.85x checks in the first two steps become correctness-only (pinned / all cards busy / chunks / decode). For one job per card, each card's best of 3 together is compared with its best of 3 alone.
• Recording: every round's time, each plan's spread and the per-round paired ratios go to the step summary and to speed_aware_timings.json in the logs artifact.
• Limits: unchanged. The measured margins are far larger than the measured noise, and the ladder limit (0.95) still fails as soon as two cards stop beating one.
• No new concurrency group with the per-card jobs. The device plugin already makes the cards exclusive. A shared group would serialize every per-card job, and GitHub cancels all but one pending member.

[elyerinfox]

Second commit: the job-level concurrency group is removed. It kept only one pending scheduling job and cancelled the rest ("Canceling since a higher priority waiting request for safewords-runners-gpu-x2 exists"). That is how develop run 37240219796 (#63) and my dispatch 37240853718 lost their scheduling jobs. The one-runner x2 set plus the device plugin already serialize the job: without the group, four dispatches queued and ran one after another, and none was cancelled.

Verification: workflow_dispatch runs on this branch. All passed. Values are best-of-3 ratios, with the per-round paired ratios in brackets.

run	ladder (≤0.95)	single 10 s (≤1.10)	archive AV1 (≤0.85)	archive H.265 (≤0.85)	job per card (≤1.15)	reuse H.264 chunks
37238570868 (1st commit)	0.784 [0.784 0.602 0.685]	1.049	0.765 [0.805 0.895 0.765]	0.650	0.99 / 0.98	{1: 4} ok
37243040360	0.788	1.063	0.708	0.683	0.99 / 0.96	{1: 4} ok
37243034316	0.786	1.065	0.705	0.632	1.00 / 1.00	{1: 3, 0: 1}
37243125722	0.786	1.048	0.708	0.687	0.99 / 0.98	{1: 4} ok
37243046364	0.785	1.060	0.706	0.684	1.00 / 0.98	{1: 3, 0: 1}

• 37238570868 ran under CPU load. Three safewords-runners test pods were scheduled onto devbox (it is also a CI node) at 22:40, 22:46 and 22:47, during its timing step (22:46-22:51). Per-plan spread reached 46% (A750 ladder alone: 6.84 / 10.01 / 7.95 s). In round 2 the archive-AV1 paired ratio was 0.895, so a single-shot check would have failed. The best of 3 gave 0.765.
• The four clean runs have per-plan spreads mostly under 2%.
• H.264 reuse: three of the five runs hit the {1: 4} split that failed develop twice. The new reuse check passes it: one session built on the A750, three reused.