Pin the executor drive's drain queue to .userInitiated QoS: a job resumed from a background thread could starve for minutes and wedge settle() under a saturated parallel run

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes are documented here. The format follows [Keep a Changelog](h
 
 ## [Unreleased]
 
+### Fixed
+
+- **`.modelTesting` tests no longer wedge or crawl under a saturated parallel run when a model task is resumed from a background-QoS thread.** The executor drive ran every test's jobs on one process-wide GCD concurrent queue that carried no QoS of its own. A queue without a QoS runs each block at the QoS of the thread that submitted it, and a drive job is submitted by whichever thread resumes the task — a `DispatchQueue.global(qos: .background)` callback, a `.background` Task, a low-QoS test double. Such a job was a background-QoS block, which macOS can leave unscheduled for minutes when a large test plan saturates the machine; while it was pending it counted as `outstanding`, so the executor reported itself busy, the inactivity watchdog never fired, `settle()` could never reach its fixpoint, and long enough the absolute ceiling fired with its "almost certainly a deadlock" wording — with no lock involved anywhere. Measured downstream in an 18-target plan as a smooth 4×–250× slowdown of `settle()`-heavy tests (1.4 s in isolation → 5–343 s) with a ~3 % chance of a 1500 s wedge per run, identical on 1.0.16 and 1.0.17. The drain queue now carries `qos: .userInitiated` as a floor: a job submitted from a lower-QoS thread is raised to it, one submitted from a higher-QoS thread keeps its QoS exactly as before, and once scheduled a job runs at its task's own priority whoever resumed it (measured with a probe that parks a `node.task` on a continuation resumed from a background thread and reads the resumed job's `qos_class_self()`: 9, `background`, before; the task's own priority after). Not `.enforceQoS`: forcing every job to one level changed the relative scheduling of drive jobs against work the drive cannot see — `Task.yield()` inside a task that prefers the drive resumes on the global pool, not on the drive — and on CI's 3-core runners that let a `settle()` fixpoint check outrun a yielding child it could not count. `DriveJobQoSTests` pins the floor as configuration; a behavioural version needs a background block to get scheduled inside the test budget, which is precisely what a saturated runner does not guarantee. This is the drive's stated "non-starvable" contract made true; no wait budget or timeout changed.
+
 ---
 
 ## [1.0.17] — Hot-path performance: index-keyed accessors, lock-free registrar lookups, 10× faster collection reconcile

Sources/SwiftModel/Internal/TestExecutorDrive.swift

@@ -133,7 +133,34 @@ func _makeTestExecutorBox() -> (any Sendable)? {
 /// runs. Each executor keeps its own `outstanding` counter, so per-test
 /// quiescence detection stays isolated.
 @available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *)
-private let _sharedDrainQueue = DispatchQueue(label: "swift-model.test-drain.shared", attributes: .concurrent)
+///
+/// The queue carries an explicit `.userInitiated` QoS as a FLOOR. A concurrent queue with
+/// no QoS of its own runs each block at the QoS of the thread that submitted it, and a
+/// drive job is submitted by whichever thread resumes the task — a
+/// `DispatchQueue.global(qos: .background)` callback, a `.background` Task, a low-QoS test
+/// double. Under a saturated parallel run a background-QoS block can stay unscheduled for
+/// minutes; while it is pending it counts as `outstanding` (the executor reports itself
+/// busy), so the test's `settle()` can never reach its fixpoint and the inactivity
+/// watchdog never fires — the exact evidence the absolute-ceiling report describes,
+/// without any lock involved. Measured downstream as a smooth 4×–250× slowdown of
+/// `settle()`-heavy tests under an 18-target plan with an occasional 1500 s wedge,
+/// identical on releases before and after the lock changes. The drive's contract is
+/// "non-starvable"; the floor is what makes that true.
+///
+/// A floor, not `.enforceQoS`: with the queue's QoS set, a block submitted from a
+/// lower-QoS thread is raised to `.userInitiated`, while one submitted from a higher-QoS
+/// thread (the main thread, a `.userInteractive` task) keeps that QoS exactly as before.
+/// Forcing every job to one level changed the relative scheduling of drive jobs against
+/// work the drive cannot see — `Task.yield()` inside a task that prefers this executor
+/// resumes on the global pool, not here — and on CI's 3-core runner that let a
+/// `settle()` fixpoint check outrun a yielding child it could not count
+/// (`ExecutorDrainSettleTests.settleIsLoadIndependentAcrossChildTasks`). The floor fixes
+/// the starvation without touching that balance.
+private let _sharedDrainQueue = DispatchQueue(label: "swift-model.test-drain.shared", qos: .userInitiated, attributes: .concurrent)
+
+/// The drain queue's QoS, for the regression test that pins the floor.
+@available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *)
+var _drainQueueQoS: DispatchQoS { _sharedDrainQueue.qos }
 
 @available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *)
 final class _DrainTestExecutor: TaskExecutor, @unchecked Sendable {
@@ -373,7 +400,26 @@ extension TestAccess {
                     let lastActivity = max(self._lastActivityNsLocked, exec.activityNs)
                     let sinceActivity = _drainMonotonicNs() &- lastActivity
                     if sinceActivity >= graceNs {
-                        return .reached   // idle, and no activity of any kind for a full grace window
+                        // Idle and quiet for a full grace window — but a task that has
+                        // YIELDED is invisible to `outstanding` while the runtime hops its
+                        // continuation through the global executor on the way back to this
+                        // executor's `enqueue`. On a starved 3-core CI runner that hop was
+                        // measured at 773 ms with the drive otherwise idle (children's
+                        // jobs ended at 0.7 ms, their yielded continuations ran at 774 ms,
+                        // settle fired at 807 ms — inside the second hop), so no grace
+                        // window bounds it. Yield from THIS task: it queues behind every
+                        // pending yielded child in the same global executor, so when
+                        // control returns those children have been re-enqueued
+                        // (outstanding > 0) or have run (activity stamped). Quiescence is
+                        // declared only if the system is still idle and quiet after that
+                        // round trip — an ordering signal, not a wall-clock one.
+                        await Task.yield()
+                        let stillIdle = exec.isExecutorIdle && bg.isIdle && main.isIdle && !self.context.hasPendingStartTask
+                        let activityAfter = max(self._lastActivityNsLocked, exec.activityNs)
+                        if stillIdle && activityAfter == lastActivity {
+                            return .reached
+                        }
+                        continue   // the yield surfaced pending work; go around again
                     }
                     // Idle but recent activity — wait out the remainder of the
                     // grace (non-starvable), then re-check; a resuming task will

Tests/SwiftModelTests/DriveJobQoSTests.swift

@@ -0,0 +1,31 @@
+import Testing
+import Foundation
+@testable import SwiftModel
+
+#if canImport(Dispatch)
+import Dispatch
+
+/// Pins the executor drive's QoS floor.
+///
+/// The drive runs every test's jobs on one process-wide GCD concurrent queue. A
+/// concurrent queue with no QoS of its own runs each block at the QoS of the thread
+/// that submitted it, and a drive job is submitted by whichever thread resumes the
+/// task — so a resumption from a background-QoS thread used to produce a background-QoS
+/// job that a saturated machine could leave unscheduled for minutes while it counted as
+/// executor activity: `settle()` never reached its fixpoint and the trait's absolute
+/// ceiling fired with no lock in sight. With the queue's QoS set to `.userInitiated`,
+/// such a job is raised to that floor (measured: the resumed job read
+/// `qos_class_self()` = background (9) before, the task's own priority after).
+///
+/// This test asserts the configuration rather than re-measuring the behaviour: a
+/// behavioural check needs a background-QoS block to get scheduled inside the test's
+/// budget, which is precisely what a saturated CI runner does not guarantee — the
+/// first cut of this test timed out under the TSan job for that reason.
+struct DriveJobQoSTests {
+    @Test func drainQueueCarriesAUserInitiatedFloor() {
+        guard #available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *) else { return }
+        #expect(_drainQueueQoS == .userInitiated,
+                "the drive's drain queue must carry a .userInitiated QoS floor; got \(_drainQueueQoS)")
+    }
+}
+#endif