diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d91f85..926af4d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ All notable changes are documented here. The format follows [Keep a Changelog](h ## [Unreleased] +### Fixed + +- **`.modelTesting` tests no longer wedge or crawl under a saturated parallel run when a model task is resumed from a background-QoS thread.** The executor drive ran every test's jobs on one process-wide GCD concurrent queue that carried no QoS of its own. A queue without a QoS runs each block at the QoS of the thread that submitted it, and a drive job is submitted by whichever thread resumes the task — a `DispatchQueue.global(qos: .background)` callback, a `.background` Task, a low-QoS test double. Such a job was a background-QoS block, which macOS can leave unscheduled for minutes when a large test plan saturates the machine; while it was pending it counted as `outstanding`, so the executor reported itself busy, the inactivity watchdog never fired, `settle()` could never reach its fixpoint, and long enough the absolute ceiling fired with its "almost certainly a deadlock" wording — with no lock involved anywhere. Measured downstream in an 18-target plan as a smooth 4×–250× slowdown of `settle()`-heavy tests (1.4 s in isolation → 5–343 s) with a ~3 % chance of a 1500 s wedge per run, identical on 1.0.16 and 1.0.17. The drain queue now carries `qos: .userInitiated` as a floor: a job submitted from a lower-QoS thread is raised to it, one submitted from a higher-QoS thread keeps its QoS exactly as before, and once scheduled a job runs at its task's own priority whoever resumed it (measured with a probe that parks a `node.task` on a continuation resumed from a background thread and reads the resumed job's `qos_class_self()`: 9, `background`, before; the task's own priority after). Not `.enforceQoS`: forcing every job to one level changed the relative scheduling of drive jobs against work the drive cannot see — `Task.yield()` inside a task that prefers the drive resumes on the global pool, not on the drive — and on CI's 3-core runners that let a `settle()` fixpoint check outrun a yielding child it could not count. `DriveJobQoSTests` pins the floor as configuration; a behavioural version needs a background block to get scheduled inside the test budget, which is precisely what a saturated runner does not guarantee. This is the drive's stated "non-starvable" contract made true; no wait budget or timeout changed. + --- ## [1.0.17] — Hot-path performance: index-keyed accessors, lock-free registrar lookups, 10× faster collection reconcile diff --git a/Sources/SwiftModel/Internal/TestExecutorDrive.swift b/Sources/SwiftModel/Internal/TestExecutorDrive.swift index f576659..801341d 100644 --- a/Sources/SwiftModel/Internal/TestExecutorDrive.swift +++ b/Sources/SwiftModel/Internal/TestExecutorDrive.swift @@ -133,7 +133,34 @@ func _makeTestExecutorBox() -> (any Sendable)? { /// runs. Each executor keeps its own `outstanding` counter, so per-test /// quiescence detection stays isolated. @available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *) -private let _sharedDrainQueue = DispatchQueue(label: "swift-model.test-drain.shared", attributes: .concurrent) +/// +/// The queue carries an explicit `.userInitiated` QoS as a FLOOR. A concurrent queue with +/// no QoS of its own runs each block at the QoS of the thread that submitted it, and a +/// drive job is submitted by whichever thread resumes the task — a +/// `DispatchQueue.global(qos: .background)` callback, a `.background` Task, a low-QoS test +/// double. Under a saturated parallel run a background-QoS block can stay unscheduled for +/// minutes; while it is pending it counts as `outstanding` (the executor reports itself +/// busy), so the test's `settle()` can never reach its fixpoint and the inactivity +/// watchdog never fires — the exact evidence the absolute-ceiling report describes, +/// without any lock involved. Measured downstream as a smooth 4×–250× slowdown of +/// `settle()`-heavy tests under an 18-target plan with an occasional 1500 s wedge, +/// identical on releases before and after the lock changes. The drive's contract is +/// "non-starvable"; the floor is what makes that true. +/// +/// A floor, not `.enforceQoS`: with the queue's QoS set, a block submitted from a +/// lower-QoS thread is raised to `.userInitiated`, while one submitted from a higher-QoS +/// thread (the main thread, a `.userInteractive` task) keeps that QoS exactly as before. +/// Forcing every job to one level changed the relative scheduling of drive jobs against +/// work the drive cannot see — `Task.yield()` inside a task that prefers this executor +/// resumes on the global pool, not here — and on CI's 3-core runner that let a +/// `settle()` fixpoint check outrun a yielding child it could not count +/// (`ExecutorDrainSettleTests.settleIsLoadIndependentAcrossChildTasks`). The floor fixes +/// the starvation without touching that balance. +private let _sharedDrainQueue = DispatchQueue(label: "swift-model.test-drain.shared", qos: .userInitiated, attributes: .concurrent) + +/// The drain queue's QoS, for the regression test that pins the floor. +@available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *) +var _drainQueueQoS: DispatchQoS { _sharedDrainQueue.qos } @available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *) final class _DrainTestExecutor: TaskExecutor, @unchecked Sendable { @@ -373,7 +400,26 @@ extension TestAccess { let lastActivity = max(self._lastActivityNsLocked, exec.activityNs) let sinceActivity = _drainMonotonicNs() &- lastActivity if sinceActivity >= graceNs { - return .reached // idle, and no activity of any kind for a full grace window + // Idle and quiet for a full grace window — but a task that has + // YIELDED is invisible to `outstanding` while the runtime hops its + // continuation through the global executor on the way back to this + // executor's `enqueue`. On a starved 3-core CI runner that hop was + // measured at 773 ms with the drive otherwise idle (children's + // jobs ended at 0.7 ms, their yielded continuations ran at 774 ms, + // settle fired at 807 ms — inside the second hop), so no grace + // window bounds it. Yield from THIS task: it queues behind every + // pending yielded child in the same global executor, so when + // control returns those children have been re-enqueued + // (outstanding > 0) or have run (activity stamped). Quiescence is + // declared only if the system is still idle and quiet after that + // round trip — an ordering signal, not a wall-clock one. + await Task.yield() + let stillIdle = exec.isExecutorIdle && bg.isIdle && main.isIdle && !self.context.hasPendingStartTask + let activityAfter = max(self._lastActivityNsLocked, exec.activityNs) + if stillIdle && activityAfter == lastActivity { + return .reached + } + continue // the yield surfaced pending work; go around again } // Idle but recent activity — wait out the remainder of the // grace (non-starvable), then re-check; a resuming task will diff --git a/Tests/SwiftModelTests/DriveJobQoSTests.swift b/Tests/SwiftModelTests/DriveJobQoSTests.swift new file mode 100644 index 0000000..bc96329 --- /dev/null +++ b/Tests/SwiftModelTests/DriveJobQoSTests.swift @@ -0,0 +1,31 @@ +import Testing +import Foundation +@testable import SwiftModel + +#if canImport(Dispatch) +import Dispatch + +/// Pins the executor drive's QoS floor. +/// +/// The drive runs every test's jobs on one process-wide GCD concurrent queue. A +/// concurrent queue with no QoS of its own runs each block at the QoS of the thread +/// that submitted it, and a drive job is submitted by whichever thread resumes the +/// task — so a resumption from a background-QoS thread used to produce a background-QoS +/// job that a saturated machine could leave unscheduled for minutes while it counted as +/// executor activity: `settle()` never reached its fixpoint and the trait's absolute +/// ceiling fired with no lock in sight. With the queue's QoS set to `.userInitiated`, +/// such a job is raised to that floor (measured: the resumed job read +/// `qos_class_self()` = background (9) before, the task's own priority after). +/// +/// This test asserts the configuration rather than re-measuring the behaviour: a +/// behavioural check needs a background-QoS block to get scheduled inside the test's +/// budget, which is precisely what a saturated CI runner does not guarantee — the +/// first cut of this test timed out under the TSan job for that reason. +struct DriveJobQoSTests { + @Test func drainQueueCarriesAUserInitiatedFloor() { + guard #available(macOS 15.0, iOS 18.0, tvOS 18.0, watchOS 11.0, *) else { return } + #expect(_drainQueueQoS == .userInitiated, + "the drive's drain queue must carry a .userInitiated QoS floor; got \(_drainQueueQoS)") + } +} +#endif