Stop the app tests asserting the host's unified memory - #1
Merged
Conversation
`AppModel.init` defaults `deviceCapabilities` to `.current()`, which reads `hw.memsize`. That flows into the eligibility gate behind `canLoadModel`, `canReloadModel`, `canInstallModel`, and `canInstallVisionPack`, so 143 test call sites were asserting the RAM of whatever machine ran the suite. The Apple-silicon CI runners have 7 GB, under the 8 GB catalog floor for the 26B. Every install, load, and companion action was refused before the state machine under test ever ran, failing eight suites for a reason unrelated to the code under test. `ConversationHistoryTests` then indexed the conversation the load had never filled and trapped, killing the process and hiding the three CI steps after it. None of these tests need the hardware: the installer and inference clients are mocks and `makeCompleteModelInstall` writes zero-byte weights. Route them through `makeAppModel`, which mirrors the initializer and pins a qualifying device. Hardware gating keeps its coverage in `ModelCatalogTests`, where each test states the device it means, and `#require` replaces the two index accesses so one failure no longer takes the run down with it. Verified with `current()` patched to the runner's 7 GB: 1376 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why CI is red
The head commit only touched README and site assets. The app test suites were gating on the RAM of whatever machine ran them.
AppModel.initdefaultsdeviceCapabilitiesto.current(), which readshw.memsize. That feeds the eligibility gate behindcanLoadModel,canReloadModel,canInstallModel, andcanInstallVisionPack. GitHub'smacos-26-arm64runners have 7 GB — under the 8 GB catalog floor for the 26B — so every install, load, and companion action was refused before the state machine under test ever ran.ConversationHistoryTeststhen indexed a conversation the load had never filled, trapped withFatal error: Index out of range, and killed the test process — which is why the symlink, Markdown-link, and app-version steps never ran. The 2026-08-27 run failed the same way at a different point.The change
None of these tests need the hardware. The installer and inference clients are mocks, and
makeCompleteModelInstallwrites zero-byte weight files — nothing downloads, loads, or allocates a working set.Tests/TUFFApp/Core/Support/AppModelTestHost.swift: amakeAppModel(...)factory mirroringAppModel.initargument for argument, pinning a qualifying device. All 143AppModel(sites inTests/TUFFApp/Coreroute through it. This follows the pattern already inAppWorkspaceSmokeTests.#require(...first)replaces the two trapping index accesses, so one failure reports instead of taking the run down.No production code changes — deliberately no env-var or runtime override, per the CONTRIBUTING guardrail against undocumented runtime switches.
Device eligibility keeps its real coverage in
ModelCatalogTests, which injects 8 GiB and expects refusal (including the exact "Requires 16 GB unified memory; this Mac has 8 GB." text) and injects 16 GiB and expects the opposite. Those now state their device deliberately rather than inheriting it from 143 sites.Verification
current()temporarily patched to the runner's exact 7 GB: 1376 tests / 231 suites pass (patch reverted;Sources/is untouched in this PR)check_tracked_symlinks.rb,check_markdown_links.rb,check_app_version.rball passswift build -c release— it already succeeded on CI for this commit and noSources/file changedKnown limit
The pinned 32 GiB fixture won't notice a future catalog floor raised above it. A 7 GB runner wouldn't catch that either — it refuses everything regardless.
RecommendedMemoryTestsandModelCatalogTestsare where that belongs. Nothing yet stops a new test from callingAppModel(directly and reintroducing host dependence; aScripts/check_*.rbguard would fit the repo's idiom if wanted.🤖 Generated with Claude Code