Repository navigation
fix: skip macOS AppleDouble files in the model library scan - #970
Merged
Merged
Conversation
Checkpoint folders on non-HFS+ volumes (exFAT/FAT external drives)
carry an AppleDouble companion ('._model-00001-of-00043.safetensors')
next to every real file, holding that file's resource fork. The scan's
directory listing includes them, so inspect() counted them as weight
shards, read their 4 KiB fork payload as a safetensors header, and
failed the checkpoint with 'The safetensors header is invalid or
incomplete.' The native engine ignores these files, and they are never
checkpoint assets, so filter them from the inspected name list.
A stray '._' companion in a selected parent matched the checkpoint detection test, so walk() stopped at the parent and never reached the real checkpoint beneath it; filter companions from detection as well and drop the undocumented '._.' exception from both filters.
Open
Owner
|
Thank you, @heitke, for applying the AppleDouble filter at discovery and inspection, with companion and nested-folder tests. We merged your three commits for 1.1.0 in d9a5d47, beside #967. A Real shards still determine completeness and weight totals; corrupt or missing real shards still fail. Your tests cover the regression follow-up we had offered. Posted by Ash's AI Assistant Synthena |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
macOS writes AppleDouble companion files (
._<name>, resource forks) next to real files on volumes that cannot store resource forks natively — the common case for large model libraries on exFAT/FAT external drives. The scan's directory listing includes them, and they are never checkpoint assets, so this change filters names starting with._in two places:inspect()— companions no longer count as weight shards, so a sharded checkpoint's shard accounting,weightsBytesandmemoryFloorBytesreflect only real assets, and a companion's fork bytes are never read as a safetensors header.walk()— checkpoint detection uses the same filtered list, so a stray._<anything>.safetensorsin a selected parent directory cannot makewalk()treat that parent as a checkpoint, stop descending, and miss the valid checkpoint beneath it.The filter keeps no exceptions: nothing in the repo or in Apple's AppleDouble naming documents a
._.file, and a model folder is never a filesystem root. Directory-entry budget accounting is unchanged —list()counts everyreaddirentry, including ignored companions. Safety checks for real checkpoint assets and Hugging Face snapshot links are untouched.A large checkpoint on an exFAT volume hits two independent causes — the AppleDouble companions and the over-cap shard index (#967) — and needs both changes; neither PR alone fixes that case.
Receipt
1.0.0-beta.6, native engine1.0.12(build.zig.zon), commite223dd3a(this PR, based on46a08c18); macOS 26.6.2, Apple M3 Ultra (Apple Silicon, Metal), Node 22.14.0, npm 10.9.8. Checkpoint: a 43-shard checkpoint on an exFAT drive, verified manually by the author of the original change on the earlier commit; its repository and revision are unrecorded.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.tools/bench_openai.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.tools/prefill_cold.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.app/with the repo's pinned Node 22.14.0 and npm 10.9.8:assertcalls respectively, 0 failures, 0 skips),npm run typecheckclean,git diff --checkclean, and the repo's private-leak and test-coverage gates pass; the fullnpm testsuite was not run locally end to end — in this environment two unrelated suites (session-search-index,mobile-security-dependencies) fail for missing local modules, the same failures as on the unmodified base commit. GitHub CI: 548/548 test files passed (thedesktopcheck).cpmaterializes AppleDouble files depends on the filesystem and copy flags); real-model end-to-end load through the engine.Regression evidence (synthetic, deterministic, through the public
ModelLibraryAPI):complete/loadable, the sidecars are excluded fromweightsBytesandmemoryFloorBytes, andresolve()succeeds. The same new tests fail against the pre-change implementation (the sidecars are counted as shards and the checkpoint is rejected).._old.safetensorsin the selected parent does not stopwalk(): the nested valid checkpoint is discovered andloadable, and the parent itself is not recorded as a model. Before this change the parent is mistaken for a checkpoint and the nested one is never discovered (demonstrated against the base commit).