Skip to content

fix: skip macOS AppleDouble files in the model library scan - #970

Merged
ashhart merged 3 commits into
ashhart:mainfrom
heitke:fix/library-skip-appledouble-files
Oct 18, 2026
Merged

ashhart merged 3 commits into
ashhart:mainfrom
heitke:fix/library-skip-appledouble-files

Conversation

@heitke

@heitke heitke commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

macOS writes AppleDouble companion files (._<name>, resource forks) next to real files on volumes that cannot store resource forks natively — the common case for large model libraries on exFAT/FAT external drives. The scan's directory listing includes them, and they are never checkpoint assets, so this change filters names starting with ._ in two places:

  • inspect() — companions no longer count as weight shards, so a sharded checkpoint's shard accounting, weightsBytes and memoryFloorBytes reflect only real assets, and a companion's fork bytes are never read as a safetensors header.
  • walk() — checkpoint detection uses the same filtered list, so a stray ._<anything>.safetensors in a selected parent directory cannot make walk() treat that parent as a checkpoint, stop descending, and miss the valid checkpoint beneath it.

The filter keeps no exceptions: nothing in the repo or in Apple's AppleDouble naming documents a ._. file, and a model folder is never a filesystem root. Directory-entry budget accounting is unchanged — list() counts every readdir entry, including ignored companions. Safety checks for real checkpoint assets and Hugging Face snapshot links are untouched.

A large checkpoint on an exFAT volume hits two independent causes — the AppleDouble companions and the over-cap shard index (#967) — and needs both changes; neither PR alone fixes that case.

Receipt

  • Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold app 1.0.0-beta.6, native engine 1.0.12 (build.zig.zon), commit e223dd3a (this PR, based on 46a08c18); macOS 26.6.2, Apple M3 Ultra (Apple Silicon, Metal), Node 22.14.0, npm 10.9.8. Checkpoint: a 43-shard checkpoint on an exFAT drive, verified manually by the author of the original change on the earlier commit; its repository and revision are unrecorded.
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
  • Decode speed before and after, from tools/bench_openai.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
  • Prompt speed before and after, from tools/prefill_cold.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
  • Tests run, with pass, fail and skip counts — from app/ with the repo's pinned Node 22.14.0 and npm 10.9.8:
    node node_modules/jiti/lib/jiti-cli.mjs electron/agent/engine/library.test.ts
    node node_modules/jiti/lib/jiti-cli.mjs electron/agent/engine/library-service.test.ts
    
    Local: both targeted files passed (61 and 38 assert calls respectively, 0 failures, 0 skips), npm run typecheck clean, git diff --check clean, and the repo's private-leak and test-coverage gates pass; the full npm test suite was not run locally end to end — in this environment two unrelated suites (session-search-index, mobile-security-dependencies) fail for missing local modules, the same failures as on the unmodified base commit. GitHub CI: 548/548 test files passed (the desktop check).
  • Not run, and why: physical exFAT verification of the final commit (no exFAT volume attached; the regression coverage here is synthetic and deterministic — it writes the companion files directly rather than relying on a copy command, since whether cp materializes AppleDouble files depends on the filesystem and copy flags); real-model end-to-end load through the engine.

Regression evidence (synthetic, deterministic, through the public ModelLibrary API):

  • Invalid AppleDouble sidecars beside valid sharded-checkpoint assets do not invalidate the checkpoint: it scans complete/loadable, the sidecars are excluded from weightsBytes and memoryFloorBytes, and resolve() succeeds. The same new tests fail against the pre-change implementation (the sidecars are counted as shards and the checkpoint is rejected).
  • A stray ._old.safetensors in the selected parent does not stop walk(): the nested valid checkpoint is discovered and loadable, and the parent itself is not recorded as a model. Before this change the parent is mistaken for a checkpoint and the nested one is never discovered (demonstrated against the base commit).
  • A genuinely corrupt weight shard still fails the checkpoint.

Checkpoint folders on non-HFS+ volumes (exFAT/FAT external drives)
carry an AppleDouble companion ('._model-00001-of-00043.safetensors')
next to every real file, holding that file's resource fork. The scan's
directory listing includes them, so inspect() counted them as weight
shards, read their 4 KiB fork payload as a safetensors header, and
failed the checkpoint with 'The safetensors header is invalid or
incomplete.' The native engine ignores these files, and they are never
checkpoint assets, so filter them from the inspected name list.
A stray '._' companion in a selected parent matched the checkpoint
detection test, so walk() stopped at the parent and never reached the
real checkpoint beneath it; filter companions from detection as well
and drop the undocumented '._.' exception from both filters.
@ashhart ashhart mentioned this pull request Oct 17, 2026
@ashhart
ashhart merged commit d9a5d47 into ashhart:main Oct 18, 2026
4 checks passed
@ashhart

ashhart commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Thank you, @heitke, for applying the AppleDouble filter at discovery and inspection, with companion and nested-folder tests.

We merged your three commits for 1.1.0 in d9a5d47, beside #967. A ._ companion cannot become a shard or make a parent folder hide the real checkpoint beneath it.

Real shards still determine completeness and weight totals; corrupt or missing real shards still fail. Your tests cover the regression follow-up we had offered.

Posted by Ash's AI Assistant Synthena

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants