[heitke]

What this changes

The local model library scan rejects large-vocabulary checkpoints because their shard index exceeds the scan's 4 MiB per-file JSON cap (JSON_LIMIT in app/electron/agent/engine/library.ts). The bundled native engine reads these files without complaint — the cap only guards the scan's bounded read loop — so this change raises JSON_LIMIT to 64 MiB per file and the scan-wide metadata budget to 256 MiB. With that, a 43-shard GLM-5.3-Flash checkpoint (~170 GiB, 11.5 MiB model.safetensors.index.json) scans complete/compatible/loadable instead of failing with The checkpoint metadata is too large to inspect safely.

• The failure this fixes is the shard index. tokenizer.json is checked through safeFile(), not the JSON cap, so an oversized tokenizer already passed; the regression tests pin that behavior.
• metadataBytes is a scan-wide budget shared by every checkpoint in one scan, not a per-checkpoint allocation.
• JSON_LIMIT also caps safetensors header sizes, so raising it widens that bound too.
• During parsing, a metadata file is held as a buffer, a decoded string, and the parsed object, so peak memory per file exceeds the file size; the caps bound what the scan is willing to read, not total parsing memory.
• The limits keep the scan bounded while covering the observed checkpoint sizes with headroom; admission still requires the safetensors header check, shard-completeness checks, and the engine's own memory gate.

A large checkpoint on an exFAT volume hits two independent causes — the over-cap shard index and the AppleDouble companions (#970) — and needs both changes; neither PR alone fixes that case.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold app 1.0.0-beta.6, native engine 1.0.12 (build.zig.zon), commit 1b32d2b7 (this PR, based on 46a08c18); macOS 26.6.2, Apple M3 Ultra (Apple Silicon, Metal), Node 22.14.0, npm 10.9.8. Checkpoint: a local glm-5.3-flash-mlx-4bit-mtp checkpoint (43 shards, ~170 GiB) on an external drive; its Hugging Face repository and revision are unrecorded, and the receipt's verification of it is the author-reported manual run on the earlier commit, not re-run for this commit.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
• Decode speed before and after, from tools/bench_openai.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
• Prompt speed before and after, from tools/prefill_cold.py: not applicable — filesystem model-library inspection only; inference and prompt-processing code are unchanged.
• Tests run, with pass, fail and skip counts — from app/ with the repo's pinned Node 22.14.0 and npm 10.9.8:

  node node_modules/jiti/lib/jiti-cli.mjs electron/agent/engine/library.test.ts
  node node_modules/jiti/lib/jiti-cli.mjs electron/agent/engine/library-service.test.ts
  

  Local: both targeted files passed (64 and 38 assert calls respectively, 0 failures, 0 skips), npm run typecheck clean, git diff --check clean, and the repo's private-leak and test-coverage gates pass; the full npm test suite was not run locally end to end — in this environment two unrelated suites (session-search-index, mobile-security-dependencies) fail for missing local modules, the same failures as on the unmodified base commit. GitHub CI: 548/548 test files passed (the desktop check).
• Not run, and why: physical exFAT verification of the final commit (no exFAT volume attached; the regression coverage here is synthetic); real-model end-to-end load through the engine (needs the ~170 GiB checkpoint attached).

Regression evidence (synthetic, through the public ModelLibrary API — scan, addFolder, resolve):

• Before this change: a checkpoint whose shard index is ~5 MiB fails with The checkpoint metadata is too large to inspect safely.; after, it scans complete/compatible/loadable, resolves, and reports the right weightsBytes/memoryFloorBytes. The same new test fails against the pre-change implementation.
• An oversized tokenizer.json alone scans loadable both before and after — confirming the tokenizer was never gated by this cap.
• A file above the new 64 MiB per-file cap is still rejected (sparse file, so the test costs a stat, not 64 MiB of writes), and malformed JSON in an index is still rejected.
• With the scan-wide metadata budget lowered through the existing configurable limits, the scan stops at a visible bound: truncated: true and a safety-limit warning.

[ashhart]

Thank you, @heitke, for the larger-metadata fix and regression fixtures.

We merged your work for 1.1.0 in 4a2d760. The scanner accepts a valid 5 MiB index, refuses a 65 MiB index before reading and still rejects corrupt metadata.

Our scan-budget fixture also checks two indexes that fit separately but exceed the shared budget together. Native model admission and shard validation remain in place.

Posted by Ash's AI Assistant Synthena