Repository navigation
tensorfold benchmark refuses Nemotron 3.5 Super VL: config.json over the 4 MiB read cap reports "Checkpoint is incomplete" #934
Description
Title: tensorfold benchmark refuses Nemotron 3.5 Super VL: config.json over the 4 MiB read cap reports "Checkpoint is incomplete"
Version: tensorfold-native 1.0.12 (v1.0.12, 833bbfc), Linux aarch64 release archive
Hardware: DGX Spark (GB10), driver 580.173.02
Model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 @ 82d93699acd3be9e4617c53360fe5b3f5dbeb1b3, the revision the 1.0.12 recipe names
What happens
tensorfold-native benchmark ~/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4/snapshots/82d93699acd3be9e4617c53360fe5b3f5dbeb1b3 \
--output ~/tf-bench/super-vl-1.0.12-gb10.jsontensorfold benchmark: Checkpoint is incomplete; run tensorfold pull MODEL
The checkpoint is complete: all 114 files of the revision are in the snapshot, tensorfold-native info on the same folder
reads it (family: Nemotron 3.5 Super VL (nemotron_h_omni), 75,063,317,678 bytes of weights), and tensorfold-native serve
on the same folder serves it.
Cause
zig/src/server/benchmark_model.zig:12 caps the config read at config_limit = 4 << 20 (4 MiB), and readConfig turns
any read error into null (line 22). Super VL's config.json is 8,668,620 bytes, so the read fails and
benchmark.zig:308 reports it as an incomplete checkpoint.
describe() reads the config through the same function (line 37). A run attached to a serving Super VL with
--server http://localhost:8022 --backend cuda --model-id super completes, but its receipt's model fields
(config_sha256, family, quantization, revision) are all null and the result is unranked.
Expected
The managed benchmark runs Super VL like other served families and gives a rankable receipt.
Possible fix
- Raise
config_limitabove Super VL's 8.7 MB (16 MiB leaves room), or read config.json through the same path
infoandserveuse, since both read this file. - Report an oversized file as its own error ("config.json is N bytes, over the M-byte limit") instead of
"Checkpoint is incomplete", so the next oversized config doesn't send people to re-download.
Activity
ashhart commentedon Oct 17, 2026
Thank you, @jkuepker, the exact 8,668,620-byte config and same-folder info/serve comparison identify the benchmark's separate read limit.
We checked 833bbfc at benchmark_model.zig:12/20-27. Its bounded read returns null on failure, and describe uses that same reader, so this large config is treated as incomplete and loses model facts.
We will carry a bounded-reader/error-reporting fix with large-config and metadata regressions. It is separate from the app scan fix in #967. We will also check repository-ID and attached-server identity explicitly; an arbitrary local path already omits repository revision by design, so raising the cap alone cannot promise every receipt becomes ranked. Your existing pins are enough to start that check.
Posted by Ash's AI Assistant Synthena
benchmark: config.json read through one 64 MiB metadata reader, its f…
ashhart commentedon Oct 18, 2026
Thank you, @jkuepker, for the exact config size and same-folder info/serve comparison.
1.1.0 fixes the 4 MiB refusal reproduced with an 8,668,620-byte config in 1e8debd. Info, serve and the managed benchmark now share one bounded 64 MiB metadata read. Missing files retain the incomplete-checkpoint message; oversized, unreadable or malformed configs get their own error. Receipt metadata comes from that same read.
Attached-server identity and ranking restrictions are separate from this cap. Please reopen if the released managed benchmark still rejects that config or loses its available metadata.
Posted by Ash's AI Assistant Synthena
benchmark: config.json read through one 64 MiB metadata reader, its f…