Feat/internvla n1 dualvln - #2
Open
hungho77 wants to merge 27 commits into
Open
Conversation
…atrix Adds recipes/internvla-n1-dualvln/, the front matter for porting a working FP8 quantization and TensorRT-Edge-LLM deployment pipeline for InternVLA-N1-DualVLN on Jetson Thor. This commit is structure and documentation only; the Python and shell entrypoints land in follow-ups. The recipe is laid out along its dependency boundary rather than by file type. InternVLA-N1-DualVLN declares model_type internvla_n1 and ships no modeling code, so it cannot be loaded with trust_remote_code -- the class has to come from the InternNav repository. A repackaging step strips the eight System-1 tensor prefixes and rewrites config.json to a stock Qwen2.5-VL, using nothing but safetensors surgery, which takes InternNav off the hot path for quantization, export, engine build and latent verification. Only System-1 export and the agent-level checks need INTERNNAV_PATH, and the layout says so per step. configs/schemes.yaml encodes the scheme x strategy validity matrix so impossible combinations are rejected before the model loads rather than forty minutes into a run. Two entries are hardware facts: NVFP4 cannot touch the vision tower because the ViT MLP intermediate_size is 3420 and 3420/16 is not an integer, and the KV cache stays FP8 under NVFP4 weights because NVFP4 KV needs sm100f while Thor is sm110. NVFP4 weights are marked experimental rather than supported: they quantize, export and generate fluent text, but z_latents cosine falls to 0.647, which breaks the System2 -> System1 bridge. For a navigation model the bridge is the acceptance metric, not text fluency, so the README leads with that distinction. The README also retracts an earlier claim carried in the source project that FP16 engines are broken on Thor and FP8 is therefore mandatory. The garbage came from a Myelin fc_h_fusion miscompile at TensorRT 10.13; with the workaround applied the unquantized FP16 engine is in fact the highest-fidelity variant measured. FP8 is recommended on size and latency, not correctness. numpy and scipy are deliberately absent from the pyproject extra. This recipe needs numpy 1.x while three other extras pin numpy==2.2.6, so rather than assert a resolution outcome that has not been tested here, they are installed as a documented post-sync step. Note the repository already carries three mutually exclusive transformers pins across extras, so this is not a new class of problem; run uv lock and check before concluding anything about it. Directory names avoid build/ and lib/, which the root .gitignore silently swallows. A recipe-level .gitignore covers the multi-gigabyte engine and ONNX artifacts, which nothing else in the repository excludes.
repackage_system2.py strips the eight System-1 tensor prefixes and rewrites config.json
to a stock Qwen2.5-VL. Verified on the real 16 GB InternVLA-N1-DualVLN checkpoint: 729
System-2 keys kept, 609 System-1 keys dropped, and the result loads through AutoConfig and
AutoProcessor as Qwen2_5_VLConfig / Qwen2_5_VLProcessor with no InternVLA fields left. That
is what takes InternNav off the hot path for everything except System-1 export.
Five bridge tensors (latent_queries, cond_projector.{0,2}.{weight,bias}) are written to a
separate bridge.safetensors instead of being dropped with the rest of System 1. They are
what the z_latents fidelity check consumes, so keeping them here lets quantize, export,
build and verify all run from the repackaged directory without reopening the 16 GB source.
quant_schemes.py loads configs/schemes.yaml and rejects impossible combinations before the
model is loaded, so a bad request costs seconds instead of dying mid-quantization. The
vision-tower check reads vision_config.intermediate_size from the checkpoint rather than
hardcoding 3420, so it stays correct for other Qwen2.5-VL sizes; on this checkpoint it
reports '3420 / 16 = 213.75' in the error. Verified against all four strategies and both
scheme families: fp8 passes everywhere, nvfp4 is blocked on s3/s4 and gated behind
--allow_experimental on s1/s2.
A free-space preflight refuses to start below 18 GB, and --free_source deletes each source
shard as its converted copy lands for machines that cannot hold both checkpoints.
…l loader Completes the System-2 quantization path. Verified end to end on the real checkpoint: 16.6 GB repackaged System 2 -> 9.4 GB FP8, 1164 TensorQuantizers, 80s quantize plus 16s export, hf_quant_config reporting FP8 with lm_head and visual* excluded, and tokenizer.model_max_length still 8192 (calibration truncates at 512 as a call-time argument only; a leak there would silently cap inference sequence length). model_loader.py drops the InternVLA-N1 branch entirely rather than carrying it forward. Post-repackage the input is always a stock Qwen2.5-VL, so the branch, its two InternNav imports, the diffusers gradient-checkpointing monkeypatch and the unused _is_system2_checkpoint helper are all dead weight -- 70 lines removed, and with them the dependency that would otherwise sit on the critical path of every quantization run. quantize.py calls the validity gate before loading the model, so an impossible request costs seconds. Confirmed: nvfp4+s3 is rejected in under a second with the checkpoint's own '3420 / 16 = 213.75' and never touches the weights, while fp8+s1 loads in 14s and reports sm_110. --cfg is renamed --scheme, kebab-case flags become snake_case to match the other recipes, and the output tag no longer hardcodes qwen2.5-vl-7b. calibration.py gains an offline path. This machine sits behind a TLS-intercepting gateway, so load_dataset() fails certificate verification even though the parquet shards are already in the hub cache -- and HF's own offline mode does not help, because it still wants the dataset script. The loader now probes reachability and reads the cached shards directly, which is the difference between a working and a non-working calibration run on any air-gapped or proxied deployment machine, not just this one.
… scales applied Accuracy benchmarking a quantized checkpoint in PyTorch needs it loaded correctly, and the obvious way is silently wrong. export_hf_checkpoint writes LLM weights as real torch.float8_e4m3fn with dequantization scales in separate weight_scale/input_scale tensors. AutoModelForImageTextToText.from_pretrained on that directory appears to succeed -- it warns the scale tensors 'were not used when initializing' and continues -- but every quantized weight then sits at its raw FP8 magnitude. Measured on this checkpoint, the naive load produces weights 630x to 1799x too large depending on the layer. Any accuracy number taken from it would be reporting a broken load as quantization damage. load_quantized.py reconstructs w_bf16 = w_fp8.to(bf16) * weight_scale over a normal from_pretrained (which is what materialises the rope buffers; a meta-device build leaves them empty). Verified against the unquantized System 2 in float64: cosine 0.99965 and 2.6-2.7% relative error per projection, which is what FP8 E4M3 per-tensor should cost, against a scale ratio of exactly 1.000. Modules excluded from quantization -- the vision tower and lm_head -- are stored bf16 and pass through untouched. Note what this does and does not model: weight quantization only. Real FP8 W8A8 also quantizes activations, which the TensorRT engine does and this does not, so a number from here is a lower bound on the engine's deviation rather than a prediction of it. The docstring says so, because the distinction is easy to lose.
… episodes
The official metrics (SR, SPL, NE, OS, nDTW) are closed-loop and need Habitat or
InternUtopia plus MP3D scenes. Neither is installed here and neither is practical on a
Jetson, so this measures what quantization actually threatens and what can be measured
offline: whether System 2 still picks the same waypoint.
Result on 42 held-out samples from two scenes, BF16 System 2 versus its FP8 quantization,
identical samples and greedy decoding:
pixel_goal_l2 BF16 mean 47.24 px / median 27.05 px
FP8 mean 50.12 px / median 26.97 px
agreement 31/42 replies byte-identical, median deviation 0.00 px
The mean moves 2.88 px while the median does not move at all, because the gap is carried
by 8 samples out of 42 rather than by a systematic shift -- worst case 89 px. That
distinction matters for a navigation model and is why both statistics are reported.
Two harness bugs were found by disbelieving the first numbers rather than publishing
them, and both are now prevented in code:
Feeding a pitched frame through a turn-1 prompt gave a 0% coordinate parse rate. The
model was right and the harness was wrong: the agent runs two turns, and the coordinate
only exists in turn 2, after the level view has been answered with a look-down token and
the tilted frame appended. The pixel goal in the level camera is the [-1,-1] sentinel on
every frame precisely because a level camera cannot see a point on the floor.
prompt_builder's 'episode_idx' is the step index within an episode, not the episode
number -- it derives the history frames from linspace(0, episode_idx-1). Passing the
episode number produced one image placeholder against nine images.
action_accuracy is now scored on turn 1 only. At turn 2 the model emits a coordinate by
design, so scoring it against a movement action reads 0% and looks like a regression.
…port/build script README now leads the results with the number a navigation model is actually judged on: FP8 keeps the median waypoint error unchanged (27.05 -> 26.97 px over 42 held-out samples) with 31 of 42 replies byte-identical and a median deviation of 0.00 px. The 2.88 px gap in the mean comes from 8 disagreeing samples rather than a systematic shift, so both statistics are given; reporting the mean alone would overstate the damage and the median alone would hide the tail. It also states plainly what the measurement does not cover -- activations are quantized in the engine but not on this PyTorch path, so the figure is a lower bound -- and that the published SR/SPL/NE table is a literature reference, not something reproduced here, since the closed-loop metrics need Habitat or InternUtopia and MP3D scenes. 03_export_build_system2.sh collapses the source project's four near-duplicate export scripts into one, driven by --no_quantization for the FP16 reference. Two things it does are load-bearing and carry comments saying so: emit_hidden_states=True before export, without which the engine returns logits only and the System 1 bridge cannot be evaluated; and __LUNOWUD=-peep:fc_h_fusion=off on the build, without which an FP16 engine on TensorRT 10.13 produces fluent gibberish from a Myelin miscompile. The visual encoder is built once at 4/4096/1024 image tokens. The source project's default of 512 came from a single-image demo and cannot hold a VLN prompt's 9-10 frames, which is why its multi-image verifications silently pointed at a hand-built engine no script produced.
…onfig.json Two build-script defects found by running it rather than reading it. visual_build appends its own 'visual' component directory under --engineDir, so passing $ENGINE_DIR/visual produced $ENGINE_DIR/visual/visual/visual.engine. Every consumer looks for $ENGINE_DIR/visual/visual.engine, so the engine was built correctly and then filed where nothing would find it. Pass the parent instead. llm_bench reads its engine configuration from base_config.json while llm_build writes config.json -- same content, different name. Without the copy the benchmark refuses to open the engine that was just handed to it, with a parse error that reads like engine corruption rather than a missing filename. Verified on the FP8 engine built from the repackaged System 2: llm.engine 7.62 GB (the figure the source project recorded), visual.engine 1.36 GB, text smoke test coherent, and llm_bench measuring 95.8 ms prefill at 1024 tokens and 37.35 ms decode at 1024 past-KV.
…ison
Both engines built from the same repackaged System 2 and measured with llm_bench on
Jetson Thor at batch 1:
LLM engine prefill 1024 decode pastKV 1024
base FP16 15.0 GB 196.22 ms 86.16 ms
FP8 7.62 GB 95.80 ms 37.35 ms
gain 1.97x smaller 2.05x faster 2.31x faster
Both emit the same text on the same prompt, and the PyTorch-side accuracy run already
showed the median waypoint error unchanged, so FP8 here is a straight win rather than a
trade: half the engine, roughly double the throughput, same decision.
Worth recording that the FP16 engine is correct only because the build applies
__LUNOWUD=-peep:fc_h_fusion=off; the build log confirms both workaround flags were passed.
Without it TensorRT 10.13 miscompiles Myelin's horizontal gate/up fusion on sm_110 and the
engine produces fluent gibberish, which is what once made FP8 look mandatory on this
platform. It is not -- FP8 is chosen here on size and speed.
… both engines verify_latents.py rebuilds the bridge end to end against a PyTorch BF16 reference -- embed, scatter image embeddings, append the four trajectory tokens, run the engine with hand-built 3D mRoPE, take the last-layer hidden states, apply the host-side norm and cond_projector -- and compares z_latents. engine hidden pre-norm post-norm z_latents rel-L2 base FP16 0.999843 0.999123 0.999471 0.0293 FP8 (s1) 0.997793 0.987343 0.991861 0.1023 Both clear the 0.99 gate, so FP8 preserves the signal System 1 actually consumes. This is the check NVFP4 fails at 0.647 while still producing fluent text, which is why it ships as experimental rather than supported. The check reads latent_queries and cond_projector from the bridge.safetensors that repackage_system2.py sets aside, so it needs neither the 16 GB original checkpoint nor InternNav. Four transformers 5.x incompatibilities were fixed with version-probing helpers rather than a version pin, since the recipe should survive either: the vision tower and get_rope_index moved onto the inner model; get_image_features returns a BaseModelOutputWithPooling whose pooler_output holds the merged embeddings while last_hidden_state is the pre-merger tensor at vision width; and get_rope_index gained a required mm_token_type_ids argument. That last one does not fail on arity -- it fails later inside with 'NoneType is not an iterator', which is a poor way to discover it. Also switched the reference forward from flash_attention_2 to sdpa, which is what is available on Jetson and what the deployed agent uses.
…nchmarks Adds traj_dit and memory-block export, their BF16 engine builds, the System-1 fidelity check and the two System-1 benchmarks, plus internvla_compat.py which carries the three patches needed to load System 1 at all: the DepthAnythingV2 checkpoint path, the diffusers gradient-checkpointing signature, and the traj_dit FFN multiplier. That last one is a real checkpoint-compatibility finding -- DualVLN was trained with ffn_dim_multiplier 2/3, and the stock build_traj_dit never passes it, so the state dict mismatches without the patch. Four defects carried over from the source project are fixed here rather than propagated: MemBlock was defined inline in the memory exporter as well as in memblock.py, so the two copies could drift -- the exporter now imports the single definition; the traj_dit engine filename said _async in one place and not the other, and neither constant was read; a dead 'if False else None' statement; and INTERNNAV_ROOT is renamed to the recipe-wide INTERNNAV_PATH. trt_torch.py keeps its original NVIDIA Apache-2.0 header rather than being restamped. It is third-party code and Apache-2.0 section 4 requires retaining the notice; it is the one file in this recipe that does not carry the VinRobotics BSD header. Documented an environment constraint found by hitting it: System 1 must run under transformers 4.51.3. The InternNav modeling code reads config.hidden_size off the top-level config, which transformers 5.x no longer flattens, so it fails with a bare AttributeError. The System-2 path is unaffected and runs on either version.
Both System-1 engines built and verified loadable with the expected I/O: traj_dit 134 MB ONNX -> 72 MB engine x, timestep, z_latents -> output memory block 200 MB ONNX -> 104 MB engine images -> memory_tokens Sizes match what the source project recorded. Both stay BF16 on purpose: they are small enough that quantizing them buys nothing, and the diffusion head is the part least tolerant of it. The memory exporter no longer exits 1 after a successful build. Its in-script parity check imports the TensorRT Python bindings, which JetPack ships for Python 3.12 only, while the export itself needs transformers 4.51 from the 3.10 environment -- so the check could never run in the same interpreter as the export. It now skips with a message pointing at verify/verify_system1.py under the 3.12 environment. Reporting failure for an artifact that was produced correctly is worse than not running the check. Two more environment gaps closed by installing rather than working around: diffusers 0.33.1 (the traj_dit scheduler) and setuptools<81 (InternNav's LongCLIP still imports pkg_resources, which newer setuptools dropped).
…hmarks Adds the five remaining fidelity checks, the System-2 benchmark, the simulator adapter and its eval entry point. verify_engine_policy.py runs green on the FP8 engine: the adapter is a proper InternVLAN1Net subclass adding only _engine_generate, and the engine's reply survives the tokenizer roundtrip exactly. lib/_trt_contract.py is deliberately not ported. It is dead code -- imported by nothing, reading an export_metadata.json that no script writes, and referencing a different project's exporter. Defects fixed rather than carried across: two scripts still defaulted to one developer's absolute layout under vln-thor/artifacts; benchmark_system2.py wrote reports/bench_e2e.json without creating the directory, so it crashed on its own last line after doing all the work; INTERNNAV_ROOT/VLN_OPT_WORK/VLN_OPT_ENGINES are renamed to the recipe-wide INTERNNAV_PATH/WORK_DIR/ENGINE_DIR; nine Vietnamese comments translated; and seven files restamped to the BSD header while trt_torch.py keeps its NVIDIA Apache-2.0 notice. benchmark_system2.py depends on a golden manifest that neither this recipe nor the source project generates. Instead of leaving that as a FileNotFoundError on the first read, it now fails up front and describes the file it wants -- the same sample shape quantize/benchmark_accuracy.py already builds from LeRobot episodes. Scripts that need prompt_builder now locate the recipe root by walking up to the directory containing quantize/, rather than counting parent levels: they sit at two different depths and a fixed count works for one and not the other.
… not the weights
Measured in PyTorch on a real NVFP4 checkpoint of this model, weights only:
weight rel-err weight cos z_latents
FP8 2.67% 0.999644 0.998020
NVFP4 9.45% 0.995534 0.987986
NVFP4 weight quantization costs 0.988, not 0.647. The weight error is 3.5x FP8's and the
bridge degrades roughly in proportion, which is unremarkable. So the 0.647 recorded for the
NVFP4 engine does not come from weight quantization, and the candidates are the two things
this measurement does not model: 4-bit activation quantization, and the engine itself.
The engine hypothesis is the one I would chase first. This platform has already produced
one 'quantization is broken' conclusion that turned out to be a TensorRT miscompile, and
there is a second known one specific to NVFP4 -- CASK miscompiling two or more fused
epilogues in an NVFP4 GEMM at batch 1, fixed by -cask_fusion:max_num_epilogues=1. A 0.647
taken from an engine built without that workaround would be measuring the miscompile rather
than the format.
Channel analysis rules out the obvious remedy for the weight-side loss: at the final layer
the top 128 channels by magnitude carry only 29.5% of the squared error, and masking them
lowers cosine rather than restoring it. The error is spread, not concentrated in outliers,
so AWQ scaling or a targeted exclusion has nothing to grip.
load_quantized.py gained NVFP4 support to make this measurable at all: unpacking two E2M1
values per byte with a per-16 FP8 block scale and a float32 global scale. The quantized path
now hands transformers an already-dequantized state_dict rather than letting it read the
checkpoint -- FP8 loads silently wrong without scales, and NVFP4 fails outright on the
halved width, so from_pretrained cannot be trusted with either.
NVFP4 stays experimental. The next step is to rebuild its engine with the CASK workaround
and re-measure end to end, not to do more weight analysis.
…t standard
The recipe carried 862 flake8 warnings while every existing recipe in this repository is
clean at zero. That gap is now closed. Most of it was VLN-Opt's dense one-liner style --
compound semicolon statements, missing whitespace, multi-import lines -- mechanically
fixable with autopep8 at the repo's 120-column setting.
Three defects were real, not cosmetic, and only surfaced because the lint pass ran:
* benchmark_system2.py did not parse at all. A sys.path shim had been inserted directly
above an import that lives inside a function, so its module-level indentation broke
the enclosing block. The file had never been executed after that edit.
* verify_latents_vln.py referenced json without importing it -- left behind when
load_ckpt_tensor was rewritten to read bridge.safetensors, since the old body carried
the import.
* Six unused imports and one dead local.
E402 is silenced with explicit noqa rather than reordered: these modules must insert a
sys.path entry before importing prompt_builder or the InternNav tree, so the import
genuinely cannot come first.
Verified the reformatting changed no behaviour: the scheme validity gate still accepts
fp8+s1, rejects nvfp4+s3 with the checkpoint's own 3420/16 arithmetic, and gates nvfp4+s1
behind --allow_experimental; the dequantizing loader still reproduces cosine 0.999647 for
FP8 and 0.995489 for NVFP4 against the unquantized weights.
Separately confirmed the NVFP4 nibble order empirically rather than trusting the spec: the
low-nibble-first unpacking gives cosine 0.995485 against the reference weights while the
swapped order gives -0.002446, so the layout is not a coin flip that happened to land.
…t of its collapse Rebuilding the NVFP4 engine with the fork's CASK workaround settles most of the question the investigation opened. Three measurements, each isolating a layer: FP8, weights only (PyTorch) 0.998020 NVFP4, weights only (PyTorch) 0.987986 NVFP4 engine, with the CASK workaround 0.931005 NVFP4 engine, no workaround (source figure) 0.647 So the 0.647 was mostly a compiler artifact. -cask_fusion:max_num_epilogues=1 -- applied automatically by the fork, gated to NVFP4 graphs at batch 1 -- moves it to 0.931, and the build log confirms all three flags fired. Weight quantization was never the problem: NVFP4 weights cost 0.988, an error 3.5x FP8's with the bridge degrading in proportion. A real gap remains between 0.988 and 0.931, and it now has a named suspect rather than being a mystery: NVFP4 is W4A4, and 4-bit activations through a 3584-wide hidden state in blocks of 16 are the part the PyTorch weights-only path cannot model. The 0.647 is quoted from the source project and was not reproduced here. What is measured here is that the same checkpoint reaches 0.931 once the workaround is applied. NVFP4 stays experimental: 0.931 is below the 0.99 gate. But it is far from broken, and the 5.5 GB engine is the smallest of the three. Adds 06_measure.sh to collect prefill/decode latency for every built engine into the JSON run_matrix.py reads, and run_matrix.py itself, which assembles sizes, latency, bridge fidelity and task accuracy into one table without recomputing anything.
All three variants built from one repackaged System 2 and measured on an idle GPU: variant ckpt LLM engine prefill decode z_latents pixel L2 mean/median BF16 16.6 GB 14.15 GB 135.8ms 56.4ms 0.999471 47.24 / 27.05 px FP8 s1 10.1 GB 7.62 GB 82.1ms 31.5ms 0.991861 46.26 / 22.51 px NVFP4 s1 7.2 GB 4.77 GB 73.2ms 20.2ms 0.931005 40.69 / 23.54 px FP8 is the recommendation: 1.86x smaller and 1.65x/1.79x faster than BF16, bridge held at 0.9919, and the median waypoint error does not worsen. It reads slightly better (27.05 -> 22.51 px), which for 42 samples should be read as unchanged rather than as a gain from quantization. NVFP4 is smaller and faster still but its bridge sits at 0.931, below the gate, so it stays experimental. Two caveats are documented alongside the table because both cost time to learn here. Latency must be measured on an idle GPU: sharing the device with another job inflated FP8 prefill from 82 ms to 117 ms, and nothing in the scripts enforces this. The 'identical replies' count is a weak indicator and is now labelled as such. It moved from 31/42 to 11/42 for FP8 between two revisions of the checkpoint loader while pixel_goal_l2 barely moved. I initially attributed that to GPU contention; a clean re-run reproduced the contended numbers exactly, so the cause was the loader change, not contention. The current loader is verified correct -- all 729 weights in the loaded model match the dequantized state dict -- so the current numbers stand. Greedy decoding over a 152k vocabulary flips on tiny logit differences; the L2 median is the number to trust.
AWQ does not help this model. End to end the bridge measures 0.986293 against plain NVFP4's 0.987986 -- the same within noise. That matches what the channel analysis predicted: the error is spread across channels rather than carried by outliers, so AWQ's per-channel rescaling has nothing to grip. Getting to that number needed two loader fixes, and the first reading was wrong in a way worth recording. AWQ-lite scales activations by a per-input-channel factor and stores the weight pre-divided by it, so a plain matmul has to multiply it back in. Ignoring the pre_quant_scale tensors entirely produced ~190% relative weight error, which reads as AWQ being catastrophically broken rather than being loaded wrong. The direction is also easy to invert: verified empirically on this checkpoint that multiplying gives cosine 0.9898, leaving it alone 0.9606, and dividing 0.8089. The second is subtler and invalidates a metric rather than a number. Only down_proj and o_proj carry an explicit pre_quant_scale, because q/k/v share one layernorm and gate/up share another -- for those, AWQ folds the scale into the layernorm weight instead. Measured here, AWQ moves the layernorm weights by 0.012x to 91x while plain NVFP4 leaves them at exactly 1.000. So comparing AWQ's linear weights against the reference's linear weights is not a comparison at all; the invariant is the composition. The weights stage now detects AWQ checkpoints and skips with that explanation rather than reporting a meaningless 180%. qat.py adds quantization-aware fine-tuning, which is the remaining lever: the gap between NVFP4 weights-only (0.988) and the engine (0.931) is 4-bit activations, and no weight-side method reaches those after the fact. ModelOpt has no separate QAT entry point -- it is mtq.quantize followed by ordinary fine-tuning with the fake-quantizers left in so gradients pass through them. It excludes scene YmJkqBEsHnH from training by default: the same MP3D building appears as calib_scenes/r2r and probe_heldout/rxr, and the source project has already reported one calibration gain that turned out to be exactly this leak. --allow_overlap disables the guard for anyone who wants to measure the effect deliberately. Note that success rate cannot be measured on this machine -- SR, SPL and NE are all closed-loop. QAT is evaluated on the proxies it can move, z_latents and pixel-goal L2, and the docstring says so.
…weight-side gap
All three NVFP4 presets measure the same end to end against the unquantized reference:
nvfp4_default 0.987986 baseline
nvfp4_awq_full 0.986293 equal within noise
nvfp4_local_hessian 0.987986 byte-identical to default
nvfp4_local_hessian is a no-op on this model, and it is worth stating plainly because it
fails silently. The preset genuinely differs -- algorithm={'method': 'local_hessian',
'fp8_scale_sweep': True} against 'max' -- and the run exits 0, but the exported weights
match nvfp4_default bit for bit: 0 of 6,422,528 bytes differ, scales included. Two runs
with different algorithms cannot produce identical output unless the algorithm never ran.
It stays selectable via --scheme, so a user would reasonably believe they had tried it.
AWQ does run, its weights genuinely differ, and it still does not help -- which is what the
channel analysis predicted: the error is spread across channels rather than carried by
outliers, so per-channel rescaling has nothing to grip.
Three weight-side methods stopping at the same 0.988 is the argument for QAT rather than a
fourth. The gap from there to the engine's 0.931 is 4-bit activations, and no
post-training method reaches those.
…ne run
Adds quantization-aware fine-tuning and reports what it actually did, which was make
things worse:
z_latents (engine) pixel L2 mean / median
NVFP4 PTQ 0.931005 40.69 / 23.54 px
NVFP4 + QAT 0.891583 41.86 / 23.16 px
The training loss rose from 0.78 to 1.13 over 16 steps, so the run pushed the weights the
wrong way rather than converging. That makes this a failed training run, not evidence that
QAT cannot help: the experiment never reached the question it was meant to answer. Saying
'QAT does not work for this model' from sixteen steps would be a conclusion the data does
not support, so the README says what would change next instead -- a much smaller learning
rate, hundreds of steps rather than sixteen, and a warmup.
Two obstacles are worth recording because both cost a run to find.
Full fine-tuning does not fit on this hardware. Weights plus gradients plus AdamW moments
for 8.29 B parameters are ~100 GB before any activations, against a 122 GB pool shared with
the host; the first attempt was killed by the OOM killer. --train_last_n_layers (default 4)
makes it fit at 932 M trainable parameters, and it targets the right place anyway, since
the bridge reads the last layer's hidden states.
Freezing combined with gradient checkpointing needs two fixes applied together. The
activations entering the first trainable layer carry no grad_fn, and reentrant
checkpointing then discards the graph, so loss.backward() fails with 'element 0 of tensors
does not require grad'. Both use_reentrant=False and enable_input_require_grads() are
required; either alone still fails.
Also documents a metric trap I walked into first: z_latents against the unquantized
reference is the wrong measure for QAT. PTQ approximates the original model, so similarity
to it is meaningful; QAT deliberately moves away from the original to absorb quantization
noise, so a good QAT run can lower that number while improving behaviour. QAT is judged
through the engine and on task accuracy instead.
…engine number
The results table carried two z_latents columns that measure different things, side by
side, with nothing saying so. That is a real reporting defect: it invites the reader to
treat 0.988 and 0.931 as disagreeing measurements of one quantity when they are correct
measurements of two.
load_for_eval reconstructs the weights into a plain model with no quantizers, so it
reproduces weight quantization error only -- activations stay bf16 and the matmuls are
bf16. But FP8 is W8A8 and NVFP4 is W4A4: the engine quantizes every activation too. The
weights-only path cannot see that half at all.
load_fake_quant now inserts live quantizers via mtq.quantize, so both halves are simulated
and a PyTorch number can be put next to an engine number honestly. load_for_eval stays for
weight-side questions, with its docstring saying which is which.
The README now shows the decomposition rather than the bare pair:
weights only engine activation cost
FP16 - 0.999471 -
FP8 0.998020 0.991861 0.006
NVFP4 0.987986 0.931005 0.057
The FP16 engine at 0.999471 is what makes this legible: TensorRT itself costs about 0.0005,
so almost none of the gap is export or runtime. Going from 8-bit activations to 4-bit costs
nine times more than going to 8-bit, which is the whole reason NVFP4 misses the gate here
while FP8 clears it.
…vation quantization Measuring fake quant directly refutes what this recipe previously claimed. With live quantizers in PyTorch -- weights and activations both simulated, as the engine does them -- NVFP4 measures 0.978631, against 0.987986 weights-only and 0.931005 through the engine. That splits the 0.057 total as: weight quantization 0.012, activation quantization 0.009, and everything else 0.048. Activations account for 16% of the gap. The remaining 84% only appears once the model runs as a TensorRT engine. I had attributed the whole gap to activations, reasoning from three numbers rather than measuring the middle one. The reasoning was plausible -- NVFP4 is W4A4, 4-bit activations through a 3584-wide hidden state are an obvious suspect -- and wrong. compare_fake_quant.py exists to make that check cheap, and its docstring states both outcomes up front so it cannot be run as a confirmation exercise. FP8 has no comparable effect: its entire PyTorch-to-engine gap is 0.006 against NVFP4's 0.048, and the FP16 engine at 0.999471 bounds TensorRT's generic cost at ~0.0005. So this is specific to the NVFP4 kernel path, the same neighbourhood as the CASK epilogue miscompile that -cask_fusion:max_num_epilogues=1 improves from 0.647 to 0.931 without evidently resolving. The practical consequence changes what to do next. NVFP4 quantization on this model is close to the gate at 0.9786; it is the engine that loses the rest. That is also why AWQ, local-Hessian and QAT all failed to move it -- all three attack quantization quality, and quantization quality was never the binding constraint.
…ut batch size diagnose_engine_gap.py compares an engine against its own fake-quant model in one process on the same inputs, so the difference is the engine alone rather than engine plus quantization. Hidden states before the final norm, text prompt: FP8 0.998256 NVFP4, maxBatchSize 1 + CASK cap 0.986790 NVFP4, maxBatchSize 2, no CASK flag 0.986790 The NVFP4 engine carries roughly eight times FP8's engine-side error, 0.013 against 0.002, and the bridge amplifies it -- 0.013 in hidden states becomes the 0.048 seen in z_latents after the final norm, GELU and cond_projector. It is not the batch-1 miscompile. The two NVFP4 engines are genuinely different builds (different checksums, maxBatchSize 1 and 2, and the fork correctly withholds -cask_fusion:max_num_epilogues=1 from the batch-2 build) yet measure identically to six decimal places. So the CASK cap fully recovers what the batch-1 path loses, and the residual is inherent to the NVFP4 kernels. My guess that a second batch-1 miscompile was hiding here was wrong. The FP8 control earned its place. The first version of this diagnostic compared against output_hidden_states[-1] and read 0.4837 for NVFP4 -- which looks exactly like a broken kernel. FP8 read 0.4949 through the same path, and FP8 is known good at 0.9919, so the harness was wrong: the engine emits hidden states before the final norm while that tensor is after it. Hooking the norm and taking its input moves FP8 to 0.9983. Without the control I would have reported a broken NVFP4 kernel path for the second time in this investigation.
…sorRT
Checked upstream first, as asked: InternNav ships no TensorRT or ONNX export at all --
nothing under internnav/ references trtexec, tensorrt or torch.onnx. The System-1
conversion is entirely this recipe's, inherited from the source project.
Two engines carry the compute, and both were verified by execution rather than by
inspection:
traj_dit (NextDiT head) 134 MB ONNX -> 72 MB engine
x[64,32,384] f32, timestep[64] i64, z_latents[64,*,768] f32 -> output[64,32,384]
memory block (DAv2 + MemoryEncoder + QFormer) 200 MB -> 104 MB
images[T,3,224,224] f32 -> memory_tokens[1,32,768]
Finite output, correct shapes. Worth noting the engine I/O is fp32/int64 despite BF16
weights; feeding bf16 trips an assertion in the wrapper rather than converting.
action_encoder, action_decoder, pos_encoding, cond_projector and the flow-matching
scheduler loop stay in PyTorch on the host. That is a design choice, not an omission --
they are tiny or control-flow heavy.
The end-to-end parity check remains unrunnable here and the README now says why rather
than leaving it as 'ready'. verify_system1.py needs InternNav and TensorRT in one
interpreter. InternNav targets transformers 4.x: under 5.x it fails first on
config.hidden_size, which transformers now nests under text_config -- patched in
internvla_compat.patch_config_flattening -- and then on apply_chunking_to_forward, removed
from modeling_utils. That chain has no natural end, and downgrading transformers inside the
TensorRT environment risks the edgellm exporter that currently works. Running the check
needs a fourth environment with transformers 4.51 and the TensorRT bindings together.
…stages The parity check needs InternNav (transformers 4.x, Python 3.10) and the TensorRT bindings (Python 3.12) at once, which no interpreter here has. Splitting it across the two environments removes the conflict: stage A dumps the PyTorch reference's inputs and outputs to a .pt, stage B feeds the engines those same tensors. Both engines reproduce PyTorch: memory_tokens 0.999981 traj_dit, one step 0.999508 full trajectory 0.999670 (rel-L2 0.0258) Stage A also captures the reference's starting noise and one real traj_dit call. Both matter: generate_traj draws its latents mid-function, so reseeding gives a different valid trajectory (cosine ~0.31), and the single-step probe is what separates a bad engine from a bad reimplementation of the sampler loop. MemBlock expects ResNet-normalized input because generate_traj normalizes before rgb_model; passing raw pixels made memory_tokens read 0.315 while each half stayed internally consistent.
…stems The results section reported System 2 and System 1 separately, with no dtype column and no System-1 latency at all, so nothing said what a whole planning step costs. Adds System-1 latency (stage B now takes --bench_iters) and the PyTorch baseline to compare it against, then restructures the matrix into three tables: System 2 by quantization scheme, System 1 as a conversion result, and both together per planning step. Every row names its dtype -- weights, activations, KV cache and vision tower separately, since s1 leaves the last two alone. System 1 engines 63.1 ms vs 175.4 ms in PyTorch 2.78x full step, FP8 709 ms vs 1806 ms all-PyTorch 2.55x weights, FP8 9.16 GB vs 15.7 GB unquantized Latency is flat in num_sample_trajs on the PyTorch side (175.4 / 175.6 / 173.1 ms at 32 / 4 / 1), so System 1 is launch-bound, which is why engines pay there. Also drops the stale duplicate table and the superseded 12-sample figures, and fixes the stage B GELU to approximate="tanh" to match the checkpoint's cond_projector.
System 1 shipped BF16 on the assumption that quantizing it was not worth it. This measures that instead of assuming it, and the assumption holds -- for a reason that is worth writing down. Unlike System 2, System 1 goes torch.onnx.export -> trtexec, and TensorRT does FP8 only through explicit quantization, so the Q/DQ has to be in the ONNX. Adds a ModelOpt PTQ pass calibrated on real tensors captured from a live System 2 -> System 1 run rather than random draws: FP8 scales are amax-based and both modules consume tensors whose scale is set upstream, so a synthetic draw is quietly wrong. FP8 works -- 328/328 Q/DQ pairs in traj_dit, 160/160 in the memory block, 1.55x smaller, 20% faster on the diffusion loop -- and is still the wrong trade: engines 176 MB -> 114 MB 0.7% of 9.16 GB deployed step 710 ms -> 698 ms 1.7%, System 2 dominates waypoint dev 0.0032 -> 0.0198 6x, p95 39% of a waypoint's reach Splitting it shows where the loss comes from: traj_dit alone costs 0.9997 -> 0.9880 even though its per-step error is only 0.999508 -> 0.997811, because the sampler runs 10 steps and each feeds the next. The memory block adds the rest and buys nothing in latency (2.09 vs 2.04 ms). Adds waypoint deviation to the parity check -- cosine says how aligned two trajectories are, not how far apart the robot ends up -- plus --engine_suffix so the same check runs against either precision. Three things that fail quietly, all now handled: ModelOpt emits trt::TRT_FP8*, not ONNX QuantizeLinear, so a naive counter reports a quantized graph as unquantized; the legacy exporter cannot infer a conv kernel shape through Q/DQ; and the QFormer's fused MHA fast path has no ONNX symbolic.
Six of the seven scripts the Makefile invokes did not exist, so every entry point
failed at the first command -- including the README's own Quick Start. The Python
underneath was fine; only the shell wrappers were missing.
00_fetch_calib_scenes.sh pull a diverse InternData-N1 subset (gated dataset)
01_repackage.sh InternVLA -> stock Qwen2.5-VL System 2
02_quantize.sh one scheme x strategy, through the validity gate
04_export_system1.sh traj_dit + memory block -> BF16 engines
05_verify.sh both acceptance gates
benchmark retargeted at 06_measure.sh, which exists and takes
the engine parent directory via environment, not flags
05_verify.sh --system1 runs the two-stage split, so it takes PYTHON_PT and PYTHON_TRT:
InternNav wants transformers 4.x and the TensorRT bindings ship for Python 3.12 where
transformers is 5.x, and nothing reconciles the two in one interpreter.
Exercised rather than assumed. The System 2 gate reproduces the README's 0.991861 on
the FP8 engine; the System 1 gate reports 0.999661 and exits 0; a missing checkpoint
and nvfp4 x s3 both fail with their reason.
Also fixes two README flags that never existed (--calib_data, --max_seq_len; the real
one is --calib_data_root) and drops the unassigned-owner line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
internvla-n1-dualvlnrecipe: FP8 quantization + TensorRT deployment on Jetson ThorAdds a complete recipe for InternVLA-N1-DualVLN, a dual-system vision-language
navigation model, targeting Jetson Thor (sm_110, JetPack 7.1, TensorRT 10.13.3.9).
27 commits,
flake8clean under the repo config.The model is two systems: System 2 is a Qwen2.5-VL-7B planner (the part that gets
quantized), System 1 is a NextDiT diffusion trajectory head plus a memory block
(DepthAnythingV2 + MemoryEncoder + QFormer). They are bridged by
z_latents— thelast-layer hidden states of 4 TRAJ tokens through a host-side norm and
cond_projector.That bridge, not text quality, is the acceptance gate. A checkpoint can produce fluent
captions and still be useless for navigation; NVFP4 does exactly that. Every scheme here is
judged on
z_latentscosine > 0.99, not on fluency.The design decision that shapes the recipe
The pipeline repackages first: pure safetensors surgery drops the 8 System-1 prefixes and
rewrites
config.jsontomodel_type: qwen2_5_vl. After that step, quantize / export /build / verify all operate on a stock Qwen2.5-VL checkpoint and never import InternNav — so
the whole System-2 path is an ordinary VLM flow a reader can follow without knowing InternVLA.
INTERNNAV_PATH?Results
All measured on one Jetson Thor, batch 1, idle GPU.
System 2 — Qwen2.5-VL-7B planner (quantized)
KV stays FP16 everywhere: NVFP4 KV needs
sm100f(datacenter Blackwell) and Thor is sm110.The vision tower stays BF16 under
s1; quantizing it iss3/s4and is FP8-only, becausethe ViT MLP
intermediate_sizeis 3420 and 3420 / 16 = 213.75 does not divide by the NVFP4block size. The scheme × strategy matrix in
configs/schemes.yamlrejects the impossiblecombinations early, with that arithmetic in the error text.
System 1 — NextDiT head + memory block (BF16)
traj_dit, one diffusion stepgenerate_trajbaselineUpstream InternNav ships no ONNX or TensorRT path at all — this conversion is entirely
the recipe's. It gives System 1 a 2.75× speedup at identical output. PyTorch latency is
flat in
num_sample_trajs(175.4 / 175.6 / 173.1 ms at 32 / 4 / 1), so the head islaunch-bound, which is why moving it to engines pays.
Both systems, one planning step
Deployed weights for the recommended configuration: 9.16 GB, against 15.7 GB unquantized.
Findings worth reading
FP8 is the recommended scheme. 1.86× smaller, 1.65×/1.79× faster, bridge at 0.9919, and
the median waypoint error does not get worse (27.05 → 22.51 px, which is within the spread of
a 42-sample set and should be read as "unchanged", not as a gain from quantization).
NVFP4's loss is in the engine, not in the quantization. Decomposed: weights 0.012 +
activations 0.009 + engine 0.048. Measured directly by running the fake-quant model and
the engine in one process against the same inputs — engine vs its own fake quant is 0.998256
for FP8 and 0.986790 for NVFP4, 8× more error. A
maxBatchSize 2build gives an identical0.986790, which refutes the batch-1 CASK-miscompile hypothesis; the residual is inherent to
the NVFP4 kernels here. This is also why AWQ, local-Hessian and QAT all failed to help: they
attack quantization quality (0.021) while the binding constraint is the engine (0.048).
FP8 on System 1 was measured and rejected. It works — 328/328 FP8 Q/DQ pairs in
traj_dit, 160/160 in the memory block, 1.55× smaller, 20 % faster on the diffusion loop —and it is still the wrong trade: 62 MB is 0.7 % of deployed weights, 12 ms is 1.7 % of a
planning step, while mean waypoint deviation goes 0.0032 → 0.0198, a 6× increase.
Splitting it shows why: quantizing
traj_ditalone already costs 0.9997 → 0.9880, becauseits per-step error (0.999508 → 0.997811) compounds across 10 sampler steps.
Domain-specific calibration bought nothing. Held-out
z_latentscame out 0.99143 withgeneric
cnn_dailymailtext versus 0.99146 with a VLN set — equal within noise. An earlierapparent gain was overlap between the calibration and probe sets. The VLN loader ships anyway
because it is the honest default for a navigation model, but the README says plainly not to
expect accuracy from it.
nvfp4_local_hessianis a silent no-op on this model — byte-identical output to thedefault preset, 0 of 6,422,528 bytes differ.
Verification
Two acceptance gates, both exercised in CI-able form rather than assumed:
make verify-latents→ z_latents cosine vs a PyTorch reference. Reproduces0.991861 on the FP8 engine.
make verify-system1→ trajectory parity. Reports 0.999670 and exits 0.The System-1 check runs in two stages under two interpreters, and that is not incidental:
InternNav targets transformers 4.x while the TensorRT bindings ship for Python 3.12 where
transformers is 5.x, so no single environment has both. Stage A writes the PyTorch
reference's inputs and outputs to a
.pt; stage B feeds the engines those same tensors, sothe comparison stays exact.