is the Laptop RTX5090 also covered?

[rs38]

only lacks some VRAM, bandwidth and cores, same architecture afaik

[MirkoCovizzi]

I just ordered a laptop to try this out, planning to hyper-optimize a fork of this. Hopefully it should match or beat the RTX 3090 desktop version of ninfer.

[rs38]

cool, pls share, I have such a Laptop to test as well...so lazy to just try it out right now :(

[MirkoCovizzi]

cool, pls share, I have such a Laptop to test as well...so lazy to just try it out right now :(

Point AI to this repo and ask it to optimize it for your laptop.

[tmark00]

I did some testing on a Razer Blade 16 (2025), RTX 5090 Laptop 24 GB, Ryzen AI 9 HX 370, running NInfer under WSL2 Ubuntu 24.04 / CUDA 13.3 / NVIDIA driver 610.88.

With Qwen3.8-27B groupwise-int:

MTP0: 31.4 tok/s
MTP3 (--draft-tokens 3 --lm-head-draft): ~51–54 tok/s
exact same MTP3 workload: 36.2 tok/s in Razer Silent mode vs 51.7 tok/s in HyperBoost + cooler, so laptop power profile alone changes decode performance by ~43%.

The NVFP4 artifact behaved much worse on this GPU:

NVFP4 + MTP3: only 16.6 tok/s on the same prompt
NVFP4 without MTP currently crashes during engine setup with:
gdn_input_proj_conv_snapshot: tensor operand overlaps live workspace
the crash is reproducible with and without CUDA Graphs.

I also tested the OpenAI-compatible server with a real ~12–15k-token prompt containing 26 tools. At 32k context CUDA Graph planning failed (~220 MiB actually needed vs ~82 MiB planned), and prefill-chunk=1024 hit cudaErrorCooperativeLaunchTooLarge. Using --no-cuda-graph --prefill-chunk 256 made it stable, with real decode around 36–43 tok/s.

So NInfer definitely runs on the RTX 5090 Laptop, but the current kernels/planner do not seem well tuned for it yet. Groupwise + MTP3 is clearly the best NInfer configuration I found. For comparison, current llama.cpp Q5_K_M + MTP on the same laptop reaches roughly 55–57 tok/s at 32k context, so at the moment NInfer does not give me a practical speed advantage.

Happy to run additional laptop-specific benchmarks if useful.

[MirkoCovizzi]

Not what I have experienced.

This is with Qwen 3.6, will update with Qwen 3.8 (both NVFP4, a bit pointless to use anything else given the unfair advantage of Blackwell arch):

As estimated, looks exactly like 50% of perf of desktop 5090.

[dmitrygalanov]

@MirkoCovizzi any updates on your experiment? Do you plan to make PR?

[rs38]

using NVFP4 is the main reason looking for llama.cpp alternatives, I failed using vLLM (using WSL).
also most NVFP4-checkpoints seem not to leave much VRAM room for K-V as being much bigger than normal Q4 ggufs.

[Neroued]

NVFP4 + MTP3: only 16.6 tok/s on the same prompt
NVFP4 without MTP currently crashes during engine setup with:
gdn_input_proj_conv_snapshot: tensor operand overlaps live workspace
the crash is reproducible with and without CUDA Graphs.

fixed yesterday

[Neroued]

I raised qwen3.8 nvfp4 artifact's size to 20GB for better accuracy (not evaluated yet), so it's too large for 24gb vram.

[rs38]

could there be a point to decrease nvfp4 artifact size instead of using int q4? anyone tried? (admitting that I have no clue how to actually do the quantisation)

[MirkoCovizzi]

@MirkoCovizzi any updates on your experiment? Do you plan to make PR?

The result is that I switched to a smaller NVFP4 of Qwen 3.8 27B (https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer) combined with #35, purely based on an informed hunch that we can be a bit aggressive with quantization and the model will be forgiving. Unfortunately, the E8 lattice Hadamard rotation does not work with current NVFP4 with FP8 attention heads (the 20GB quant), only rk8v4 works well with this NVFP4. It looks like reasoning starts falling apart with that KV quantization, probably because of the FP8 attention heads not playing well with the E8 lattice.

Ostfralla's NVFP4 + E8 lattice KV -> full context size of 262144 plus a leftover slack of ~1GiB (this is important, there is potential for squeezing in the vision heads). I'm currently testing it with GPQA Diamond just to "quickly" check the quality of that quant, but looking at various threads it seems like this model is very resilient to quantization (some users report Q4_K_M performing on par with Q8, no one knows why). This model is truly a lightning in a bottle.

Speeds I'm getting are a bit worse than Qwen 3.6 27B for some reason (despite same size), but still very good: ~70-80 tok/s decode (with peaks of ~100-110 in code), ~3-4k tok/s prefill. Feels good to use, reasoning traces look intelligent and clean, tool calling looks consistent, understands user's intent, TTFT extremely responsive (feels better than cloud).

These numbers, especially decode, are bound to get even better once we add adaptive MTP (ggml-org/llama.cpp#27210) and n-gram mod (https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md#n-gram-mod-ngram-mod). These are all low hanging fruit that pay off in long running agentic workflows where the model thinks code it has seen before.

Overall, at least for me personally, looks like the RTX 5090 laptop version is a sleeper build at the moment if you can find it at a good price (randomly found it 2 days ago for 3799 eur on Amazon new; on par with current Strix Halo prices, but better perf in ComfyUI and Qwen), and will get even better with time (Qwen3.9 27B, better quantization algos, etc.).

I will consider opening a PR or publishing a fork once I have confident results, especially benchmarks, which take a long time.

[Don-Moahskarton]

I have a 5090 laptop (XMG Zenith) as well, @MirkoCovizzi feel free to ping me if you want testing on a different laptop. for now I'll try this Ostfralla quant :)
Any advice on best tuning places for this repo appreciated, and I can report as well on my experience

[tmark00]

Nice to see that you @MirkoCovizzi independently ended up with essentially the same configuration I have been testing.

I can confirm similar results on a Razer Blade 16 (2025), RTX 5090 Laptop 24 GB, Ryzen AI 9 HX 370, WSL2 / CUDA 13.3.

I switched from the newer official ~20 GB Qwen3.8-27B NVFP4 artifact to Ostfralla's smaller NVFP4 as well, using a compatible pre-official NInfer runtime.

With 32k context, INT8 KV, MTP3, CUDA Graphs and --prefill-chunk 256, I get:

~16.8 GiB GPU weights
~2.8 GiB free VRAM after startup
~59 tok/s in a simple 1k-token synthetic test
real agent workload with 26 tools and 12–19k prompts: typically ~64–79 tok/s decode
first 12.5k-token prompt: ~2899 tok/s prefill
prompt-cache reuse can bring TTFT down to ~0.1–0.8 s
a 4342-token generation sustained ~66.3 tok/s

So your ~70–80 tok/s decode / 3–4k tok/s prefill numbers look very consistent with what I am seeing independently.

I have not tested PR #35 / E8 lattice KV yet, so I am currently limited to 32k context, but your 262k result is very interesting. It also reinforces the impression that the smaller Ostfralla NVFP4 is a particularly good fit for the 24 GB laptop 5090: the newer official NVFP4 leaves much less VRAM headroom and, in my testing, MTP performance was substantially worse.

At this point I would also call the RTX 5090 Laptop a surprisingly strong NInfer platform. The main constraint seems to be VRAM layout rather than raw compute.

[MirkoCovizzi]

Nice to see that you @MirkoCovizzi independently ended up with essentially the same configuration I have been testing.

I can confirm similar results on a Razer Blade 16 (2025), RTX 5090 Laptop 24 GB, Ryzen AI 9 HX 370, WSL2 / CUDA 13.3.

I switched from the newer official ~20 GB Qwen3.8-27B NVFP4 artifact to Ostfralla's smaller NVFP4 as well, using a compatible pre-official NInfer runtime.

With 32k context, INT8 KV, MTP3, CUDA Graphs and --prefill-chunk 256, I get:

~16.8 GiB GPU weights ~2.8 GiB free VRAM after startup ~59 tok/s in a simple 1k-token synthetic test real agent workload with 26 tools and 12–19k prompts: typically ~64–79 tok/s decode first 12.5k-token prompt: ~2899 tok/s prefill prompt-cache reuse can bring TTFT down to ~0.1–0.8 s a 4342-token generation sustained ~66.3 tok/s

So your ~70–80 tok/s decode / 3–4k tok/s prefill numbers look very consistent with what I am seeing independently.

I have not tested PR #35 / E8 lattice KV yet, so I am currently limited to 32k context, but your 262k result is very interesting. It also reinforces the impression that the smaller Ostfralla NVFP4 is a particularly good fit for the 24 GB laptop 5090: the newer official NVFP4 leaves much less VRAM headroom and, in my testing, MTP performance was substantially worse.

At this point I would also call the RTX 5090 Laptop a surprisingly strong NInfer platform. The main constraint seems to be VRAM layout rather than raw compute.

One suggestion I would have is to try MTP k=5 instead of k=3, I've been seeing better perf, validated with the bench. ~90 tok/s with speeds above ~140 tok/s while writing code.

@Neroued Is it ok if we keep this thread going for specialized discussion over the RTX 5090 laptop variant, or should we move to somewhere else? I would like to leave the door open to upstreaming contributions since the chip arch is effectively the same, just cut down. Looking forward to hear your opinion about this.