(Tutorial) - Serve recipes for 5090

[CommanderTurtle]

Have a 5090 and ultraspecced gaming rig? I've been modeling and exploring variants for a while now. Worthwhile 5090 vLLM flags here.

Just edit serve command as needed:

INT4 (Agents A1):

(click to expand)

CUDA_VISIBLE_DEVICES=0
vllm serve /path/to/.cache/huggingface/hub/models--compute1--Agents-A1-GPTQ-INT4-Sym/snapshots/somestring/
--served-model-name compute1/Agents-A1-GPTQ-INT4-Sym
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--max-model-len 196608
--gpu-memory-utilization 0.87
--kv-cache-dtype fp8_e5m2
--max-num-seqs 2
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enable-auto-tool-choice
--trust-remote-code

This one is - compute1/Agents-A1-GPTQ-INT4-Sym

NVFP4 (Agents A1):

(click to expand) Slightly beefier, nvfp4 over int4. My current serve command

CUDA_VISIBLE_DEVICES=0
VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve /path/to/.cache/huggingface/hub/models--cyankiwi--Agents-A1-AWQ-NVFP4/snapshots/somestring/
--served-model-name cyankiwi/Agents-A1-AWQ-NVFP4
--host 0.0.0.0
--port 8000
--moe-backend marlin
--tensor-parallel-size 1
--max-model-len 172032
--gpu-memory-utilization 0.88
--kv-cache-dtype fp8
--max-num-seqs 2
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enable-auto-tool-choice
--trust-remote-code

Non-negotiable moe-backend and flashinfer sampler env var - discovered from (this model - link to highlight) - where it's shown the env vars needed for compressed-tensors on consumer blackwell (tl;dr sm120 = 5090 ... sm121 = RTX6000+, datacenter blackwell)

This one is - cyankiwi/Agents-A1-AWQ-NVFP4

INT4 (AgentWorld):

Second model - not as-perfect in results, but still qwen-based and godly. Some abliterated-s had been made and worth researching if needed (bigger community): ↓↓↓

(click to expand)

CUDA_VISIBLE_DEVICES=0
vllm serve groxaxo/Qwen-AgentWorld-35B-A3B-GPTQ-Pro-Int4
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--max-model-len 131072
--gpu-memory-utilization 0.73
--dtype auto
--max-num-seqs 2
--language-model-only
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enable-auto-tool-choice
--trust-remote-code

The second one is - groxaxo/Qwen-AgentWorld-35B-A3B-GPTQ-Pro-Int4

Specs: 64gb ddr5, and 5090.

---

get the cude on wsl? (generic debian-fork instructions):

• Cuda install made easy, not reliant on .debs..

sudo apt update
sudo mkdir -p /etc/apt/keyrings
# Correct Add-Cuda-Repo Method:
deb [signed-by=/etc/apt/keyrings/cuda-wsl.gpg] https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /
# Verify your keyring file exists
ls -l /etc/apt/keyrings/cuda-wsl.gpg
# If it’s missing, recreate it..
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub \
| sudo gpg --dearmor -o /etc/apt/keyrings/cuda-wsl.gpg
# Fix the repo file cleanly
sudo nano /etc/apt/sources.list.d/cuda-wsl.list
# Replace the contents with:
deb [signed-by=/etc/apt/keyrings/cuda-wsl.gpg] https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /
# Save → exit → then run:
sudo apt update
# This installs the **WSL‑optimized CUDA toolkit**, not the generic Linux one:
sudo apt install -y cuda
# Policy against SHA1 warning?
# https://neilzone.co.uk/2026/01/dealing-with-apts-warning-policy-will-reject-signature-within-a-year-see---audit-for-details/
# "The “correct” way of solving this is for the repository provider to update their signing key to something which is compliant. 
# However, I have no control over what a repository provider does, or when they will do it. For instance, the warning message above 
# suggests to me that I will have a problem on 1 February 2026, so under a month away. I can suppress this warning - and tell apt to accept 
# the key - by adding -o APT::Key::GPGVCommand=1 to apt update: apt update -o APT::Key::GPGVCommand=1 Or, to avoid having to add 
# that each time, I can added a slightly-tweaked version of it to an apt config file, in /etc/apt/apt.conf.d. For instance, I can put this into 
# /etc/apt/apt.conf.d/25keypolicy: APT::Key::GPGVCommand "1"; Hopefully though, repository providers will update their keys (which will 
# then need re-importing)."
sudo nano /etc/apt/apt.conf.d/25keypolicy
# Add line, save and exit:
APT::Key::GPGVCommand "1";

nvidia-smi

Thank me later bois.