Description
All inference produces garbage or repetitive output (token repetition collapse) on Jetson AGX Thor across all tested model formats (FP16, BF16, NVFP4).
The AttentionPlugin CuTe DSL FMHA kernel appears to load successfully, but inference is incorrect from the very first generated token. Every tested model eventually degenerates into repetitive or nonsensical text.
Additionally, the runtime reports:
This suggests either:
the prompt is not being tokenized correctly before execution,
an incorrect FMHA kernel is being selected,
or the attention computation is incorrect on SM110.
Steps to Reproduce
1. Build TensorRT Edge-LLM
cmake .. \
-DBUILD_PYTHON_BINDINGS=ON \
-DTRT_PACKAGE_DIR=/usr \
-DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-12 \
-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda \
-DCUDA_DIR=/usr/local/cuda
make -j$(nproc)
2. Export Qwen3-1.7B
tensorrt-edgellm-export
Qwen/Qwen3-1.7B
/models/qwen3-1.7b-onnx
3. Build engine
EDGELLM_PLUGIN_PATH=/TensorRT-Edge-LLM/build/libNvInfer_edgellm_plugin.so
./build/examples/llm/llm_build
--onnxDir ~/models/qwen3-1.7b-onnx/llm
--engineDir /models/qwen3-1.7b-engine
--maxInputLen 4096
--maxKVCacheCapacity 8192
--maxBatchSize 1
4. Run inference
EDGELLM_PLUGIN_PATH=/TensorRT-Edge-LLM/build/libNvInfer_edgellm_plugin.so
./build/examples/llm/llm_inference
--engineDir ~/models/qwen3-1.7b-engine
--inputFile input.json
--outputFile output.json
--dumpOutput
Input JSON
{
"batch_size": 1,
"temperature": 0.7,
"top_p": 0.9,
"top_k": 50,
"max_generate_length": 100,
"requests": [
{
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
]
}
]
}
Expected Behavior
The model should generate a coherent response, for example:
The capital of France is Paris.
Actual Behavior
The very first generated token is already corrupted, and generation quickly collapses into repetitive garbage, for example:
CTIONSitorndataوبر鸠icielوبرnéeوبروبر几盟几[RNR...
Generation continues until max_generate_length is reached.
Models Tested
The issue reproduces consistently with all tested models and formats:
Qwen3-1.7B FP16 (exported from Hugging Face)
Qwen3-1.7B BF16 (--dtype bf16)
nvidia/Qwen3-8B-NVFP4 (pre-quantized)
Relevant Logs
[INFO] AttentionPlugin FMHA path: CuTe DSL FMHA
[INFO] CuTe DSL LLM FMHA kernel modules loaded (FP16 + FP8 + paged)
[INFO] CuTe DSL FMHA kernel loaded for SM101
[INFO] Batch 0 finished, total tokens=100, reason: max_length
[INFO] prompt_tokens: 0
Notable observations:
The tokenizer appears to initialize successfully.
prompt_tokens is reported as 0.
The FMHA plugin reports SM101, while the target hardware is SM110.
Environment
| Component |
Version |
| Platform |
NVIDIA Jetson AGX Thor |
| JetPack |
7.1 (L4T R38.2.2) |
| Architecture |
aarch64 |
| GPU |
SM110 |
| Device Memory |
122 GiB |
| TensorRT Edge-LLM |
commit 7f061f2 (v0.9.1) |
| CUDA |
13.0 |
| TensorRT |
10.13.3 |
| Build Type |
Release |
| Host Compiler |
GCC 13.3.0 |
| CUDA Host Compiler |
GCC 12 |
| TRT_PACKAGE_DIR |
/usr |
| EMBEDDED_TARGET |
jetson-thor |
Additional Notes
The behavior is identical across FP16, BF16, and NVFP4 models, suggesting that quantization is unlikely to be the root cause.
The two observations that appear most suspicious are:
prompt_tokens: 0, indicating the prompt may not be reaching the model correctly.
CuTe DSL FMHA kernel loaded for SM101 on an SM110 device, which may indicate an incorrect kernel selection or architecture detection issue.
Please let me know if additional logs, engine artifacts, tokenizer outputs, or debug traces would be helpful.
Description
All inference produces garbage or repetitive output (token repetition collapse) on Jetson AGX Thor across all tested model formats (FP16, BF16, NVFP4).
The AttentionPlugin CuTe DSL FMHA kernel appears to load successfully, but inference is incorrect from the very first generated token. Every tested model eventually degenerates into repetitive or nonsensical text.
Additionally, the runtime reports:
prompt_tokens: 0even though the tokenizer loads successfully.CuTe DSL FMHA kernel loaded for SM101although the target GPU is SM110.This suggests either:
the prompt is not being tokenized correctly before execution,
an incorrect FMHA kernel is being selected,
or the attention computation is incorrect on SM110.
Steps to Reproduce
1. Build TensorRT Edge-LLM
2. Export Qwen3-1.7B
3. Build engine
4. Run inference
Input JSON
Expected Behavior
The model should generate a coherent response, for example:
Actual Behavior
The very first generated token is already corrupted, and generation quickly collapses into repetitive garbage, for example:
Generation continues until
max_generate_lengthis reached.Models Tested
The issue reproduces consistently with all tested models and formats:
Qwen3-1.7B FP16 (exported from Hugging Face)
Qwen3-1.7B BF16 (
--dtype bf16)nvidia/Qwen3-8B-NVFP4 (pre-quantized)
Relevant Logs
Notable observations:
The tokenizer appears to initialize successfully.
prompt_tokensis reported as 0.The FMHA plugin reports SM101, while the target hardware is SM110.
Environment
Additional Notes
The behavior is identical across FP16, BF16, and NVFP4 models, suggesting that quantization is unlikely to be the root cause.
The two observations that appear most suspicious are:
prompt_tokens: 0, indicating the prompt may not be reaching the model correctly.CuTe DSL FMHA kernel loaded for SM101on an SM110 device, which may indicate an incorrect kernel selection or architecture detection issue.Please let me know if additional logs, engine artifacts, tokenizer outputs, or debug traces would be helpful.