Skip to content

Bug: Garbage / repetitive output on Jetson AGX Thor (SM110) across all quantization formats using CuTe DSL FMHA #164

Description

@SatishSurya1

Description

All inference produces garbage or repetitive output (token repetition collapse) on Jetson AGX Thor across all tested model formats (FP16, BF16, NVFP4).

The AttentionPlugin CuTe DSL FMHA kernel appears to load successfully, but inference is incorrect from the very first generated token. Every tested model eventually degenerates into repetitive or nonsensical text.

Additionally, the runtime reports:

  • prompt_tokens: 0 even though the tokenizer loads successfully.

  • CuTe DSL FMHA kernel loaded for SM101 although the target GPU is SM110.

This suggests either:

  • the prompt is not being tokenized correctly before execution,

  • an incorrect FMHA kernel is being selected,

  • or the attention computation is incorrect on SM110.


Steps to Reproduce

1. Build TensorRT Edge-LLM

cmake .. \
    -DBUILD_PYTHON_BINDINGS=ON \
    -DTRT_PACKAGE_DIR=/usr \
    -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-12 \
    -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda \
    -DCUDA_DIR=/usr/local/cuda

make -j$(nproc)

2. Export Qwen3-1.7B

tensorrt-edgellm-export 
Qwen/Qwen3-1.7B
/models/qwen3-1.7b-onnx

3. Build engine

EDGELLM_PLUGIN_PATH=/TensorRT-Edge-LLM/build/libNvInfer_edgellm_plugin.so 
./build/examples/llm/llm_build
--onnxDir ~/models/qwen3-1.7b-onnx/llm
--engineDir /models/qwen3-1.7b-engine
--maxInputLen 4096
--maxKVCacheCapacity 8192
--maxBatchSize 1

4. Run inference

EDGELLM_PLUGIN_PATH=/TensorRT-Edge-LLM/build/libNvInfer_edgellm_plugin.so 
./build/examples/llm/llm_inference
--engineDir ~/models/qwen3-1.7b-engine
--inputFile input.json
--outputFile output.json
--dumpOutput

Input JSON

{
"batch_size": 1,
"temperature": 0.7,
"top_p": 0.9,
"top_k": 50,
"max_generate_length": 100,
"requests": [
{
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
]
}
]
}

Expected Behavior

The model should generate a coherent response, for example:

The capital of France is Paris.

Actual Behavior

The very first generated token is already corrupted, and generation quickly collapses into repetitive garbage, for example:

CTIONSitorndataوبر鸠icielوبرnéeوبروبر几盟几[RNR...

Generation continues until max_generate_length is reached.


Models Tested

The issue reproduces consistently with all tested models and formats:

  • Qwen3-1.7B FP16 (exported from Hugging Face)

  • Qwen3-1.7B BF16 (--dtype bf16)

  • nvidia/Qwen3-8B-NVFP4 (pre-quantized)


Relevant Logs

[INFO] AttentionPlugin FMHA path: CuTe DSL FMHA
[INFO] CuTe DSL LLM FMHA kernel modules loaded (FP16 + FP8 + paged)
[INFO] CuTe DSL FMHA kernel loaded for SM101
[INFO] Batch 0 finished, total tokens=100, reason: max_length
[INFO] prompt_tokens: 0

Notable observations:

  • The tokenizer appears to initialize successfully.

  • prompt_tokens is reported as 0.

  • The FMHA plugin reports SM101, while the target hardware is SM110.


Environment

Component Version
Platform NVIDIA Jetson AGX Thor
JetPack 7.1 (L4T R38.2.2)
Architecture aarch64
GPU SM110
Device Memory 122 GiB
TensorRT Edge-LLM commit 7f061f2 (v0.9.1)
CUDA 13.0
TensorRT 10.13.3
Build Type Release
Host Compiler GCC 13.3.0
CUDA Host Compiler GCC 12
TRT_PACKAGE_DIR /usr
EMBEDDED_TARGET jetson-thor

Additional Notes

The behavior is identical across FP16, BF16, and NVFP4 models, suggesting that quantization is unlikely to be the root cause.

The two observations that appear most suspicious are:

  1. prompt_tokens: 0, indicating the prompt may not be reaching the model correctly.

  2. CuTe DSL FMHA kernel loaded for SM101 on an SM110 device, which may indicate an incorrect kernel selection or architecture detection issue.

Please let me know if additional logs, engine artifacts, tokenizer outputs, or debug traces would be helpful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions