Using customvision onnx on Nvidia (TensorRT/Deepstream)

[amarrmb]

I am experimenting to get a customvision model (exported as ONNX) to run on a nvidia device using deepstream SDK (TensorRT engine for accelerating ONNX).

1. I was able to follow the steps in this repository and train a model using CustomVision.AI and run it on Nvidia device.
   This works great for a object detection type of model.

2. When i am using a classification model trained using customvision and exported as ONNX, i get several warnings during conversion:
   [W1]:Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
   [W2]:Calling isShapeTensor before the entire network is constructed may result in an inaccurate result.

3. Even though the TensorRT model gets generated, its not working and i get incorrect results (possibly due to the weights cast down).

So wanted to check if Microsoft has tried this setup? Any information on how customvision uses ONNX internally?

[stevenlix]

TensorRT doesn't support INT64 and will convert INT64 weights to INT32. Usually we don't see accuracy loss due to the conversion, but it depends on how big the INT64 value is. You might also try OnnxRuntime for onnx model inferencing. You can run the model by different execution providers to check the latency and accuracy.