Slow inference speed

[stephenleo]

Training time is definitely faster!
But exactly following the documentation from the readme, I am getting inference times of ~2.8s on a T4 GPU on Google Colab which is much slower than SetFit (~20ms).
What am I doing wrong?

%%timeit
print(classifier("I love this package!"))

2.82 s ± 796 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[elronbandel]

Thanks for noting. We tested it with the batched inference code that is part of the training code (trainer.evaluate()). Will check why this is slower with the pipeline we might did something wrong.

[nishkalavallabhi]

I noticed much slower inference speed than setfit in the batched inference mode too.

[elronbandel]

We didnt really get into it in the paper (correct me if im wrong), but i think the current version of fast-fit is not faster than set-fit, yet on a100-80gb we managed to produce results in the same ballpark as other encoders.

If you want inference speed that is on par with set-fit (and be even faster) use inference_type=clf, this is esentially using regular classification head that was trained with fast-fit loss. Same training time, and infernce speed of regular classification head. We currently working on a technical report claiming this inference has similar accuracy on FewMany as fast-fit while having faster inference.

You can adjust the code here in order to save the fast-fit model with its classification head as a regular RobertaForSequenceClassification of huggingface transformers. This will allow users to save the fast-fit model with the classificaiton head and even push them to huggingface, and finally ofcourse enjoy the fast inference.

If you are intereseted on contributing to this would be amazing @stephenleo @nishkalavallabhi, also if you have any other inputs or concerns.

[stephenleo]

Thanks for the update @elronbandel. The paper has this statement in section 5.3

It’s important to note that, due to the model’s size and the length of the few-shot prompt, inference time can be slow, with
throughput exceeding 1 second per input, in contrast to about 1 millisecond with FastFit

I can take a look at the model saving code. Can you pls link to where I should start looking?

[elronbandel]

Thanks @stephenleo ! The FastFit model inmodeling.py has a base model under self.encoder and a fully connected classifcation head under self.clf. if we could somehow link them together and save it with the appropirate ModelXForSequenceClassification model it should work.

[hyx07]

In the inference code we need to move the model to GPU?
model = FastFit.from_pretrained("fast-fit")
model.to(torch.device('cuda'))
otherwise my GPU shows no sign of working, and the inference time does decrease a lot adding the last code.