LLMDet is now available in 🤗 Transformers

[NielsRogge]

Hi,

The LLMDet model is officially integrated in the Hugging Face Transformers library together with the MM-Grounding DINO model 🤗 It enables easy inference in a few lines of code.

Resources

• Models: https://huggingface.co/collections/iSEE-Laboratory/llmdet-688475906dc235d5f1dc678e
• Docs: https://huggingface.co/docs/transformers/main/en/model_doc/mm-grounding-dino

I recommend trying out the zero-shot object detection arena where you can compare LLMDet against OWLv2 and Grounding DINO: https://huggingface.co/spaces/ariG23498/zero-shot-od.

Happy prompting!

Related to #7 #24

[lsch0lz]

@NielsRogge are there any plans to provide an ONNX version of this model? I tried to convert it using Optimum, but it seems that the MM-Grounding-Dino architecture is not implemented, so conversion is not possible.

The author mentioned that he is not capable of doing it, but this would be crucial for industrial usage.

optimum-cli export onnx --model iSEE-Laboratory/llmdet_base llmdet_base/
/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/torch/onnx/_internal/registration.py:162: OnnxExporterWarning: Symbolic function 'aten::scaled_dot_product_attention' already registered for opset 14. Replacing the existing function with new function. This is unexpected. Please report it on https://github.com/pytorch/pytorch/issues.
  warnings.warn(
config.json: 2.70kB [00:00, 1.56MB/s]
Traceback (most recent call last):
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1218, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 914, in __getitem__
    raise KeyError(key)
KeyError: 'mm-grounding-dino'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lukasscholz/repositorys/privat/test/.venv/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/optimum/commands/optimum_cli.py", line 208, in main
    service.run()
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/optimum/commands/export/onnx.py", line 276, in run
    main_export(
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/optimum/exporters/onnx/__main__.py", line 310, in main_export
    config = AutoConfig.from_pretrained(
  File "/Users/lukasscholz/repositorys/privat/test/.venv/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1220, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `mm-grounding-dino` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

[NielsRogge]

Could you open an issue at https://github.com/huggingface/optimum/issues?