Future Roadmap: Native LibreYOLO Model Architecture

[gokulkato2001]

Hi, I have a question regarding the future roadmap of LibreYOLO.

Are there any plans or ideas to develop a native LibreYOLO model/architecture in the future, similar to how Ultralytics has its own YOLO model implementations?

Currently, LibreYOLO supports architectures such as YOLOX, YOLOv9, and RF-DETR under a unified API. I’m wondering whether the long-term goal is to continue supporting existing architectures, or whether there is a plan to build a LibreYOLO-native model that is fully developed and maintained within the project.

If there are any future plans or discussions around this, I’d be interested to know.

[gboeer]

What do you mean exactly?

The models in libreyolo are native allready. You can see them beeing defined here:

libreyolo/models

[gokulkato2001]

Thanks for the clarification.

What I meant is slightly different. I understand that the models are implemented natively within the LibreYOLO codebase.

My question is more about the architecture itself.

Currently, LibreYOLO provides a unified API/framework around existing architectures such as YOLOX, YOLOv9, and RF-DETR. Even though their implementations are maintained within libreyolo/models, the underlying model architectures originate from those respective projects.

What I’m asking is whether there are any plans or discussions to eventually design and develop a completely new object-detection architecture that is LibreYOLO-native from the ground up, rather than integrating or adapting an existing architecture.

For example, something like:

YOLOX / YOLOv9 / RF-DETR → existing architectures integrated into LibreYOLO

vs.

LibreYOLO → its own independently designed detection architecture

I hope that clarifies what I meant.

[EHxuban11]

Hello @gokulkato2001 ,

Yes!

I want to create a model called LibreYOLO27 this year. There needs to be funding for the training runs though, so I'll probably start looking for sponsors for this project. LibreYOLO27 will be a SOTA CNN model, exportable to every relevant format and target hardware. It will include detection, segmentation, OBB, classification and keypoint variants. The goal of LibreYOLO27 is to generalize well when finetuned in different datasets, as measured with the RF100 benchmark. COCO mAP will be a secondary target.

[EHxuban11]

Any ideas welcome @gokulkato2001 : )

[thi23459112]

Hello @gokulkato2001 ,

Yes!

I want to create a model called LibreYOLO27 this year. There needs to be funding for the training runs though, so I'll probably start looking for sponsors for this project. LibreYOLO27 will be a SOTA CNN model, exportable to every relevant format and target hardware. It will include detection, segmentation, OBB, classification and keypoint variants. The goal of LibreYOLO27 is to generalize well when finetuned in different datasets, as measured with the RF100 benchmark. COCO mAP will be a secondary target.

I have one suggestion: besides building a truly open-source version of Ultralytics, I think we should also think more deeply about the productization and deployment side.

For example, if the goal is to deploy the system on Jetson devices, we will likely need to consider NVIDIA DeepStream. DeepStream itself has multiple versions and different environment combinations, so simply maintaining compatibility across different versions could become a significant cost. However, I think this is something we should consider early in the planning stage, because based on my experience in the industry, there is a real demand for this type of productization, and I have already seen similar products deployed in production.

Also, besides the tasks you mentioned, such as object detection, segmentation, and OBB, I would strongly recommend including object tracking as well. Tracking is also a very common approach in industrial applications.

Different tracking algorithms have different requirements for Re-ID. Some require a Re-ID model, while others do not. If we eventually want to deploy the solution through DeepStream, this could introduce additional integration and compatibility challenges, so I think it would be better to take this into consideration from the beginning.

Another point is that DeepStream deployment typically relies on native libraries such as .so files and executes the underlying components in C/C++, which generally provides better runtime performance and efficiency compared with Python. Therefore, if productization and edge deployment are part of the long-term goal, I think it would be beneficial to design the underlying architecture with this deployment scenario in mind from the beginning, rather than having to make major architectural changes later.

For the long-term roadmap, it may also be worth considering 3D Bounding Box (3D BBox) detection. It may not need to be a priority in the first stage, but if we eventually want to expand toward more comprehensive 3D vision and intelligent edge applications, keeping the architecture flexible enough to support 3D tasks could be valuable.

[EHxuban11]

Hello @thi23459112 ,
Thank you very much for caring and for doing these suggestions.

Regarding 3D bounding box detection, I agree that it is a very interesting task to add. I saw the other issue you created recommending WildDet3D, and ported it to LibreYOLO: #845

I also took the initiative to add another 3D detection model: #846

Regarding the deepstreem integrations, there has already been some discussion in the past: #648
I didn't remember that it was with you, but yeah. It resulted on https://www.libreyolo.com/docs/export/deepstream
Which might very well be insuficient for real life deployments.

Regarding tracking, we already have tracking: https://www.libreyolo.com/docs/tasks/object-tracking
The supported techniques are bytetrack, botsort, ocsort, deepocsort
I'm sure it can be improved.

I agree that we can improve the deployment side, but we need concrete tasks. The problem I have is that I've never used deepstream at work, so I have very bad intuitions when building those solutions. Do you see gaps there that we could attack?

[thi23459112]

Hello @thi23459112 , Thank you very much for caring and for doing these suggestions.

Regarding 3D bounding box detection, I agree that it is a very interesting task to add. I saw the other issue you created recommending WildDet3D, and ported it to LibreYOLO: #845

I also took the initiative to add another 3D detection model: #846

Regarding the deepstreem integrations, there has already been some discussion in the past: #648 I didn't remember that it was with you, but yeah. It resulted on https://www.libreyolo.com/docs/export/deepstream Which might very well be insuficient for real life deployments.

Regarding tracking, we already have tracking: https://www.libreyolo.com/docs/tasks/object-tracking The supported techniques are bytetrack, botsort, ocsort, deepocsort I'm sure it can be improved.

I agree that we can improve the deployment side, but we need concrete tasks. The problem I have is that I've never used deepstream at work, so I have very bad intuitions when building those solutions. Do you see gaps there that we could attack?

I took a look at the link you provided. There are indeed several commonly used tracking methods listed there, but I believe there are actually more options worth considering.

I’d also like to share the DeepStream setup that I currently have deployed. In addition to NVIDIA’s built-in tracking methods, I have also integrated some custom tracking algorithms myself. These custom implementations are mainly based on algorithms from the open-source community. You could also use Claude to help locate the relevant source code and files if needed.

https://reurl.cc/90bxdd

My current application is mainly focused on traffic scenarios, and I’m using FastTracker. The main reason I chose FastTracker is that it is relatively lightweight and does not require additional Re-ID model weights, which puts less load on the edge device and makes it more suitable for long-term operation.

As for NVIDIA’s built-in NVDCF, its FPS performance is indeed very impressive, and it can maintain good performance even with multiple objects. However, in actual traffic scenarios, I found that its tracking performance is still slightly worse, so I decided not to use NVDCF and chose FastTracker instead.

During the actual deployment, I also discovered a small bug in the original ByteTrack /BotSort / FastTracker implementation that the original authors may not have noticed: it can cause a memory leak during long-term operation.

I only discovered this after running the system continuously for several days. As the memory keeps leaking, the FPS gradually decreases over time, and in severe cases, the system can eventually become completely stuck. For edge devices like ours, this can be a significant issue.

I have already fixed this issue. If everything looks good after your review, feel free to use my fixed version directly.

One thing I want to point out is that these tracking methods are currently still implemented in Python. I’m using pyds to communicate with Jetson / DeepStream.

This is different from NVIDIA’s built-in trackers. For example, NVDCF implements the tracking algorithm in C, so its FPS performance is much better. Even when there are dozens of objects in the frame, it can still maintain a stable FPS.

Therefore, if you have the capability to further optimize it, I would recommend rewriting the tracking algorithm in C / C++. This should make it more suitable for future deployment and further reduce the computational load on the edge device.

[EHxuban11]

Hello @thi23459112,

Thank you for the detailed follow-up and for the long-run feedback, that is very valuable.

On the memory leak in the original ByteTrack/BoT-SORT: our ports already prune removed tracks, so I don't think we carry that one, but if you can point at the exact spot in the upstream code you fixed, I'll double-check ours. If it's a different pattern, a short note on what grows would be enough.

On FastTracker: I checked the paper. It's a few HOTA points above ByteTrack on MOT17, never compared against BoT-SORT, and has no speed numbers, so I'm not adding it for now. If you need it you can plug your own tracker behind the detector.

For the C/C++ rewrite, I don't think it's significantly faster than python. I think that we would need to make some profiling to see if the tracking algos add any latency but a priori it should not. If you run the tests that show that the tracking algos are too slow on python, happy to work on that.

[thi23459112]

Hello @thi23459112,

Thank you for the detailed follow-up and for the long-run feedback, that is very valuable.

On the memory leak in the original ByteTrack/BoT-SORT: our ports already prune removed tracks, so I don't think we carry that one, but if you can point at the exact spot in the upstream code you fixed, I'll double-check ours. If it's a different pattern, a short note on what grows would be enough.

On FastTracker: I checked the paper. It's a few HOTA points above ByteTrack on MOT17, never compared against BoT-SORT, and has no speed numbers, so I'm not adding it for now. If you need it you can plug your own tracker behind the detector.

For the C/C++ rewrite, I don't think it's significantly faster than python. I think that we would need to make some profiling to see if the tracking algos add any latency but a priori it should not. If you run the tests that show that the tracking algos are too slow on python, happy to work on that.

I also added an additional condition to prevent removed_stracks from continuously growing and potentially causing memory issues. You can take a look and see if this approach is the same as yours:

_MAX_REMOVED_STRACKS = 1000

if len(self.removed_stracks) > _MAX_REMOVED_STRACKS:
    self.removed_stracks = self.removed_stracks[-_MAX_REMOVED_STRACKS:]

For evaluating the performance of different tracking algorithms, you can also refer to this benchmark:

https://motchallenge.net/results/MOT17/

It provides a good reference for comparing different tracking methods in terms of accuracy and performance. You can also use it to look for newer and more SOTA algorithms that can provide better accuracy while still maintaining good inference speed.

At the time, I integrated FastTracker because it was considered one of the SOTA approaches then. However, there are now newer tracking algorithms that are both more advanced and computationally efficient, so I would recommend checking the benchmark to see if there is a better option for our use case.

Regarding whether we should rewrite the tracking implementation in C/C++, my current comparison is between NVIDIA's built-in tracking method and my Python-based tracking implementation.

For a single-camera setup, the performance difference between C++ and Python may not be very significant. However, considering our actual deployment scenario, I would recommend using C++ if possible.

In real-world deployments, one Edge Device will typically handle multiple CCTV streams simultaneously. As the number of cameras increases, Python can become more of a bottleneck in terms of CPU usage, memory consumption, and overall resource management. Therefore, if feasible, I would recommend rewriting the tracking component in C++ to improve performance and stability when handling multiple CCTV streams.

Currently, one of our main bottlenecks is the FPS of the CCTV input streams. For now, we try to keep the input below 15 FPS. Once it goes above 15 FPS, the overall computational load starts to become more challenging.

However, our long-term goal is to increase the supported FPS while maintaining real-time performance and avoiding frame drops or latency, especially when multiple CCTV streams are processed simultaneously.

Therefore, in addition to optimizing the tracking algorithm itself, I think rewriting the tracking component in C++ and optimizing the overall processing pipeline should also be considered as part of the future performance optimization plan.

[EHxuban11]

Hey @thi23459112, thanks for sharing the code. We already handle that removed_stracks growth in ByteTrack and BoT-SORT: once the list exceeds 1,000 entries, we retain the latest 500.

I’ve also merged custom tracker support in #869 into dev. You can now use model.track(source, tracker=your_tracker) with an adapter implementing reset() and update(results, image=None) The interface requirements are documented in the PR and will be documented in the normal docks in the next release.

That’s the extension path I intend to support for additional trackers. I’m not planning to add FastTracker as a built-in or maintain C/C++ tracker implementations and native wheels, but you can plug in your own implementation through this API without forking LibreYOLO.