Real-Time C++ TensorRT 10.x Inference Pipeline for RF-DETR

A production-grade, highly optimized C++ inference pipeline designed specifically for TensorRT 10.x executing RF-DETR object detection. The pipeline provides dual processing configurations: a high-throughput multi-threaded Live Stream Engine featuring low-latency ring queue management, and a sequential Batch/Video File Processor with automated, visually polished overlays.

---

🎬 Live Inference Demo

---

🌟 Key Features

• TensorRT 10.x Native API: Utilizes modern enqueueV3 asynchronous execution interfaces, eliminating all legacy destroy() paradigms and using zero-copy dynamic binding buffer queries (getNbIOTensors, getIOTensorName).
• Hungarian Type & Hungarian Prefix Naming Convention: Strictly structured for enterprise codebases, with clean Hungarian type qualifiers (e.g., si32_width, fp32_confidence, vdet_detections, m_uptr_context).
• Multi-Threaded Live Stream Engine (trt_stream_pipeline): Spawn-isolated capture and execution threads operating over a thread-safe SafeQueue with a Drop-Oldest congestion control policy to guarantee zero-latency frame handling.
• Sequential Video File Processor (trt_video_pipeline): Optimized for frame-by-frame processing of local media files (MP4, AVI, MKV) with custom-color overlay visualization, smooth bounding boxes, and performance metrics drawing.
• Premium Aesthetics overlays: Employs sleek drop-shadow text overlays and handpicked, modern, high-contrast BGR palettes (Golden Amber, royal violet, emerald green) instead of standard harsh primary colors.
• Clean SOLID/DRY Architecture: Strategies for pre-processing (OpenCV-backed Letterboxing) and post-processing (Sigmoid un-scaling and Non-Maximum Suppression) are completely decoupled from the engine runtime.

---

📊 Performance & Hardware Benchmarks

The following benchmarks demonstrate the inference speed (frames per second) of RF-DETR model configurations on local hardware (optimized with TensorRT 10.x and FP16 precision):

Model Variant	Resolution	Precision	Inference Speed (FPS)	Latency per Frame
RF-DETR Nano	640x640	FP16	350 FPS	~2.86 ms
RF-DETR Small	640x640	FP16	230 FPS	~4.35 ms
RF-DETR Base	640x640	FP16	180 FPS	~5.56 ms

Note: Latency is computed end-to-end including custom OpenCV letterbox preprocessing and GPU upload/download operations.

---

🏗️ System Architecture

graph TD
    subgraph "Input Sources"
        RTSP["RTSP Stream / Local Camera"]
        VideoFile["Local Video File (MP4/AVI)"]
        Synth["Synthetic Frames Generator"]
    end

    subgraph "Orchestration Targets"
        StreamProc["trt_stream_pipeline (Multi-Threaded)"]
        VideoProc["trt_video_pipeline (Sequential)"]
    end

    subgraph "Thread Isolation (Streaming Mode)"
        CapThread["Capture Thread (SafeQueue Push)"]
        SafeQ[("SafeQueue (Max Size: 3, Drop-Oldest)")]
        InferThread["Inference Thread (SafeQueue Pop)"]
    end

    subgraph "Processing Pipeline"
        Prep["RFDetrPreprocessor (Letterbox Padding)"]
        TRT["TrtEngine (TensorRT 10.x Context enqueueV3)"]
        Post["RFDetrPostprocessor (Sigmoid & NMS)"]
        Visual["Visual Annotation Engine (Sleek Overlays)"]
    end

    RTSP --> StreamProc
    Synth --> StreamProc
    VideoFile --> VideoProc

    StreamProc --> CapThread
    CapThread --> SafeQ
    SafeQ --> InferThread
    InferThread --> Prep

    VideoProc --> Prep

    Prep --> TRT
    TRT --> Post
    Post --> Visual
    Visual --> Output["Annotated Output File / Console Streams"]

Loading

---

🛠️ Technology Stack & Requirements

• Operating System: Linux (Ubuntu 20.04/22.04 recommended)
• Compiler: GCC 9+ (supporting C++17/C++20 standards)
• Inference SDK: NVIDIA TensorRT 10.x (Strict dependency)
• Graphics API/Compute: CUDA Toolkit 12.x+ / cuDNN 9.x+
• Visual Library: OpenCV 4.x (linking core, imgproc, highgui, videoio, dnn modules)

---

⚙️ Compilation & Build Guide

The project includes both a standalone CMakeLists.txt configuration and an integrated Docker development environment.

🐳 Using Docker (Recommended for TensorRT 10)

1. Build the container:

   docker build -t trt_inference_pipeline:latest -f DockerFile .

2. Run with GPU support:

   docker run --gpus all -it --net=host -v $(pwd):/workspace trt_inference_pipeline:latest

🔨 Compiling via CMake (Inside Container/Host)

mkdir -p build && cd build
cmake ..
make -j$(nproc)

This compiles two target binaries inside the build directory:

• trt_stream_pipeline: The real-time, multi-threaded streaming binary.
• trt_video_pipeline: The sequential video processing binary.

---

🚀 Running the Pipeline

1. Live Streaming & Camera Inputs (trt_stream_pipeline)

Run with a serialized model engine and an input stream path (RTSP link, Camera index, or synthetic to generate mock data):

./trt_stream_pipeline <path_to_engine> <stream_source>

Examples:

• Synthetic Emulator (No camera/file needed):

  ./trt_stream_pipeline model.engine synthetic

• USB Web Camera:

  ./trt_stream_pipeline model.engine 0

• IP Camera / RTSP Link:

  ./trt_stream_pipeline model.engine rtsp://admin:pass@192.168.1.100:554/h264

2. Local Video Files Processing (trt_video_pipeline)

Processes local video files frame-by-frame and exports the annotated output to output/:

./trt_video_pipeline <path_to_engine> <path_to_video>

Example:

./trt_video_pipeline model.engine test.mp4

Outputs will be created under output/test_annotated.mp4.

---

📦 Directory Structure

├── core
│   ├── interfaces.hpp   # Decoupled interface wrappers (IEngine, IPreprocessor, IPostprocessor)
│   └── scheme.hpp       # Data structures (Frame, Detection) with Hungarian notations
├── engine
│   ├── trt.hpp          # TrtEngine definition (TensorRT 10.x only)
│   └── trt.cpp          # TensorRT 10.x enqueueV3 deserialization and inference implementation
├── plugins
│   ├── preprocessor.hpp # Letterbox resize and padding header
│   ├── preprocessor.cpp # preprocessor implementation using cv::dnn
│   ├── postprocessor.hpp# Box reconstruction and custom NMS header
│   └── postprocessor.cpp# Sigmoid-activated logit extraction and IoU NMS implementation
├── utils
│   ├── annotations.hpp  # Premium overlays drawing tools header
│   ├── annotations.cpp  # Sophisticated BGR palette drawing functions
│   ├── pipeline.hpp     # General CMake orchestration helpers header
│   ├── pipeline.cpp     # Signal handling and setup logic
│   └── safe_queue.hpp   # Drop-oldest congestion control template
├── main_stream.cpp      # Multi-threaded RTSP entry point
├── main_video.cpp       # Sequential video file entry point
├── CMakeLists.txt       # Build system definition
├── DockerFile           # Reproducible execution environment
└── LICENSE              # Apache 2.0 Licensing

---

🛡️ License

This project's codebase is open-source software licensed under the Apache License, Version 2.0.

🎬 Demo Media License Exclusions

The sample video and detection demo media located within the repository (e.g., from the MOT17 dataset) are used strictly for non-commercial illustration and evaluation. They are subject to the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license, and are excluded from the permissive Apache 2.0 software license.