inference_bench

A C++ benchmarking tool for ONNX Runtime inference, with a Python reference implementation for cross-validation.

Features

• High-resolution latency measurement using std::chrono::high_resolution_clock
• Configurable warmup and benchmark iterations
• Statistical reporting: mean, min, max, P50, P95, P99, and throughput (inf/s)
• Automatic model introspection — works with many ONNX models
• Python reference script for result validation

Requirements

• CMake 3.20+
• Ninja
• C++17-compatible compiler (GCC or Clang)
• ONNX Runtime 1.20.1 (pre-built binaries included under third_party/)

Build

cmake -B build -G Ninja
ninja -C build

Usage

./build/inference_bench --model <path> [--runs N] [--warmup N]

Option	Default	Description
--model	(required)	Path to the ONNX model file
--runs	100	Number of timed inference iterations
--warmup	10	Number of warmup iterations (not measured)

Example

./build/inference_bench --model models/mobilenetv2.onnx --runs 200 --warmup 20

Example output

--- Inference Benchmark ---
model: ../models/mobilenetv2.onnx
warmup: 10
runs: 100
--- Model Info ---
Input name: input
Input shape: [1, 3, 224, 224]
--- Benchmark Results ---
Total time: 327.74 milliseconds
Mean: 3.28 milliseconds
Min: 2.05 milliseconds
Max: 12.36 milliseconds
P50: 2.71 milliseconds
P95: 6.21 milliseconds
P99: 12.11 milliseconds
Throughput(mean latency): 305.12 inf/s

Python reference

A Python implementation is included for result validation:

pip install -e .
python scripts/bench_python.py --model models/mobilenetv2.onnx

Project structure

inference_bench/
├── src/
│   └── main.cpp              # C++ benchmarking application
├── scripts/
│   └── bench_python.py       # Python reference implementation
├── models/
│   └── mobilenetv2.onnx      # Sample model for testing
├── third_party/
│   └── onnxruntime/          # ONNX Runtime v1.20.1 headers and shared library
└── CMakeLists.txt

Tech stack

• Language: C++17
• Build: CMake + Ninja
• Runtime: ONNX Runtime 1.20.1
• Platform: Linux x86-64 (WSL2 supported)

What I learned building this

• Warmup iterations: At the start of a run the system is unstable — caches are cold, the runtime hasn't fully initialized. Running a few unmeasured iterations first ensures results reflect steady-state performance.

• Percentile metrics (P95/P99): Mean latency hides outliers. P95/P99 reveal the worst-case latencies a percentage of requests will hit, showing whether slow runs are random noise or a pattern.

• ONNX Runtime model introspection: The model exposes its own input names and shapes at runtime through the session object — no hardcoded tensor dimensions needed, works with any ONNX model.

• Navigating third-party C++ headers: The ORT HTML docs are confusing. Reading onnxruntime_cxx_api.h directly was more reliable for understanding types and method signatures.

For deeper notes on C++, benchmarking, and ONNX Runtime internals, see learning_notes.md.