🚀 Edge AI Real-Time Inference Engine (Latency-Aware Runtime)

A CPU-only, real-time inference runtime for object detection using YOLO nu variants, designed with latency guarantees, backpressure, and graceful degradation under overload.

Unlike typical ML demos that maximize throughput or accuracy, this project treats latency as a first-class constraint and prioritizes freshness and system stability — the way real edge and robotics systems do.

I built a CPU-only, real-time Edge AI inference engine for object detection using YOLO (nu variant), focused on latency guarantees rather than raw accuracy. The system uses a multi-threaded pipeline, bounded queues, drop-oldest backpressure, dynamic batching, INT8 quantization with ONNX Runtime, and ingress frame throttling to remain stable under overload. I empirically measured throughput–latency tradeoffs (batch size, precision, queue pressure) and enforced real-time correctness by dropping stale frames instead of producing late results.

---

✨ Key Features

• Real-time streaming pipeline (not offline inference)
• Multi-threaded architecture (preprocess + inference)
• Bounded queues with drop-oldest backpressure
• Dynamic batching with empirically derived optimal batch size
• ONNX Runtime inference (FP16 and INT8)
• Post-training INT8 quantization with custom calibration
• Ingress frame throttling to match compute capacity
• Strict latency-aware execution (drops stale frames instead of producing late results)
• Benchmarking & diagnostics (throughput, drops, bottlenecks)

---

🧠 Design Philosophy

• Fresh results > Complete results
• Bounded latency > Maximum throughput
• Drop early, not late

The system is intentionally allowed to drop frames under load to preserve real-time guarantees.
This mirrors production edge systems used in robotics, CCTV analytics, and embedded vision.

---

🏗️ High-Level Architecture

Camera / Stream ↓ Ingress Control • Frame throttling (FPS cap) • Optional deadline checks ↓ Preprocess Thread • Resize & normalize • Drop-oldest on congestion ↓ Inference Thread • Dynamic batching (B=2 optimal) • ONNX Runtime (INT8) • SLA-aware execution ↓ Postprocess / Output

yaml Copy code

---

⚙️ Runtime Components

runtime/pipeline.py

Core real-time engine:

• Thread coordination
• Bounded queues
• Drop-oldest backpressure
• Deadline-aware execution
• Dynamic batching
• Frame throttling

runtime/onnx_infer.py

Abstraction over ONNX Runtime:

• Clean separation of inference backend
• Easy switching between FP16 / INT8 models

runtime/calibration_reader.py

Custom INT8 calibration pipeline:

• Feeds real images for post-training quantization
• Mirrors production INT8 workflows

---

📊 Benchmarks & Experiments

The benchmarks/ folder contains multiple baselines and experiments:

• Static (single-image) inference
• Streaming baseline (no backpressure)
• Threaded real-time pipeline
• FP16 vs INT8 comparison
• Batch size tradeoff analysis

Key Empirical Findings

• FP16 improved kernel speed but was insufficient alone
• Batch size has a non-linear optimum (B=2 best on tested CPU)
• INT8 reduced inference time but required policy tuning to realize gains
• Ingress throttling was more effective than increasing queues
• Queue-pressure drops are healthier than deadline violations

---

📈 Example Final Behavior

(INT8 + Batch = 2 + Frame Throttling)

Processed frames: 27 Dropped frames: 33 Dropped deadline: 0 Dropped q_pre: 0 Dropped q_inf: 4

yaml Copy code

This demonstrates:

• Stable inference
• Bounded latency
• Controlled, intentional frame dropping
• No stale outputs

---

🧪 Models

• yolov5nu.onnx — FP16 dynamic-batch ONNX model
• yolov5nu_int8.onnx — INT8 quantized ONNX model (post-training)

Note: All experiments use YOLO nu variants for better edge performance.

---

🚀 How to Run (Quick Start)

# create environment
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

pip install -r requirements.txt

# run real-time benchmark
python -m benchmarks.threaded_stream

Runtime Configuration

Adjust behavior in pipeline.py: batch_size

STRICT_REALTIME

min_frame_interval_ms

🧾 Project Structure

Copy code
edge-inference-runtime/
├── benchmarks/          # Baselines & streaming tests
├── runtime/             # Core runtime components
├── scripts/             # Export & quantization scripts
├── calib_images/        # INT8 calibration images
├── models/              # ONNX artifacts
├── metrics/             # Latency utilities
├── README.md
└── requirements.txt

🎯 What This Project Demonstrates

Systems-level thinking applied to ML inference

Understanding of real-time constraints, not just model accuracy

Practical use of ONNX Runtime and INT8 quantization

Ability to reason about throughput–latency tradeoffs

Production-style design choices for edge AI

📌 Suitable For

Edge AI / Robotics internships

Computer Vision roles

ML Systems / Inference Engineering roles

Research projects involving real-time perception

🔚 Final Note

This project intentionally avoids “perfect accuracy” or “max FPS” claims. Instead, it focuses on honest, bounded, and stable real-time behavior, which is what real systems require.

---