[tracking] Hardware acceleration roadmap and qualification matrix

[eric8810]

Purpose

Record what the Paddle/ONNX OCR ecosystem actually supports, how production-oriented projects use hardware backends, and which paths are relevant to light-ocr.

This tracker separates two goals:

• Interactive/offload: minimize OCR's process CPU consumption so the foreground application remains responsive.
• Batch/throughput: use available CPU/GPU resources to finish a queue faster.

Those goals require different defaults and benchmarks.

Upstream capability matrix

Platform/hardware	Officially supported paths	Common engineering pattern	Relevance to light-ocr
Apple Silicon GPU/ANE	ONNX Runtime CoreML EP; native Core ML	Static/bounded or enumerated shapes, MLProgram specialization/cache, verify compute plan and compute unit	Primary interactive direction. Current models reach GPU only after shape specialization in local probes; ANE is not currently demonstrated.
Apple Silicon via Paddle	Paddle macOS CPU build	CPU threads/MKL-style tuning only; no official macOS GPU Paddle build	Not an offload solution. Paddle upstream currently leaves this gap.
NVIDIA GPU (Linux/Windows)	Paddle Inference CUDA/cuDNN and TensorRT; PaddleX HPI; ORT CUDA/TensorRT	Explicit min/opt/max dynamic-shape profiles, FP16/INT8 qualification, engine/timing cache, I/O binding, CUDA streams	Strong future desktop/server backend, but requires separate packaging and quality/performance gates.
Intel CPU/GPU/NPU	OpenVINO and ORT OpenVINO EP	Device selection, streams/requests, precision selection, unsupported-node/device fallback	Candidate for Intel desktop/server deployments.
Windows DirectX 12 GPU	ORT DirectML / Windows ML	Sequential session constraints, known shapes, broad vendor coverage	Broad compatibility path; DirectML is in sustained engineering and new Windows work should evaluate WinML.
Qualcomm Snapdragon	ORT QNN EP (CPU/GPU/HTP NPU)	Quantized models/context cache plus power/performance profiles	Separate Windows/Android device target; requires quantization and device qualification.
Huawei Ascend	Paddle custom-device/CANN and ORT CANN EP	Explicit device/provider package and graph configuration	Relevant only if Ascend becomes a product target.
Mobile/embedded GPU	Paddle Lite OpenCL; ncnn Vulkan; MNN	Convert models to a mobile engine, FP16/INT8, backend-specific tuning	A separate mobile architecture, not a drop-in desktop ORT switch.

What upstream projects actually do

PaddleOCR / PaddleX

• High-performance inference (enable_hpi) automatically selects among Paddle Inference, OpenVINO, ONNX Runtime, and TensorRT where supported, but the documented HPI environment is Linux x86-64 and its gpu path is NVIDIA/CUDA-oriented.
• TensorRT/Paddle Inference uses FP32/FP16/INT8, explicit dynamic-shape ranges, serialized engine caches, and GPU-memory tuning.
• Paddle2ONNX exports OCR models with dynamic shapes to preserve deployment flexibility and parity. That portability choice conflicts with CoreML's best-performance path, so Apple shape specialization must be deliberate and quality-tested.

References:

• https://www.paddleocr.ai/main/en/version3.x/inference_deployment/local_inference/high_performance_inference.html
• https://paddlepaddle.github.io/PaddleX/main/en/pipeline_deploy/high_performance_inference.html
• https://www.paddlepaddle.org.cn/inference/guides/nv_gpu_infer/index_nv_gpu_infer.html
• https://github.com/PaddlePaddle/PaddleOCR/blob/main/deploy/paddle2onnx/readme.md
• https://www.paddlepaddle.org.cn/documentation/docs/en/install/compile/macos-compile-make_en.html

ONNX Runtime

• Execution Providers partition a graph by backend capability and priority. Provider availability is not proof that all expensive nodes execute on that device.
• CoreML exposes compute-unit selection, static-shape requirements, specialization strategy, model cache, and compute-plan profiling.
• CUDA recommends I/O binding to avoid CPU↔GPU copies; CUDA graphs add stronger fixed-address/fixed-shape constraints.
• TensorRT requires optimization profiles for dynamic inputs and relies heavily on engine/timing caches; unsupported subgraphs can fall back to CUDA.
• OpenVINO and QNN expose device/performance modes and may fall back when the target accelerator cannot execute a model.

References:

• https://onnxruntime.ai/docs/execution-providers/
• https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html
• https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
• https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html
• https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html
• https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html
• https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html

RapidOCR / mobile runtimes

RapidOCR makes the backend an explicit per-model choice. Its current configuration includes ORT CPU/CUDA/DirectML/CANN/CoreML, OpenVINO, Paddle CUDA/NPU, PyTorch CUDA/MPS/NPU, TensorRT, and MNN. Its TensorRT implementation declares detection and recognition min/opt/max shape profiles, defaults to FP16 when supported, and caches the built engine. It also verifies requested-vs-actual ORT providers and warns on fallback.

• RapidOCR repository: https://github.com/RapidAI/RapidOCR
• Backend configuration: https://github.com/RapidAI/RapidOCR/blob/main/python/rapidocr/config.yaml
• ORT provider validation: https://github.com/RapidAI/RapidOCR/blob/main/python/rapidocr/inference_engine/onnxruntime/provider_config.py
• TensorRT shape/engine builder: https://github.com/RapidAI/RapidOCR/blob/main/python/rapidocr/inference_engine/tensorrt/engine_builder.py
• ncnn uses ARM CPU/NEON plus Vulkan GPU with FP16/INT8-oriented deployment: https://github.com/Tencent/ncnn

Recommended order for this repository

1. Apple interactive path: qualify actual CoreML GPU placement and bounded/static/enumerated shapes ([performance][macOS] Qualify true CoreML GPU execution with bounded/static OCR shapes #3).
2. Make execution truthful: add profiles, placement/fallback observability, and CPU-seconds/cold/RSS metrics ([performance] Add execution profiles and observable accelerator fallback #4).
3. CPU batch path: add opt-in bounded concurrent recognition only for throughput use cases ([performance] Add bounded concurrent recognition for opt-in CPU throughput #5).
4. Only when product targets require them: add separate NVIDIA (CUDA/TensorRT), Intel (OpenVINO), Windows (WinML/DirectML), or mobile backends. Each needs its own binary packaging, model/precision/shape contract, cache policy, and corpus gate.

Cross-backend definition of done

• Support matrix identifies OS, architecture, accelerator, runtime/package variant, model precision, and stability level.
• Requested and actual device placement are observable; fallback is never silently described as acceleration.
• Each backend has cold/warm benchmarks with wall time, process CPU seconds, peak RSS/VRAM, cache size/build time, and per-stage timing.
• Each backend passes the same OCR text/confidence/geometry/reading-order corpus, with documented tolerances.
• Dynamic/static/bucketed shape policy and supported input limits are explicit.
• Quantization/FP16 is opt-in until accuracy gates pass.
• Host scheduling distinguishes interactive CPU-preserving mode from batch-throughput mode.
• Unsupported platforms fail clearly and retain a documented CPU fallback.

Non-goals

• Implement every backend in this tracker.
• Assume an upstream backend name implies good performance for these exact models.
• Use one benchmark image or average seconds/page as the qualification standard.

[eric8810]

当前加速路线与资格矩阵已由 #11 和仓库 Roadmap/Apple/Linux/Windows acceleration 文档承接。0.3.0 已完成 Apple Direct Core ML 与 Linux/Windows Native WebGPU 的 API、真实 placement、质量、性能、资源、fallback 和发布边界记录。CUDA、OpenVINO、DirectML 或其他专用 backend 如需启动，将按各自 Provider Gate 建立独立 issue。