Performance issue with single-image processing on macOS

[go-restream]

System Configuration: macOS with M4 chip and 24GB RAM

../target/release/examples/paddleocr_vl --model-dir models/PaddleOCR-VL-1.5 --task ocr demo_p.png
2026-02-06T02:19:38.000180Z  INFO paddleocr_vl: PaddleOCR-VL Recognition Example (Candle-based)
2026-02-06T02:19:38.000289Z  INFO paddleocr_vl: Task: Ocr
2026-02-06T02:19:38.000324Z  INFO paddleocr_vl: Using device: Cpu
2026-02-06T02:19:38.000327Z  INFO paddleocr_vl: Loading PaddleOCR-VL model from: models/PaddleOCR-VL-1.5
2026-02-06T02:19:42.940209Z  INFO paddleocr_vl: Model loaded in 4939.86ms
2026-02-06T02:19:42.940231Z  INFO paddleocr_vl: 
=== Processing 1 images ===
2026-02-06T02:19:42.940234Z  INFO paddleocr_vl: 
Processing: demo_p.png
2026-02-06T02:20:16.076940Z  INFO paddleocr_vl:   Inference time: 33133.90ms

[go-restream]

metal Vs cpu ( 14012.68ms Vs 33133.90ms )

../target/release/examples/paddleocr_vl --model-dir models/PaddleOCR-VL-1.5 --task ocr demo_p.png -d metal
2026-02-06T02:30:11.854158Z  INFO paddleocr_vl: PaddleOCR-VL Recognition Example (Candle-based)
2026-02-06T02:30:11.854207Z  INFO paddleocr_vl: Task: Ocr
2026-02-06T02:30:11.912882Z  INFO paddleocr_vl: Using device: Metal(MetalDevice(DeviceId(1)))
2026-02-06T02:30:11.912933Z  INFO paddleocr_vl: Loading PaddleOCR-VL model from: models/PaddleOCR-VL-1.5
2026-02-06T02:30:15.594557Z  INFO paddleocr_vl: Model loaded in 3681.61ms
2026-02-06T02:30:15.594580Z  INFO paddleocr_vl: 
=== Processing 1 images ===
2026-02-06T02:30:15.594582Z  INFO paddleocr_vl: 
Processing: demo_p.png
2026-02-06T02:30:29.610072Z  INFO paddleocr_vl:   Inference time: 14012.68ms

[GreatV]

Thanks for reporting this performance issue! This is indeed a known limitation.

From the logs, you're experiencing incomplete Candle Metal backend support. While you used the -d metal flag, the oar-ocr-vl crate currently lacks explicit Metal feature support, causing some operations to fall back to CPU execution.

1. Immediate Solution

Please try modifying oar-ocr-vl/Cargo.toml to add the metal feature:

[features]
default = []
cuda = ["candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
# Add this line
metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"]

Then rebuild with:

cargo build -p oar-ocr-vl --features metal --release --example paddleocr_vl

2. Temporary Workaround

If you need immediate speed improvement:

• Use the official Python version with MLX-VLM backend (see official docs)
• Or try reducing the max_tokens parameter (default 512, try 256)

I'll submit a PR soon to add Metal feature support. If you'd like to test the fix, feel free to ping me!

/cc @go-restream Please try the above solution and report back, especially the performance after adding metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"].

[go-restream]

Metal support has been implemented locally （-d metal）, but its performance is not yet ideal.

[GreatV]

Fixed in PR #92

[go-restream]

Pulled the latest code, the performance comparison test results are as follows:

Test Result Summary

Metal (GPU) Mode
- Average Time: 20.22 seconds
- Fastest: 18.33 seconds
- Slowest: 22.66 seconds

CPU Mode
- Average Time: 50.38 seconds
- Fastest: 49.11 seconds
- Slowest: 52.39 seconds

@GreatV The processing time for a single image is >18sec. Is it due to the underlying architecture of the Candle framework?

[GreatV]

Hi @go-restream. Could you please provide the image you used, so we can test the performance on our side?

[go-restream]

@GreatV sure , I send u email. pls check.

[GreatV]

❯ bash run_paddleocrvl15.sh
    Finished `release` profile [optimized] target(s) in 0.28s
     Running `target/release/examples/doc_parser --model-name paddleocr-vl-1.5 --model-dir ../PaddlePaddle/PaddleOCR-VL-1.5 --layout-model .oar/pp-doclayoutv3.onnx --layout-model-name pp-doclayoutv3 --device metal demo_p.png`
2026-02-07T10:56:47.504810Z  INFO doc_parser: Unified Document Parser Example
2026-02-07T10:56:47.505000Z  INFO doc_parser: Model: PaddleOcrVl15
2026-02-07T10:56:47.554086Z  INFO doc_parser: Device: Metal(MetalDevice(DeviceId(1)))
2026-02-07T10:56:47.554129Z  INFO doc_parser: Loading layout model...
2026-02-07T10:56:47.732870Z  INFO doc_parser: Loading PaddleOCR-VL model...
2026-02-07T10:56:50.838946Z  INFO doc_parser: PaddleOCR-VL loaded in 3106.06ms
2026-02-07T10:56:50.838964Z  INFO doc_parser: 
=== Processing 1 images ===
2026-02-07T10:56:50.838966Z  INFO doc_parser: 
Processing: demo_p.png
2026-02-07T10:57:21.344805Z  INFO doc_parser:   Parsed in 30.50s
2026-02-07T10:57:21.344828Z  INFO doc_parser:   Elements: 29

--- Markdown ---
...
--- End ---

   Compiling bitflags v2.9.1
   Compiling sysctl v0.6.0
   Compiling raw-cpuid v11.6.0
   Compiling onig v6.5.1
   Compiling hayro-interpret v0.5.0
   Compiling tokenizers v0.22.2
   Compiling gemm-common v0.19.0
   Compiling gemm-f32 v0.19.0
   Compiling gemm-f64 v0.19.0
   Compiling gemm-c32 v0.19.0
   Compiling gemm-c64 v0.19.0
   Compiling gemm-f16 v0.19.0
   Compiling hayro v0.5.0
   Compiling gemm v0.19.0
   Compiling candle-core v0.9.2
   Compiling oar-ocr-core v0.6.1 (/Users/wangxin/repos/oar-ocr/oar-ocr-core)
   Compiling candle-nn v0.9.2
   Compiling candle-transformers v0.9.2
   Compiling oar-ocr-vl v0.6.1 (/Users/wangxin/repos/oar-ocr/oar-ocr-vl)
    Finished `release` profile [optimized] target(s) in 48.93s
     Running `target/release/examples/doc_parser --model-name paddleocr-vl-1.5 --model-dir ../PaddlePaddle/PaddleOCR-VL-1.5 --layout-model .oar/pp-doclayoutv3.onnx --layout-model-name pp-doclayoutv3 demo_p.png`
2026-02-07T10:58:11.463193Z  INFO doc_parser: Unified Document Parser Example
2026-02-07T10:58:11.463360Z  INFO doc_parser: Model: PaddleOcrVl15
2026-02-07T10:58:11.463413Z  INFO doc_parser: Device: Cpu
2026-02-07T10:58:11.463416Z  INFO doc_parser: Loading layout model...
2026-02-07T10:58:11.660078Z  INFO doc_parser: Loading PaddleOCR-VL model...
2026-02-07T10:58:15.822035Z  INFO doc_parser: PaddleOCR-VL loaded in 4161.94ms
2026-02-07T10:58:15.822068Z  INFO doc_parser: 
=== Processing 1 images ===
2026-02-07T10:58:15.822076Z  INFO doc_parser: 
Processing: demo_p.png
2026-02-07T11:00:12.115924Z  INFO doc_parser:   Parsed in 116.29s
2026-02-07T11:00:12.115955Z  INFO doc_parser:   Elements: 29

--- Markdown ---
...
--- End ---

After digging deeper, I believe the performance bottleneck is likely caused by Candle's Metal backend limitations. Specifically, the lack of native support for key operations like arange and broadcast_* forces these computations to fall back to CPU, resulting in frequent CPU↔GPU data transfers that become the main bottleneck. The on_compute_device workaround in PR #92 helps, but the overhead from tensor movement between devices still impacts overall performance. Native Metal support for these ops in future Candle updates would likely resolve this.