Weekend Mobile Tests (LLM) — 2026-07-26

[github-actions]

Automated weekend mobile run for @qvac/llm-llamacpp.

Overall: ❌ failures detected

Platform	Result	Total	Passed	Failed
Android	❌ failed	45	43	2
iOS	✅ passed	30	30	0

Full per-test tables and Device Farm run links are on the workflow run summary.

Weekly coverage: OCR (LightOn + Paddle), Elephant + HighResAurora image functional, and vlmPerf{Gemma4,Qwen35}.

[localhost41]

I inspected the Android console and performance artifacts from the linked run. The two reported failures are the same underlying Unlimited-OCR correctness failure, exercised by runOcrUnlimitedTest and runUnlimitedOcrPerfTest; they are not two independent OCR regressions.

Reproduction boundary:

• Run head: 19f540e, @qvac/llm-llamacpp 0.38.2.
• Galaxy S25 Ultra / Adreno 830: the loader registers Vulkan and OpenCL, logs Chosen GPU OpenCL, and auto-selects the GPU multimodal projector. Model load and inference complete without a crash, but generation stops after 37 tokens with unrelated multilingual output and none of the expected document terms.
• Galaxy S26 Ultra / Adreno 840: it follows the same OpenCL + GPU-projector selection path and passes with 918 generated tokens containing the expected document text.
• Pixel 9 Pro / Mali G715: OpenCL is removed, the LLM uses Vulkan with the projector on CPU, and it also passes with 918 generated tokens.

There is also a diagnostics mismatch in the artifact: the performance record says backend: vulkan for the S25 and S26 even though the runtime selection log says Chosen GPU OpenCL. That makes this class of backend-specific failure harder to spot from the summary.

The next useful step is a controlled S25 matrix using the identical model, image, and generation parameters:

1. Current auto path: OpenCL LLM + GPU projector, repeated to establish determinism.
2. OpenCL LLM + mmproj-use-gpu=false, to isolate projector corruption from decoder/backend corruption.
3. A Vulkan-only overlay build, with CPU and GPU projector variants if both are supported.
4. CPU-only reference.

If the CPU-projector OpenCL leg passes, the issue is in the Adreno 830 projector path. If both OpenCL legs fail while Vulkan/CPU pass, the problem is in the OpenCL model path or its device-specific kernels. Only then would I narrow the auto-selection policy; a broad Adreno fallback would discard the confirmed-good Adreno 840 path.

I have not proposed a code change yet because the existing artifacts cannot distinguish those two OpenCL boundaries. A follow-up should also make the performance report record the resolved backend rather than the generic requested GPU label.

[costin-busioc]

Closing as a fix coming from: #3952