do plan integrate GPU/NPU accelaration ?

[Marigo001]

Hi man do plan integrate GPU/NPU accelaration ?

[Spike8086]

hi,now the supports to those acceleration not quite well in phone,unless i change the framwork to mnn or LiteRT,otherwise it's not worthwhile to do so,so i may wait till those truly get better performance

[Marigo001]

This guys create a app with GPU acceleration. Maybe check this (https://github.com/jegly/Box) and (https://github.com/jegly/offlinellm)

[abyllon]

Unfortunately,it using LiteRT(which Google ai edge gallery using),and i try the APP Box you said, it only support cpu for regular gguf,as for npu/gpu it only suppory LiteRT as Google do

[Spike8086]

sorry dude, but yeah,just like he said, also, OfflineLLM using Vulkan,this is exactly the frame work i wait for a robust performance, i wanna sth truly could enhance and speed up instead of another option but cannot have real acceleration

[Marigo001]

OK. I understand. Maybe you must go directly with lireRT. Because google make the best LLM for device. All we need is GEMMA

[Marigo001]

So ? You are now working in implementing vulkan gpu accelaration (spcecially for snapdragon) for GGUF or not yet?

[Spike8086]

still not work yet, thanks for your consideration

[Marigo001]

thank for your answer