Does paddler support big modells to be splitted to more then 1 GPU?

[rzerres]

To me it is unclear, if llama-sys-2 is capable to broadcast requests to more then 1 GPU, eg. for MoE LLMs.

• What will happen if a Model dosn't fitting into a given GPU VRAM?
• Is it possible to move Layers to RAM? And how to advertise it via MetaData?
• Howto bind multiple GPU's to a Fleed or given Balancer?

[mcharytoniuk]

1. You can't, because the way llama.cpp works it creates a memory map.
2. Yeah, but it's either RAM or VRAM.
3. You can start multiple agents on the same device, bind each agent to a separate GPU.

[malzag]

@rzerres Thanks for starting this discussion! Just a quick note: we're changing how we organize things in this repo and moving away from Discussions toward tracking everything through Issues. I've moved this conversation over to an issue here: #145. Feel free to continue the conversation there. Thanks!