Really interesting project! I’ve been looking through the code and the architecture, and I really like the approach you’re taking with MoE models and memory management.
I have a technical question: on a typical consumer PC with a reasonable amount of system RAM and a consumer GPU, how large of a model can FreeToken realistically handle with the current architecture? For example, is there a practical parameter limit where performance becomes too slow, or could the current implementation scale to models in the 200B–300B+ range depending on the model’s MoE structure and hardware?
Also, are there any specific parts of the codebase you would recommend looking at to better understand how the expert caching and CPU/GPU scheduling work?
Great work — really interesting project.
Really interesting project! I’ve been looking through the code and the architecture, and I really like the approach you’re taking with MoE models and memory management.
I have a technical question: on a typical consumer PC with a reasonable amount of system RAM and a consumer GPU, how large of a model can FreeToken realistically handle with the current architecture? For example, is there a practical parameter limit where performance becomes too slow, or could the current implementation scale to models in the 200B–300B+ range depending on the model’s MoE structure and hardware?
Also, are there any specific parts of the codebase you would recommend looking at to better understand how the expert caching and CPU/GPU scheduling work?
Great work — really interesting project.