Hi,
I think you have not made it clear at all that this code as-is cannot be run locally, and that it relies on HF remote inference. I only realized that when I finally got it to run and it asked me to enter a HF API key.
I'm not sure what your goal is with this project, but if the intent is to have it in the hands of the enthusiasts and get more contributors for it, then we need something more in line with what the global community already uses (4bit models, local gpu/cpu inference, etc).
Right now it's out of the reach for most people to even evaluate the capabilities.
Thanks.
Hi,
I think you have not made it clear at all that this code as-is cannot be run locally, and that it relies on HF remote inference. I only realized that when I finally got it to run and it asked me to enter a HF API key.
I'm not sure what your goal is with this project, but if the intent is to have it in the hands of the enthusiasts and get more contributors for it, then we need something more in line with what the global community already uses (4bit models, local gpu/cpu inference, etc).
Right now it's out of the reach for most people to even evaluate the capabilities.
Thanks.