demo: Use Kroko ASR, show icon in tray

[Mathnerd314]

I wanted to show you what I have in my fork, in case you have any ideas on project evolution. My thoughts are:

• there definitely needs to be a better configuration mechanism, like a JSON file or something, the current environment variable-per-setting approach is not scalable
• resampling like I did here seems like a good idea, especially if the device capture rate is configurable
• supporting multiple models / APIs is maybe possible with the Go "build tags" / "build constraints" system? Or just compiling in support for all of them, but that could lead to dependency hell.

Also I went for a kind of minimalist approach with the GUI, just a tray icon that changes, I see you have gone in a different direction with the toaster notifications

[richiejp]

Thanks for sharing! This looks really interesting and actually what you are doing with Sherpa here could be really useful in LocalAI as well where the Kokoro backend is presently implemented with Python, so a Sherpa backend could be really nice. I've only had a brief look so far, but here are my thoughts on the topics this brings up in general.

• I'd be in favor of having JSON/YAML/TOML etc. as soon as we have a setting that requires hierarchical data (as in an array of structs) and would still want to keep env vars for the settings that are flat data. Env vars are good for external system managers (e.g. SystemD, Kubernetes/Docker compose, Nix (see https://github.com/NixOS/nixpkgs/pull/475019/changes#diff-f79247f9a5a519d871c7c9df661fe337a1235da96ecf612fcaef3ff4df252872R17 voxinput is configured completely inline within someones NixOS configuration using standard SystemD and Nix mechanisms).
• Resampling is done in LocalAI presently
• Also having multiple models/backends is currently outsourced to LocalAI or whatever implements the OpenAI API, I'll go into that more below
• The tray only approach is good; I didn't try it because Fyne's tray doesn't work on Sway and I was getting annoyed with Fyne. Having said that, I'd still accept the tray only approach as an option in its own PR.

As to having multiple API's and running models actually inside VoxInput;

• I have nothing against supporting multiple APIs and I think they could all be compiled in at once unless some of them use CGO and then yes build tags sound like a good idea to support distro's that don't have the underlying C libraries. Having said that my personal current focus is on the realtime API.
• I don't want any mandatory or default features that require running an AI model inside VoxInput. The basic assumption is that the machine running VoxInput is a low spec laptop with an already saturated CPU and no GPU/TPU.
• I'm not against having the option of running models inside VoxInput on CPU. Especially if it is for something like wake-words where possibly you don't want to stream all your mic audio to OpenAI or even your own LocalAI instance to implement wake-words.
• I don't want to maintain a large number of models in VoxInput. If we adopt kokoro, then a better model comes along, then we replace it. Having a large choice of models is outsourced to LocalAI (or the user can directly pass through settings to Sherpa).
• AFAICT you are using ONNX runtime with Sherpa which is not pure Go? So should be behind a build tag for distros that don't have the underlying libraries.

[richiejp]

Implemented in LocalAI

[Mathnerd314]

I haven't touched this in a while, it's been working well for me. But seeing as you closed it, I figured a late reply is better than none. So, jumping in:

• Direct sherpa-onnx integration is the main feature here. I can appreciate LocalAI, but spinning up a Docker container is more heavyweight than simply downloading models to a folder and having things work. The memory leak situation also comes to mind — being able to Ctrl-C and restart in a few seconds is quite useful when my PC uptime is measured in weeks.
• Env vars: Having 20+ environment variables is a sign that a config file is warranted. Much of the config is already hierarchical with duplicated sub-structures, so there's definitely a point where it's worth biting the bullet and making the switch. The longer it's delayed, the more painful it becomes — though admittedly this is bikeshedding if everything works.
• CGO: It looks like you've settled on a solution using purego for LocalVQE. What do you think about having the sherpa-onnx backend as a loadable dependency in a similar fashion?
• Model proliferation: sherpa-onnx is a cross-model library, so I'd expect future model support to land there rather than here. The real question is LocalAI vs. sherpa-onnx, but again, they serve different profiles (Docker web service vs. shared library).
• Resampling: I noticed you've added linear resampling for LocalVQE (here). There's obviously a tradeoff compared to the polyphase FIR filtering with Kaiser window design that go-audio-resampler uses, but since the ML model is the bottleneck anyway, it would be interesting to benchmark the two and see if there's a noticeable quality difference.
• Tray-only UI: Agreed, this is orthogonal. For me, Fyne works well with KDE. I opened a separate issue Tray-only UI #67 to explore GUI options.

[richiejp]

Since I added LocalVQE with Purego anyway it's not much of a big deal to add Sherpa and the LocalAI integration uses Purego as well with Sherpa as a point of reference. I'm tempted to suggest https://github.com/mudler/parakeet.cpp instead of Sherpa, but it's not the same thing and also they are not mutually exclusive. I don't like how Sherpa bundles the ONNX runtime and in general I think there is some supply chain risk with that project, so I'd like Sherpa behind a build tag, but otherwise it's fine.

Env vars: I basically have launch scripts for different profiles that set the env vars and don't see much need for something different, but it could be nicer to have a composable config format built in so I'd consider it

Resampling; I've never looked into it, this is just the naive implementation Claude spat out. Feel free to make a pr for that or perhaps I will because the upstream library looks interesting.

[Mathnerd314]

I don't mind changing out the model. I think what I will do is wait for mudler/parakeet.cpp#10 for Nemotron 3.5 ASR 0.6b to be merged. It is larger than Kroko (756 MB for int8 quantization vs. 147 MB), but you do get 40 languages, and I don't mind the size. Per k2-fsa/sherpa-onnx#2918 the offline parakeet models are not really streaming capable. And then I'll do a hacking session and if all goes well I'll close this and try to make a more mergeable PR. I just hope latency with CPU is reasonable.