Use Laya and other fast local models instead of paid APIs

[TomLucidor]

Coming from Pi being portable, could Jev be replaced with some of the alternatives that use Diffuion/BERT/etc maybe to speed up decision-making?

[DevMortimer]

Very much possible. I already tried testing with laya-mlx, but from my usage, it was very bad. I will visit it again when I'm free. Other folks can try.

#28

[TomLucidor]

What about the others like openjev or whataver, can they be included as well?
And I don't quite trust this either yet https://benchmarkheaven.com/jev-models

[DevMortimer]

The endpoint support from #134 is in pi-warden 0.74.0: any server that speaks the System One protocol (POST /v1/systemone) can now be the judge, including upstream Laya's laya-serve. See #134 for the config.

I measured Laya with pi-warden's judge benches on an M5 and on a Linux machine. It is fast (50–190 ms per request), but it caught 0% of rule violations at the guard's cutoff, where Jev catches 91%, and it was at chance on the stuck guard (0.53 against 0.99). Full numbers are in #134. As in #28, my conclusion is that it is not good enough for the guards; speed is its only advantage.

I genuinely want a local judge to be good, so here is what I found and what it would take.

One finding for anyone who tries: pi-warden's rule questions start with a shared frame of about 190 tokens, and Laya reads only 192 tokens of each question (head_max_len). The rule itself never reached the model, so all 12 rule questions in a request got the same answer, in 161 of 161 cases. With the rule moved first and the frame cut short, the answers differ per rule, but the ranking only rose from AUC 0.58 to 0.62 (Jev: 0.99), and recall at the 0.7 cutoff stayed at 0%. The wording was a real problem; it was not the main one.

The bar a local model has to clear before pi-warden recommends it, measured with npm run eval:rules and npm run eval:judge through the endpoint setting:

• Rules bench: AUC of 0.95 or more over all asked cases, and recall of 80% or more at a false-alarm rate of 5% or less at some cutoff.
• Judge bench: stuck accuracy of 0.90 or more, and done accuracy of 0.85 or more.

Post the command, the checkpoint, the hardware, and the results.json summary here. If a model clears the bar, I will document it as a supported local option.