Repository navigation
Conversation
Optional backend: a ~260-line proxy speaking jev's /v1/systemone wire format, answers computed by llama.cpp decoding a quantized GGUF instead of the OpenVINO graph. Reuses jev's own wire/prompt/vocab/choice/score sources unchanged; one decode per question on the shared state prefix with KV rolled back in between; P(yes) from raw logits. Receipts (RTX 4080 Laptop, q4_k_m, llama.cpp b11541): - 1-question p50 22 ms vs 292 ms CPU (13x) - ~300-token state, 5 questions: 73 ms vs 1372 ms (19x) - tokenizer parity exact (usage.input_tokens match on every request) - 89% overall decision agreement vs jev serve (100% danger/dodge, 95-100% fire/spatial, mean |dP| 0.02-0.16); results.example.json Includes validate.py (seeded agreement + latency vs jev serve) and a Makefile pinning llama.cpp b11541. Caveats in README: single client, no KV snapshots across requests, cold-vs-warm GPU clocks.
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From #40 (and as discussed): an optional GPU backend for
jev serve.What.
contrib/gpu-llama/— a ~260-line C++ proxy that speaks the same/v1/systemonewire format and computes answers with llama.cpp decoding a quantized GGUF instead of the OpenVINO graph. It reuses jev's ownwire/prompt/vocab/choice/score/request_tokenssources unchanged (-I src), so the wire semantics, prompt rendering and answer math are literally the upstream code. The only new logic: one llama.cpp decode per question on top of the shared state prefix, with the KV cache rolled back in between (every question sees the state alone, the way jev's attention mask does), and P(yes) read from raw logits:sigmoid(logit("1") - logit("0")).Why. In a real-time consumer of jev (a browser game loop judging 10-15x/s), CPU OpenVINO latency is the bottleneck. This lane makes batched 5-question judgments ~19x faster, which turns a 0.9B classifier into a genuinely real-time decision engine.
Receipts (RTX 4080 Laptop GPU, q4_k_m, llama.cpp
b11541):jev serveusage.input_tokens)jev servevalidate.pyreproduces the agreement + latency study against a runningjev serve(seeded scenarios,results.example.jsonincluded).Makefilebuilds against any llama.cpp CUDA prefix (pinb11541). Full runbook and caveats incontrib/gpu-llama/README.md; serving notes also on the wiki (serving jev via llama.cpp).Caveats (documented in the README): single client at a time (requests serialize; a second client turns 42 ms p50 into ~487 ms — matches jev's
NUM_STREAMS=1), no KV snapshots across requests, and cold-vs-warm GPU clocks (idle 210 MHz, warm 1.6 GHz — a 68 -> 43 ms p50 swing in benchmarks).Happy to adjust structure (e.g.
gpu/instead ofcontrib/), swap httplib for whatever the project prefers, or split docs out of the code — whatever fits the repo best.