[PLNech]

Adds --hint {latency,throughput} (default latency, the released behavior) and --streams N (default 1) to jev serve/jev decide, wired into the compile config's PERFORMANCE_HINT / NUM_STREAMS, and reported by /health.

Bench on a shared 20-core x86-64 box (p50 of 20 sequential runs, short 90-token 1-question request / grouped 235-token 4-question request, ms):

	short	grouped
default (latency, 1 stream)	40	135
throughput, 2 streams	113	375
throughput, 4 streams	126	314

Under a single sequential client the defaults win; streams pay off only under concurrent load (not measured here). Happy to adjust flag naming to match project conventions.

[PLNech]

Extracting parsing for unit tests there seemed overkill, but lmk if you want that :)

[PLNech]

Ran a multi-day validation campaign with this build (a browser demo where jev judges 10-15x/s in a game loop) — some field numbers that motivate exposing these knobs:

• Single client, batched 4-5 questions per request: p50 ~224 ms on CPU OpenVINO, ~46 ms on a llama.cpp q4 lane (13-19x, see the wiki page on serving jev via llama.cpp).
• Two concurrent clients on one llama.cpp instance serialize into ~10x latency (42 ms -> 487 ms p50). Same class of problem the throughput hint / stream count address on OpenVINO — without the knobs, a second tab silently poisons latency for the first.
• Cold vs warm matters when benchmarking: an idle GPU sits at 210 MHz; sustained request traffic ramps it to ~1.6 GHz and p50 drops 68 -> 43 ms. Idle-machine numbers oversell, busy-desktop numbers undersell — worth a warmup phase in any bench.
• Backend agreement for anyone switching lanes: 89% overall decision agreement between the GPU lane and reference jev serve (100% on danger/dodge questions, 95-100% on fire/spatial, mean |dP| 0.02-0.16 per question).

Happy to share the validation harness (headless browser, URL-parameter driven, per-episode score/latency export) if useful.

[feder-cr]

Thanks for the PR and for measuring it. I can't merge it as it is: with this engine the two knobs can only make a request slower.

• jev serve has a single model user: one InferRequest (src/model.cpp) driven by one scheduler thread (src/scheduler.hpp). Calls never run concurrently, so a second stream never gets work of its own; it only takes cores away from the one call that is running. Your table shows exactly that: 40 -> 113 ms and 135 -> 375 ms, and there is no concurrent throughput to win back.
• Concurrent clients are handled by the scheduler instead: requests waiting at the same time are read together in one call (--batch-tokens; batched_requests in /health counts them).

Streams would only pay off with several infer requests, several scheduler workers and the state snapshots shared safely between them, which is a scheduler redesign rather than a flag. If you have a concurrent workload where batching loses, an issue with the numbers is the place to start.

[PLNech]

Agreed, and your explanation matches the table exactly: with a single InferRequest on a single scheduler thread, the second stream never sees a request of its own — the 40 → 113 ms regression is it stealing cores from the one call in flight, and there is no concurrent throughput to win it back.

I'd also missed that concurrent clients already merge through --batch-tokens (visible as batched_requests in /health) — the mechanism I was trying to expose flags for was already there. Closing this.

For the record, our concurrent-latency pain point lives on the llama.cpp lane (PR #43), where two clients serialize ~10x inside that server. If a workload ever shows batching losing on this side, I'll open the issue with numbers first, as you suggest.