Concurrent local-provider inference: fair pool coordinator

[e3ndr]

Concurrency for local providers — fair pool coordinator

Local providers (Ollama, vLLM) can now satisfy multiple requests without
blocking each other, while the VRAM invariant is preserved.

What changed

Before: one fair ReentrantLock per resource pool, taken per request.
Exactly one request across all providers in a pool could run at a time,
and each request slept every other provider in the pool — including
providers on other pools (a real cross-pool bug).

After: a per-pool coordinator (Pools / Pool) that:

• keeps a single FIFO queue, strictly by arrival — a resource hog can't
  stomp out requests already in line (no skipping the line);
• lets the queue head start when the pool is empty, or when it's on the
  active provider with a free concurrency() slot (and, if
  concurrencyMatchesModel(), all in-flight requests share its model);
• enforces at most one provider with in-flight requests per pool (the
  VRAM invariant). A provider switch sleeps the previous provider and
  wakes the new one — done outside the pool lock;
• keeps providers warm: a provider is only slept on a switch, never
  because its queue went empty (load times can be penalizing);
• makes pools independent — requests on pool A never sleep or block
  providers on pool B (fixes the cross-pool bug).

New config

Per provider (defaults shown):

• concurrency (int, default 1) — max parallel requests on this provider.
• concurrencyMatchesModel (bool, default false) — only same-model
  requests run in parallel; a different model waits for the others to drain.

Exposed on the provider list API and editable in the UI. Parsed/serialized
by the Ollama and vLLM providers; InferenceProvider carries the defaults
so all other provider types are unaffected.

Routes

Both /inference/v1/* and /proxy/:provider/* inference endpoints go
through the same pool gate — the proxy route can no longer bypass the pool
limits. Cloud providers (resourcePool: "cloud") stay ungated.

Verification

Built with JDK 21 (mvn package, Svelte UI compiles clean). Scheduler
logic verified with a standalone harness driving the real Pool against
stub providers — 5/5 pass:

1. same provider, concurrency=2 → two requests in flight at once (peak 2)
2. different provider waits for a full drain, then runs
3. model-match serializes different models on the same provider
4. no line-cutting: a queued other-provider request blocks even a free slot
5. (S1–S4 above; the first-request-on-empty-pool path is exercised by all)

The harness also caught a real deadlock on the first run — acquire()
enqueued then blocked, but dispatch() was only called from release() /
interruptQueued(), so the first request on an empty pool never started.
Fixed by dispatching immediately after enqueue (see "Fix deadlock" commit).

Signed off by ~Ash