Tracks the C++ side of ddvnguyen/hydra_vortex#747 (llama-server: dynamic parallel-slot admission by combined-ctx threshold).
What
Add a llama-server param --parallel-ctx-threshold <tokens> to the fork. When a slot is free and a request is queued (cont_batching's existing wait queue), only admit it into a slot this round if:
(sum of ctx already resident in active slots) + (candidate's prompt ctx) < threshold
Otherwise leave it queued until an active slot frees enough headroom — i.e. fall back to the existing serialize-and-queue behavior.
Why
Every 2-slot-concurrency config tested in hydra_vortex#740 so far pays a permanent CPU-FFN-offload tax (arm128: 7.90 tok/s single vs arm090's ~37-40 tok/s) or has no real concurrency (arm090, parallel=1). Admission control by combined active ctx lets the pool stay sized for one full ~148K session (arm090 parity for the common single-request case) while still admitting a 2nd concurrent request when combined ctx stays under the threshold (~100K).
Constraints
- Scheduling/admission gating ONLY — no KV buffer allocation/sizing changes; kv_unified pool stays statically sized at boot.
- UM (GGML_CUDA_ENABLE_UNIFIED_MEMORY) stays on as safety net.
- Default: disabled (0) → behavior identical to today.
Test plan
Full plan in ddvnguyen/hydra_vortex#747:
- Single request only: tg at arm090 parity (~37-40 tok/s) — no regression of common case.
- n=2 concurrent, combined ctx well under threshold: genuine parallel decode.
- n=2 concurrent, deep request pushing combined over threshold: 2nd request queues (verified via slot-assignment/queue logs).
- 4-phase VRAM margin capture (boot/warm/fill/decode) + PCIe dmon.
Parent issue: ddvnguyen/hydra_vortex#747
Tracks the C++ side of ddvnguyen/hydra_vortex#747 (llama-server: dynamic parallel-slot admission by combined-ctx threshold).
What
Add a llama-server param
--parallel-ctx-threshold <tokens>to the fork. When a slot is free and a request is queued (cont_batching's existing wait queue), only admit it into a slot this round if:Otherwise leave it queued until an active slot frees enough headroom — i.e. fall back to the existing serialize-and-queue behavior.
Why
Every 2-slot-concurrency config tested in hydra_vortex#740 so far pays a permanent CPU-FFN-offload tax (arm128: 7.90 tok/s single vs arm090's ~37-40 tok/s) or has no real concurrency (arm090, parallel=1). Admission control by combined active ctx lets the pool stay sized for one full ~148K session (arm090 parity for the common single-request case) while still admitting a 2nd concurrent request when combined ctx stays under the threshold (~100K).
Constraints
Test plan
Full plan in ddvnguyen/hydra_vortex#747:
Parent issue: ddvnguyen/hydra_vortex#747