Start branch for 0.7.2

CHANGELOG.md

@@ -807,7 +807,10 @@ States the part that is not automatic: `$SLURM_JOB_GPUS` must be passed as
 including cards allocated to another job, and the failure surfaces later as a
 CUDA OOM in whichever job loses.
 
-## Version 0.7.1 - Unreleased
+## Version 0.7.2 - Unreleased
+
+
+## Version 0.7.1 - Released 2026-09-24
 
 
 ## Version 0.7.0 - Released 2026-08-28

dev/journals/gpt-5.6-sol-rtx8000.md

@@ -0,0 +1,13 @@
+## 2026-09-24 14:10:00 -0400
+
+Summary of user intent: preserve the Quadro RTX 8000 / Qwen3.8-27B HyperQwen profiling work as durable infer-stack developer knowledge and add a third profiling round that measures the behavior that matters for a persistent Pi-style coding-agent session: long-context prefix reuse and decode speed as context grows. The user wants every tested variation represented as an explicit infer-stack endpoint and wants the benchmark harness to survive failures without disturbing unrelated leases. `BASE_URL=0.0.0.0:14042` is a manually supplied gateway address and must not be assumed to come from `OPENAI_BASE_URL`.
+
+Model: GPT-5.6 Sol. Configuration: tool-enabled reasoning session.
+
+The durable finding from rounds 1 and 2 is that the 48-GiB Turing card has a capacity surplus but a cold-prefill performance problem. W4A16 with float16 activations, FP16/auto KV, `TRITON_ATTN`, one sequence, prefix caching, and `max_model_len=262144` boots and reports roughly 415K-443K KV tokens depending on the exact arm. Short decode is roughly 33-35 tok/s; a 32K cold prompt takes roughly 230 s. Activation-int8 did not improve the measured workload, FP8 KV is rejected on the sm75 Triton path, the tested int4/KVarN paths never become useful servers, and Flex attention is blocked by the hybrid model's 400-token page geometry. The prepared `-fast` target improved short decode by about 3% without materially changing 8K/32K prefill. Round-2 MTP/DFlash2 failures are not evidence that speculation is impossible on Turing: the pinned single-user launcher explicitly refuses speculation with `--dtype=half` because its speculative verify path is BF16-specific.
+
+The first near-full experiment also taught an important measurement lesson: a long request routed through LiteLLM hit its 600-second timeout/retry policy while the vLLM server remained healthy, so gateway wall time cannot be used as evidence about maximum context or cold-prefill completion. Long benchmark traffic must go directly to vLLM while infer-stack continues to own the endpoint and lease.
+
+Round 3 deliberately avoids another broad configuration sweep. It creates only the standard and `-fast` W4A16 full-context endpoints on the known-working Triton/8192 path. At each context depth it measures a cold base request, an exact cached repeat, an appended turn that should reuse almost all of the prefix, and a 256-token generation from the fully cached appended prompt. This separates cold-prefill cost from the warm interactive path and records `prompt_tokens_details.cached_tokens` when vLLM exposes it. Default depths stop at 128K; 196K/240K are opt-in because cold prefill may take hours. A cold failure stops deeper probes for that endpoint by default. All benchmark requests bypass LiteLLM and go directly to the managed vLLM container. The script releases only its own experiment leases and leaves the catalog endpoints for inspection.
+
+The reusable design takeaway is that capacity, cold ingestion, cached incremental turns, and decode-at-depth are separate properties. Once memory capacity is already above the model's native context, further KV compression is not automatically useful; measure the workload-specific bottleneck before optimizing the memory format. For long-lived agent sessions, exact-prefix cache reuse may dominate practical usability even when cold prefill is poor.