From ec124e49a71a9b2ea188daf421f51fabbd12eb4d Mon Sep 17 00:00:00 2001 From: joncrall Date: Thu, 24 Sep 2026 13:08:34 -0400 Subject: [PATCH 1/2] Start branch for 0.7.2 --- CHANGELOG.md | 5 ++++- infer_stack/__init__.py | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d9819b8..003d63cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -798,7 +798,10 @@ States the part that is not automatic: `$SLURM_JOB_GPUS` must be passed as including cards allocated to another job, and the failure surfaces later as a CUDA OOM in whichever job loses. -## Version 0.7.1 - Unreleased +## Version 0.7.2 - Unreleased + + +## Version 0.7.1 - Released 2026-09-24 ## Version 0.7.0 - Released 2026-08-28 diff --git a/infer_stack/__init__.py b/infer_stack/__init__.py index 5073941f..f6465ef9 100644 --- a/infer_stack/__init__.py +++ b/infer_stack/__init__.py @@ -1,2 +1,2 @@ __all__ = [] -__version__ = '0.7.1' +__version__ = '0.7.2' From 9e2061c5a934050902513cf70620a28ca045a6cf Mon Sep 17 00:00:00 2001 From: joncrall Date: Thu, 24 Sep 2026 15:04:20 -0400 Subject: [PATCH 2/2] Add RTX 8000 Qwen profiling round 3 Co-authored-by: GPT-5.6 Sol --- dev/journals/gpt-5.6-sol-rtx8000.md | 13 + dev/profile_qwen38_rtx8000_round3.sh | 454 +++++++++++++++++++++++++++ dev/qwen38_rtx8000_findings.md | 304 ++++++++++++++++++ 3 files changed, 771 insertions(+) create mode 100644 dev/journals/gpt-5.6-sol-rtx8000.md create mode 100755 dev/profile_qwen38_rtx8000_round3.sh create mode 100644 dev/qwen38_rtx8000_findings.md diff --git a/dev/journals/gpt-5.6-sol-rtx8000.md b/dev/journals/gpt-5.6-sol-rtx8000.md new file mode 100644 index 00000000..c40c8527 --- /dev/null +++ b/dev/journals/gpt-5.6-sol-rtx8000.md @@ -0,0 +1,13 @@ +## 2026-09-24 14:10:00 -0400 + +Summary of user intent: preserve the Quadro RTX 8000 / Qwen3.8-27B HyperQwen profiling work as durable infer-stack developer knowledge and add a third profiling round that measures the behavior that matters for a persistent Pi-style coding-agent session: long-context prefix reuse and decode speed as context grows. The user wants every tested variation represented as an explicit infer-stack endpoint and wants the benchmark harness to survive failures without disturbing unrelated leases. `BASE_URL=0.0.0.0:14042` is a manually supplied gateway address and must not be assumed to come from `OPENAI_BASE_URL`. + +Model: GPT-5.6 Sol. Configuration: tool-enabled reasoning session. + +The durable finding from rounds 1 and 2 is that the 48-GiB Turing card has a capacity surplus but a cold-prefill performance problem. W4A16 with float16 activations, FP16/auto KV, `TRITON_ATTN`, one sequence, prefix caching, and `max_model_len=262144` boots and reports roughly 415K-443K KV tokens depending on the exact arm. Short decode is roughly 33-35 tok/s; a 32K cold prompt takes roughly 230 s. Activation-int8 did not improve the measured workload, FP8 KV is rejected on the sm75 Triton path, the tested int4/KVarN paths never become useful servers, and Flex attention is blocked by the hybrid model's 400-token page geometry. The prepared `-fast` target improved short decode by about 3% without materially changing 8K/32K prefill. Round-2 MTP/DFlash2 failures are not evidence that speculation is impossible on Turing: the pinned single-user launcher explicitly refuses speculation with `--dtype=half` because its speculative verify path is BF16-specific. + +The first near-full experiment also taught an important measurement lesson: a long request routed through LiteLLM hit its 600-second timeout/retry policy while the vLLM server remained healthy, so gateway wall time cannot be used as evidence about maximum context or cold-prefill completion. Long benchmark traffic must go directly to vLLM while infer-stack continues to own the endpoint and lease. + +Round 3 deliberately avoids another broad configuration sweep. It creates only the standard and `-fast` W4A16 full-context endpoints on the known-working Triton/8192 path. At each context depth it measures a cold base request, an exact cached repeat, an appended turn that should reuse almost all of the prefix, and a 256-token generation from the fully cached appended prompt. This separates cold-prefill cost from the warm interactive path and records `prompt_tokens_details.cached_tokens` when vLLM exposes it. Default depths stop at 128K; 196K/240K are opt-in because cold prefill may take hours. A cold failure stops deeper probes for that endpoint by default. All benchmark requests bypass LiteLLM and go directly to the managed vLLM container. The script releases only its own experiment leases and leaves the catalog endpoints for inspection. + +The reusable design takeaway is that capacity, cold ingestion, cached incremental turns, and decode-at-depth are separate properties. Once memory capacity is already above the model's native context, further KV compression is not automatically useful; measure the workload-specific bottleneck before optimizing the memory format. For long-lived agent sessions, exact-prefix cache reuse may dominate practical usability even when cold prefill is poor. diff --git a/dev/profile_qwen38_rtx8000_round3.sh b/dev/profile_qwen38_rtx8000_round3.sh new file mode 100755 index 00000000..86b004f5 --- /dev/null +++ b/dev/profile_qwen38_rtx8000_round3.sh @@ -0,0 +1,454 @@ +#!/usr/bin/env bash +# Round 3: measure whether prefix caching makes full-context Qwen3.8-27B practical +# for an interactive coding-agent session on a Quadro RTX 8000 (Turing / sm75). +# +# Round 1 established that the W4A16 target with FP16/auto KV can expose the full +# 262144-token model context on this 48 GiB card. Round 2 established that TRITON_ATTN +# is the viable attention backend in the pinned image, 4096/8192 prefill chunks are +# effectively tied, and the prepared "-fast" target improves short decode modestly. +# See dev/qwen38_rtx8000_findings.md for the measurements and known dead ends. +# +# This round answers a different question: as a Pi-like conversation grows, can the +# server reuse the existing prompt prefix cheaply enough that long sessions remain +# interactive even though a cold long-context prefill is expensive? +# +# For each endpoint/context depth: +# 1. cold base prompt, max_tokens=1; +# 2. exact repeat, max_tokens=1 (pure prefix-cache reuse); +# 3. same prefix plus a small appended turn, max_tokens=1; +# 4. exact repeat of the appended turn, max_tokens=256 (decode at context depth). +# +# All benchmark traffic goes directly to vLLM inside its container. LiteLLM is not in +# the benchmark path because its 600 s request timeout/retry behavior invalidated the +# first near-full probe. Every model variant is an explicit infer-stack endpoint. +# Failures are recorded and the script continues; only this script's leases are released. +# Endpoints remain in the catalog for manual inspection. + +set -uo pipefail + +BASE_URL="${BASE_URL:-0.0.0.0:14042}" # metadata / infer-stack sanity only +MODEL_NAME="qwen3.8-27b-dbirks-hyperqwen" +MODEL_SOURCE="hf://dbirks/Qwen3.8-27B-W4A16-AutoRound" +IMAGE="ghcr.io/syv-ai/hyperqwen:sha-684e927" +GPU_NAME="${GPU_NAME:-Quadro RTX 8000}" +GPU_UTIL="${GPU_UTIL:-0.90}" +BOOT_TIMEOUT="${BOOT_TIMEOUT:-1200}" +WAIT_INTERVAL="${WAIT_INTERVAL:-5}" +MAX_MODEL_LEN="262144" +MAX_NUM_BATCHED_TOKENS="8192" +APPEND_MAX_TOKENS="${APPEND_MAX_TOKENS:-256}" +WARM_TIMEOUT="${WARM_TIMEOUT:-1800}" +STOP_AFTER_COLD_FAILURE="${STOP_AFTER_COLD_FAILURE:-1}" +RUN_DEEP="${RUN_DEEP:-0}" +CONTEXT_TARGETS="${CONTEXT_TARGETS:-8192 32000 65536 131072}" +if [[ "$RUN_DEEP" == 1 ]]; then + CONTEXT_TARGETS="$CONTEXT_TARGETS 196608 240000" +fi +RESULT_DIR="${RESULT_DIR:-$PWD/dev/benchmark-results/qwen38-rtx8000-round3-$(date +%Y%m%dT%H%M%S)}" + +mkdir -p "$RESULT_DIR" +SUMMARY_TSV="$RESULT_DIR/summary.tsv" +RUN_LOG="$RESULT_DIR/run.log" +exec > >(tee -a "$RUN_LOG") 2>&1 + +log() { printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; } +need_cmd() { command -v "$1" >/dev/null 2>&1 || { log "ERROR: missing command: $1"; exit 2; }; } +for cmd in infer-stack nvidia-smi docker curl jq awk sed grep timeout python3; do need_cmd "$cmd"; done + +GPU_INDEX="$(nvidia-smi --query-gpu=index,name,memory.total --format=csv,noheader,nounits | awk -F',' -v want="$GPU_NAME" ' + index($2, want) { gsub(/[[:space:]]/, "", $1); print $1; exit } +')" +[[ -n "$GPU_INDEX" ]] || { log "ERROR: could not find GPU matching: $GPU_NAME"; exit 2; } + +log "Using GPU $GPU_INDEX" +nvidia-smi -i "$GPU_INDEX" --query-gpu=index,name,compute_cap,memory.total,memory.used,memory.free,power.limit --format=csv,noheader || true + +{ + echo "date=$(date --iso-8601=seconds)" + echo "base_url=$BASE_URL" + echo "gpu_index=$GPU_INDEX" + echo "gpu_name=$GPU_NAME" + echo "gpu_util=$GPU_UTIL" + echo "max_model_len=$MAX_MODEL_LEN" + echo "max_num_batched_tokens=$MAX_NUM_BATCHED_TOKENS" + echo "context_targets=$CONTEXT_TARGETS" + echo "run_deep=$RUN_DEEP" + echo "infer_stack_git=$(git rev-parse HEAD 2>/dev/null || true)" + echo "host=$(hostname)" + nvidia-smi -i "$GPU_INDEX" --query-gpu=index,name,compute_cap,memory.total,driver_version --format=csv,noheader || true +} > "$RESULT_DIR/metadata.txt" + +log "Ensuring model entry exists" +infer-stack catalog model add "$MODEL_NAME" --source "$MODEL_SOURCE" --force + +# Fields: endpoint|model_path|notes +VARIANTS=( + "qwen38-rtx8000-r3-standard-262k|/app/models/Qwen3.8-27B-W4A16-AutoRound|standard prepared W4A16 target" + "qwen38-rtx8000-r3-fast-262k|/app/models/Qwen3.8-27B-W4A16-AutoRound-fast|HyperQwen fast target checkpoint" +) + +cleanup_endpoint_leases() { + local endpoint="$1" ids id + ids="$(infer-stack leases --json 2>/dev/null | jq -r --arg ep "$endpoint" ' + .leases[]? | select(.state == "active") | select((.endpoints // []) | index($ep)) | .id + ' 2>/dev/null || true)" + while IFS= read -r id; do + [[ -n "$id" ]] || continue + infer-stack release "$id" --evict --yes >/dev/null 2>&1 || true + done <<< "$ids" +} + +CURRENT_ENDPOINT="" +CURRENT_ENV_FILE="" +cleanup_current() { + if [[ -n "$CURRENT_ENV_FILE" && -f "$CURRENT_ENV_FILE" ]]; then + infer-stack release --env-file "$CURRENT_ENV_FILE" --evict --yes >/dev/null 2>&1 || true + fi + [[ -n "$CURRENT_ENDPOINT" ]] && cleanup_endpoint_leases "$CURRENT_ENDPOINT" + CURRENT_ENDPOINT="" + CURRENT_ENV_FILE="" +} +trap cleanup_current EXIT INT TERM + +add_endpoint() { + local endpoint="$1" model_path="$2" + # batch/start_qwen.sh does not have a native FP16/auto-KV profile: its default + # KV=fp8 is an Ampere+/FlashInfer-oriented profile. Keep the launcher's preparation, + # tool parser and prefix-cache integration, but override the KV dtype/backend in the + # trailing EXTRA_ARGS, which is deliberately expanded last by HyperQwen. + local extra env_yaml + extra="--served-model-name={served_model_name} --dtype=half --kv-cache-dtype=auto --attention-backend=TRITON_ATTN --max-num-batched-tokens=${MAX_NUM_BATCHED_TOKENS}" + env_yaml="{PORT: \"{port}\", VERIFY: 0, MODEL: \"$model_path\", KV: fp8, PREFIX_CACHE: 1, MAX_LEN: \"{max_model_len}\", MAX_SEQS: 1, GPU_UTIL: \"{gpu_memory_utilization}\", INT8_ACT: \"\", INT8_LAYERS: \"\", EXTRA_ARGS: \"$extra\"}" + + infer-stack catalog endpoint add "$endpoint" \ + --engine vllm \ + --model "$MODEL_NAME" \ + --min-vram-gib 40 \ + --gpu "$GPU_INDEX" \ + --max-model-len "$MAX_MODEL_LEN" \ + --gpu-mem "$GPU_UTIL" \ + --reclaim stop \ + --force \ + --runtime \ + "image=$IMAGE" \ + 'enable_prefix_caching=true' \ + 'max_num_seqs=1' \ + 'command=[batch]' \ + "env=$env_yaml" \ + 'mounts={"/app/models": "hyperqwen/qwen3.8-27b/models", "/cache": "hyperqwen/qwen3.8-27b/cache"}' +} + +for row in "${VARIANTS[@]}"; do + IFS='|' read -r endpoint model_path notes <<< "$row" + log "Catalog: $endpoint -- $notes" + add_endpoint "$endpoint" "$model_path" +done +infer-stack catalog validate + +printf 'endpoint\tmodel_path\ttarget_tokens\tstatus\tboot_seconds\tkv_cache_tokens\tmax_concurrency\tcold_prompt_tokens\tcold_cached_tokens\tcold_wall_s\trepeat_cached_tokens\trepeat_wall_s\tappend_prompt_tokens\tappend_cached_tokens\tappend_wall_s\tdecode_prompt_tokens\tdecode_cached_tokens\tdecode_completion_tokens\tdecode_wall_s\tdecode_tok_s\tnotes\terror\n' > "$SUMMARY_TSV" + +sanitize() { printf '%s' "$1" | tr '\t\r\n' ' '; } +append_result() { + local endpoint="$1" model_path="$2" target="$3" status="$4" boot="$5" kv="$6" conc="$7" + local cold_pt="$8" cold_ct="$9" cold_w="${10}" repeat_ct="${11}" repeat_w="${12}" + local append_pt="${13}" append_ct="${14}" append_w="${15}" decode_pt="${16}" decode_ct="${17}" + local decode_comp="${18}" decode_w="${19}" decode_tps="${20}" notes="${21}" error="${22}" + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ + "$(sanitize "$endpoint")" "$(sanitize "$model_path")" "$(sanitize "$target")" "$(sanitize "$status")" \ + "$(sanitize "$boot")" "$(sanitize "$kv")" "$(sanitize "$conc")" "$(sanitize "$cold_pt")" \ + "$(sanitize "$cold_ct")" "$(sanitize "$cold_w")" "$(sanitize "$repeat_ct")" "$(sanitize "$repeat_w")" \ + "$(sanitize "$append_pt")" "$(sanitize "$append_ct")" "$(sanitize "$append_w")" "$(sanitize "$decode_pt")" \ + "$(sanitize "$decode_ct")" "$(sanitize "$decode_comp")" "$(sanitize "$decode_w")" "$(sanitize "$decode_tps")" \ + "$(sanitize "$notes")" "$(sanitize "$error")" >> "$SUMMARY_TSV" +} + +find_container() { + docker ps -a --filter "name=infer-stack-vllm-$1" --format '{{.Names}}' | head -n 1 +} + +capture_logs() { + local container="$1" outfile="$2" + [[ -n "$container" ]] && docker logs "$container" > "$outfile" 2>&1 || true +} + +parse_boot_facts() { + local logf="$1" + KV_CACHE_TOKENS="$(grep -Eo 'GPU KV cache size: [0-9,]+ tokens' "$logf" | tail -1 | sed -E 's/.*size: ([0-9,]+) tokens/\1/' | tr -d ',' || true)" + MAX_CONCURRENCY="$(grep -Eo 'Maximum concurrency for [0-9,]+ tokens per request: [0-9.]+x' "$logf" | tail -1 | sed -E 's/.*request: ([0-9.]+)x/\1/' || true)" +} + +wait_with_crash_detection() { + local endpoint="$1" wait_log="$2" deadline=$(( $(date +%s) + BOOT_TIMEOUT )) wait_pid container status restarts + infer-stack wait "$endpoint" --timeout "$BOOT_TIMEOUT" --interval "$WAIT_INTERVAL" > "$wait_log" 2>&1 & + wait_pid=$! + while kill -0 "$wait_pid" >/dev/null 2>&1; do + if (( $(date +%s) >= deadline )); then + kill "$wait_pid" >/dev/null 2>&1 || true + wait "$wait_pid" >/dev/null 2>&1 || true + echo 'timeout waiting for readiness' >> "$wait_log" + return 1 + fi + container="$(find_container "$endpoint")" + if [[ -n "$container" ]]; then + status="$(docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null || true)" + restarts="$(docker inspect -f '{{.RestartCount}}' "$container" 2>/dev/null || echo 0)" + if [[ "$status" == restarting || "$status" == dead || "$status" == exited ]]; then + sleep 8 + status="$(docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null || true)" + restarts="$(docker inspect -f '{{.RestartCount}}' "$container" 2>/dev/null || echo 0)" + if [[ "$status" == restarting || "$status" == dead || "$status" == exited ]]; then + echo "container failure: status=$status restarts=$restarts" >> "$wait_log" + kill "$wait_pid" >/dev/null 2>&1 || true + wait "$wait_pid" >/dev/null 2>&1 || true + return 1 + fi + fi + fi + sleep 10 + done + wait "$wait_pid" +} + +# Direct request to vLLM. This intentionally bypasses LiteLLM, whose 600-second +# request timeout/retry policy made the round-1 near-full measurement unusable. +direct_request_file() { + local container="$1" request_file="$2" response_file="$3" timeout_s="$4" + local meta="$response_file.curl.txt" + if ! timeout "$((timeout_s + 30))" bash -c \ + 'docker exec -i "$1" curl -sS --max-time "$2" -o /tmp/infer-stack-r3-response.json -w "%{http_code}\t%{time_total}\n" -H "Content-Type: application/json" --data-binary @- http://127.0.0.1:8000/v1/chat/completions < "$3" > "$4" && docker exec "$1" cat /tmp/infer-stack-r3-response.json > "$5"' \ + _ "$container" "$timeout_s" "$request_file" "$meta" "$response_file"; then + return 1 + fi + [[ "$(cut -f1 "$meta" 2>/dev/null || true)" == 200 ]] || return 1 + jq -e '.choices | length > 0' "$response_file" >/dev/null 2>&1 +} + +make_request() { + local endpoint="$1" prompt_file="$2" max_tokens="$3" request_file="$4" + jq -n --arg model "$endpoint" --rawfile prompt "$prompt_file" --argjson max_tokens "$max_tokens" \ + '{model:$model,messages:[{role:"user",content:$prompt}],temperature:0,max_tokens:$max_tokens}' > "$request_file" +} + +make_prompt() { + local container="$1" target="$2" tag="$3" outfile="$4" + log "Generating tokenizer-measured ~${target}-token prompt: $tag" + docker exec -i "$container" /app/venv/bin/python - "$target" "$tag" <<'PY' > "$outfile" +import sys +from transformers import AutoTokenizer + +target = int(sys.argv[1]) +tag = sys.argv[2] +tok = AutoTokenizer.from_pretrained('/app/models/Qwen3.8-27B-W4A16-AutoRound') +tok.model_max_length = 10**12 +unit = ( + f'[{tag}] A software repository contains source files, tests, API contracts, ' + 'design notes, invariants, dependency metadata, and repeated deterministic context ' + 'for a prefix-cache benchmark. Preserve exact earlier context across turns.\n' +) + +def count(n): + return len(tok.encode(unit * n, add_special_tokens=False)) + +lo, hi = 0, max(2, target // max(1, count(1)) * 2 + 16) +while count(hi) < target: + hi *= 2 +while lo + 1 < hi: + mid = (lo + hi) // 2 + if count(mid) <= target: + lo = mid + else: + hi = mid +text = unit * lo +body = len(tok.encode(text, add_special_tokens=False)) +text += f'\n[{tag} target={target} generated_body_tokens={body}]\n' +sys.stdout.write(text) +PY +} + +make_appended_prompt() { + local base="$1" target="$2" outfile="$3" + cat "$base" > "$outfile" + cat >> "$outfile" </dev/null || printf '0' +} + +cold_timeout_for_target() { + local target="$1" + if (( target <= 8192 )); then echo "${COLD_TIMEOUT_8K:-900}" + elif (( target <= 32000 )); then echo "${COLD_TIMEOUT_32K:-1800}" + elif (( target <= 65536 )); then echo "${COLD_TIMEOUT_64K:-3600}" + elif (( target <= 131072 )); then echo "${COLD_TIMEOUT_128K:-10800}" + elif (( target <= 196608 )); then echo "${COLD_TIMEOUT_192K:-18000}" + else echo "${COLD_TIMEOUT_240K:-21600}" + fi +} + +run_request() { + local container="$1" endpoint="$2" prompt_file="$3" max_tokens="$4" response="$5" timeout_s="$6" + local request="$response.request.json" + make_request "$endpoint" "$prompt_file" "$max_tokens" "$request" + direct_request_file "$container" "$request" "$response" "$timeout_s" +} + +run_context_probe() { + local container="$1" endpoint="$2" model_path="$3" target="$4" outdir="$5" boot="$6" notes="$7" + local prompt="$RESULT_DIR/prompts/${target}.txt" + local appended="$RESULT_DIR/prompts/${target}-appended.txt" + local timeout_s + timeout_s="$(cold_timeout_for_target "$target")" + mkdir -p "$RESULT_DIR/prompts" "$outdir" + [[ -s "$prompt" ]] || make_prompt "$container" "$target" "round3-${target}" "$prompt" || return 1 + [[ -s "$appended" ]] || make_appended_prompt "$prompt" "$target" "$appended" + + local cold="$outdir/cold-${target}.json" + local repeat="$outdir/repeat-${target}.json" + local append="$outdir/append-${target}.json" + local decode="$outdir/decode-${target}.json" + local error="" status="ok" + + log " context=$target: cold base request (timeout=${timeout_s}s)" + if ! run_request "$container" "$endpoint" "$prompt" 1 "$cold" "$timeout_s"; then + error="cold request failed or timed out" + status="cold_failed" + append_result "$endpoint" "$model_path" "$target" "$status" "$boot" "$KV_CACHE_TOKENS" "$MAX_CONCURRENCY" \ + "" "" "" "" "" "" "" "" "" "" "" "" "" "$notes" "$error" + return 1 + fi + + local cold_pt cold_ct cold_w repeat_ct repeat_w append_pt append_ct append_w decode_pt decode_ct decode_comp decode_w decode_tps + cold_pt="$(usage_field "$cold" '.usage.prompt_tokens')" + cold_ct="$(usage_field "$cold" '.usage.prompt_tokens_details.cached_tokens')" + cold_w="$(cut -f2 "$cold.curl.txt" 2>/dev/null || true)" + + log " context=$target: exact cached repeat" + if run_request "$container" "$endpoint" "$prompt" 1 "$repeat" "$WARM_TIMEOUT"; then + repeat_ct="$(usage_field "$repeat" '.usage.prompt_tokens_details.cached_tokens')" + repeat_w="$(cut -f2 "$repeat.curl.txt" 2>/dev/null || true)" + else + repeat_ct=""; repeat_w=""; status="partial"; error="exact repeat failed" + fi + + log " context=$target: append a small new turn over the cached prefix" + if run_request "$container" "$endpoint" "$appended" 1 "$append" "$WARM_TIMEOUT"; then + append_pt="$(usage_field "$append" '.usage.prompt_tokens')" + append_ct="$(usage_field "$append" '.usage.prompt_tokens_details.cached_tokens')" + append_w="$(cut -f2 "$append.curl.txt" 2>/dev/null || true)" + else + append_pt=""; append_ct=""; append_w=""; status="partial"; error="${error:+$error; }append turn failed" + fi + + log " context=$target: 256-token decode from the fully cached appended prompt" + if run_request "$container" "$endpoint" "$appended" "$APPEND_MAX_TOKENS" "$decode" "$WARM_TIMEOUT"; then + decode_pt="$(usage_field "$decode" '.usage.prompt_tokens')" + decode_ct="$(usage_field "$decode" '.usage.prompt_tokens_details.cached_tokens')" + decode_comp="$(usage_field "$decode" '.usage.completion_tokens')" + decode_w="$(cut -f2 "$decode.curl.txt" 2>/dev/null || true)" + decode_tps="$(awk -v t="$decode_comp" -v w="$decode_w" 'BEGIN{if(w>0)printf "%.3f",t/w}')" + else + decode_pt=""; decode_ct=""; decode_comp=""; decode_w=""; decode_tps="" + status="partial"; error="${error:+$error; }cached decode failed" + fi + + append_result "$endpoint" "$model_path" "$target" "$status" "$boot" "$KV_CACHE_TOKENS" "$MAX_CONCURRENCY" \ + "$cold_pt" "$cold_ct" "$cold_w" "$repeat_ct" "$repeat_w" "$append_pt" "$append_ct" "$append_w" \ + "$decode_pt" "$decode_ct" "$decode_comp" "$decode_w" "$decode_tps" "$notes" "$error" + + log " context=$target: cold=${cold_w:-NA}s repeat=${repeat_w:-NA}s repeat_cached=${repeat_ct:-NA}/${cold_pt:-NA} append=${append_w:-NA}s append_cached=${append_ct:-NA}/${append_pt:-NA} decode=${decode_tps:-NA} tok/s" + return 0 +} + +run_variant() { + local row="$1" + local endpoint model_path notes + IFS='|' read -r endpoint model_path notes <<< "$row" + local outdir="$RESULT_DIR/$endpoint" env_file="$RESULT_DIR/$endpoint/lease.env" + mkdir -p "$outdir" + cleanup_current + cleanup_endpoint_leases "$endpoint" + CURRENT_ENDPOINT="$endpoint" + CURRENT_ENV_FILE="$env_file" + rm -f "$env_file" + + log "======================================================================" + log "$endpoint -- $notes" + infer-stack catalog endpoint show "$endpoint" > "$outdir/catalog.yaml" 2>&1 || true + local start boot container err + start="$(date +%s)" + if ! timeout "$BOOT_TIMEOUT" infer-stack acquire "$endpoint" --allowed_gpus "$GPU_INDEX" --env-file "$env_file" --no-wait --yes > "$outdir/acquire.log" 2>&1; then + boot=$(( $(date +%s) - start )) + append_result "$endpoint" "$model_path" "boot" acquire_failed "$boot" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "$notes" "acquire failed" + cleanup_current + return 1 + fi + if ! wait_with_crash_detection "$endpoint" "$outdir/wait.log"; then + boot=$(( $(date +%s) - start )) + container="$(find_container "$endpoint")" + capture_logs "$container" "$outdir/container.log" + parse_boot_facts "$outdir/container.log" + err="$(grep -E 'ERROR|ValueError|RuntimeError|OutOfMemory|not supported|requires' "$outdir/container.log" | tail -n 5 | tr '\n' ' ')" + append_result "$endpoint" "$model_path" "boot" readiness_failed "$boot" "$KV_CACHE_TOKENS" "$MAX_CONCURRENCY" "" "" "" "" "" "" "" "" "" "" "" "" "" "$notes" "$err" + cleanup_current + return 1 + fi + + boot=$(( $(date +%s) - start )) + container="$(find_container "$endpoint")" + capture_logs "$container" "$outdir/container.log" + parse_boot_facts "$outdir/container.log" + infer-stack test "$endpoint" --max-tokens 32 --timeout 300 > "$outdir/infer-stack-test.log" 2>&1 || true + + local target failed=0 + for target in $CONTEXT_TARGETS; do + if ! run_context_probe "$container" "$endpoint" "$model_path" "$target" "$outdir" "$boot" "$notes"; then + failed=1 + if [[ "$STOP_AFTER_COLD_FAILURE" == 1 ]]; then + log " stopping deeper contexts for $endpoint after cold failure at $target" + break + fi + fi + done + + capture_logs "$container" "$outdir/container.log" + cleanup_current + return "$failed" +} + +log "Removing active leases for round-3 endpoints" +for row in "${VARIANTS[@]}"; do + IFS='|' read -r endpoint _ <<< "$row" + cleanup_endpoint_leases "$endpoint" +done + +for row in "${VARIANTS[@]}"; do + run_variant "$row" || true +done + +python3 - "$SUMMARY_TSV" "$RESULT_DIR/summary.csv" <<'PY' +import csv +import sys +src, dst = sys.argv[1:] +with open(src, newline='') as f, open(dst, 'w', newline='') as g: + csv.writer(g).writerows(csv.reader(f, delimiter='\t')) +PY + +# A compact view sorted by context then cached-decode throughput descending. +{ + head -n1 "$SUMMARY_TSV" + tail -n +2 "$SUMMARY_TSV" | sort -t $'\t' -k3,3n -k20,20nr +} > "$RESULT_DIR/ranking.tsv" + +log "======================================================================" +log "Round 3 complete" +log "Summary: $SUMMARY_TSV" +log "Ranking: $RESULT_DIR/ranking.tsv" +log "Logs/responses: $RESULT_DIR" +log "All round-3 experiment leases have been released; endpoints remain in the catalog." +column -t -s $'\t' "$RESULT_DIR/ranking.tsv" 2>/dev/null || cat "$RESULT_DIR/ranking.tsv" diff --git a/dev/qwen38_rtx8000_findings.md b/dev/qwen38_rtx8000_findings.md new file mode 100644 index 00000000..fed59a39 --- /dev/null +++ b/dev/qwen38_rtx8000_findings.md @@ -0,0 +1,304 @@ +# Qwen3.8-27B on Quadro RTX 8000: infer-stack profiling notes + +This document records the measured Qwen3.8-27B / HyperQwen behavior on a +48-GiB Quadro RTX 8000 so future work does not have to rediscover the same +Turing-specific constraints. It is intentionally empirical: measured results +are separated from hypotheses and from follow-up ideas. + +## Scope and pinned setup + +The experiments use infer-stack custom endpoints around the HyperQwen image +`ghcr.io/syv-ai/hyperqwen:sha-684e927`, based on HyperQwen commit +`684e9277f163d1701d6179194c7f6bc1b9175d44`. The target model identity in +infer-stack is `qwen3.8-27b-dbirks-hyperqwen`, sourced from +`dbirks/Qwen3.8-27B-W4A16-AutoRound`. + +Measured host GPU: + +- NVIDIA Quadro RTX 8000 +- 49,152 MiB physical VRAM (about 47.27 GiB visible to vLLM) +- compute capability 7.5 (Turing / sm75) +- 260 W power limit in the recorded runs + +The performance goal is a single-user coding-agent endpoint with the full +Qwen3.8 model context (`262144` tokens), prioritizing usable long context over +maximum short-prompt throughput. + +The benchmark scripts deliberately create explicit infer-stack endpoints. The +hardware is consulted while constructing the endpoints, but acquire/apply does +not silently retune them. Benchmark requests that may exceed LiteLLM's gateway +timeout go directly to the vLLM API inside the managed container; infer-stack +still owns endpoint catalog data, placement, lease lifecycle, persistent model +mounts, and routing. + +## Executive summary + +The card has enough memory for the full 262,144-token model context with the +prepared W4A16 target and ordinary FP16/`auto` KV. Low-bit KV is not required +for capacity. The working path is `dtype=float16`, `TRITON_ATTN`, prefix +caching on, one sequence, and no speculative decoding. + +The principal limitation is cold long-prompt prefill performance, not memory. +A measured ~32K prompt takes about 230 seconds on the working sm75 Triton path. +Short decode is about 33-35 tokens/s. HyperQwen's prepared `-fast` target gives +about a 3% short-decode improvement without materially changing 8K/32K prefill. + +Round 3 therefore measures the metric that matters for a persistent coding +agent: whether prefix caching makes subsequent turns cheap as the conversation +history grows. It compares the normal and `-fast` target at increasing context +depths and records cold prefill, exact cached reuse, an appended-turn reuse, +and 256-token decode from a fully cached long prompt. + +## Round 1: capacity and compatibility matrix + +Script: `dev/profile_qwen38_rtx8000.sh` + +### Full-context W4A16 + FP16/auto KV works + +The conservative endpoint booted with: + +- `dtype=torch.float16` +- `max_seq_len=262144` +- `speculative_config=None` +- `kv_cache_dtype=auto` +- prefix caching enabled +- `max_num_seqs=1` +- `TRITON_ATTN` +- Marlin W4A16 linear kernels +- Triton/FLA GDN prefill and Triton GDN decode + +A representative boot reported a 443,110-token GPU KV cache and maximum +concurrency 1.69x at 262,144 tokens. At `gpu_memory_utilization=0.90`, vLLM +reported about 27.34 GiB of KV cache while the model/other consumed memory was +about 14.71 GiB. There is therefore ample capacity for one full-context request +without low-bit KV compression. + +Measured short decode for the plain W4A16 endpoint was 33.790 completion +tokens/s. A 32,048-token request completed in 228.47 seconds. + +### Turing backend behavior + +FlashAttention 2 is not available on sm75. vLLM automatically selected +`TRITON_ATTN` from the compatible candidates. Qwen's fused GDN decode kernel +also requires compute capability 8.0+, so the model fell back to the Triton GDN +decode path. These fallbacks are expected for this card and are not boot +failures. + +### Activation int8 did not help + +Two activation-int8 variants were tested against the same FP16/auto-KV base: + +| Variant | short decode | 32K wall | KV tokens | +| --- | ---: | ---: | ---: | +| W4A16 baseline | 33.790 tok/s | 228.47 s | 443,110 | +| int8 gate-up | 33.240 tok/s | 230.92 s | 416,183 | +| int8 MLP | 32.709 tok/s | 225.45 s | 413,807 | + +The int8 variants were not faster in the measured interactive decode workload +and reduced available KV capacity. They are not current candidates for the +RTX 8000 endpoint. + +### Low-bit KV paths were dead ends for this pinned stack + +FP8 KV failed explicitly because the Triton attention backend on this Quadro +RTX 8000 does not support that FP8 KV path; the reported native FP8 requirement +was SM89+. + +`int4_per_token_head` produced very large nominal KV pools (~1.66-1.68M tokens) +but the engine later died and never became a usable endpoint. That path is not +needed for capacity and is not worth pursuing until there is a separate reason +to debug its sm75 failure. + +KVarN similarly produced ~1.98M nominal KV tokens, but the tested path ran into +HyperQwen/vLLM assumptions that select an FA2-dependent route requiring compute +capability 8.0+. MTP/DFlash2 + KVarN combinations also failed. Since ordinary +FP16 KV already exceeds the model's maximum context by a wide margin, these +compression paths solve no current capacity problem on the 48-GiB card. + +### The first near-full test did not prove a model failure + +The first ~260K probe went through LiteLLM. LiteLLM returned HTTP 408 after its +600-second timeout and the harness spent about 2409 seconds because gateway +retry behavior repeated the long request. During that period the vLLM +container remained healthy and continued returning HTTP 200 to `/health`. + +Therefore the round-1 `near-full context probe failed` result must not be read +as evidence that 262K is unsupported. It is a gateway-timeout artifact. Later +long-context benchmark traffic should go directly to vLLM. + +## Round 2: attention backend and prefill chunk size + +Script: `dev/profile_qwen38_rtx8000_round2.sh` + +Round 2 held W4A16 + FP16/auto KV + 262,144 context fixed and swept attention +backend and `max_num_batched_tokens`. Benchmark traffic went directly to the +vLLM container. + +### Triton is the working attention backend + +The successful base measurements were: + +| Endpoint shape | short decode | ~8K wall | ~32K wall | KV tokens | +| --- | ---: | ---: | ---: | ---: | +| Triton / 2048 | 26.151 tok/s* | 23.533 s | 231.348 s | 417,371 | +| Triton / 4096 | 33.418 tok/s | 23.483 s | 230.728 s | 416,579 | +| Triton / 8192 | 33.443 tok/s | **23.296 s** | **230.547 s** | 414,995 | +| Triton / 16384 | 33.386 tok/s | 23.770 s | 233.624 s | 392,424 | + +`*` The 2048 short-decode result conflicts with the earlier ~33.8 tok/s +measurement at essentially the same configuration and is likely contaminated +by first-arm JIT/cache effects. Do not infer that 2048 intrinsically harms +decode from this one number. + +4096 and 8192 are effectively tied. 8192 was marginally fastest at 8K and 32K, +while 16384 was slower and reduced KV capacity. Round 3 uses 8192 as the +provisional operating point. + +### FLEX_ATTENTION was blocked by hybrid page geometry + +All tested Flex endpoints failed before serving. The actionable error was: + +```text +flex_attn_kv_block_size must be a power of 2 and divisible by flex_attn_block_n, +got 400, None +``` + +Qwen's hybrid attention/Mamba geometry caused vLLM to select a 400-token page, +which is incompatible with this Flex requirement. This is not a benchmark +showing Flex is slower on Turing; Flex was never successfully measured. Fixing +that geometry may be interesting later, but it is not required to reach full +context. + +### The HyperQwen `-fast` target improves decode modestly + +On Triton / 8192, the prepared fast target measured: + +| Target | short decode | ~8K wall | ~32K wall | KV tokens | +| --- | ---: | ---: | ---: | ---: | +| standard | 33.443 tok/s | 23.296 s | 230.547 s | 414,995 | +| `-fast` | **34.554 tok/s** | 23.443 s | 230.868 s | **425,291** | + +The ~3.3% short-decode improvement is real enough to keep the fast target as a +performance candidate; no meaningful prefill improvement was measured. Quality +parity was not established by these performance experiments, so retain the +standard target as the conservative comparison rather than silently replacing +it. + +### Round-2 speculation failures were configuration refusals, not a Turing verdict + +The round-2 MTP/DFlash2 endpoints failed in 13-21 seconds, before normal model +loading. The script used `--dtype=half`, while the pinned HyperQwen +`single-user/start_qwen.sh` explicitly refuses FP16 with speculation: + +```text +--dtype float16 needs SPEC=off: this repo's speculative path is bf16-only. +``` + +HyperQwen documents a BF16 assumption in its speculative verify attention +kernel. Therefore those failures do **not** establish that MTP or DFlash2 is +fundamentally impossible on sm75. Efficient speculation on this Turing card +would likely require changing/validating HyperQwen's speculative kernels for +FP16. Merely switching this card to BF16 is not obviously desirable because +Turing lacks Ampere-class native BF16 tensor-core support. Treat speculation as +a separate future kernel-porting project, not part of the current endpoint +configuration search. + +## What is established versus still unknown + +Established: + +- The RTX 8000 has enough VRAM for Qwen3.8-27B W4A16 at `max_model_len=262144` + with FP16/auto KV. +- `TRITON_ATTN` is a working attention backend in the pinned image on sm75. +- Short decode is approximately 33-35 tok/s on the viable configurations. +- Cold 32K prefill is approximately 230 seconds. +- 4096/8192 prefill chunks are effectively tied; 16384 is worse. +- The `-fast` target improves short decode by about 3% without improving cold + prefill. +- FP8 KV, the tested int4 KV path, and the tested KVarN path are not useful + candidates on this card with the pinned stack. +- The first near-full LiteLLM result was a gateway timeout, not proof of a vLLM + context-capacity failure. + +Not yet established: + +- cold prefill time at 64K, 128K, 196K, or 240K; +- how much of a long prompt vLLM's hybrid prefix cache actually reuses on this + model/card; +- latency of an appended coding-agent turn after a large prefix is cached; +- decode throughput at 32K/64K/128K/240K context depth; +- quality parity of the normal and `-fast` prepared targets; +- whether a modified Flex page geometry would improve sm75 prefill; +- whether HyperQwen's speculative kernels can be ported profitably to FP16/sm75. + +Do not extrapolate the 32K cold-prefill time to 262K as a claimed measurement. +The observed scaling is strongly nonlinear, so a naive fit can suggest hours, +but no near-full direct-vLLM cold run has completed yet. + +## Round 3: prefix-cache and context-depth experiment + +Script: `dev/profile_qwen38_rtx8000_round3.sh` + +Round 3 intentionally narrows the matrix to two endpoints: + +- standard W4A16 target; +- HyperQwen `-fast` target. + +Both use: + +- `TRITON_ATTN`; +- `max_num_batched_tokens=8192`; +- `dtype=float16`; +- FP16/`auto` KV; +- `max_model_len=262144`; +- prefix caching enabled; +- one sequence; +- no speculative decoding. + +At each context depth the script performs four direct-vLLM requests: + +1. a cold base prompt with one generated token; +2. the exact base prompt again, measuring cached-token reuse and warm wall time; +3. the same long prefix plus a small appended turn, again measuring cached + tokens and wall time; +4. the appended prompt again with 256 generated tokens, measuring decode speed + when the long prompt is fully cached. + +Default context targets are 8K, 32K, 64K, and 128K. The expensive 196K and +240K cold probes are opt-in with `RUN_DEEP=1`. If a cold request times out, the +script records the failure and by default stops trying deeper contexts for that +endpoint instead of wasting additional hours. + +Run the default experiment with: + +```bash +cd ~/code/infer_stack +BASE_URL=0.0.0.0:14042 ./dev/profile_qwen38_rtx8000_round3.sh +``` + +After the ordinary run establishes whether warm-prefix behavior remains useful +through 128K, request the deep proof with: + +```bash +cd ~/code/infer_stack +BASE_URL=0.0.0.0:14042 RUN_DEEP=1 ./dev/profile_qwen38_rtx8000_round3.sh +``` + +Results are written under +`dev/benchmark-results/qwen38-rtx8000-round3-/`, including +`summary.tsv`, `summary.csv`, per-endpoint request/response files, container +logs, and the exact rendered endpoint catalog entries. + +## Current endpoint guidance + +For a conservative manually selected endpoint, use the standard W4A16 target +with Triton, FP16/auto KV, 8192 batched tokens, prefix caching, and full 262K +model length. For a performance candidate, use the otherwise identical `-fast` +target; it has the best measured short decode so far, but its quality parity +should be checked independently. + +Do not choose low-bit KV merely to obtain full context on this 48-GiB card: the +working FP16/auto-KV path already has substantially more KV capacity than one +262K request needs. The next decision should be based on round-3 warm-prefix +latency and decode-at-depth, because those measurements directly represent the +persistent coding-agent workload.