[job] AMD launchers: one thinking switch and a serve-flag cleanup #146

Activity

1. AzeezIsh

   added

   jobA piece of work anyone can pick up. Shows on atlasinference.dev/#jobsA piece of work anyone can pick up. Shows on atlasinference.dev/#jobs

   on Oct 1, 2026
2. AzeezIsh commented on Oct 1, 2026

   

   AzeezIsh

   on Oct 1, 2026

   ContributorAuthor

   More actions

   Flag inventory for step 3 (research only; nothing changed yet). Sources: the three Windows recipes on #80's run-mode branch, serve-amd.sh on main plus the thinking switch, the tester's command line, and engine defaults from serve_args.rs. Rows marked "launcher-derived" are platform sizing; "MODEL.toml default" rows are per-model behaviour to freeze; "diagnostic-only" rows stay flags and never defaults.

   flag / env	Win 27B MTP (first_run)	Win DF2	Win FNext	Linux serve-amd.sh	Tester	Engine default / what it does	Evidence for recipe value	Proposed home
   MODEL (positional)	$ModelDir	$ModelDir	$ModelDir	nvidia/Qwen3.8-27B-NVFP4 or $1	(implied path)	none — model to serve	—	launcher arg
   --model-name	$ModelName (env ATLAS_MODEL_NAME)	nvidia/Qwen3.8-27B-NVFP4	nvidia/Qwen3.8-Flash-Next-NVFP4	MODEL_NAME env if set	—	positional MODEL, else config _name_or_path (serve_args.rs:118-121)	API name for local snapshots	launcher-derived
   --host/--bind	$BindHost (env ATLAS_BIND, default 127.0.0.1; wildcard 0.0.0.0 refused)	$BindHost param/ATLAS_BIND (127.0.0.1)	$BindHost (127.0.0.1, wildcard refused)	HOST env (default 0.0.0.0)	—	127.0.0.1 (:997)	bind-safety convention, perf-leg driver	launcher-derived
   --port	$Port/ATLAS_PORT (8081)	$Port/ATLAS_PORT (8096)	$Port (8095)	PORT env (8081)	—	8888 (:144-145)	recipe separation	launcher-derived
   --no-fast-load	yes	yes	yes	— (absent)	yes	false (:866); mmap/fast weight path off	frozen recipe bit for determinism; tester carries it	MODEL.toml? diagnostic-only
   --no-tui	—	yes	yes	—	—	false (:872); plain logs for harnesses	recipe is headless-driven	launcher-derived
   --max-seq-len	ATLAS_MAX_SEQ_LEN (4096)	SeqLen 49152	SEQ_LEN env (8192)	MAX_SEQ_LEN env (4096)	65536	32768 (:152-153)	first_run 4096 = memory-conservative fingerprint recipe (first_run.ps1:534); DF2 49152 for 48k transcripts	launcher-derived (per-model default in MODEL.toml?)
   --max-prefill-tokens	ATLAS_MAX_PREFILL_TOKENS	2048	PREFILL_TOKENS env (2048)	MAX_PREFILL_TOKENS env (2048)	2048	8192 (:630-631)	2048 everywhere: BC=32 prefill kernel keeps long prompts single-chunk (serve-amd.sh comment ~L98)	MODEL.toml default
   --gpu-memory-utilization	ATLAS_GPU_UTIL (0.95)	Util 0.90 on main (0.80 in #80's pre-merge copy)	ATLAS_UTIL env (0.90, 0.86 serial)	GPU_UTIL env (0.88, 0.80 under DFLASH)	0.548	0.90 (:416-417)	0.95/0.88/0.80 measured headroom per config (first_run.ps1:535-537; serve-amd.sh ~L185 dflash OOM note). Strix cap: ≤0.80 + ctx≤12288 + slots≤32 after the 02:57Z OOM-reboot.	launcher-derived (hard platform bound)
   --kv-cache-dtype	bf16	bf16	bf16	bf16	bf16	engine fallback fp8 (DEFAULT_KV_CACHE_DTYPE, serve_args.rs:25); MODEL.toml may override	bf16 on all four recipes — fp8 KV is the accuracy-risk path on these gates	MODEL.toml default
   --lm-head-dtype	bf16	bf16	(absent → default)	LM_HEAD env (nvfp4; bf16 for unsloth)	bf16	"default" (:401-402) → MODEL.toml	nvfp4 = +~35% vocab projection on NVIDIA ckpt (serve-amd.sh ~L195); bf16 required on unsloth/FP8 rows	MODEL.toml default (per-checkpoint)
   --max-batch-size	1	1	1	MAX_BATCH env (1)	1	8 (:568-569)	all recipes pin 1: bench determinism + memory	MODEL.toml default? launcher
   --max-num-seqs	—	—	1	—	—	128 (:420-421)	FNext memory pin	launcher-derived
   --vision-max-pixels	262144	—	—	—	—	0 (:898-899) = disabled	27B VL input cap	MODEL.toml default
   --speculative + --num-drafts	3	— (dflash instead)	1 (SERIAL=0)	NUM_DRAFTS env (3)	— (dflash)	--speculative off (:510); num_drafts MODEL.toml/engine fallback 1 (:16-21)	K=4 MTP validated recipe (serve-amd.sh header: 28.3-28.6 tok/s, bfcl 83.02/80.41, run-1789370409958768995)	launcher-derived (mode switch)
   --mtp-quantization	bf16	—	—	(engine default bf16)	—	bf16 (:577-578)	same as default — explicit for readability	remove (defaults suffice)
   --mtp-vocab	100000	—	—	(engine default 100000)	—	100000 (:585-586)	ditto	remove
   --dflash + --draft-model	—	yes + $DraftDir	—	DFLASH=1 + DRAFT_MODEL env	yes +?	off (:528)	DF2 recipe is the perf-leg record	launcher-derived
   --dflash-gamma	—	Gamma 8	—	DFLASH_GAMMA env (16 default)	8	16 (:547-548)	γ=8 the measured DF2 recipe	MODEL.toml default
   --dflash-window-size	—	0	—	—	0	4096 (:555-556); 0 = full window	recipe	MODEL.toml default
   --disable-tool-grammar	true	—	—	—	—	Option (:477); unset = grammar on	bench determinism pin of the frozen accuracy recipe (serve-amd.sh ~L192 comment)	MODEL.toml / diagnostic-only
   --ssm-cache-slots	64	SsmSlots 24	SSM_SLOTS env (0)	SSM_SLOTS env (0)	24	16 (:695-696)	slots cover ctx at ckpt granularity; first_run 64 matches proven profile (first_run.ps1:537). Strix cap ≤32.	launcher-derived (memory fn of ctx)
   --ssm-checkpoint-interval	16	SsmInterval 128	—	SSM_CKPT_INTERVAL env (16)	128	256 (:708-709)	16 = certified warm-prefill recipe; 128 = coarse snapshots for long ctx (serve-amd.sh ~L83)	MODEL.toml default
   --enable-prefix-caching	—	yes	opt-in env PREFIX_CACHE=1	—	yes	false (:592-593)	DF2 long-transcript reuse	launcher-derived
   --disable-thinking / --stupify	yes (now conditional via ATLAS_THINKING)	yes (conditional)	DISABLE_THINKING=1 → conditional	THINKING=off opt	yes	false (:430-431); engine kill switch, outranks requests	thinking-off was bench pin for the frozen recipes; #146 switch landed 0219291/0bb0bedce	launcher (-Thinking/THINKING) ✓ done
   --default-chat-template-kwargs	(new: on only)	(new: on only)	(new: on only)	THINKING=on	—	None (:492); precedence flag→request→MODEL.toml	#146 thinking-on switch	launcher ✓ done
   --request-timeout	0	900	0	—	900	300 s (:813-814)	0 = long-run legs must not cut responses; 900 the DF2 harness bound	MODEL.toml default / launcher
   --dangerously-allow-unresolved-kernel-lookups	deliberately absent (first_run.ps1:525-530)	absent	absent	ALLOW_FALLBACKS always on	—	false (:86-87); refuse-on-missing audit	Linux needs it: 92 pre-existing fallbacks (serve-amd.sh ~L170). Windows recipes correctly omit it	diagnostic-only — do NOT ship in MODEL.toml
   --check-kernels	—	—	—	—	—	false; dry-run audit, exit=unresolved count	gate tool	diagnostic-only
   --scheduling-policy	—	—	—	—	—	fifo (:614-615)	—	engine default
   --swap-space-gb, --tbt-deadline-ms, --oom-guard-mb	—	—	—	—	—	3 GB / 100 ms / 4096 MB	—	engine defaults
   ATLAS_W4A16_VARIANT=v1	—	—	—	yes	—	BF16-MMA NVFP4 GEMM; SCALE FP8 encode broken on gfx1151	serve-amd.sh ~L75	engine/platform — leave env
   ATLAS_W4A16_DP4A	—	—	—	:-1 export	—	DP4A int8 decode GEMV arm; on by default in source	accuracy-validated (83.02/80.41 record)	remove eventually (default-on in code)
   ATLAS_SSM_TAIL_MIDCHUNK=1	—	—	—	yes	—	mid-chunk SSM tail capture	3.6 certified recipe carried (serve-amd.sh ~L115)	MODEL.toml / engine default?
   ATLAS_MTP_GATE_REPROBE=64	—	—	—	yes	—	re-probe accept gate every 64 tok	3.6 recipe	MODEL.toml
   ATLAS_DFLASH_OPTION_B	—	=1	—	:-1	—	paged Option-B drafter (legacy propose ~1.0 s/step measured)	serve-amd.sh ~L178	engine default eventually
   ATLAS_DFLASH_CTX_WINDOW	—	=$SeqLen	—	(job scripts set 12288)	—	drafter ctx window — must be ≤12288 on strix post-OOM	win recipe sets = SeqLen (memory footgun)	launcher-derived + platform cap
   ATLAS_MTP_ACCEPT_DEBUG=1	—	yes	—	(job scripts)	—	accept-stats logging	harness measurement	diagnostic-only
   ATLAS_KV_EXTERNAL_RESERVE_GB=0	(implied)	—	—	yes	—	self-relative KV accounting — correct on unified memory	serve-amd.sh ~L105; STRIX_WINDOWS_HIP.md "must be 0"	engine default on unified mem?
   ATLAS_QSA_VERIFY_ACTIVE	—	—	:-1 default on	—	—	MTP-past-QSA verify arm; NVIDIA-only default	measured on gfx1151 (#80 records)	MODEL.toml / engine default
   ATLAS_NO_MTP_DRAFTER_CONTEXT	—	—	—	(job scripts set 1)	—	disables default-on drafter ctx prefill/carry	diag	diagnostic-only
   FP8_DEQUANT/ATLAS_FP8_DEQUANT_*/ATLAS_GDN_BF16_WEIGHTS	—	—	—	auto for unsloth ckpt	—	dequant per-row FP8→BF16 once (w8a16 can't take [N,1] scales)	serve-amd.sh ~L95-115; faults other ckpts	launcher-derived (checkpoint sniff) ✓ already conditional
   DFLASH/DRAFT_MODEL/DFLASH_GAMMA (env)	—	—	—	drafter selection envs	—	serve-amd.sh-only interface	—	launcher
   GPU_UTIL/MAX_SEQ_LEN/MAX_PREFILL_TOKENS/SSM_SLOTS/SSM_CKPT_INTERVAL/LM_HEAD/HOST/PORT/MODEL_NAME/MAX_BATCH/NUM_DRAFTS (env)	—	—	—	launcher env overrides	—	—	knobs	launcher
   ATLAS_PORT/ATLAS_BIND/ATLAS_THINKING/ATLAS_UTIL/SEQ_LEN/PREFILL_TOKENS/SSM_SLOTS/SERIAL/PREFIX_CACHE/DISABLE_THINKING (env, Windows)	recipe params	recipe params	recipe params	—	—	—	—	launcher

   Reading

   • Everything in the right-hand columns "launcher-derived" is platform/recipe
     sizing (memory, bind, port, slots) — belongs in the launcher scripts.
   • "MODEL.toml default" = per-model behaviour worth freezing as the engine's
     declared default for this model×quant (kv bf16, lm-head dtype, dflash
     gamma/window, ssm ckpt interval, prefill tokens, request-timeout).
   • "diagnostic-only" = stay flags, never defaults: kernel-audit bypass,
     accept-debug, MTP ctx toggles.
   • "remove" candidates: --mtp-quantization bf16, --mtp-vocab 100000 —
     passed explicitly but equal to engine defaults on every recipe.
3. AzeezIsh commented on Oct 1, 2026

   

   AzeezIsh

   on Oct 1, 2026

   ContributorAuthor

   More actions

   Why configurations crash the host instead of failing at startup (investigation for the memory item; two strix OOMs on 2026-10-01).

   The KV budget is computed once, in crates/spark-model/src/factory/build.rs:391-465: total × util − used_so_far − inference_reserve. Here used_so_far is a self-relative measurement taken at that instant (build.rs:355-389), so anything allocated after it falls outside --gpu-memory-utilization.

   The 02:57Z strix boot (27B + DFlash2, ctx window 32768, 24 Marconi slots @ 128, util 0.80) logged 48.0 GB budget = 45.2 GB pre-KV + 2.8 GB KV. These allocations then landed outside that budget:

   allocation	site	size in that boot
   DFlash scratch: rows = γ + ctx window, plus logits	dflash_head/from_weights.rs:238-284	≈ 4.5–5 GB at ctx 32768 (≈ 250 MB at 4096)
   DFlash drafter weights + selector	load_dflash_weights	≈ 0.8–1 GB
   DFlash KV pool: ceil(ctx/16)+1 blocks × 5 layers	dflash_head/from_weights.rs	0.67 GB
   second 1-layer KV pool	paged_impl	0.2 GB
   GDN two-phase prefill buffers	impl_a1_init	88 MB
   ctx accumulator for each sequence after the first: max_seq_len × 5 × hidden × 2	dflash_head.rs:463-468 (at request admission)	1.68 GB per extra sequence; slot 0 is primed before KV and counted

   That's roughly 6–7 GB on a box with nothing left, and the kernel OOM-killer rebooted it. On unified memory these allocations take physical RAM, so the util math never gets a chance to refuse.

   Separately, the Marconi snapshot pool is about 151.5 MB per slot (ssm_snapshot.rs:117-203). The checkpoint interval only sets how often a state is saved, not the slot size. The recipe comment saying the engine raises slots to cover max_seq_len is stale: --ssm-cache-slots flows through unmodified. preflight.rs:169 reserves the pool on paper, which shrinks KV, but the allocation itself still runs against whatever physical memory is left.

   Proposed fix (smallest correct):

   • Hoist the DFlash drafter load and head construction (weights, KV pool, scratch) and the GDN prefill buffers so they run before used_so_far is sampled. That's the same pattern the ctx-acc priming already uses at build.rs:254. The self-relative measurement then counts them, with no second copy of size formulas to drift.
   • Reserve the per-request ctx accumulator for max_batch_size sequences in preflight.rs, next to ssm_snapshot_bytes.
   • Then the existing kv_budget == 0 bail (build.rs:437) fires with real numbers for any configuration that doesn't fit.
   • The second half belongs to this issue's launcher item: on unified memory, budget against memory actually free at boot (co-tenants subtracted), not total × util.
4. AzeezIsh commented on Oct 2, 2026

   

   AzeezIsh

   on Oct 2, 2026

   ContributorAuthor

   More actions

   Memory item: Devin's KV-budget fix passed its strix validation (job 140, 2026-10-02 00:16–00:19Z). Posting from the job log because it hadn't been reported.

   Branch fix/kv-budget-uma-headroom @ 215e3e534 (stacked on fix/kv-budget-counts-drafter; still local), strix-hip qwen3.8-27b nvfp4, bin 3cc011eb:

   leg	config	expected	result
   U-074	DFlash2 recipe, util 0.80	up, drafter counted pre-KV	up, 0 OOM kills. "DFlash drafter hoisted pre-KV"; KV 4.3 GB → 68,224 tokens
   U-CRASH	the 10-01 02:57Z crash config (ctx window 32768, Marconi 24 @ 128, util 0.80)	refuse at startup with numbers	refused, 0 OOM kills: DFlash drafter does not fit: 47.0 GB used + 6.6 GB drafter (weights + KV pool + ctx-window …)
   U-076	MTP recipe at util 0.88	up, UMA footprint cap	up, 0 OOM kills. KV 12.4 GB → 197,792 tokens

   So the class of host crash this issue describes (allocations landing outside --gpu-memory-utilization) now becomes a startup refusal on the config that rebooted strix. The teardown_errors=1 / sweep_fails=282 in U-074/U-076 is #122's SSM-pool half; this branch doesn't include #148.

   Still open before it ships:

   • the Windows recipes (re-check the Flash-Next KV line with strix-windows: Flash-Next launchers + prefix caching on by default (stacked on #80) #149; arithmetic on strix-windows: Flash-Next launchers + prefix caching on by default (stacked on #80) #149);
   • a GB10 boot, waiting on reiner;
   • pushing the branch as a PR.
5. AzeezIsh commented on Oct 2, 2026

   

   AzeezIsh

   on Oct 2, 2026

   ContributorAuthor

   More actions

   #146 memory fix: the AMD Linux platform gate passed. That's ST-995 plus the 1007-turn replay, the full AGENTS.md suite this fix still owed.

   Run: strix job 142, 2026-10-02 06:02–13:05Z, run dir ~/dp4a-ab/out/kvbudget-gate-20261002T0602Z. Branch fix/kv-budget-uma-headroom @ 215e3e534 (main be435875c + 6 commits, still local), bin 3cc011eb, nvidia/Qwen3.8-27B-NVFP4, DFlash2 γ8. The recipes and references are the same ones the #156 gate used.

   leg	recipe	KV under the fix	result	main reference
   ST-995	073: util 0.82, seq 26624, ctx 8192, SSM 4 @ 512, pcache	6165 blocks / 98.6K tokens	85.53 / 78.38 (run-1790935156817722493.json), 994/995 responses byte-identical to 073	85.43 / 78.19
   1007-turn replay	074: util 0.80, seq 32768, ctx 12288, SSM 4 @ 512, pcache	3977 blocks / 63.6K tokens (main: ~98K)	1007/1007, 0 failed, 0 missing, score 0.6183, 10,979 s	0.6189, 10,711 s

   • Health: 0 KV cache exhausted, 0 prefill errors, 0 faults, 0 OOM kills, clean teardowns.
   • ST-995: the one differing response is the same item that varies main-vs-main (main run 072 scored exactly 85.53 / 78.38).
   • Wall time is +2.5% against 074. That's one run per side, and recent replays of this recipe have spread from 10,488 to 10,979 s, so I'm not calling it a regression. The obvious suspect would be more prefix-cache evictions from the ~35% smaller KV; I'll check the prefill-token totals of both runs before the PR goes up.

   Still owed before this ships:

   • A rebase onto main after Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80. The conflict is in win_serve_dflash2_nvfp4.ps1.
   • The Windows DFlash2 recipe check (CtxWindow 12288, util 0.95).
   • A Windows Flash-Next KV check together with strix-windows: Flash-Next launchers + prefix caching on by default (stacked on #80) #149/prefill: refuse mid-chunk tail capture on aux-carrying models (Flash-Next warm path never engaged on Windows) #157. Under this fix the MTP drafter pool counts toward the budget, so the Flash-Next recipe needs util ≈ 0.91 to keep its KV.
   • GB10, once reiner is back.
6. AzeezIsh commented on Oct 5, 2026

   

   AzeezIsh

   on Oct 5, 2026

   ContributorAuthor

   More actions

   GB10 boots of the KV-budget fix: outputs identical, but KV collapses on GB10. Do not merge as-is. Reiner job 695, 2026-10-04: main 4405bce vs main + this branch rebased (eb493f3c4, local), job 659's recipes, util 0.88.

   model	main KV	branch KV	ratio	greedy (5 prompts)
   Flash-Next NVFP4	14,449 blocks (5.3 GB; pre-KV 98.8 GB)	8,827 blocks (3.2 GB; pre-KV 100.8 GB)	0.61×	5/5 identical
   Qwen3.8-27B NVFP4	50,227 blocks (49.1 GB)	49,316 blocks (48.2 GB)	0.98×	5/5 identical
   27B + DFlash2 (--dflash --dflash-gamma 8)	45,580 blocks (44.5 GB)	257 blocks (0.1 GB)	0.006×	5/5 identical

   • 0 faults everywhere, and boot times are unchanged.
   • The DFlash2 arm is the bug: the drafter footprint estimate (or the UMA headroom applied on top of it) leaves GB10 with 0.1 GB of KV.
   • Flash-Next loses 39%: pre-KV is accounted ~2 GB higher.
   • The branch's Linux strix gate passed (job 142), so the estimate looks calibrated for Strix's memory model and wrong for GB10's.

   Next: fix the GB10 path (likely the headroom/drafter term should key on the platform's memory model, not on unified memory alone). Re-boot the three arms before this goes anywhere.
7. AzeezIsh

   added a commit that references this issue on Oct 18, 2026

   Merge pull request #185 from Atlas-Inf/launch/thinking-switch-linux

   ...

   d7306ea
8. AzeezIsh commented on Oct 18, 2026

   

   AzeezIsh

   on Oct 18, 2026

   ContributorAuthor

   More actions

   Status update (2026-10-18):

   • Linux thinking switch shipped: serve-amd.sh THINKING=off|on merged via serve-amd.sh: THINKING=off|on switch (issue #146) #185 (0bb0bedce → main). Both launcher families now have the same off|on meaning; engine kill switch verified across OpenAI/Anthropic/Responses surfaces.
   • Thinking-off cost measured (framework think-ab, Flash-Next NVFP4 @ 7ffd183, n=995): --disable-thinking scored 76.28 overall / 69.76 normalized vs 83.92 / 82.93 default — ~7.6/~13 pt cliff on tool workloads → measured default for tool-facing recipes = thinking on (checkpoint default); off is a throughput-leg setting.
   • Memory item: the [job] Strix Halo: multi-session serving (parallel or sequential) and long context, Flash-Next and the 27B #145 suballocator resolved the boot-side of this — FNext now boots on Linux (see [job] Strix Halo: multi-session serving (parallel or sequential) and long context, Flash-Next and the 27B #145 status). The refuse-with-numbers KV path is live (35B-A3B correctly refused at util 0.80: 51.5 GB pre-KV > 48 GB budget).

   Still open: flag-inventory fold into MODEL.toml, and the winbox legs (tester-gap A/B, Windows smoke) — winbox has been offline ~4 days.
9. maplepaladin73

   added a commit that references this issue on Oct 18, 2026

   serve-amd.sh: THINKING=off|on switch (issue Atlas-Inf#146)

   ...

   0bb0bed