fix(engine): floor memory fit-check against a real weights+KV-cache estimate - #2
Merged
Conversation
added 2 commits
September 6, 2026 22:21
…ce icons STT server swapped from Nemotron ASR to Qwen3-ASR-0.6B for improved accuracy and cleaner output (no leaked language-tag tokens, no truncated sentences). Fixed-service vendor icons (STT/Embedding/Aligner/TTS) moved from a compiled-in Go map to a store-backed setting (infra.service_icons) with a live GET/PUT API — changing one no longer requires a rebuild or restart. Claude-Session: https://claude.ai/code/session_01C8p4KLqr9S2MJzQET8dZSc
…stimate A real production incident: a config's curated memory estimate (weight- adjacent, no KV-cache term) let the fit check admit a model load that actually needed far more memory once its large-context, full-SWA KV cache materialized, while another already-loaded model's real footprint left little genuinely free headroom. The host OOM-killed several unrelated services before self-healing. The fit check's need-estimate had no context-dependent term on any non-profiled path (profile -> curated catalog figure -> bare weight-file size), so any large-context, unprofiled config was exposed the same way. Extends the GGUF parser (block_count, head_count[_kv], key/value_length [+_swa variants], sliding_window[_pattern], and ssm./indexer hybrid signals) and adds a KV-cache byte estimator ported directly from llama.cpp's own hparams/KV-cache sizing logic (verified against source). A curated or file-size figure can no longer undercut the computed weights+KV floor. Architectures the formula can't model (hybrid SSM/attention designs) are detected and left on the existing curated-or-refuse path rather than guessed at.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
or bare-file-size estimate with no context-dependent term for any
non-profiled config, so a large-context config could be admitted well
under its real memory need.
dimensions, sliding-window pattern, and hybrid SSM/attention signals) and
adds a KV-cache byte estimator ported from llama.cpp's own hparams/KV-cache
sizing logic, so a curated or file-size figure can never undercut the
computed weights+KV floor.
detected and left on the existing curated-or-refuse path rather than
guessed at — no regression for those.
Test plan
go build/go vet/go test ./...(incl.-race) cleansliding-window-attention model (full formula match against a
hand-calculated expected value), the embedding/head-count fallback,
and hybrid-architecture abstention
(curated figure below the real need) — now correctly refuses