convert: default to group-scaled int4 (gs64) — the #455 tracking item - #579
Merged
Merged
Conversation
…tracking item and stop steering new users into the per-row trap
This was referenced Jul 24, 2026
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First brick of the 1.2.0 "serve path from working to reliable" arc: stop handing new users the per-row quality trap by default.
Why now
Per-row int4 is the most-reproduced failure source in the tracker: −9.3pp mean acc_norm in the ablation (#453), the root cause of the think-mode loops and EOS starvation in #455 (gs64 cured 5/5 capture-matrix cells), and the very first thing the newest field report (#549) tripped over — that user's first conversion was per-row, their "quality worse than Qwen" was exactly the measured per-row deficit, and #455 closed with "make the converter default to group scales" as tracking item (a). This PR is that item.
Changes
convert_fp8_to_int4.py:--group-sizedefault0→64. gs64 is the community-validated point (the GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe #455-curing container, the tools: IQ3_XXS-codebook scheme in the quant ablation — settles #452's codebook decision #453 ablation arm at −2.2..−3.4pp vs fp16). Per-row stays available as an explicit--group-size 0opt-out.coli convert:--group-sizepassthrough (default 64) so the launcher path is controllable without dropping to the tool.docs/quickstart.md: the "Easiest" recommendation now points at the gs64 container (mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp) instead of the per-row one it recommended until now — the docs were steering every new user into the exact container class that GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe #455 documented as broken. The per-row cards stay mentioned, labeled with what they cost.Safety
check_or_record_paramsalready recordsgroup_sizein the outdir manifest, so a pre-default per-row conversion resumed under the new default refuses loudly with the existing "use a fresh --outdir" message (convert_fp8_to_int4.py: --mtp is silently ignored with --indir and overwrites the finished container with a full int8 conversion #355-class guard doing its job) rather than interleaving formats.bits=8) never enters the grouped-int4 branch, andcoli convert's step-2 re-invocation inherits the flag harmlessly.