Skip to content

convert: default to group-scaled int4 (gs64) — the #455 tracking item - #579

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:fix/converter-default-gs64
Jul 24, 2026
Merged

convert: default to group-scaled int4 (gs64) — the #455 tracking item#579
JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:fix/converter-default-gs64

Conversation

@ZacharyZcR

Copy link
Copy Markdown
Contributor

First brick of the 1.2.0 "serve path from working to reliable" arc: stop handing new users the per-row quality trap by default.

Why now

Per-row int4 is the most-reproduced failure source in the tracker: −9.3pp mean acc_norm in the ablation (#453), the root cause of the think-mode loops and EOS starvation in #455 (gs64 cured 5/5 capture-matrix cells), and the very first thing the newest field report (#549) tripped over — that user's first conversion was per-row, their "quality worse than Qwen" was exactly the measured per-row deficit, and #455 closed with "make the converter default to group scales" as tracking item (a). This PR is that item.

Changes

  1. convert_fp8_to_int4.py: --group-size default 064. gs64 is the community-validated point (the GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe #455-curing container, the tools: IQ3_XXS-codebook scheme in the quant ablation — settles #452's codebook decision #453 ablation arm at −2.2..−3.4pp vs fp16). Per-row stays available as an explicit --group-size 0 opt-out.
  2. coli convert: --group-size passthrough (default 64) so the launcher path is controllable without dropping to the tool.
  3. docs/quickstart.md: the "Easiest" recommendation now points at the gs64 container (mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp) instead of the per-row one it recommended until now — the docs were steering every new user into the exact container class that GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe #455 documented as broken. The per-row cards stay mentioned, labeled with what they cost.

Safety

…tracking item and stop steering new users into the per-row trap
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants