[q] open to community quantized checkpoints?

[sady4850]

hi, been playing with quantizing granite (int4, measuring MMLU-Pro vs bf16) to learn how quant works in lalamo.

quick one before i dig deeper - are you open to community-contributed quantized checkpoints at all, and if so what's roughly the bar to accept one? just trying to find the direction

thanks

[norpadon]

Hi! Sorry for the late reply!

An interest in lalamo is greatly appreciated, and it is very cool that you managed to make the quantization work without instructions

The quantization pipeline depends on the very important step which the computation of the Fisher information matrix (the preconditioner argument to the quantization methods). We haven't released the code to compute the Fisher yet, but we probably will soon. I think community checkpoints are a good idea. We'll have an internal conversation about how to do it well.

What would be the ideal setup in your opinion?

[sady4850]

good to know you use the Fisher as preconditioner. i used a Hessian/Gram preconditioner (LDLQ style) instead, and on granite int4 that already lands within ~2.3 pt MMLU-Pro of bf16 on the full set. it would be interesting to see what the results would be with the fisher.

on the ideal setup, my take:

1. a checkpoint shouldn't be a blackbox blob. each one ships a recipe card: method, per-layer bit allocation, RHT seed, calibration set + size, preconditioner source (fisher vs gram/none), base model rev and lalamo git rev and any other relevant info. so the maintainer or an independent party can reproduce the checkpoint.

2. one eval contract. the contributor reports on your metric (MMLU-Pro,reasoning-enabled, fixed limit) plus PPL against the bf16 baseline, with a max degradation threshold to accept. again a maintainer or third party can reproduce from the card.

eventually it'd be fun to have a public leaderboard of checkpoints that pass the threshold ;-)

[norpadon]

That sounds interesting. Our internal quantization codebase is still not very mature, and the pipeline still requires some manual tuning per-checkpoint, so it is not 100% automated and reproducible (although we are moving towards that state)

But we have the leaderboard internally, and we are going to release it at some point

If you upload the checkpoint to huggingface, we will endorse it and also add to our registry. We'll probably make some kind of community space on HF for the checkpoints, need to figure out the right way to do it

Btw, for gram matrix, did you use one-sided yaqa?

[sady4850]

yeah, one-sided, and even the input side is just the activation gram X^T X not a fisher factor (output block = identity). gptq/ldlq curvature through lalamo's yaqa_round_blockwise. i tried the two-sided grad/fisher version but it diverged without the hadamard stabilization on the hessian, so i stayed on the one-sided gram.

happy to upload granite int4 to HF with the recipe card + MMLU-Pro/PPL numbers whenever there's a place for it. i guess there should be a mirai community space for this? could you point me to it once it's set up, thanks

[norpadon]

We have created a community org on HF. You can upload your checkpoints with model cards there: https://huggingface.co/mirai-community

Please tell if you have any issues in the process

Regarding two-sided YAQA: it is indeed quite unstable, you need to add some small value to the diagonal to make sure that blocks are well conditioned. We are following the YAQA approach and adding damping_factor * mean(diag(fisher_block)) to both input and output blocks, where damping_factor is typically set to 0.03, but can be tuned up if some layers still fail to converge