Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
huggingface ·
Researchers from Multiverse Computing introduce Quantization-Aware Healing (QAH), a method for recovering large language models that have been both structurally compressed and quantized to 4-bit precision. Traditional healing approaches like quantization-aware training (QAT) are costly and unstable, while quantization-aware distillation (QAD) fails after structural compression because no full-precision version of the smaller architecture exists. QAH instead distills directly from the original pre-compression model, using KL divergence on logits, which removes the accuracy ceiling imposed by the recovered checkpoint. Applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, the QAH model outperforms its own bfloat16 version on 7 of 9 benchmarks, including large gains in long-context reasoning and math. In a head-to-head comparison on a 9B model, QAH matches QAT's peak accuracy but reaches it about 7 times faster and remains stable, while QAT degrades sharply after its peak. The resulting 4-bit model uses about four times less weight memory than the 16-bit version and roughly half the compute per token of the 120B teacher. The authors conclude that compressed 4-bit models can be smaller, cheaper, and more accurate than their full-precision counterparts, making quantization an additional teaching opportunity rather than a performance tax.
huggingface