Bringing Nunchaku 4-bit Diffusion Inference to Diffusers
huggingface ·
This article introduces Nunchaku Lite, a new integration path that brings 4-bit diffusion inference to the Hugging Face Diffusers library. Based on the SVDQuant method, Nunchaku runs transformer layers with 4-bit weights and activations to reduce VRAM usage and accelerate denoising. With Nunchaku Lite, users can load pre-quantized checkpoints using a standard `from_pretrained()` call, without needing a custom pipeline, separate inference engine, or local CUDA compilation. The system uses SVDQ W4A4 layers for attention and MLP projections and AWQ W4A16 layers for modulation projections, with NVFP4 kernels supporting Blackwell GPUs and INT4 kernels supporting earlier Turing/Ampere/Ada architectures. Benchmarks on an RTX PRO 6000 show roughly 30% end-to-end speedup and up to 50% peak VRAM reduction; combining with `torch.compile` improves the speedup to about 1.8x. The article also explains how to quantize custom models using the diffuse-compressor toolkit, which can scan, calibrate, package, and publish new quantized Diffusers repositories.
huggingface