Transformers now runs llama.cpp quants
huggingface ·
Hugging Face Transformers has added native support for running GGUF models, allowing users to leverage quantized checkpoints optimized for local inference. This integration enables efficient execution on Apple Silicon by reusing ggml Metal kernels directly within the PyTorch ecosystem, significantly reducing overhead. The update focuses initially on Qwen3.5 architectures, offering various quantization levels like Q4_K_M to balance memory usage and precision. Users can load these models using standard APIs or serve them via an OpenAI-compatible interface for use with tools like Jan or LM Studio. Benchmarks indicate that performance is close to that of llama.cpp, though transformers includes prefill time in its measurements. Beyond GGUF, this kernel reuse strategy opens pathways for accelerating other model types, including multimodal and research architectures. Current limitations include a focus on MPS-only packed inference and limited architecture coverage, with plans to expand support gradually.
huggingface