Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
huggingface ·
Researchers from Multiverse Computing have developed a novel method for pruning Large Language Models by treating block removal as an Ising optimization problem. Instead of evaluating transformer blocks independently, this approach models the selection process as a constrained binary optimization task that accounts for interactions between blocks. By mapping these interactions to a spin system's energy landscape, the team can efficiently identify low-energy configurations that correspond to high-performing pruned models. This technique avoids the need for expensive benchmarking during the search phase, relying instead on a one-time Hessian calculation to evaluate millions of candidate configurations. The method demonstrates significant performance gains, particularly in deep compression scenarios, such as achieving a 23-point MMLU advantage over baselines when removing half the blocks from Llama-3.3-70B. Importantly, the best results often come from excited states rather than the absolute ground state, allowing for more robust model selection. The framework is also shown to generalize effectively to heterogeneous architectures, including hybrid models with mixed attention and MoE layers. This physics-inspired strategy integrates seamlessly with other compression techniques like quantization and low-rank compression.
huggingface