BenchMIRT: What are LLM benchmarks actually measuring?
huggingface ·
The article introduces BenchMIRT, a new method from Ai2 for auditing LLM benchmarks at the level of individual prompts. It uses multidimensional item response theory to separate the underlying capabilities that drive performance, independently recovering safety and general reasoning as two dominant dimensions across 16 benchmarks and 34K questions. BenchMIRT revealed that some benchmarks mix signals: BBQ aligned more with general reasoning than safety, and WMDP scores were more strongly associated with general reasoning, with stronger reasoning linked to lower scores. It can also identify the most informative questions, showing that keeping only 10% of questions often preserved nearly the same picture of model capabilities as the full set. BenchMIRT predicted held-out question performance with 79% accuracy, compared to 70% for a simpler baseline. The authors note limitations, including that all models were released by March 2025 and that discovered dimensions depend on the benchmark set used. They also acknowledge trade-offs, such as the risk that question-level transparency could be used to remove safety-related questions and create weaker evaluations.
huggingface