New Artificial Analysis index widens the gap between open and closed models
decoder ·
Artificial Analysis has overhauled its Intelligence Index, significantly widening the performance gap between open and closed AI models. As a result of these changes, Kimi K3.1 Pro dropped five spots in the rankings, falling from fourth to ninth place behind eight closed models. The updated index now places the disparity at 12 points, a substantial increase from the previous 3-point margin. To combat benchmaxxing, where models are trained on known test questions, 60 percent of the new index relies on private test sets. This shift suggests that previous rankings may have overstated the capabilities of certain models, as evidenced by Grok 4.7 also losing significant points. However, researchers note that sophisticated reinforcement learning infrastructure remains a key differentiator for top performers. While closed tests prevent gaming, they also hinder external verification, leading Artificial Analysis to revise flawed questions in benchmarks like AA-Omniscience.
decoder