What We Learned by Reproducing 2,200 papers from ICML
huggingface ·
A recent community-driven hackathon successfully attempted to reproduce over two thousand machine learning papers presented at ICML 2026. More than twelve hundred participants utilized various AI coding agents to generate thousands of publicly auditable reproduction logs covering roughly a third of the conference’s accepted work. The collective effort revealed that slightly more than half of the examined papers contained at least one independently verified claim, while nearly a quarter featured falsified or heavily contested results. Independent teams also identified several critical flaws, including mathematical proof errors, mismatched loss functions, and artificially inflated performance metrics caused by dataset padding. When researchers contacted the original authors, most responded constructively by publishing corrections or clarifying their methodologies. Despite the impressive scale and speed of automated verification, the organizers emphasized that human oversight remains indispensable for guiding experimental design, challenging underlying assumptions, and evaluating subjective outcomes like visual quality. Ultimately, the initiative demonstrated that combining scalable agent-based auditing with strategic human direction could significantly strengthen scientific rigor in rapidly expanding research fields.
huggingface