The Open ASR Leaderboard Adds Its First Global South Language
huggingface ·
The Open ASR Leaderboard has added its first Global South language by introducing two evaluation sets, Monsoon en-IN (Indian English) and Monsoon hi-IN (Hindi), making Hindi the first Indic language on the leaderboard's multilingual tab. These sets were designed to address the lack of speaker diversity in ASR benchmarks, which often show performance disparities across different racial, gender, and accent groups. The data collection intentionally varied along nine axes, including geography, age, gender, devices, and acoustic environments, and involved recording spontaneous conversations from over 4,800 speakers across hundreds of Indian districts. Each set includes public and private splits, and every clip carries detailed metadata about the speaker, such as occupation, education, income, and device model. The design prioritizes broad speaker coverage over long recordings, ensuring that no single voice or region dominates the results. Quality control measures, including language verification and anti-fraud checks, were applied to maintain reliability while preserving natural acoustic conditions. This initiative enables researchers to analyze ASR performance across demographic and geographic variations using a public leaderboard test set.
huggingface