Measuring benchmark optimization in speech recognition
huggingface ·
The article discusses benchmark optimization in speech recognition, where models perform well on public tests but may not generalize to real-world conditions. The authors from HumeAI introduced held-out sets in their ASR leaderboards to better measure practical performance. They designed three probes to detect benchmark overfitting: reference disagreement, masked entity retrieval, and orthographic switching. Testing 11 open-source ASR models, they found several high-scoring systems reproduced benchmark transcripts even when audio contradicted them or when relevant words were silenced. For example, many models omitted an audible "Thank you" from a VoxPopuli clip to match the erroneous reference transcript, and some recovered masked numbers from benchmark audio. The effect weakened on freshly collected or newly voiced audio, suggesting models used acoustic cues tied to the benchmark itself. Models with the lowest word error rates were often the most likely to reproduce benchmark errors, indicating their scores overstated real transcription ability.
huggingface