How UK AISI and EvalEval Are Making Benchmark Results Reproducible
huggingface ·
The UK AI Security Institute (AISI) is collaborating with the EvalEval Coalition to enhance the reproducibility of AI benchmark results through shared infrastructure. This partnership utilizes EvalEval's Evaluation Cards and the Every Eval Ever schema to openly publish verified evaluation data, addressing the common issue of inconsistent reporting formats in the field. AISI has released detailed findings from five major benchmarks, including HealthBench and SWE-Bench Pro, covering six frontier models such as Claude Opus 4 and GPT-5 variants. These releases accompany AISI's research paper on how inference compute impacts LLM evaluation performance. By providing context, configuration details, and transcript-level transparency, the initiative allows researchers to verify results and compare studies more effectively. The collaboration aims to standardize evaluation science, making it easier to interpret scores under different conditions. This effort supports broader meta-research by establishing reliable reference points for understanding model capabilities and risks.
huggingface