Piloting the world's first double-blind AI evaluations
deepmind ·
Google DeepMind has announced a pilot described as the world’s first double-blind evaluation of a proprietary, frontier-class AI model. The project tests a Gemini Flash Lite model against confidential benchmarks with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. It addresses benchmark contamination, in which a model’s prior exposure to evaluation questions can artificially improve its scores and weaken confidence in the results. Using Google Cloud’s Confidential Space and cryptographic verification, the system keeps the evaluator’s test prompts hidden from Google and the model weights hidden from the evaluator. This approach is intended to protect intellectual property, data sovereignty, and sensitive evaluations such as those involving cybersecurity or government bodies. DeepMind says the pilot could support more independent oversight and help produce safer, more reliable, and more trusted AI systems.
deepmind