AI benchmarks have a trust problem and Google wants to fix it
decoder ·
Google DeepMind is addressing the problem of AI benchmark contamination by using cryptographic methods to prevent models from seeing test questions in advance. The company has launched a pilot double-blind evaluation of a proprietary Gemini Flash Lite model in collaboration with the Singapore AI Safety Institute and other partners. Previously, external evaluations required a compromise: evaluators had to share their test prompts or model providers had to share their weights, risking intellectual property. Google's solution uses Confidential Space from its cloud confidential computing portfolio to keep both the external test data and the model private, with cryptographic verification for both sides. As a result, evaluators never see the model weights, and Google never sees the test prompts. This approach is especially important for sensitive evaluations in areas like cybersecurity and for government agencies, as it enables independent rigorous testing without sacrificing data sovereignty. Google hopes this effort will set a new standard for model oversight and help build more trusted AI systems.
decoder