CWE-bench v2: More coverage, quality, and transparency
huggingface ·
Collinear AI has released CWE-bench v2, a public benchmark designed to test the cybersecurity capabilities of frontier AI models. The update features a new set of 100 tasks that cover 91 different Common Weakness Enumeration (CWE) categories, aiming for exhaustive coverage of vulnerability classes. Each task requires an agent to identify and fix security weaknesses in real open-source repositories without breaking existing functionality. The evaluation process is fully programmatic and deterministic, ensuring that results are reproducible by anyone without additional costs. Current leaderboards show that Gemini 4 Argon leads with a 67.8% pass rate, while even top models struggle with a long tail of difficult tasks. The benchmark includes a held-out set of unseen tasks to track ongoing progress and prevent memorization. Users can run evaluations directly from Hugging Face using the Harbor tool, which supports both testing and training defensive agents.
huggingface