New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
decoder ·
A new open-source benchmark called CloudOpsBench has identified reasoning as the primary obstacle for AI agents performing cloud operations tasks. Developed by a team at the University of Michigan, Version 1.0 evaluates how well models handle typical DevOps activities such as fixing services and managing access rights on AWS. The benchmark includes 45 real-world tasks executed in both actual and emulated cloud environments. Despite their capabilities, leading models like Claude Fable 5.1 and GPT-6 Astra only solved 60% and 53% of the tasks, respectively. Researchers found that approximately 80% of failures stem from poor reasoning, where agents misjudge task scope or stop prematurely. Only about 20% of errors involve incorrect actions, some of which have irreversible consequences for infrastructure. The project is available on GitHub under an Apache 2.0 license, with extensions for other cloud providers planned.
decoder