The Agent Said It Was Done. The Database Disagreed.
huggingface ·
Microsoft and Hugging Face have released ThinkingBox, a new benchmark designed to evaluate AI agents based on their actual impact on backend database states rather than just their generated text or tool calls. The study analyzed 507 stateful business workflows across 18 LLM models, running each task 20 times to measure consistency alongside capability. Results revealed that while some models like Kimi-K3 could solve many tasks at least once, they often failed to do so reliably in subsequent attempts. In contrast, models such as Claude Opus 5.5 demonstrated higher dependability by passing all 20 trials for a significant portion of the tasks. The research highlights a critical gap where agents appear successful through valid tool usage but leave incorrect terminal states, such as closing tickets prematurely. To address this, the authors introduced metrics for cost per dependable task, showing that achieving consistent accuracy incurs a higher financial price than merely obtaining a single correct answer. The benchmark is now available via OpenEnv, allowing developers to test agent reliability against isolated sessions with executable checks on side effects.
huggingface