AI agents have bad taste in long-term decisions and a new benchmark proves it
decoder ·
Researchers from Microsoft and City University of Hong Kong have introduced Taste-Bench, a new benchmark designed to evaluate how well AI agents make long-term decisions during complex, multi-step tasks. Unlike standard benchmarks that only measure final task completion, this tool assesses an agent's "taste" by analyzing its ability to choose the most promising path at critical decision forks within a trajectory. The study tested 14 leading models, including GPT-5.6 Sol and GPT-5.5, revealing that even the best performers barely exceeded 60 percent accuracy on these binary choices. Performance dropped significantly when the evidence favoring one path only appeared after extensive later work, indicating that current models struggle to predict future outcomes effectively. Increasing reasoning time or computational budget did not improve results, suggesting that the necessary clues are often missing from the immediate context. However, the researchers demonstrated that this judgment capability can be trained by distilling insights from a teacher model into a student model, which improved success rates on held-out coding tasks. This finding highlights a significant gap in current AI capabilities regarding long-horizon planning and offers a pathway for improvement through targeted training.
decoder