Your Agent Aced the Task. Will It Do It Again?
huggingface ·
IBM researchers highlight a critical reliability issue in AI agents where high average accuracy masks significant inconsistency across repeated runs. They introduce the Consistency Analyzer, a diagnostic tool that identifies unstable decision points by resampling agent trajectories without needing ground truth or full replays. This analysis feeds into ALTK-Evolve to generate consistency guidelines that stabilize these 'flat' probability distributions. In evaluations on AppWorld using GPT-4.1, this approach halved the consistency gap from 24.4 percentage points to 12.0 points. The method improved Pass^5 rates significantly while maintaining or slightly increasing mean accuracy. These guidelines proved generalizable, transferring effectively to similar tasks and weaker models like gpt-oss-120b. The authors urge developers to report Pass^k alongside Mean@k to better assess agent reliability for mission-critical applications.
huggingface