ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
huggingface ·
IBM Research has introduced ALTK-Evolve, a method that enables customer service agents to achieve top-tier performance on the τ²-bench leaderboard without requiring model weight updates or fine-tuning. By using GPT-4.1 to analyze agent trajectories and distill behavioral rules into a learned playbook, the system appends these guidelines directly to the system prompt. This approach significantly improves both mean accuracy and reliability, measured by Pass^4 scores, across retail and airline domains. In the retail sector, this technique allowed a basic ReAct agent with GPT-4.1 to surpass all frontier models and even outperform a fine-tuned RAFT-30B-A3B system. For airline tasks, the same methodology topped the leaderboard when paired with the more advanced GPT-5.5 model. The study highlights that while input token costs increase slightly due to the larger prompt, conversation length actually decreases, resulting in lower latency and faster resolution times. Furthermore, the benefits of learned guidelines extend to frontier models, demonstrating that prompt-level memory is a scalable solution for enhancing agent capability and reliability.
huggingface