Agent Evaluation Metric for multi-turn conversations
aws ·
An AWS Machine Learning Blog technical post by Surafel Lakew, Sailik Sengupta, and Sina Torfi introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level method for measuring agent quality in multi-turn conversations, with correctness as its first implemented dimension. The authors argue that multi-turn agents fail in ways single-turn or outcome-level evaluation misses, because a single early mistake cascades through later turns; in an example five-turn sales-report conversation, the agent passes "profit" instead of "revenue" in turn 2, corrupting turns 3 through 5, yet a task-level check only marks the whole interaction as failed. Existing tools such as goal-completion scoring, LLM-as-judge assessment, and trace-level root cause analysis are described as valuable but limited because they treat quality as one signal rather than separately measurable sub-metrics. AEM addresses this by computing correctness per turn through two sub-metrics: truthfulness, whether values in tool-call parameters and natural-language responses are factually consistent with expectations, and completeness, whether all required elements such as parameters or requested information are present. The same sub-metrics apply to both response turns and action turns, with tool and action selection acting as structural checks beneath them, and a turn's correctness is treated as binary pass or fail in this post. Because exact string matching is too brittle, the metric uses semantic comparison via embedding-based similarity or an LLM-as-judge, with a default threshold of 0.5 that controls strictness. A failure taxonomy names specific reasons including inconsistent_response, incomplete_response, tool_mismatch, action_mismatch, missing_parameters, extra_parameters, and inconsistent_parameter_values, while a prior_action_failed label separates root causes from turns that merely inherited an earlier failure; the composed AEM score is the proportion of turns that pass. The post states that the metric runs in a repeatable pipeline that takes annotated dialogs and returns a decomposed per-turn correctness score for use across the agent lifecycle, and that the decompose-evaluate-compose pattern is designed to extend to further dimensions such as safety, instruction retention, and reasoning depth.
aws