Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations
aws ·
Amazon Bedrock AgentCore Evaluations is a new service that decouples agent evaluation from the specific agent framework, SDK, or LLM client used. It works by reading OpenTelemetry trace data from Amazon CloudWatch, which is automatically collected for agents running on Amazon Bedrock AgentCore runtime. The service classifies each span into three essential roles—invoke agent, inference, and execute tool—to reconstruct a user session and score it with evaluators like GoalSuccessRate, Correctness, and Helpfulness. It supports both the OpenTelemetry GenAI semantic conventions and the OpenInference specification, automatically detecting the correct parsing method via the instrumentation library's scope name. Currently supported frameworks include Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, with generic support for any library using a compliant opentelemetry.instrumentation.* or openinference.instrumentation.* scope. To work, spans must carry a matching session.id attribute, and message content must be available in the data source, which is automatic under unified observability but requires separate event records for older agents. The design is forward-compatible, as unfamiliar span kinds are treated as extra context rather than errors, making the evaluation process framework-agnostic.
aws