Monitoring production agent lifecycle with AWS DevOps Agent and AgentCore Evaluations
aws ·
This AWS blog post explains that production multi-agent systems fail in ways traditional infrastructure monitoring cannot detect, such as missing IAM permissions that return empty responses instead of 500 errors, or supervisor agents misrouting requests while infrastructure metrics remain healthy. The authors describe a dual-layer monitoring approach combining Amazon Bedrock AgentCore Evaluations for continuous quality assessment and AWS DevOps Agent for autonomous infrastructure incident investigation. AgentCore Evaluations samples a configurable percentage of live interactions and scores them using an LLM-as-a-Judge methodology on dimensions such as helpfulness, correctness, and goal completion, providing reasoning for each score and running pattern analysis on low-scoring sessions to surface failure modes and concrete recommendations. AWS DevOps Agent acts as an autonomous on-call engineer, pulling CloudWatch logs, building a resource topology graph, correlating errors across IAM, Bedrock, and the agent runtime, and tracing failure paths to deliver remediation steps. To demonstrate the architecture, the team built an airline reservation system with four specialized agents—Supervisor, Flight, User, and Reservation—using the Swarm pattern, where control passes dynamically between peers with no fixed execution graph to instrument. The system runs on AgentCore runtime with a React frontend hosted on AWS Amplify, Amazon S3 for session storage, and AgentCore Identity for authentication, while AgentCore Observability uses OpenTelemetry to send traces and metrics to Amazon CloudWatch so operational metrics, traces, and quality scores land in one place. The complete source code, including CDK infrastructure, an evaluation dashboard, and the DevOps Agent integration, is available in a GitHub repository and was built on the Fullstack AgentCore Solution Template.
aws