On 11 November 2026, Amazon Web Services announced the Universal Agent Evaluation Framework (UAEF), a system that evaluates AI agents from different frameworks using a consistent set of metrics and stores results for version comparison.
UAEF converts agent execution traces from various frameworks into a single canonical format called AgentTrace. The framework‑agnostic system scores agent performance using 36 built‑in metrics across seven quality dimensions: tool calling, response quality, responsible AI, performance, multi‑turn conversation, multi‑agent coordination and reasoning.
Metric Design and Scoring
All metrics return normalised scores between 0 and 1, where 1 represents optimal performance. Deterministic metrics compute scores directly from trace data and ground truth, while LLM‑as‑a‑judge metrics use Amazon Bedrock models to evaluate subjective qualities. Judge metrics are designed to prevent agent influence over their own scores through content escaping and strict parsing of responses.
Regression Detection and Reporting
UAEF detects and classifies regressions in performance metrics using a configurable threshold, defaulting to 5 percent, with severity levels of minor, moderate and severe. Results are stored in Amazon DynamoDB for metadata and Amazon S3 for full evaluation data, including per‑metric scores and judge reasoning. Teams can designate a baseline run for comparison and compare multiple runs against it to detect changes in agent performance.
The system includes a web UI that allows filtering experiments by framework or dataset and highlights regressions and improvements in a comparison view. A detailed statistical report is generated from stored results, showing confidence intervals, outliers, score distributions and correlations between metrics. LLM‑generated insights from Amazon Bedrock analyse reports to identify strengths, weaknesses, root causes and recommendations for agent improvements.
The framework supports adapters for LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore, enabling teams in telecom, healthcare and financial services to benchmark and optimise their agents.