Evaluations & Auditability

By the time you reach this lesson you already have Observability wired up: AgentCore auto-instruments your Runtime-hosted agent with OpenTelemetry, spans land in CloudWatch as aws/spans, and the GenAI Observability dashboard shows you sessions, latency, token usage, and error breakdowns. That tells you a great deal — but it only answers operational questions.

There is a second class of question it does not answer:

A trace can be perfectly healthy by every operational metric — 200 ms latency, no throttles, clean span tree — and still be wrong: the agent hallucinated a refund policy, picked the wrong tool, or answered a question it should have refused. Latency cannot see that. Quality scoring can.

The key architectural insight is that both pillars ride the same OTEL traces. You do not emit a second, separate telemetry stream for evaluation. Observability is the data foundation; Evaluations is a consumer that reads those traces, applies LLM-as-a-judge and deterministic scorers, and writes the resulting scores back into the same Observability dashboard. Nothing new to instrument.

The headline workflow is online evaluation. Instead of building a separate offline test harness, you tell AgentCore to sample a configurable percentage of the traces your agent is already producing in production, score that sample, and surface the scores in the Observability dashboard. The critical property for a Builder: no code changes to the agent. Because evaluation rides the existing OTEL traces, you are not adding a judge call to your hot path or redeploying.

Conceptually the loop is:

Production traffic
   |
   v
AgentCore Runtime  --(OTEL traces, GenAI semantic conventions)-->  CloudWatch (aws/spans)
   |                                                                      |
   |                                                            sample X% of traces
   |                                                                      v
   |                                                          AgentCore Evaluations
   |                                                       (built-in + custom evaluators)
   |                                                                      |
   +<------------------ scores written back to the GenAI dashboard ------+

The sampling percentage is the dial that controls cost versus signal. Evaluators are themselves LLM-backed for the judgment-based scores, so a higher sample rate means more judge tokens. The AgentCore CLI exposes evaluators as a first-class resource you can attach to a project:

# New AgentCore CLI (@aws/agentcore) — evaluator is an attachable resource
agentcore add evaluator
agentcore deploy

# Inspect traces the evaluators will score
agentcore traces list

You then read results in CloudWatch under the GenAI Observability surface (the same #gen-ai-observability view, Bedrock AgentCore tab) where you already view sessions and traces — the scores appear alongside the trajectory and timeline for each sampled session.

AgentCore ships 13 built-in evaluators organized by the level of the trace they score. Picking the right level matters: a session-level metric judges the whole conversation, a trace-level metric judges a single turn's response, and a tool-level metric judges the agent's tool-use decisions.

Beyond those, two more categories round out coverage:

• Trajectory match — did the agent take the expected path through tools/steps? Variants: Exact Order Match, In-Order Match, and Any Order Match. Use these when the sequence of actions matters (e.g. "check inventory before charging the card").
• Custom evaluators — define your own scorer for domain rules the built-ins do not cover (e.g. "did the agent cite a knowledge-base article ID?").

A few of these deserve a Builder's attention because they map to real failure modes:

• Faithfulness vs Correctness — Faithfulness asks "is the answer grounded in the context the agent was given?" (the anti-hallucination check), while Correctness asks "is the answer actually right?". A RAG agent can be faithful to a wrong document.
• Harmfulness, Stereotyping, Refusal — the safety triad. Refusal flags when the agent declined; pairing it with Harmfulness tells you whether refusals are protecting users or just being unhelpful.
• Tool Selection Accuracy / Tool Parameter Accuracy — the two tool-level scorers separate "called the wrong tool" from "called the right tool with bad arguments," which are very different bugs to fix.

Evaluations does not invent its own data format — it reads OTEL traces that follow the GenAI semantic conventions. Those conventions standardize how an agent's LLM calls, tool invocations, and inputs/outputs appear as span attributes, which is what lets a generic evaluator understand a trace it has never seen before.

There are two sources it can read from:

1. A Runtime endpoint — your agent deployed on AgentCore Runtime, auto-instrumented, emitting traces as part of normal operation.
2. A CloudWatch log group — traces already collected in CloudWatch (e.g. aws/spans via Transaction Search, or a vended-log delivery group), including agents running outside Runtime that you instrumented with ADOT.

Because it operates at the OTEL layer, Evaluations is framework-agnostic: it consumes Strands and LangGraph traces (also OpenInference-format traces) as long as they emit the GenAI semantic conventions. For Strands you must enable the OTEL extra so the framework actually emits traces:

# Strands agent — emit OTEL traces that Evaluations can read
# pip install 'strands-agents[otel]'
from strands import Agent

agent = Agent()  # Runtime auto-instruments; traces carry GenAI semantic conventions

The non-negotiable prerequisite is the same one from the Observability setup lesson: CloudWatch Transaction Search must be enabled first so spans are searchable, and the framework must actually be emitting traces (the strands-agents[otel] extra or an auto-instrumentor). If no traces flow, there is nothing to sample and nothing to score.

A quality score is only trustworthy if you can go back and inspect the exact interaction it scored. AgentCore's auditability story is built on the same telemetry plumbing.

Durable request/response logs. AgentCore Runtime emits APPLICATION_LOGS that capture the full request_payload and response_payload for each invocation, stamped with trace_id, span_id, and session_id. That means every score the evaluator produces is correlatable back to the literal input the agent received and the literal output it returned, joined by trace and session IDs.

Durable destinations. These logs and spans are durable and can be delivered to CloudWatch Logs, Amazon S3, or Amazon Data Firehose via vended-log delivery — so you can retain an audit trail for as long as compliance requires:

# Vended-log delivery: send Runtime application logs to a destination (CWL / S3 / Firehose)
aws logs put-delivery-source \
  --name my-agent-app-logs \
  --resource-arn <agent-runtime-arn> \
  --log-type APPLICATION_LOGS

aws logs put-delivery-destination \
  --name my-agent-dest \
  --delivery-destination-configuration 'destinationResourceArn=<s3-or-firehose-or-loggroup-arn>'

aws logs create-delivery \
  --delivery-source-name my-agent-app-logs \
  --delivery-destination-arn <delivery-destination-arn>

The default Runtime log group is /aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint>/...; non-Runtime resources use /aws/vendedlogs/bedrock-agentcore/{resource-type}/APPLICATION_LOGS/{resource-id}.

Control-plane audit via CloudTrail. Operational/management actions against AgentCore (creating runtimes, attaching evaluators, etc.) are recorded in AWS CloudTrail, giving you a who-did-what trail separate from the data-plane request/response logs.

Filter sensitive data — this is on you. Because APPLICATION_LOGS capture full request/response payloads, they will capture PII and secrets unless you filter them before emission. This is the single most important auditability gotcha: a durable, queryable record of every prompt and response is exactly what an auditor wants and exactly what a breach wants. Strip or redact sensitive fields in your agent before they reach the log payload.

Evaluations is consumption-priced, and the dimensions matter for how aggressively you sample.

• Built-in evaluators bill per input + output token consumed by the judge model.
• Custom evaluators bill per 1,000 evaluations.

All concrete rates are [VOLATILE] — teach the dimension, verify the number at https://aws.amazon.com/bedrock/agentcore/pricing/. (As of writing the pricing page lists built-in at roughly per-thousand-token rates and custom at a per-1k-evals rate — confirm the live figures before budgeting.)

Observability itself has no service-specific charge — its telemetry is billed at standard CloudWatch rates (Transaction Search first 1% of indexed spans free, then per indexed span; plus Logs ingestion/storage). So your total quality-and-audit bill is: CloudWatch (the trace/log substrate) + Evaluations (the scoring on top).

The gotchas that bite Builders:

• Sampling rate drives cost. Online evaluation runs judges over sampled traffic; a high sample rate over high-volume traffic multiplies judge tokens. Tune deliberately.
• No traces, no scores. Evaluations cannot score what was never emitted — confirm Transaction Search is on and the framework emits OTEL.
• PII in logs. Covered above — filter before emission.
• GA since March 31 2026. Still confirm region availability and pricing before depending on it in production governance.
• Don't conflate the two pillars. A green Observability dashboard (low latency, no errors) is not a quality guarantee; only evaluators answer correct/helpful/safe.