The Observability Model

Once an agent leaves your laptop and starts serving real traffic, the question stops being "does it work?" and becomes "how fast, how much did it cost, where did it fail?" That is the job of AgentCore Observability — the operations and telemetry pillar that lets you trace, debug, and monitor agents in production with per-step visualizations.

Two facts anchor everything else in this lesson:

• It is powered by Amazon CloudWatch. AgentCore does not invent a new monitoring backend. Every metric, span, and log lands in CloudWatch (CloudWatch Metrics, CloudWatch Logs, X-Ray/Transaction Search). If you already operate on CloudWatch, you already know where this data lives and how to alarm on it.
• It emits OTEL-compatible telemetry. The data is OpenTelemetry-shaped, so it plugs into your existing observability stack. Runtime-hosted agents are auto-instrumented with OpenTelemetry, and you can route that telemetry to a non-AWS backend if you prefer (covered in a later lesson).

By default, Observability outputs built-in metrics for agents, gateways, and memory; spans and logs are also available (memory can emit them once enabled). Across the board it surfaces the signals you actually page on: session count, latency, duration, token usage, and error rates.

Your agent (Runtime / Gateway / Memory / Tools / Policy)
        │   OpenTelemetry-shaped telemetry
        ▼
   Amazon CloudWatch
   ├── Metrics  (namespace: bedrock-agentcore)
   ├── Logs     (/aws/bedrock-agentcore/runtimes/...)
   └── Spans/Traces (aws/spans  →  Transaction Search)
        │
        ▼
   GenAI Observability dashboard  (Runtime + Policy only)

The rest of this lesson is about that diagram: which signal goes where, what each resource emits by default, and which of them you can actually see in the polished GenAI dashboard versus having to open raw CloudWatch.

Observability produces five kinds of telemetry, and the most useful thing you can memorize is where each one lives in CloudWatch. When something breaks at 2am, knowing the exact namespace and log-group path is the difference between a five-minute fix and an hour of clicking around.

A few clarifications worth internalizing:

• Metrics are your dashboards and alarms. Agent-emitted metrics use Embedded Metric Format (EMF) under the namespace bedrock-agentcore. If you publish custom metrics, you write to that same namespace.
• Spans are the individual units of work (one LLM call, one tool invocation) and land as log events in aws/spans — but you only get to search them once CloudWatch Transaction Search is enabled. That one-time enablement is the most common setup miss (covered in the setup lesson).
• Traces are spans stitched into the end-to-end request path. They are what powers the Trace View (trajectory + timeline) in the GenAI dashboard.
• Logs are the raw stdout/stderr and application logs. For Runtime agents they auto-flow to /aws/bedrock-agentcore/runtimes/....
• Sessions group everything by the session.id attribute, giving you a per-conversation view in the Sessions View.

# Sessions are correlated by the session.id attribute.
# For Runtime-hosted agents, pass a runtimeSessionId on invoke and AgentCore
# stamps it onto the telemetry (the header is
# X-Amzn-Bedrock-AgentCore-Runtime-Session-Id under the hood):
response = client.invoke_agent_runtime(
    agentRuntimeArn=agent_arn,
    runtimeSessionId="session-7f3a...",   # ties spans/logs/metrics to one session
    payload=b'{"prompt": "summarize the incident"}',
)

Observability is not uniform across AgentCore. The default telemetry you get depends on which resource type you are looking at, and — critically — where you can view it. Here is the breakdown:

The pattern: Agent (Runtime) is the first-class citizen — you get metrics surfaced in the rich GenAI dashboard out of the box. Everything else defaults to metrics in plain CloudWatch. For Memory, spans and logs are available but require explicit enablement (you turn on Tracing on the resource; non-Runtime resources do not even auto-create a log group). Policy is the odd one out: it shows in the GenAI dashboard, but under the Gateway tab rather than its own.

This is why "does AgentCore give me observability?" is the wrong question. The right questions are: which resource, what default telemetry, and which surface (dashboard vs raw CloudWatch). Get that mental model right and you will never go looking for Gateway metrics in a dashboard that doesn't show them.

This is the single most important thing to take away from the lesson, and the mistake most teams make first.

Gotcha: Only Agent (Runtime) — and Policy, which appears under the Gateway tab — shows up in the CloudWatch GenAI Observability dashboard. Memory, Gateway, and Tools are in plain CloudWatch only.

The failure mode is predictable: you deploy a Runtime agent, see beautiful Agents/Sessions/Trace views in the GenAI dashboard, then attach a Gateway and Memory and go looking for them in the same dashboard. They are not there — and nothing is broken. Their telemetry is flowing into CloudWatch Metrics (and, once you enable it, Logs/Spans), it just is not surfaced in the curated GenAI experience.

Where to actually look:

# The GenAI Observability dashboard (Runtime + Policy)
CloudWatch console → #gen-ai-observability → "Bedrock AgentCore" tab
    ├── Agents View
    ├── Sessions View      (group by session.id)
    └── Trace View         (trajectory + timeline, span details, error breakdowns)

# Everything else (Memory / Gateway / Tools) — plain CloudWatch:
CloudWatch console → Metrics → namespace: bedrock-agentcore
CloudWatch console → Logs → /aws/bedrock-agentcore/runtimes/...   (Runtime)

The raw locations are worth bookmarking, because debugging often drops below the dashboard:

• Standard logs: /aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint>/[runtime-logs]...
• OTEL logs: .../runtime-logs (one doc shows .../otel-rt-logs — treat the suffix as volatile and verify on the live page)
• Traces/spans: /aws/spans/default, via Transaction Search
• Metrics: namespace bedrock-agentcore

When a teammate says "the dashboard doesn't show my Gateway," the answer is not a bug report — it is "correct, open CloudWatch Metrics and filter on the bedrock-agentcore namespace."

For Runtime agents — the resource you will monitor most — AgentCore emits a precise set of metrics in 1-minute batches under the bedrock-agentcore namespace. Know these names; they are what you alarm and dashboard on.

Activity and errors:

WebSocket / streaming: ActiveStreamingConnections, InboundStreamingBytesProcessed, OutboundStreamingBytesProcessed.

Resource usage (vended): CPUUsed-vCPUHours, MemoryUsed-GBHours, carried on dimensions Service=AgentCore.Runtime, Resource=<Agent ARN>, Name=AgentName::EndpointName.

Underpinning the dashboard, the span operation is InvokeAgentRuntime (with attributes including session.id, latency_ms, and error_type ∈ throttle/system/user), written to aws/spans. Application logs (APPLICATION_LOGS) capture the full request_payload/response_payload plus trace_id/span_id/session_id.

The error codes behind User Errors/System Errors/Throttles are precise:

User errors:        InvocationError.Validation        (400)
                    InvocationError.ResourceNotFound   (404)
                    InvocationError.AccessDenied        (403)
                    InvocationError.Conflict            (409)
System errors:      InvocationError.Internal            (500)
Throttling:         InvocationError.Throttling          (429)
                    InvocationError.ServiceQuota        (402)

Two critical caveats on the resource-usage metrics:

1. CPUUsed-vCPUHours / MemoryUsed-GBHours are NOT your bill. Resource-usage telemetry ≠ billing. Use it to understand utilization and right-size, but never reconcile it against an invoice — consult Cost Explorer / the pricing page for actual charges.
2. They can lag ~60 minutes. These vended metrics are not real-time. If you are watching a load test, do not panic when CPU/memory numbers trail the traffic by up to an hour.

Step back and the whole model collapses into a short decision procedure you can run in your head whenever you need a signal:

1. What signal do I want? A number over time → Metric (bedrock-agentcore namespace). A single unit of work → Span (aws/spans). The full request path → Trace (Transaction Search / GenAI dashboard). Raw output → Logs (/aws/bedrock-agentcore/runtimes/...). Per-conversation grouping → Session (session.id).
2. Which resource is it on? Agent (Runtime) is rich-by-default; Memory/Gateway/Tools default to metrics only and need extra setup for spans/logs.
3. Where do I look? Runtime (and Policy, under the Gateway tab) → the GenAI Observability dashboard. Everything else → plain CloudWatch.

A concrete walk-through of a latency complaint on a Runtime agent:

Complaint: "the agent feels slow"
  → Metric:  Latency (bedrock-agentcore) — confirm + quantify the regression
  → Trace:   GenAI dashboard → Trace View — open a slow trace, read the timeline
  → Span:    find the slow span (LLM call? tool call?) in the trajectory
  → Session: Sessions View — is it one bad session or systemic?
  → Logs:    /aws/bedrock-agentcore/runtimes/... — read request/response payloads

Notice you never needed a new tool. Because Observability is powered by CloudWatch and the telemetry is OTEL-compatible, every step above is either a CloudWatch view or your existing OTEL stack. The mental model — five telemetry types, five homes, per-resource defaults, and the Runtime/Policy-only dashboard rule — is the whole lesson.