Setup & Viewing Your Data

Before you look at a single trace, there is one switch you must flip — and forgetting it is the #1 observability gotcha. CloudWatch Transaction Search is what makes spans searchable. AgentCore emits spans to the CloudWatch Logs group aws/spans (a.k.a. /aws/spans/default), but until Transaction Search is enabled, X-Ray won't index those spans into traces and the GenAI dashboard's Trace view will be empty.

This is a one-time, account-level action. After you enable it, allow ~10 minutes before spans become searchable.

Console path: CloudWatch → Settings → Account → X-Ray traces → Transaction Search → Enable, then set the index percentage. The first 1% of indexed spans is free; you can raise the percentage later (everything above 1% is billed at standard CloudWatch rates).

API / CLI path — three calls. First, a resource policy that lets X-Ray write events into the span log groups; then point trace segments at CloudWatch Logs; then set the indexing rule:

# 1) Allow xray.amazonaws.com to PutLogEvents into the span log groups
aws logs put-resource-policy \
  --policy-name TransactionSearchAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Service": "xray.amazonaws.com" },
      "Action": "logs:PutLogEvents",
      "Resource": [
        "arn:aws:logs:<region>:<account-id>:log-group:aws/spans:*",
        "arn:aws:logs:<region>:<account-id>:log-group:/aws/application-signals/data:*"
      ]
    }]
  }'

# 2) Send trace segments to CloudWatch Logs (the Transaction Search store)
aws xray update-trace-segment-destination --destination CloudWatchLogs

# 3) Set the indexing rule (1% is free). The exact --name/--rule
#    shape is an X-Ray API detail — confirm it against the live
#    configure-observability docs before scripting it.
aws xray update-indexing-rule \
  --name "Default" \
  --rule '{ "Probabilistic": { "DesiredSamplingPercentage": 1 } }'

The two grant targets above — aws/spans and /aws/application-signals/data — are exactly where AgentCore's spans and Application Signals data land. Get this wrong and traces silently never index.

If your agent runs on AgentCore Runtime, you are mostly done after Step 1. Deploying via the CLI auto-instruments the agent with OpenTelemetry — no extra libraries, no env vars, no log-group creation. You get baseline traces, spans, sessions, and metrics for free, and Runtime auto-creates its log group.

# Deploy the auto-instrumented Runtime agent (starter toolkit verb shown; the
# newer @aws/agentcore CLI uses `agentcore deploy` the same way)
agentcore deploy

# Invoke it — pass a session id to group spans in the Sessions view
agentcore invoke '{"prompt":"summarize my open tickets"}' --session-id my-session-0001

The critical detail for viewing data is the session id. Passing runtimeSessionId (the --session-id flag on the CLI, or runtimeSessionId in a boto3 InvokeAgentRuntime call) is what drives the Sessions view — it's the key the dashboard groups spans under. Omit it and every invocation looks like an orphan; pass a stable id across turns of one conversation and you get a clean session timeline.

Under the hood the auto-instrumentation captures the agent invocation sequence, LLM calls and responses, tool invocations and results, error paths, and your framework integration (Strands, LangChain, LangGraph, etc.). The span operation for the invocation is InvokeAgentRuntime, carrying attributes including session.id, latency_ms, and error_type.

Here is the asymmetry that trips people up. Runtime auto-creates a log group. The other resources do NOT. Memory, Gateway, Built-in Tools, and Identity will not deliver spans or logs anywhere until you configure vended-log delivery yourself. Enable nothing and you'll wonder why your Gateway shows metrics but no traces.

Vended-log delivery is a standard CloudWatch Logs mechanism made of three pieces, wired with the logs client:

import boto3

logs = boto3.client("logs")
resource_arn = "arn:aws:bedrock-agentcore:<region>:<account-id>:gateway/<gateway-id>"

# 1) The delivery SOURCE: which resource + what kind of telemetry
logs.put_delivery_source(
    name="agentcore-gw-traces",
    resourceArn=resource_arn,
    logType="TRACES",          # or "APPLICATION_LOGS"
)

# 2) The delivery DESTINATION: where it goes (CloudWatch Logs / X-Ray / S3 / Firehose)
logs.put_delivery_destination(
    name="agentcore-gw-dest",
    deliveryDestinationType="XRAY",   # "CWL" for CloudWatch Logs, "XRAY" for traces
    deliveryDestinationConfiguration={
        "destinationResourceArn": "arn:aws:logs:<region>:<account-id>:log-group:/aws/vendedlogs/..."
    },
)

# 3) The DELIVERY: bind source to destination
logs.create_delivery(
    deliverySourceName="agentcore-gw-traces",
    deliveryDestinationArn="arn:aws:logs:<region>:<account-id>:delivery-destination:agentcore-gw-dest",
)

The two logType values you'll use are APPLICATION_LOGS (stdout/stderr + app logs, including full request/response payloads) and TRACES (spans). Destinations can be CloudWatch Logs (CWL), X-Ray (XRAY), S3, or Data Firehose. When delivered to CloudWatch Logs, the default group follows the pattern:

Memory has an extra step: Memory spans require enabling Tracing on the Memory resource (a toggle in the console) — and, like everything else, that depends on Transaction Search already being on. So the order for Memory tracing is: Transaction Search → enable Tracing on the memory store → configure vended-log delivery for the spans.

With data flowing, the primary lens is the GenAI Observability dashboard in CloudWatch:

CloudWatch → #gen-ai-observability → Bedrock AgentCore tab → Agents View / Sessions View / Trace View

• Agents View — your Runtime agents with rolled-up metrics: session count, latency, duration, token usage, error rates.
• Sessions View — invocations grouped by session id. This is why passing runtimeSessionId matters; without it the grouping collapses.
• Trace View — the per-request trace trajectory + timeline, with span details and error breakdowns. This is where you click into one slow request and see exactly which tool call or LLM step ate the latency.

A hard limit to internalize: only Agent (Runtime) shows up in the GenAI dashboard. (Policy also appears, but under the Gateway tab.) Memory, Gateway, and Built-in Tools live in plain CloudWatch only — you'll find their metrics and any spans/logs you enabled in regular CloudWatch Metrics and Logs, not in the GenAI dashboard.

Raw CloudWatch locations (when you need to grep logs or build a custom widget):

A single user request might hop through Runtime → Gateway → a tool → Memory. To see it as one story, two kinds of identifiers must propagate: a session id (groups many requests in a conversation) and trace context (stitches the spans of one request together).

Session id. On Runtime, it rides the header X-Amzn-Bedrock-AgentCore-Runtime-Session-Id (the runtimeSessionId parameter in boto3). For code running outside Runtime, set it via OTEL baggage so it lands on the session.id span attribute:

from opentelemetry import baggage, context

# Outside Runtime: attach the session id so spans carry session.id
ctx = baggage.set_baggage("session.id", "my-session-0001")
context.attach(ctx)

Trace context. The headers AgentCore honors for trace propagation are:

X-Amzn-Trace-Id
traceparent
tracestate
baggage
X-Amzn-Bedrock-AgentCore-Runtime-Session-Id
mcp-session-id

X-Amzn-Trace-Id is the AWS-native form; traceparent / tracestate are the W3C Trace Context standard; baggage carries arbitrary key/values like session.id. These same custom headers are accepted on the Built-in Tools APIs (StartCodeInterpreterSession, InvokeCodeInterpreter, StartBrowserSession, StopBrowserSession) and the Identity APIs (GetWorkloadAccessToken*, GetResourceOauth2Token, GetResourceAPIKey) — so if you forward the trace headers when your agent calls those services, their spans join the same trace.

In the Trace View, correlation shows up as a single trajectory: the InvokeAgentRuntime span at the root, child spans for each LLM call and tool invocation, all sharing one trace id and tagged with the same session.id.

AgentCore's APPLICATION_LOGS are deliberately rich for auditability: they capture the full request_payload and response_payload alongside trace_id, span_id, and session_id. That is exactly what you want when reconstructing what an agent did — and exactly the place a secret or a customer's PII leaks if you aren't careful.

The payloads are verbatim. If your prompt contains an API key, a Social Security number, a full email thread, or a bearer token a user pasted in, it lands in CloudWatch Logs (and, if you configured S3/Firehose delivery, anywhere those flow). Treat application logs as a data-classification surface, not just a debugging convenience.

The fix is to redact before emission, inside your entrypoint, before anything is logged:

import re
from bedrock_agentcore.runtime import BedrockAgentCoreApp

app = BedrockAgentCoreApp()

SECRET_RE = re.compile(r"(sk-[A-Za-z0-9]{20,}|Bearer\s+\S+|\b\d{3}-\d{2}-\d{4}\b)")

def redact(text: str) -> str:
    return SECRET_RE.sub("[REDACTED]", text)

@app.entrypoint
def handler(payload):
    prompt = redact(payload.get("prompt", ""))
    # ... run the agent on the sanitized prompt; never log the raw payload ...
    return {"result": prompt}

For a managed layer, combine this with Bedrock Guardrails (PII filters) so sensitive content is caught even when your regex misses it. The principle from the security model is blunt: filter PII/secrets before emission — once it's in the logs it's durable.