Observability & Tracing

A traditional bug announces itself: an exception, a 500, a red line in the logs. Agents are worse than that. The defining failure mode is the silent failure — the agent returns a clean HTTP 200 carrying an answer that is confidently, completely wrong. Nothing crashed. The model just chose the wrong tool, retrieved the wrong document, or hallucinated a fact, and then wrote a fluent paragraph around it.

Three properties compound to make this hard:

1. Nondeterminism. The same prompt can produce different tool calls on different runs. You cannot reliably reproduce the failure by re-running it.
2. Long, multi-step traces. A deep agent can run hundreds of steps over several minutes — model call, tool call, retrieval, reflection, repeat. The mistake might be in step 37.
3. Silent failures. Because there is no error, you do not even know which run went wrong unless you are scoring outputs.

The only way to debug a system like this is to record what it actually did, step by step, in a form you can inspect after the fact. That recording is a trace, and producing it is the entire discipline of agent observability.

Agent observability borrows its core abstraction from microservices distributed tracing. There are exactly two nouns you need.

• A trace is one complete agent execution — the whole run, end to end, for a single request.
• A span is one discrete operation inside that trace: an LLM call, a tool invocation, a retrieval step, a state transition, or a memory read/write.

Spans nest. The top-level span is the agent invocation; inside it sits a chat span (the planning model call), which spawns an execute_tool span, which contains an HTTP span, and so on. The parent/child links let you reconstruct the exact tree of what called what, in order, with timing. This is why the unit of debugging is the trace tree, not a flat log file.

Think of it like a flame graph for reasoning. When the answer is wrong, you do not re-run the agent — you open the trace, walk down to the span where the decision went sideways (the model picked the wrong tool, the retriever returned junk), and read the exact input and output at that node. The bug stops being a mystery and becomes a coordinate.

A span is only as useful as the data attached to it. Logging the final answer is not enough — the most valuable debug signal lives in intermediate steps: the tool arguments the model chose, which documents it retrieved (and which it didn't), and why it picked one action over another.

The minimum viable schema per span:

The linking IDs are what stitch isolated spans into a navigable tree and let you group a multi-turn conversation (a session) or trace a complaint back to a specific user.

For high-volume production you rarely keep full detail on everything: log baseline metrics (tokens, cost, latency) for 100% of requests, but capture full-detail traces for only 10–20% — using tail-based sampling so you always keep every error trace.

Imagine two tools both record "which model ran," but one calls the field model_name and the other calls it llm.model. Now your dashboards, your queries, and your switch-to-a-new-vendor plan all break on a naming mismatch. A semantic convention is just an agreed-upon dictionary of field names so this never happens — everyone writes gen_ai.request.model and everything interoperates.

To avoid every vendor inventing its own names, the industry is converging on the OpenTelemetry GenAI Semantic Conventions (OTel GenAI SemConv) — a CNCF-backed, vendor-neutral standard for LLM and agent telemetry. (CNCF, the Cloud Native Computing Foundation, is the open-source body that also stewards Kubernetes, so this is an industry standard, not one company's API.) Standardizing the names means you can switch backends without re-instrumenting.

Key attribute and operation names:

# Attributes
gen_ai.request.model            -> "gpt-4o" / "claude-sonnet-..."
gen_ai.usage.input_tokens       -> prompt tokens
gen_ai.usage.output_tokens      -> completion tokens
gen_ai.response.finish_reasons  -> ["stop"], ["tool_calls"]
gen_ai.provider.name            -> "openai" / "anthropic"

# Span operation names
invoke_agent | chat | execute_tool | invoke_workflow

Provider-specific extras are gated by gen_ai.provider.name — e.g. OpenAI adds gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens for o-series models. Since v1.39, the conventions also cover MCP (mcp.method.name, mcp.session.id, mcp.protocol.version) and use W3C Trace Context propagation so an agent trace and the MCP server's trace are linked rather than two disconnected islands.

The conventions decide what names to record; a platform is the product that collects, stores, and shows you those traces in a UI built for LLMs — searchable trace trees, token and cost rollups, side-by-side run comparison. You could write traces to a generic database, but these platforms save you from building the viewer yourself. Four matter:

• LangSmith (LangChain, commercial + on-prem). A 3-tier model — runs (steps) → traces (one run) → threads (full conversation). 2025 added Polly, an AI assistant that answers "why did the agent loop?" from trace data, and langsmith-fetch, a CLI for debugging traces from inside coding agents. Despite the name, it works with any framework via its SDK — not just LangChain.
• Langfuse (open-source, YC W23, self-hostable). Built on OpenTelemetry. Hierarchy: Sessions → Traces → Observations (Generations, Spans, Events). Strong combo of prompt management + eval + tracing; added full LLM-as-judge execution tracing in Oct 2025 so you can audit the evaluator too.
• Arize Phoenix (fully open-source, self-hostable, no feature gates). The only major platform built entirely on OpenTelemetry with no proprietary tracing layer, using OpenInference conventions and 10 span kinds including GUARDRAIL and EVALUATOR. Best for eval rigor and data sovereignty. (Managed tier is Arize AX.)
• MLflow Tracing. As of 3.6.0 it has full OTel support and GenAI SemConv compatibility — a natural fit for teams already using MLflow for model lifecycle.

A second convention layer, OpenInference (Arize-originated), extends OTel with richer RAG metadata and explicit retriever/reranker/embedding span types; Phoenix uses it, and many tools emit both.

Tracing is not an end in itself — its payoff is a closed loop that turns reactive debugging into continuous improvement. The trace-to-fix workflow has four stages:

1. Instrument spans at every LLM, tool, and retrieval step (see the schema above).
2. Score traces online with LLM-as-judge or rule-based scorers — correctness, groundedness, did-it-finish.
3. Harvest failures into evals. Every failed or low-scoring trace becomes a new entry in your eval dataset. Real production failures are the highest-signal test cases you will ever get.
4. Gate releases in CI. Run the eval set on every change; block any release that degrades a quality metric.

This is the bridge from the Evaluating Agents lesson: traces are how you find the failures, evals are how you prevent their return.

A minimal instrumented step looks like this:

from openai import OpenAI
from langfuse import observe, get_client

client = OpenAI()

@observe(as_type="generation")
def plan_step(user_goal: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_goal}],
    )
    # Attach model and token counts to the current generation span
    langfuse = get_client()
    langfuse.update_current_observation(
        model="gpt-4o",
        usage_details={
            "input": resp.usage.prompt_tokens,
            "output": resp.usage.completion_tokens,
        },
    )
    return resp.choices[0].message.content

The decorator opens a span, records input and output automatically, and links it to its parent — turning an opaque step into a coordinate you can inspect when the answer is wrong.