Ch.18 · Observability & the OTel GenAI Stack — The Agentic Platform

OPS · Observability Chapter 18

Observability & the OTel GenAI Stack

A non-deterministic system you cannot observe is a system you cannot operate. When an agent does something wrong — and it will — you must be able to reconstruct exactly what it saw, what it decided, which tools it called, what they returned, how many tokens it burned, and where the latency went. Observability is the difference between a platform you can debug and improve and one you can only apologize for.

18.1 Why traditional monitoring falls short

Classic metrics — request count, error rate, p99 latency — tell you the service is up, not whether the agent reasoned well. Agent execution is a tree of nested, non-deterministic steps: a run calls the model, which calls a tool, which triggers a sub-agent, which retrieves and calls the model again. You need to see that whole tree, with the prompts, completions, tool I/O, and token/cost/latency attributes attached to each node. This is distributed tracing, specialized for LLMs.

Fig 18.1 · An agent run as a span tree. The root span covers the whole run; child spans capture each model call, retrieval, and tool execution, nested to show causality. Per-span token and cost attributes make the trace simultaneously a debugging tool and the raw material for the cost accounting of Chapter 20.

18.2 OpenTelemetry GenAI semantic conventions

The strategic choice is to standardize on OpenTelemetry (OTel) rather than a proprietary agent: it is the vendor-neutral standard for traces, metrics, and logs, and it now defines GenAI semantic conventions — a shared vocabulary for model-call attributes (model name, token counts, parameters), agent spans, and tool invocations. Instrumenting to these conventions means your telemetry is portable across backends and your traces are interoperable. Emit OTel spans from the gateway, the orchestrator, and the tool layer, and you get the whole tree for free.

Fig 18.2 · The telemetry pipeline. The gateway, orchestrator, and tool layer all emit OpenTelemetry spans to a shared collector, whose processor chain redacts PII (Ch. 17), samples, and batches before fanning out to a trace/eval backend, a metrics store, and the audit log. Redaction happens in the collector — before anything is persisted — so one span stream safely feeds debugging, cost, evaluation, and compliance at once.

First principle · Instrument for the question "what did it do, and why?"

Build observability to reconstruct an agent's full decision path — inputs, reasoning, actions, outcomes — for any run, after the fact. Standardize on OpenTelemetry GenAI conventions so that data is portable, vendor-neutral, and shared across every tool that consumes it (evaluation, cost, debugging).

18.3 The tooling landscape

Table 18.1 — LLM observability platforms (representative)
Tool	Shape	Notable for
Langfuse	Open-source LLM observability	Self-hostable tracing, evals, prompt mgmt; OTel-friendly
Arize Phoenix	Open-source, OTel-native	Tracing + evaluation, strong on retrieval/agent analysis
LangSmith	Managed (LangChain)	Deep tracing + eval, tight LangGraph integration
Braintrust	Managed eval + observability	Experimentation and eval-centric workflows
LangWatch	Open-source, eval- & optimization-centric	Tracing, evals, datasets, guardrails, prompt optimization; spans gateway + observability + eval

Because the agent's tree spans model, tools, and sub-agents, observability is also where you first see cost (Ch. 20) and gather the production traces that feed online evaluation (Ch. 19). Treat these three — observe, evaluate, cost — as one connected feedback system rather than separate tools.

Hazard · Traces are full of secrets and PII

Prompts and completions captured in traces routinely contain personal data and credentials. Observability pipelines are therefore in scope for the compliance rules of Chapter 17: redact PII before persistence, control access to trace data, and apply residency and retention policies. A debugging convenience must not become a data-protection breach.

· · ·