What is agent observability?
Agent observability is the practice of recording every step an AI agent takes, including model calls, tool calls, retrievals, inputs, outputs, cost and latency, as structured traces a team can search, score and debug.
Checked
Classic application monitoring answers whether a service is up and how fast it responds. An agent can return a clean response in two seconds and still have called the wrong tool, looped six times, leaked a customer's email address or given a confident wrong answer. None of that shows up as an error code.
Agent observability captures the whole run instead: each model call, each tool call with its arguments and result, each retrieval, and the reasoning that connected them. That record is what engineers read when something goes wrong, and it is the raw material every eval and production check is built on.
What an agent trace contains
Spans for each operation
The agent invocation, every model call, every tool execution, retrievals and memory reads and writes, nested so the run can be replayed in order.
Inputs and outputs
The prompt and context the model saw, the tool arguments it chose and what each tool returned.
Tokens, cost and latency
Per step and per run, so an expensive or slow path can be traced to the step that caused it.
Session and user context
Identifiers that tie a trace to a conversation, a user or a business record, with personal data redacted.
Scores
Eval results attached to the trace, from code checks, LLM judges or human review.
Where the field moved in 2026
The OpenTelemetry GenAI semantic conventions now define standard operations for agents (create_agent, invoke_agent), tools (execute_tool), retrieval and memory, plus a gen_ai.evaluation.result event for eval scores. As of October 2026 all of them are still at Development status, not Stable.
Vendors have relabelled around agents: Datadog's LLM Observability product is now titled Agent Observability. The newer product trend is clustering production traces into named failure modes: LangSmith Engine (beta since May 2026) scans traces, groups failures into issues and generates evaluators, and Laminar's Signals and Braintrust's Topics do similar grouping.
Observability and evals
Observability records what happened; evals judge whether it was good. The two meet in trace grading and online evals, where the same graders used in development score live production traces. Anthropic's agent eval guide treats production monitoring as one layer among several, alongside automated evals, A/B tests, user feedback and manual transcript review, and notes that no single layer catches everything.
How Motiontic's engineers apply it
Motiontic's engineers instrument an agent before its first user sees it, on whichever tracing backend the client's team already runs or on OpenTelemetry, so the trace format is not tied to one vendor. Personal data is redacted at capture, not after.
Every failed or flagged trace is a candidate eval task: once read and labelled, it joins the regression suite so the same failure is caught on the next change.
Application monitoring compared with agent observability
| Application monitoring (APM) | Agent observability | |
|---|---|---|
| Unit of record | A request and its service calls | An agent run: model calls, tool calls, retrievals |
| Failure signal | Errors, timeouts, status codes | Wrong tool, bad arguments, loops, wrong or unsafe answers |
| Quality | Not measured | Scored by code checks, LLM judges and people |
| Cost | Infrastructure spend | Tokens and spend per step, run and user |
| Typical question | Is the service up and fast? | Why did the agent do that, and was it right? |
Common questions
What is agent observability?
Agent observability is recording each step an AI agent takes, its model calls, tool calls, inputs, outputs, cost and latency, as structured traces, so a team can see why the agent behaved as it did and score whether it was right.
How is agent observability different from LLM observability?
LLM observability traces single model calls. Agent observability traces multi-step runs in which a model chooses tools, reads results and decides what to do next, so it adds tool spans, nested steps and per-run scoring.
Is OpenTelemetry enough for agent observability?
OpenTelemetry provides the transport and, through the GenAI semantic conventions, standard names for agent, tool and eval data. The conventions are still at Development status, so attribute names can change, and teams still need a backend to store, search and score traces.
Which tools offer agent observability?
Options include LangSmith, Langfuse, Braintrust, Arize Phoenix, Pydantic Logfire, MLflow and Datadog Agent Observability. They differ mainly on open source versus hosted, self-hosting terms, pricing model and how tightly they couple to one agent framework.
Related
OpenTelemetry GenAI semantic conventions
Standard gen_ai.* names for agent and model telemetry, still at Development status.
Trace grading
Scoring a whole recorded agent run, step by step.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
LangSmith vs Langfuse
Hosted platform from LangChain against the open-source, self-hostable platform now owned by ClickHouse.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
AgentOps
DevOps and MLOps, adapted for systems that decide their own next step.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.