What is trajectory evaluation?
Trajectory evaluation scores the sequence of steps an AI agent took, such as which tools it called, in what order and with which arguments, against a reference path or rubric, rather than scoring only the final answer.
Two agents can reach the same correct answer by very different routes. One checks the customer's account, reads the policy and issues a refund. The other issues the refund straight away and happens to be right. A final-answer eval scores them the same; a trajectory eval does not.
Trajectory evaluation looks at the path: the tools called, their order, their arguments and how many steps it took. It matters most where the process is itself a requirement, and where an inefficient or risky path is a problem even when the outcome is fine.
Common trajectory metrics
Exact match
The agent's tool calls match the reference sequence exactly.
In-order match
All reference calls appear in the right order, with extra calls allowed.
Any-order match
All reference calls appear, in any order.
Precision and recall
How many of the agent's calls were in the reference, and how many reference calls the agent made.
Single tool use
Whether one specific tool was called at all, such as a sanctions check before a payment.
What the labs say
Google's Gemini Enterprise Agent Platform evaluation service ships the metrics above (trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, trajectory_precision, trajectory_recall and trajectory_single_tool_use) and evaluates the final response alongside them. Google's ADK adds tool_trajectory_avg_score and a rubric-based, model-graded tool use quality criterion.
OpenAI's agent eval guidance checks tool selection and argument precision for single agents and handoff accuracy for multi-agent systems. Anthropic takes a different line: grade the outcome the agent produced, not the exact path, so the agent is free to find valid solutions no one planned for.
When the path matters
Mandatory steps
Regulated work where a check must happen before an action, whatever the outcome. See the compliance agents page.
Cost and latency
An agent that reaches the right answer in 30 tool calls instead of 4 is a cost problem.
Risky side effects
A path that touched a system it did not need to is worth flagging even when the answer is right.
Debugging
When an outcome eval fails, the trajectory shows which step went wrong.
How Motiontic's engineers apply it
Motiontic's engineers grade the outcome first and add trajectory checks only for steps that are genuinely required, such as an approval or a policy lookup, using single-tool or in-order checks rather than exact matches. Exact-match trajectories penalise valid alternative routes and break every time a tool is renamed, so they are reserved for fixed, regulated sequences.
Outcome grading compared with trajectory evaluation
| Outcome grading | Trajectory evaluation | |
|---|---|---|
| Scores | The final answer or end state | The tool calls, order and arguments |
| Needs | A reference outcome or rubric | A reference trajectory or step rubric |
| Strength | Leaves room for valid, unplanned solutions | Catches skipped checks, waste and risky steps |
| Main risk | Misses a right answer reached the wrong way | Penalises valid alternative paths |
| Fits | Most tasks with a checkable result | Regulated processes, cost control, debugging |
Common questions
What is trajectory evaluation?
Trajectory evaluation is scoring the steps an AI agent took, its tool calls, their order and arguments, against a reference path or rubric, as well as or instead of scoring its final answer.
Should agent evals grade the trajectory or the outcome?
Most teams weight the outcome and add trajectory checks for steps that are required. Anthropic recommends outcome grading so agents can find unplanned valid paths; Google and OpenAI also evaluate the path.
What is a reference trajectory?
A reference trajectory is the expected sequence of tool calls for a task, written by someone who knows the process. Agent runs are compared with it using exact, in-order, any-order, precision or recall metrics.
Related
Trace grading
Scoring a whole recorded agent run, step by step.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Compliance agents
Agents that do compliance work, and the guardrails that keep a company's other agents within policy.
Tool calling
How a model asks software to do something: a named tool, typed arguments, a result.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.