What is trace grading?
Trace grading is the practice of assigning scores or labels to a recorded agent trace, the end-to-end log of model calls, tool calls, handoffs and outputs, to judge how well the agent performed and where it went wrong.
Checked
A final answer says whether an agent got there. A trace says how. Trace grading puts graders on the full record of a run, so a team can see not just that a support agent gave a wrong refund amount, but that it read the wrong order at step three.
OpenAI made the term standard in its agent tooling, where the recommended flow runs from traces, to trace grading while debugging, to datasets and repeatable eval runs once failure patterns are understood.
What gets graded
Individual steps
Was the right tool chosen, were its arguments correct, was its result read properly?
The whole run
Did the agent stay on task, follow policy, avoid exposing personal data and finish in a reasonable number of steps?
Handoffs
In multi-agent systems, did control pass to the right agent with the right context?
The outcome
Was the final answer or end state correct?
Who does the grading
Graders can be code, such as deterministic checks that a required tool appears in the trace, an LLM judge applying a rubric, or a person. OpenAI's skill eval guidance pairs deterministic checks on traces with a schema-constrained rubric grader. LLM judges on traces need the same calibration against human labels as any other judge.
Trace grading in 2026
OpenAI deprecated its hosted Evals platform on 2026-06-03: existing evals went read-only on 2026-10-31 and the dashboard and API shut down on 2026-11-30, with Promptfoo named as the migration path. The technique itself is not tied to one platform. LangSmith, Langfuse, Braintrust, Arize Phoenix and other tools score traces, and the OpenTelemetry GenAI conventions define a standard event for attaching eval results to telemetry.
How Motiontic's engineers apply it
Motiontic's engineers grade traces first when an agent is new and its failure modes are unknown, then turn each recurring failure into a dataset task with a fixed grader. Trace grades are stored with the trace, so a reviewer can move from a low score straight to the step that caused it.
Three ways to grade an agent run
| Outcome grading | Trajectory evaluation | Trace grading | |
|---|---|---|---|
| Looks at | Final answer or end state | Sequence of tool calls | The full recorded run |
| Needs a reference | Usually | Yes, a reference path | Not always: rubrics work without one |
| Finds | Whether it worked | Skipped or extra steps | Where and why it failed |
| Typical use | Release gates | Required-step checks | Debugging and production review |
Common questions
What is trace grading?
Trace grading is scoring a recorded agent run, its model calls, tool calls, handoffs and output, with code checks, an LLM judge or a person, to measure quality and locate the step where things went wrong.
How is trace grading different from trajectory evaluation?
Trajectory evaluation compares the tool-call sequence with a reference path. Trace grading is broader: it can score any aspect of the run, including tone, policy, data exposure and cost, often without a reference.
Is OpenAI trace grading still available?
OpenAI's hosted Evals platform shuts down on 2026-11-30, with Promptfoo as the named migration path. Teams relying on hosted trace grading should check OpenAI's deprecations page; other observability tools also score traces.
Related
Trajectory evaluation
Grading the route an agent took, not just where it ended up.
Agent observability
Step-by-step traces of what an agent did, why, and what it cost.
LLM as a judge
A model grading model output against a rubric, checked against human labels.
Online evals
Grading live traffic in production, not only test sets before release.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.