85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is trace grading?

Trace grading is the practice of assigning scores or labels to a recorded agent trace, the end-to-end log of model calls, tool calls, handoffs and outputs, to judge how well the agent performed and where it went wrong.

Checked

A final answer says whether an agent got there. A trace says how. Trace grading puts graders on the full record of a run, so a team can see not just that a support agent gave a wrong refund amount, but that it read the wrong order at step three.

OpenAI made the term standard in its agent tooling, where the recommended flow runs from traces, to trace grading while debugging, to datasets and repeatable eval runs once failure patterns are understood.

What gets graded

  • Individual steps

    Was the right tool chosen, were its arguments correct, was its result read properly?

  • The whole run

    Did the agent stay on task, follow policy, avoid exposing personal data and finish in a reasonable number of steps?

  • Handoffs

    In multi-agent systems, did control pass to the right agent with the right context?

  • The outcome

    Was the final answer or end state correct?

Who does the grading

Graders can be code, such as deterministic checks that a required tool appears in the trace, an LLM judge applying a rubric, or a person. OpenAI's skill eval guidance pairs deterministic checks on traces with a schema-constrained rubric grader. LLM judges on traces need the same calibration against human labels as any other judge.

Trace grading in 2026

OpenAI deprecated its hosted Evals platform on 2026-06-03: existing evals went read-only on 2026-10-31 and the dashboard and API shut down on 2026-11-30, with Promptfoo named as the migration path. The technique itself is not tied to one platform. LangSmith, Langfuse, Braintrust, Arize Phoenix and other tools score traces, and the OpenTelemetry GenAI conventions define a standard event for attaching eval results to telemetry.

How Motiontic's engineers apply it

Motiontic's engineers grade traces first when an agent is new and its failure modes are unknown, then turn each recurring failure into a dataset task with a fixed grader. Trace grades are stored with the trace, so a reviewer can move from a low score straight to the step that caused it.

Three ways to grade an agent run

Three ways to grade an agent run
Outcome gradingTrajectory evaluationTrace grading
Looks atFinal answer or end stateSequence of tool callsThe full recorded run
Needs a referenceUsuallyYes, a reference pathNot always: rubrics work without one
FindsWhether it workedSkipped or extra stepsWhere and why it failed
Typical useRelease gatesRequired-step checksDebugging and production review

Common questions

What is trace grading?

Trace grading is scoring a recorded agent run, its model calls, tool calls, handoffs and output, with code checks, an LLM judge or a person, to measure quality and locate the step where things went wrong.

How is trace grading different from trajectory evaluation?

Trajectory evaluation compares the tool-call sequence with a reference path. Trace grading is broader: it can score any aspect of the run, including tone, policy, data exposure and cost, often without a reference.

Is OpenAI trace grading still available?

OpenAI's hosted Evals platform shuts down on 2026-11-30, with Promptfoo as the named migration path. Teams relying on hosted trace grading should check OpenAI's deprecations page; other observability tools also score traces.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.