What is a grader in AI evals?
A grader, in AI evals, is the logic that scores some aspect of an agent's or model's attempt at a task, using code checks, a model following a rubric, or a human reviewer.
An eval task says what should happen; the grader decides whether it did. Anthropic's January 2026 guide defines a grader as logic that scores some aspect of performance, and notes that one task can carry several graders, each with several assertions: was the customer verified first, was the refund issued, was the reply grounded in the tool results.
Graders are where many eval results go wrong. A score is only as good as the check behind it, so a grader needs testing and calibration of its own, and someone has to read the transcripts it judged.
Three kinds of grader
Code-based
String matching, unit tests, static analysis, checks on the state of the environment and checks on which tools were called. Fast, cheap, reproducible and easy to debug, but brittle when a valid answer takes an unexpected form, and weak on subjective work.
Model-based
Rubric scoring, natural-language assertions, pairwise comparison, reference-based checks and multi-judge consensus. Flexible and able to handle open-ended output, but non-deterministic, more expensive than code and in need of calibration against human judgement.
Human
Subject-matter-expert review, spot-check sampling, A/B tests and inter-annotator agreement. The gold standard, and the source used to calibrate model graders, but slow and expensive at scale.
Grade the outcome, give partial credit
Anthropic's advice is to grade what the agent produced rather than the exact sequence of steps it took, because rigid step-order checks penalise valid solutions no one anticipated. The outcome is the final state of the environment: whether a reservation exists in the database, not whether the agent said it made one.
For multi-step tasks, partial credit separates near misses from early failures. An agent that verifies the customer and identifies the problem but fails to issue the refund has done better than one that fails at the first step, and a binary grader hides the difference. Where the path itself is a requirement, such as a compliance check that must happen first, trajectory evaluation adds checks on the steps.
How graders fail
Too strict
Anthropic's CORE-Bench example: a grader rejected 96.12 when the expected value was 96.124991. Fixing grading bugs and ambiguous specs, with a less constrained scaffold, moved Claude Opus 4.5 from 42% to 95%.
Misconfigured
METR found tasks that required exceeding a stated score threshold, which penalised models that followed the instructions and rewarded those that ignored them.
Checking the claim, not the state
A grader that reads the agent's final message rewards agents that say they finished. Checking the system of record removes the incentive.
Uncalibrated judges
Model-based graders drift from human judgement unless their verdicts are compared with human labels on the same transcripts, repeatedly.
How Motiontic's engineers apply it
Motiontic's engineers use code graders wherever the outcome can be checked in the system itself, such as a database row, a passing test or a saved draft, and model graders only for qualities code cannot check. Model graders use rubrics calibrated against labels from the client's own reviewers, and every new grader is checked by reading a sample of the transcripts it passed and failed.
Grader types compared
| Code-based | Model-based | Human | |
|---|---|---|---|
| Examples | Unit tests, state checks, tool-call checks | Rubric scoring, pairwise comparison | Expert review, spot checks |
| Speed and cost | Fast and cheap | Slower, costs model calls | Slow and expensive |
| Consistency | Deterministic | Non-deterministic | Varies by reviewer |
| Weakness | Brittle to valid variations | Needs calibration against humans | Hard to scale |
| Fits | Verifiable outcomes | Open-ended output, tone, groundedness | Calibration and subjective quality |
Common questions
What is a grader in an AI eval?
A grader is the part of an eval that scores an attempt at a task. It can be code that checks the result, a model applying a rubric, or a person reviewing the output.
What is the difference between a grader and an LLM judge?
An LLM judge is one kind of grader, the model-based kind. Graders also include deterministic code checks and human reviewers, and many tasks use more than one.
Should an agent grader check the steps or the result?
Anthropic recommends grading the outcome, so the agent is free to find valid paths no one anticipated. Step checks are added where the path is itself a requirement, such as a verification that must come before a payment.
How is a model-based grader calibrated?
By scoring the same transcripts with the model grader and with human reviewers, adjusting the rubric until they agree closely enough, and repeating the comparison as the agent and model change.
Related
LLM as a judge
A model grading model output against a rubric, checked against human labels.
Task, trial and transcript (eval vocabulary)
The basic units every agent eval is built from.
Trajectory evaluation
Grading the route an agent took, not just where it ended up.
Eval harness
The machinery that runs evals, separate from the agent it tests.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.