85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is LLM as a judge?

LLM as a judge is an evaluation method in which a language model grades another model's or agent's output against a written rubric, with its scores calibrated against human graders before they are trusted.

Some qualities can be checked with code: did the test pass, is the JSON valid, does the number match. Many cannot: is the answer faithful to the source, is the tone right, did the agent follow policy. Human review handles those well but does not scale to every change and every release.

An LLM judge fills that gap. A model is given the output, a rubric and sometimes a reference answer, and returns a grade. It is fast and flexible, and it is only as reliable as its agreement with people.

How to make a judge reliable

  • Narrow questions

    Pass or fail, or a pairwise choice between two outputs, is more stable than an open 1 to 10 scale.

  • Reason before scoring

    Ask the judge to explain its reasoning, then give the grade.

  • Control known biases

    Swap the order of options to offset position bias, and watch for a preference for longer answers.

  • Calibrate against people

    Score a labelled sample with both the judge and human graders, and scale up only once they agree.

  • Keep the judge separate

    A model that grades its own output tends to rate it too highly, so the judge runs as a separate step.

What the labs say

Anthropic's "Demystifying evals for AI agents" (January 2026) describes model-based graders as flexible but non-deterministic, and says they need calibration against human graders. Its Agent SDK guidance orders verification as rules first, such as tests and linters, then visual checks, then an LLM judge.

OpenAI's evaluation guidance recommends pass or fail or pairwise judgements, controlling for position and verbosity bias, and having the judge reason before it scores. Google's ADK uses rubric-based, model-graded criteria for subjective quality, groundedness and safety.

How Motiontic's engineers apply it

Motiontic's engineers use code graders wherever a check can be written as code, and add an LLM judge only for qualities code cannot measure. Each judge ships with a small human-labelled set, and its agreement rate is tracked like any other metric.

When a judge and a person disagree, the case is read in full and the rubric is tightened before the judge is trusted again.

Three kinds of grader compared

Three kinds of grader compared
Code-based graderLLM judgeHuman grader
Speed and costFast and cheapFast, moderate costSlow and expensive
ConsistencyDeterministicVaries between runsVaries between people
FitsExact checks: tests, formats, valuesFaithfulness, tone, policy, relevanceCalibration and the hardest cases
Main riskBrittle to valid variationsBias and drift from human judgementDoes not scale to every change

Common questions

What is LLM as a judge?

LLM as a judge is using a language model to grade the output of another model or agent against a rubric, as part of an eval. It covers qualities that code cannot check, and is calibrated against human graders.

Can an LLM judge replace human review?

Not entirely. It extends human review to more cases and more often, but its grades are trusted only after they agree with human labels on a sample, and disagreements still go to a person.

What are the known biases of LLM judges?

Two common ones are favouring the option shown first and favouring longer answers. Swapping the order of options, short rubrics and asking the judge to reason before scoring reduce them.

Should the judge be the same model as the one being graded?

It is safer to keep them separate. Lab guidance notes that a model grading its own work tends to over-rate it, so the judge runs as an independent step with its own prompt and rubric.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.