85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is a regression eval?

A regression eval is a suite of tasks an AI system already handles correctly, rerun on every prompt, tool, model or data change to confirm the change has not broken behaviour that used to work.

Fixing one failure in an AI system often causes another. A prompt edit that helps refunds can hurt cancellations; a model upgrade that improves reasoning can change output formats. Regression evals exist to catch those side effects.

The suite holds tasks the system has already solved. It should pass at or near 100%, so any drop is a signal that something broke rather than noise.

Capability evals and regression evals

Anthropic's agent eval guide separates the two. Capability evals target what the system cannot yet do reliably and start with a low pass rate that should climb. Regression evals guard what already works and should sit near 100%. As capability tasks are solved, they graduate into the regression suite.

The other labs describe the same idea in their own terms: OpenAI's guidance is to run evals on every change, and Google's ADK runs conformance tests against recorded baselines to catch regressions.

Running regression evals on non-deterministic systems

  • Several trials per task

    Run each task more than once and set a threshold, since one failed run may be noise and one passing run may be luck.

  • Fixed infrastructure

    Anthropic found that container resource limits alone moved agentic coding scores by up to 6 points, so record and pin the eval environment.

  • Thresholds, not equality

    Compare against the last accepted baseline with an agreed tolerance rather than expecting identical output.

  • Every kind of change

    Prompts, tools, retrieval data and model versions all trigger a run, not only code commits.

How Motiontic's engineers apply it

Motiontic's engineers wire the regression suite into CI so it runs on every pull request that touches a prompt, a tool definition or a model setting, and before any model upgrade is switched on. Each production failure that is fixed becomes a new regression task, so the suite grows with the system and stays the client's to own after launch.

Capability evals compared with regression evals

Capability evals compared with regression evals
Capability evalRegression eval
PurposeMeasure progress on hard tasksProtect behaviour that already works
Expected pass rateLow at first, climbingAt or near 100%
A drop meansUsually noise or a harder task mixSomething broke
Task sourceNew and unsolved casesSolved cases graduated from the capability suite
RunsDuring development of a featureOn every change, as a release gate

Common questions

What is a regression eval?

A regression eval is a set of tasks an AI system already passes, rerun after each change to make sure nothing that worked before has broken.

What pass rate should a regression eval have?

At or near 100%. Anthropic's guidance describes regression suites that way, which is what makes any drop meaningful.

When should regression evals run?

On every change to prompts, tools, retrieval data or model version, and before any model upgrade goes live, not only when application code changes.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.