What is a capability eval?
A capability eval is a set of test tasks built to measure what an AI model or agent can do well, deliberately starting at a low pass rate so the team has a hill to climb.
Most eval suites answer one question: did the last change break anything? A capability eval answers a different one: what can the agent not yet do, and is it getting closer? Anthropic's January 2026 guide to agent evals separates the two explicitly. Capability evals, which it also calls quality evals, should start at a low pass rate; regression evals should sit near 100%.
A suite where everything passes cannot show progress, so a capability eval is designed to be failed at first. It is where a new feature is specified before the agent can perform it, which Anthropic describes as eval-driven development: write the tasks, then iterate until the agent passes them.
How a capability eval is built
Tasks the agent should learn
Drawn from real failures, feature requests and edge cases. Anthropic's guidance is that 20 to 50 simple tasks from real failures are enough to start, rather than waiting for hundreds.
Clear success criteria
Each task states its inputs and what success looks like, precisely enough that two engineers would grade the same attempt the same way.
Several trials per task
Agent output varies from run to run, so each task is attempted more than once and reported as pass@k, pass^k or both.
Graders matched to the task
Code checks where the outcome can be verified, model-based graders for open-ended output, and human review to calibrate the model graders.
Graduation, and what a 0% score means
After launch, capability tasks that reach a high pass rate graduate into the regression suite, which runs continuously. The question changes from whether the agent can do the task at all to whether it still does it reliably. Harder tasks replace the graduates, so the capability suite keeps a hill to climb instead of saturating.
A task that scores 0% across many trials is a warning as much as a result. Anthropic notes that 0% pass@100 most often signals a broken task, not an incapable agent. Its example: Claude Opus 4.5 first scored 42% on CORE-Bench, and fixing grading bugs, clarifying ambiguous specs and using a less constrained scaffold took it to 95%.
The other meaning: dangerous-capability evals
In frontier AI safety, capability evals also test for abilities a lab does not want a model to have. Google DeepMind's paper Evaluating Frontier Models for Dangerous Capabilities (March 2024) covers persuasion and deception, cyber-security, self-proliferation and self-reasoning. The mechanics are the same, tasks, trials and graders, but the goal is inverted: a low score is the result the lab hopes for.
Teams building product agents almost always mean the first sense: a suite that tracks progress on the work the agent is being built to do.
How Motiontic's engineers apply it
Motiontic's engineers write capability tasks for a feature before building it, from the client's real cases, so progress is measured from the first week. Transcripts are read before any score is trusted, and tasks that pass consistently move into the regression suite that runs in CI.
Capability eval compared with regression eval
| Capability eval | Regression eval | |
|---|---|---|
| Question | What can the agent not do well yet? | Does it still do what it used to? |
| Expected pass rate | Low at first, rising over time | Near 100% |
| When it runs | While a feature or model change is developed | On every change, in CI |
| A failure means | Work still to do, or a broken task | A regression that blocks the release |
| Lifecycle | Tasks graduate out once they pass reliably | Grows as tasks graduate in |
Common questions
What is the difference between a capability eval and a regression eval?
A capability eval measures what an agent cannot yet do reliably and starts with a low pass rate. A regression eval checks that solved tasks stay solved and should pass close to 100% of the time.
How many tasks does a capability eval need?
Anthropic's guidance is to start with 20 to 50 simple tasks drawn from real failures rather than waiting for hundreds. Tasks are added as the agent improves and as new failures appear.
When does a capability task move to the regression suite?
When it passes consistently across trials. At that point it protects a solved behaviour, and a harder task takes its place in the capability suite.
Is a capability eval the same as a benchmark?
A public benchmark is a shared capability eval used to compare models. A team's own capability eval uses its own tasks and success criteria, so it measures the agent on the work it will actually do.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Regression eval
The suite that catches what a change broke before users do.
Grader (AI evals)
The part of an eval that decides pass or fail.
Eval saturation
When every score is near 100% and the eval stops telling anyone anything.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.