85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What are pass@k and pass^k?

pass@k is the probability that at least one of k attempts at a task succeeds, while pass^k is the probability that all k attempts succeed; the first measures capability and the second measures reliability.

Agents do not give the same result every run. A task that succeeds three times out of four can look solved or broken depending on how the runs are counted. pass@k and pass^k are two ways of counting, and they answer different questions.

pass@k is generous: it counts a task as solved if any of k tries works, which suits a setting where a person or a test can pick the good attempt. pass^k is strict: every one of k tries must work, which is closer to what a customer sees when the agent runs on every request.

How the two are calculated

If each trial succeeds independently with probability p, then pass@k equals 1 minus (1 minus p) to the power k, and pass^k equals p to the power k. The two move in opposite directions: as k grows, pass@k rises towards 100% and pass^k falls towards zero.

Anthropic's agent eval guide gives the example of a 75% per-trial success rate: pass^3 is about 42%. On the same numbers, pass@3 is about 98%. One agent, one task, and the headline figure ranges from 42% to 98% depending on the metric.

Where the metrics come from

pass@k comes from code generation benchmarks, where a model writes several candidate programs and a task counts as solved if any of them passes the unit tests. pass^k was proposed in the τ-bench paper (June 2024) specifically to measure how reliably an agent behaves over repeated trials of the same task.

Anthropic's January 2026 guide recommends tracking both, and using pass^k where customer-facing consistency matters.

Which one to report

  • Exploring capability

    pass@k shows whether a task is within reach at all, which helps when choosing a model or deciding whether an approach is worth pursuing.

  • Shipping to customers

    pass^k shows how often every user would get a correct result, which is the number a support or finance workflow has to live with.

  • Always alongside

    Report the per-trial success rate and the value of k next to either metric; without them the figure cannot be compared.

How Motiontic's engineers apply it

Motiontic's engineers run each eval task several times and report pass^k for any workflow a customer or employee relies on without checking. A task with a high pass@k and a low pass^k is a sign the agent can do the work but not consistently, which usually points to ambiguous instructions, a fragile tool or missing context rather than a weak model.

pass@k compared with pass^k

pass@k compared with pass^k
pass@kpass^k
QuestionDoes at least one of k attempts succeed?Do all k attempts succeed?
MeasuresCapabilityReliability and consistency
As k growsRises towards 100%Falls towards zero
At 75% per trial, k = 3About 98%About 42%
FitsResearch, model selection, human-picked outputsCustomer-facing and unattended agents

Common questions

What does pass@k mean?

pass@k is the share of tasks where at least one of k attempts succeeds. It shows what a model or agent is capable of when several tries are allowed.

What does pass^k mean?

pass^k is the share of tasks where all k attempts succeed. It shows how consistently an agent gets a task right, which matters when every run reaches a user.

Which metric should a production agent be judged on?

For agents that act without a person checking each result, pass^k is the more honest figure. Anthropic's eval guidance recommends it where customer-facing consistency matters.

Why does pass^k fall so quickly?

Because every attempt must succeed, the per-trial rate is multiplied by itself k times. At 90% per trial, pass^5 is about 59%, so small gains in per-trial reliability make a large difference.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.