What is infrastructure noise in AI evals?
Infrastructure noise is variation in an AI agent's eval score caused by the environment the eval runs in, such as CPU, memory, time limits and how they are enforced, rather than by the model or agent.
A static benchmark scores a model's output directly, so the machine it runs on hardly matters. An agentic eval is different: the agent writes code, installs packages, runs tests and waits on processes, all inside a container. Anthropic's February 2026 post describes agentic coding evals as end-to-end system tests by construction, which means the system is part of the score.
Anthropic measured how much. On Terminal-Bench 2.0, the gap between the most and least resourced setups was 6 percentage points (p < 0.01), which is larger than the gap between many models on public leaderboards.
What Anthropic found
Resource limits drive error rates
Infrastructure errors fell from 5.8% under strict per-task resource specs (1x) to 2.1% at 3x headroom and 0.5% with no cap.
Moderate headroom mostly fixes errors
From 1x to 3x, success scores moved within noise (p = 0.40). The extra resources mainly stopped containers being killed.
Generous headroom changes the task
From 3x to uncapped, success rose almost 4 more points, about 6 in total over 1x, because agents could solve problems that need more memory or CPU.
Smaller, same direction, on SWE-bench
Varying RAM up to 5x across 227 problems with 10 samples each raised scores by 1.54 points.
Time of day
Anthropic observed that pass rates may vary with time of day, but only anecdotally; it did not quantify the effect.
How to control it
Two numbers per resource
Specify a guaranteed allocation and a separate hard kill threshold per task, rather than one pinned value with no margin for brief spikes.
Calibrate the band
On Terminal-Bench 2.0, a 3x ceiling cut infrastructure errors by about two-thirds while keeping the score lift within noise.
Report the configuration
Publish the multiplier used and how limits were enforced, alongside the score.
Spread the runs
Run at different times and on different days, and average the results.
Reading leaderboards and internal results
Anthropic's guidance is that leaderboard differences below 3 percentage points deserve skepticism unless the eval configuration is documented and matched. Naive binomial confidence intervals already span 1 to 2 points before infrastructure is considered. A lead of a few points may be a real capability gap, or a larger virtual machine.
The same applies inside a team. A model upgrade that gains two points on an internal suite, run on different machines from the baseline, has not been shown to help.
How Motiontic's engineers apply it
Motiontic's engineers record the eval environment, including image, CPU, memory, limits and timeouts, with every result, and rerun the baseline on the same configuration before comparing a model or prompt change. Infrastructure errors are counted separately from task failures, so a run killed by a memory limit is not reported as the agent getting the task wrong.
Terminal-Bench 2.0 by resource setting (Anthropic, February 2026)
| Setting | Infrastructure error rate | Score effect compared with 1x |
|---|---|---|
| 1x (strict per-task spec) | 5.8% | Baseline |
| 3x headroom | 2.1% | Within noise (p = 0.40) |
| Uncapped | 0.5% | About 6 points higher (p < 0.01) |
Common questions
What causes infrastructure noise in agent evals?
Mainly CPU and memory limits and how strictly they are enforced, along with time limits. Anthropic also saw signs that time of day affects pass rates, though it did not quantify that.
How large can infrastructure noise be?
Anthropic measured a 6 percentage point spread on Terminal-Bench 2.0 between its most and least resourced setups, and 1.54 points on SWE-bench when RAM was varied up to 5x.
How should small leaderboard differences be read?
With caution. Anthropic suggests differences below 3 percentage points deserve skepticism unless both results used a documented, matched eval configuration.
Is infrastructure noise the same as model non-determinism?
No. Non-determinism is variation in the agent's output in the same environment, handled with several trials and pass^k. Infrastructure noise comes from the environment itself and is controlled by fixing and recording the configuration.
Related
Task, trial and transcript (eval vocabulary)
The basic units every agent eval is built from.
Eval harness
The machinery that runs evals, separate from the agent it tests.
Regression eval
The suite that catches what a change broke before users do.
Eval awareness
When the model notices the test, the score stops meaning what it should.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.