What is an attention budget in LLMs?
An attention budget is the finite capacity a large language model has to attend across its context window, so every token added draws on it and leaves less focus for the tokens that matter.
Anthropic introduced the phrase in "Effective context engineering for AI agents" (2025-09-29) to explain a pattern agent builders kept running into: models get less precise as context grows, well before the window is full. The post treats context as a resource with diminishing returns, not a container to fill.
The idea changes the question a builder asks. "Does it fit?" is about the context window. "Is it worth the focus it costs?" is about the attention budget. A tool result that fits easily can still crowd out the one instruction it was meant to serve.
Why the budget is finite
Pairwise attention
In a transformer every token attends to every other token, so n tokens create n squared pairwise relationships. As n grows, the model's ability to track each relationship is stretched thinner.
Training distribution
Models see short sequences far more often than long ones in training, so they have less experience with, and fewer specialised parameters for, dependencies that span a whole long context.
Position interpolation
Techniques that let a model handle inputs longer than it was trained on work by adapting them to the shorter trained range, at some cost to how precisely token positions are understood.
A gradient, not a cliff
Anthropic describes the result as a performance gradient. Models stay capable at long lengths, but retrieval and long-range reasoning get less precise as context grows.
Where an agent spends it
In a long agent run the instructions are rarely the big spender. Tool definitions loaded up front, raw tool results and the growing message history usually dominate, and each one dilutes the system prompt and the current task.
That is why the remedies the labs publish are mostly about keeping things out of view: references in place of payloads, stale tool results cleared, side tasks pushed to subagents. Anthropic's post notes that a subagent may use tens of thousands of tokens on its own work but hands back a condensed summary, often 1,000 to 2,000 tokens.
What it is often confused with
Context window
A hard limit on how many tokens fit in one call. The attention budget is about how well the tokens that fit are used.
Context rot
The observed symptom: recall and accuracy fall as context fills. The attention budget is the explanation the labs give for it.
Token cost
Tokens also cost money and latency, but a cheap, cached token still spends attention. Prompt caching lowers the bill, not the distraction.
How Motiontic's engineers apply it
Motiontic's engineers break each step's context down by component, such as system prompt, tool definitions, history and tool results, and cut the largest low-value line first. Changes are kept only when eval scores hold or improve, because a smaller context that drops a needed fact is not a saving.
Context window compared with attention budget
| Context window | Attention budget | |
|---|---|---|
| What it is | Maximum tokens per call | Capacity to use those tokens well |
| Limit type | Hard: the request fails or is cut | Soft: quality degrades gradually |
| Grows with | New model versions | Not directly; better training helps at the margin |
| Main lever | Choosing a model | Choosing what goes into context |
| Typical symptom | An error or truncation | Skipped steps, forgotten constraints, weaker recall |
Common questions
What does attention budget mean in AI?
It is a way of describing a model's limited capacity to attend across its context. Each extra token takes a little of that capacity, so long, noisy contexts leave less focus for the information the task needs.
Does a one million token context window remove the attention budget?
No. A larger window raises how much can fit, but Anthropic's guidance still treats context as a finite resource and recommends the smallest set of high-signal tokens that does the job.
How do agents save attention budget?
By loading data just in time instead of up front, clearing tool results once used, compacting long history, keeping tool sets small and distinct, and giving side tasks to subagents that return short summaries.
Related
Context rot
Why agents get worse the longer they run, and what keeps context clean.
Context engineering
Curating everything in the model's view, not just the wording of the prompt.
Context window
The model's working memory for one request, measured in tokens, and finite.
Just-in-time context
Keep the address in context, fetch the contents when the step needs them.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.