85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is an attention budget in LLMs?

An attention budget is the finite capacity a large language model has to attend across its context window, so every token added draws on it and leaves less focus for the tokens that matter.

Anthropic introduced the phrase in "Effective context engineering for AI agents" (2025-09-29) to explain a pattern agent builders kept running into: models get less precise as context grows, well before the window is full. The post treats context as a resource with diminishing returns, not a container to fill.

The idea changes the question a builder asks. "Does it fit?" is about the context window. "Is it worth the focus it costs?" is about the attention budget. A tool result that fits easily can still crowd out the one instruction it was meant to serve.

Why the budget is finite

  • Pairwise attention

    In a transformer every token attends to every other token, so n tokens create n squared pairwise relationships. As n grows, the model's ability to track each relationship is stretched thinner.

  • Training distribution

    Models see short sequences far more often than long ones in training, so they have less experience with, and fewer specialised parameters for, dependencies that span a whole long context.

  • Position interpolation

    Techniques that let a model handle inputs longer than it was trained on work by adapting them to the shorter trained range, at some cost to how precisely token positions are understood.

  • A gradient, not a cliff

    Anthropic describes the result as a performance gradient. Models stay capable at long lengths, but retrieval and long-range reasoning get less precise as context grows.

Where an agent spends it

In a long agent run the instructions are rarely the big spender. Tool definitions loaded up front, raw tool results and the growing message history usually dominate, and each one dilutes the system prompt and the current task.

That is why the remedies the labs publish are mostly about keeping things out of view: references in place of payloads, stale tool results cleared, side tasks pushed to subagents. Anthropic's post notes that a subagent may use tens of thousands of tokens on its own work but hands back a condensed summary, often 1,000 to 2,000 tokens.

What it is often confused with

  • Context window

    A hard limit on how many tokens fit in one call. The attention budget is about how well the tokens that fit are used.

  • Context rot

    The observed symptom: recall and accuracy fall as context fills. The attention budget is the explanation the labs give for it.

  • Token cost

    Tokens also cost money and latency, but a cheap, cached token still spends attention. Prompt caching lowers the bill, not the distraction.

How Motiontic's engineers apply it

Motiontic's engineers break each step's context down by component, such as system prompt, tool definitions, history and tool results, and cut the largest low-value line first. Changes are kept only when eval scores hold or improve, because a smaller context that drops a needed fact is not a saving.

Context window compared with attention budget

Context window compared with attention budget
Context windowAttention budget
What it isMaximum tokens per callCapacity to use those tokens well
Limit typeHard: the request fails or is cutSoft: quality degrades gradually
Grows withNew model versionsNot directly; better training helps at the margin
Main leverChoosing a modelChoosing what goes into context
Typical symptomAn error or truncationSkipped steps, forgotten constraints, weaker recall

Common questions

What does attention budget mean in AI?

It is a way of describing a model's limited capacity to attend across its context. Each extra token takes a little of that capacity, so long, noisy contexts leave less focus for the information the task needs.

Does a one million token context window remove the attention budget?

No. A larger window raises how much can fit, but Anthropic's guidance still treats context as a finite resource and recommends the smallest set of high-signal tokens that does the job.

How do agents save attention budget?

By loading data just in time instead of up front, clearing tool results once used, compacting long history, keeping tool sets small and distinct, and giving side tasks to subagents that return short summaries.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.