The usual answer is "because attention is quadratic". True. And incomplete: on a model in the 70 billion parameter class, the quadratic term only becomes dominant past 100,000 tokens, a long way from the regime where almost every request runs.
The real cost sits elsewhere, in the decoding regime. Long context fills accelerator memory, which collapses the number of requests that can be handled together, and that collapse is what pushes energy per token up. Providers bill for it explicitly, as it happens.
What attention does, and what it costs
Take "the bank approved the loan in under a week" and "the bank flooded after the storm". The word enters the model with the same vector in both cases. Attention drags it towards the lender or towards the riverside, mixing in part of the information carried by "approved" or by "flooded".
Each token puts out a query, what it is looking for, and a key, what it offers. It also passes on a value. Every query is multiplied with every key, which gives a score matrix of size n by n, then that matrix is used to take a weighted average of the values.
This n by n matrix is where the quadratic cost comes from: doubling the length of the context quadruples the number of scores to compute.
Except attention is not the only item. The block that follows it, holding two thirds of the parameters, grows linearly with length. As long as the linear term dominates, the observed behaviour is linear.
Where the crossover falls depends on conventions that are rarely spelled out. With causal masking and the published dimensions of Llama 3:
| Model | Attention = 10% of the computation | 25% | 50%, the crossover |
|---|---|---|---|
| Llama 3 8B | 2,958 tokens | 8,875 | 53,248 |
| Llama 3 70B | 5,803 tokens | 17,408 | 104,448 |
| Llama 3.1 405B | 10,809 tokens | 32,427 | 194,560 |
At 8,000 tokens of context on a 70B, quadratic attention is worth about 8% of the computation. At 32,000, about 31%. The "it is quadratic so it explodes" line is wrong in the everyday regime. It becomes true further out, which is where the commercial race is heading.
The real wall: the KV cache
So that the keys and values of every previous token do not have to be recomputed at each new token produced, the model keeps them in memory. This store grows linearly with the length of the context and has to stay in high performance memory for the whole generation.
It decides how many requests fit on an accelerator at once, so how many can be handled in the same pass, so the energy per token.
| Model | Per token | At 128k tokens | At 1M tokens |
|---|---|---|---|
| Llama 3 8B | 131.1 kB | 17.2 GB | 131.1 GB |
| Llama 3 70B | 327.7 kB | 42.9 GB | 327.7 GB |
| Llama 3.1 405B | 516.1 kB | 67.6 GB | 541.2 GB |
| DeepSeek-V3 (compressed attention) | 70.3 kB | 9.2 GB | 73.7 GB |
An H100 carries 80 GB of memory, an H200 141 GB, a B200 180 GB. A single Llama 3 70B request at one million tokens asks for 328 GB of cache, more than double the 140 GB of model weights, and fits on no single accelerator.
The effect can be measured directly. On a real agent trace with a median input of around 67,000 tokens, a classic cache layout saturates memory at 64 concurrent requests and tops out at 1,863 tokens per second per GPU. A layout that splits the cache across the accelerators reaches 6,091 tokens per second at 512 concurrent requests, while using only 82% of the memory. A factor of 3.3, without touching the model.
The commercial evidence
A provider does not invent a pricing tier if its marginal cost is linear.
| Provider | Model | Short input | Long input | Output |
|---|---|---|---|---|
| Gemini 3.1 Pro | $2.00 | $4.00 (>200k) | ×1.5 | |
| Gemini 2.5 Pro | $1.25 | $2.50 (>200k) | ×1.5 | |
| OpenAI | gpt-6-astra | $10 | $20 | ×1.5 |
| OpenAI | gpt-5.6-sol | $4 | $8 | ×1.5 |
Two independent providers, the same multiplier: ×2 on input, ×1.5 on output. Google also bills storage of its explicit cache at up to $4.50 per million tokens per hour on its top-end models.
The counter-example exists and has to be given: Anthropic scrapped its tier, and bills a 900,000 token request at the same unit rate as a 9,000 token one. That does not prove the cost has become linear, only that the company absorbs it into its average price. Mistaking a pricing decision for a physical property is the error to watch for.
Question
On a 70 billion parameter model served at 16,000 tokens of context, which item explains most of the extra cost compared with 4,000 tokens?
Choisissez une réponse pour voir l'explication.
RAG is a demand reduction technique
If the cost comes from the length of the context, then putting only what is useful into it is a lever on efficiency, not only on relevance.
A Google DeepMind study measures the gain across nine datasets: routing each question to a targeted search rather than to the full context cuts token volume by 61.6% on Gemini 1.5 Pro and 61.2% on GPT-3.5-Turbo, with performance nearly held.
Two caveats to state, or you will overpromise. These are tokens, not joules: nobody has published an energy comparison between a retrieval pipeline and a full context on the same task. And the pipeline is not free, since the corpus has to be indexed, the vectors stored and the index queried on every call, costs that amortise over the number of requests the way training does.
What to take away
The million token window is a feat of engineering paid for in operations, every day, on every request. Stretching a window costs little at training time. Serving it costs a lot.
For an audit, ask what median input length the calls actually carry, not what context size the model accepts. The answer is in the logs.
FAQ
Is an LLM's attention really quadratic?
Yes in theory, but it only dominates the cost past a high threshold. On Llama 3 70B, quadratic attention reaches 50% of the computation at around 104,000 tokens of context, and is worth only about 8% at 8,000 tokens. In the regime where most requests run, the observed behaviour is essentially linear.
What actually makes long context expensive?
The KV cache, meaning the keys and values of every previous token held in memory during generation. It grows linearly with length, occupies accelerator memory, and cuts the number of requests that can be handled at once by the same amount. And energy per token depends directly on that batching.
How much space does the KV cache of a one million token request take?
About 328 GB for Llama 3 70B at 16-bit precision, more than double the 140 GB of the model weights, and well beyond the 80 GB of an H100 or the 180 GB of a B200. A compressed attention architecture like DeepSeek-V3's brings that figure down to about 74 GB.
Why do some providers charge more for long context?
Because their marginal cost is not linear. Google and OpenAI both apply a multiplier of 2 on input tokens beyond their context tier and 1.5 on output. Anthropic went the other way and bills the whole window at the standard rate, which is a commercial policy rather than a physical difference.
Does RAG reduce the carbon footprint compared with a long context?
It reduces the volume of tokens sent to the model, measured at about 61% in a Google DeepMind study with performance nearly held. No published study compares the energy of the two approaches on the same task, and the calculation has to include indexing the corpus and running the search queries, which amortise over usage volume.