Because inference has two distinct physical regimes. The prompt is read in a single parallel pass that saturates the processor. The answer is written token by token, rereading all the model's weights each time, on a processor that sits idle 88% of the time.
The measurement: 6.05 × 10⁻⁵ Wh per input token against 2.13 × 10⁻³ Wh per output token, a factor of about 35. The price shows a factor of only 5.
The two phases of an inference
Every call to a model, whatever it is, goes through both.
The whole prompt is available at once, so every token crosses the network in parallel, in a single pass. The multiplications are matrices against matrices, which the hardware loves.
Prefill is compute-bound: the bottleneck is the number of arithmetic operations. It is the only regime where cutting numerical precision saves energy.
One token goes in, and the model's entire set of weights has to be reread to produce a single one. The multiplications become matrix against vector. Tens of gigabytes get read for a few billion operations.
Decoding is memory-bound: the bottleneck is memory bandwidth, not computation. On an H200, more than 88% of the compute units sit idle and the chip draws only 137 to 300 W out of a 700 W budget.
| Prefill | Decoding | |
|---|---|---|
| Input handled | the whole prompt | 1 token |
| Parallelism | total | none, sequential by construction |
| Physical bottleneck | computation | memory bandwidth |
| Measured GPU utilisation | 95.7 to 97.9% | 28.6 to 56.9% |
| Power drawn (700 W TDP) | up to 684 W | 137 to 300 W |
| Effect of a power cap | latency badly degraded | none, the cap never triggers |
| Cost per token | low | about 35 times prefill, at batch size one |
One striking benchmark, measured by Sarathi-Serve (Agrawal et al., OSDI 2024, arXiv:2403.02310) on Mistral-7B, an A100, with a 1,024 token prompt: the linear operations for a single decode token cost as much as 128 prefill tokens.
The figure, and where it comes from
The measurement comes from Delavande, Pierrard and Luccioni, Small Talk, Big Impact (January 2026). Protocol: dedicated H100 SXM 80 GB server with no competing workload, GPU measured through NVML and CPU through RAPL, LLaMA 3.1-8B in FP32, batch size one. The two phases are isolated by subtraction, comparing single-token generations against full generations.
| Term | Measured coefficient | What it represents |
|---|---|---|
| Per input token | 6.05 × 10⁻⁵ Wh | reading the prompt |
| Per output token | 2.13 × 10⁻³ Wh | producing one token |
| Cross term | 2.87 × 10⁻⁷ Wh/token² | rereading the context at every step |
The ratio between the first two is about 35. It is not written as such in the paper, it derives from the two published coefficients. Vellaisamy et al. (Carnegie Mellon), on H200 and a different protocol, land on an order of magnitude of 30 to 80 depending on the configurations compared.
The three coefficients form one single model, the one the paper fits:
E = A × T_input + C × T_output + D × T_context × T_output
Wh = Wh/tk × tk + Wh/tk × tk + Wh/tk² × tk × tk
The second line is the dimensional check: each product yields watt-hours, including the third, whose token-squared unit exists only to be multiplied twice.
The third term deserves a stop. It is not a separate item alongside decoding: it is the part of the decoding cost that depends on context length, where C is the part that does not. At every token produced, the model rereads the KV cache of everything before it, hence the product of the two lengths. T_context is the entire context, fresh tokens and tokens reread from cache alike, since rereading makes no distinction.
Its weight depends entirely on the profile, at batch size one:
| Profile | Input | Output | Cross term |
|---|---|---|---|
| Short chat, 1,000 tokens for 150 | 14% | 76% | 10% |
| Long chat, 10,000 for 500 | 20% | 34% | 46% |
| Agent session, 126,180 for 500 | 28% | 4% | 68% |
That is the subject of the chapter on long context.
The price confirms the gap, much more weakly
| Provider | Model | Input | Output | Ratio |
|---|---|---|---|---|
| OpenAI | gpt-6-astra | $10 | $50 | 5.0 |
| OpenAI | gpt-5 | $1.25 | $10 | 8.0 |
| Anthropic | Opus 5 | $5 | $25 | 5.0 |
| Anthropic | Sonnet 5 | $2 | $10 | 5.0 |
| Gemini 3.8 Flash | $0.75 | $3.75 | 5.0 |
The market converges on 5, in a range of 4 to 8.3. The measured physics gives 35.
Both point the same way, an order of magnitude apart. The price bakes in margin, the memory tied up for the whole duration of decoding, contractual latency and competitive positioning.
Using price as a proxy for energy therefore under-estimates the weight of output by a factor of 7. It is the most available and the most wrong proxy in the field. It serves to rank usages against each other, not to size a footprint.
Question
A client wants to cut the footprint of their internal assistant. The team proposes compressing prompts by 40%. What is the maximum gain you can promise on a classic chat usage?
Choisissez une réponse pour voir l'explication.
What this changes in a recommendation
Three direct consequences, in the order they come up on an engagement.
The lever depends on the profile, and most teams get the profile wrong. On chat, output carries three quarters of the load: constrain the length of answers, switch off reasoning when the task does not warrant it, pick a less talkative model. On an identical task, two models produce answers of different lengths, and that verbosity appears in no comparison grid.
On agentic work, output drops to a few per cent and the weight moves into the length of context reread at every step. The lever becomes the size of the context carried from one turn to the next, not the concision of the answers.
Batching is the provider's job. Since decoding is memory-limited, handling several requests in the same pass amortises the reading of the weights. The measured gain runs from 17 to 25 times between a batch of 1 and a batch of 32. It is the most effective lever in the whole chain, and a customer has almost no grip on it, beyond preferring a shared API to an underloaded dedicated instance.
Optimise joules per task, never per token. The trap is mechanical: fixed costs amortise over length, so energy per token falls as you generate more. Vellaisamy et al. measure on Qwen3-8B and MATH-500, on an H200, 7.46 J per token at 10 output tokens against 0.72 J at 512 tokens, while the energy of the whole request is multiplied by 5. An indicator in joules per token rewards verbosity.
FAQ
Does an output token cost more than an input token?
Yes, about 35 times more in energy. The reference measurement gives 6.05 × 10⁻⁵ Wh per input token against 2.13 × 10⁻³ Wh per output token on an H100 with an 8 billion parameter model at batch 1. On price, the three major providers bill output at 5 times input in 2026, a far smaller gap than the physical one.
Why is generating text more expensive than reading it?
Because the two phases do not share the same bottleneck. Reading the prompt handles every token in parallel and saturates the compute units. Generation produces one token at a time and has to reread the model's entire set of weights for each one, which saturates memory bandwidth and leaves more than 88% of the compute units idle.
Can API price be used as a proxy for the carbon footprint?
For ranking usages against each other, yes. For sizing a footprint, no. The price ratio between output and input is about 5, where the energy ratio measured at batch size one is about 35. Using price therefore under-estimates the weight of output by a factor of 7, and price also bakes in margin, contractual latency and commercial positioning.
Does shortening prompts reduce the footprint?
Moderately. On a classic chat usage, input accounts for about a quarter of the total energy, so compressing prompts by 40% gains at best 10%. On an agent session, the genuinely compressible share drops to around 29% of the footprint, which caps a perfect compressor's gain at under a third.
Does capping GPU power reduce inference consumption?
Barely, and this is counter-intuitive. During generation an H200 draws only 137 to 300 W out of a 700 W budget, so no cap ever triggers. Published measurements conclude that an operator relying on power capping alone gets no energy saving at all on that phase.