Your sentence is first cut into pieces. Each piece is turned into numbers, which the model adjusts according to the words around it, then sends through dozens of layers of computation. The model makes that trip in two phases: it first reads your message in one go, then writes its answer word after word.
There are six steps, and each one maps to a place where energy goes. They explain why French costs more than English, why writing costs more than reading, what a long conversation costs, and how a giant model can burn less than a small one.
Step 1: your text is cut into tokens
A model works on neither letters nor whole words. It works on pieces of words, drawn from a fixed dictionary built before its training.
Frequent pieces have their own entry. "Hello" fits in a single piece. A rare word like "antidisestablishmentarianism" gets cut into six.
The piece of text the model handles. On OpenAI's current tokenizers, one token is worth about 4.9 characters in English and 4.3 in French.
What you pay, what the machine computes and the electricity it uses are all counted in tokens.
French has two handicaps. It fits fewer characters into each token, and it needs about 20% more characters than English to say the same thing. The result: the same text needs about 37% more tokens in French, on OpenAI's current tokenizers. In Burmese it is more than three times as many, and up to nearly fourteen times for the least represented languages. You pay the difference, in euros as in carbon.
The detail and the measurement: does French cost more than English.
Step 2: each token becomes a list of numbers
You cannot compute anything with a dictionary number. Token 4,827 is not "bigger" than token 312. So the model replaces each number with a long list of numbers, 8,192 on Llama 3 70B, a Meta model with 70 billion parameters, which places the token in a space where words with close meanings are neighbours.
This starting list is the same in every sentence. "Bank" comes in with the same numbers in "she paid the cheque into the bank" and in "we sat fishing on the river bank". The model does not yet know which bank is meant.
It does not know the order of the words either. So the position of each token is added, these days most often with a method called RoPE. It lets you stretch the context window of an existing model without retraining it from scratch: Llama 3 went from 8,000 to 128,000 tokens with about 5% of its training data. Stretching the window is cheap at training time. Using it is expensive, and step 6 explains why.
Step 3: attention adjusts each word to its context
Attention is the central mechanism of these models. It moves the numbers for "bank" towards the lender or towards the riverside, depending on the words before it.
Each token produces three lists of numbers: a query (what it is looking for), a key (what it offers the others) and a value (what it passes on). The model compares each token's query to the keys of the tokens before it, and each token gets back a mix of the values, weighted by those comparisons.
Two consequences matter for energy.
First, the number of comparisons grows with the square of the text length. A hundred tokens make about five thousand comparisons, two hundred make twenty thousand. This is the so-called quadratic cost of attention. Yet it only becomes the largest compute cost past about 50,000 tokens of context on Llama 3 70B, because the rest of the model weighs much more. An ordinary chat stays far below that threshold, but agent sessions often go past it.
Second, a token only looks at the ones before it, since the model learns to predict what comes next. Its representation therefore stops changing once computed, and the model can keep it in memory instead of recomputing it. Step 6 comes back to this.
The detail: why long context costs so much.
Step 4: most of the computation, in the dense layers
Attention moves information between words, but it holds only a small share of the parameters. On Llama 3 70B, about 80% of them sit in the layers that follow each attention block, and which transform each token separately.
The work can now be put in figures. Producing one token takes about two operations per parameter crossed. A 70 billion parameter model therefore does roughly 140 billion operations for each token.
A FLOP is one computing operation: an amount of work, like a distance. A FLOPS is a number of operations per second: the power of a machine, like a speed. You often see one written for the other.
This unit has regulatory weight. The European AI Act asks providers of general-purpose models for their model's energy consumption and accepts, when it is unknown, an estimate based on the computing resources used.
One architecture changed the link between a model's size and its consumption.
Instead of one big dense layer, the model holds dozens, sometimes hundreds, of smaller specialised ones: the experts. For each token, a router picks a few of them, and the others do no computing.
Kimi K2, for example, has 1,000 billion parameters and activates only 32 billion per token. Computation drops by as much, but every expert has to stay loaded in memory, since there is no telling in advance which ones will be needed.
So the headline parameter count is no longer enough to compare two models. The useful figure is the number of active parameters per token, sometimes thirty times smaller than the press release figure.
The detail: what Mixture of Experts changes.
The transformer block
An attention layer followed by a dense layer make up a block, called a transformer since the 2017 paper that proposed it. Llama 3 70B stacks eighty of them. Every model in this course is a variant of this pattern: they differ in the number of blocks, the length of the lists of numbers, and whether the dense layers are split into experts.
At the output of the last block, the model gives a score to every entry in its dictionary. Those scores become probabilities, and the next token is drawn at random according to them. Temperature controls that draw. Low, it almost always picks the most likely token, and answers vary little. High, it lets less likely tokens through, and answers vary more.
Step 5: reading and writing, two opposite regimes
The eighty blocks are crossed in two ways, depending on whether the model is reading or writing.
The model reads your whole message in one block. All the tokens cross the layers at the same time, and the compute units run at full capacity. It is the cheapest phase per token.
The model writes its answer token after token, since each word depends on the one before. For each token, it has to reread all its parameters from memory. The compute units spend most of their time waiting: the tensor cores, which do the matrix maths, sit more than 88% idle in the measurements of Ma et al. on an H200.
Hence a gap that runs through the whole course. When the machine handles only one request at a time, a written token uses about 35 times a read token, in the reference measurement made on an 8 billion parameter model. The billed price most often shows a ratio of 5. On a short conversation, the length of the answers is what weighs. On an agent session, the context reread for every written token takes over, and its length is what to watch.
The detail and the measurements: why an output token costs more than an input token.
Step 6: what the model keeps in memory
To write its fiftieth word, the model needs everything that came before. So it keeps in memory what it computed in step 3 for each token, instead of recomputing everything at every word.
The keys and values of every token already seen, kept in the chip's fast memory for the whole answer. This store grows with the length of the conversation. On Llama 3 70B, one million tokens would take up more than 300 GB, more than double the 140 GB of the model itself.
This sharing weighs a lot. A chip handling thirty-two conversations at once rereads the model's parameters once for thirty-two answers. On models of about 4 billion parameters with 4,000 tokens of context, Ma et al. measure a gap in energy per token of a factor of 17 to 25 between a single conversation and a full batch. The gain shrinks as context grows, since each conversation takes up more room.
This lever belongs to the provider, almost never to the customer. That is why self-hosting a model used at low volume burns more than calling an API shared by many users.
Question
You write a 2,000 word prompt and the model answers in 100 words, on a machine handling only your request. Where did most of the energy go?
Choisissez une réponse pour voir l'explication.
What to take away before moving on
Cutting text into tokens fixes the unit of account, and it penalises poorly represented languages. Each token becomes a list of numbers, which attention adjusts to the context. Most of the parameters sit in the dense layers, and expert models activate only part of them.
The model reads the whole prompt at once and writes one token at a time, which makes writing much more expensive than reading. The room each conversation takes up in memory decides how many conversations a chip can run side by side, and so the energy of each one.
The rest of the course puts sourced figures on these six steps, then derives an emission factor per token from them.
The logical next step: why count tokens rather than requests.
FAQ
How does an LLM work, step by step?
Your text is first cut into tokens, pieces of words drawn from a fixed dictionary. Each token is replaced by a list of several thousand numbers that places it in a space of meaning, and its position is added. The attention mechanism then adjusts those numbers according to the surrounding words. The dense layers, which hold most of the parameters, transform each token. The model crosses these layers once to read the whole prompt, then once per token to write the answer.
What is the attention mechanism in a language model?
The mechanism that lets each word adjust its meaning according to the others. Each token compares what it is looking for with what the tokens before it offer, and gets back a mix of their information, weighted by those comparisons. "Bank" comes in with the same numbers in "the bank approved the loan" and "the bank flooded after the storm"; attention tips it towards the lender or towards the riverside.
Why is attention said to be quadratic?
Because each token compares itself to all the ones before it. A hundred tokens make about five thousand comparisons, two hundred make twenty thousand: doubling the length of the text quadruples the number of comparisons. On Llama 3 70B, though, this only becomes the largest cost past about 50,000 tokens of context, because the dense layers, which grow only in proportion to the length, weigh much more below that threshold.
What is an embedding and why does the model need one?
You cannot compute anything with a dictionary number. The embedding replaces that number with a list of several thousand numbers, learned during training, which puts words with close meanings at close positions. This starting list is the same in every sentence: attention is what then adjusts it to the context.
Why does writing cost more than reading for an AI?
Because the two phases do not run the same way. Reading handles all the prompt's tokens at the same time and runs the compute units at full capacity. Writing produces one token at a time, and the model has to reread all its parameters from memory for each one: the chip spends most of its time waiting. When the machine handles only one request, a written token uses about 35 times a read token, while the billed price most often shows a ratio of 5.