Your sentence gets cut into fragments, each fragment becomes a list of numbers, those numbers shift according to the words around them, then they cross layers of computation. The whole thing runs twice, in two very different regimes: first to read your message in one go, then to write the answer word after word.
Six mechanisms, and each one turns into a place where energy goes. The middle two form the pattern known as a transformer, stacked dozens of times over. They explain why French costs more than English, why writing costs 35 times more than reading, why a long conversation gets expensive, and why a giant model can burn less than a small one.
This chapter walks through them with the bare minimum of figures. Each one gets its own chapter afterwards.
Step 1: your text becomes fragments
A model does not read letters. It does not even read words. It reads line numbers in a fixed dictionary, built once and for all before training.
That dictionary holds fragments rather than whole words. "Hello" probably fits in a single fragment because it turns up constantly. A rare word like "antidisestablishmentarianism" gets broken into five or six pieces.
The unit the model actually handles. One token is worth roughly 4.9 characters in English and 4.3 in French. French fits less into a token, so it needs more tokens for the same text.
Everything is counted in tokens: what you pay, what the machine computes, and the electricity it burns. It is the only unit common to all three.
So the same text needs about 37% more tokens in French than in English, and up to three times more in a poorly represented language. You pay the difference, in euros as in carbon.
The detail and the measurement: does French cost more than English.
Step 2: each fragment becomes a list of numbers
A line number in a dictionary means nothing to a calculation. Token 4,827 is not "bigger" than token 312. So each number has to be swapped for something that carries meaning.
That something is a long list of numbers, several thousand of them, which places the fragment somewhere in a space. Fragments with close meanings end up close together, because training put them there.
The list of numbers that stands for a token inside the model. On a model in the 70 billion parameter class, it holds 8,192 values.
The model keeps these lists in one big table, one row per dictionary entry. With 128,000 entries of 8,192 numbers, that table alone weighs a little over a billion parameters. It is learned like everything else.
One point that matters for what follows: this vector is frozen. The word "bank" has one single list of numbers at the model's input, the same one in "the bank approved the loan" and in "the bank flooded after the storm". The dictionary has no idea which bank is meant. The next step is what settles it.
The model has no idea about order either
Second gap to fill. The mechanism that comes next handles every token at the same time, with no notion of before or after. "The dog bites the man" and "the man bites the dog" would reach it identical.
So position has to be injected. Since 2021 the dominant method rotates the vectors by an angle proportional to their position, like the hands of a clock.
A rotation applied to the vectors according to where the token sits in the sentence.
Its useful property: when two tokens compare themselves, the result depends only on the gap between their positions, never on the absolute position. The information becomes relative, for free, by geometry.
Direct commercial consequence. Because position is an angle rather than a learned value, a model's context window can be stretched without retraining the whole thing. Windows went from 4,000 to a million tokens in three years.
Hold on to that last sentence, it explains a good part of the subject. Stretching the window has become nearly free at training time. Serving it is not free at all, and step 6 will say why.
Step 3: attention moves those vectors according to context
Here is the central mechanism, the one that made these models possible.
The problem it solves: a single vector cannot carry both the high street branch and the muddy riverside. So there has to be something that, depending on the surrounding words, drags the vector for "bank" towards one region of the space or the other.
The mechanism is formalised with three lists of numbers, all derived from the same vector.
Every token produces three things. A query, what it is looking for. A key, what it offers the others. A value, what it passes on if it gets picked.
The model compares each token's query against every other token's key. That gives a score for each pair. Those scores become weights, and each token gets back a weighted average of what its neighbours hold.
Three consequences, in the order that matters for energy.
The number of comparisons climbs with the square. Each token compares itself to every other one: a hundred tokens make ten thousand comparisons, two hundred make forty thousand. This is the famous quadratic cost of attention. It is real, and it dominates less than people say. On a 70 billion class model it only becomes the main cost past 100,000 tokens, a long way from the regime where almost every request actually runs.
The model does this several times over, in parallel. A single comparison only captures one kind of relation, so several of them sit side by side, each free to learn a different pattern: subject-verb agreement, what a pronoun refers back to, matching brackets. These are the attention heads. A 70 billion model typically carries 64 of them per layer, across 80 layers.
Each token only looks at the past. The model predicts the next word, so during training it must not be able to read the answer. Looking ahead is forbidden.
That last constraint has a consequence worth keeping in mind for step 6. Since a token's representation never depends on what comes after it, it stops changing once computed. So it can be kept rather than recomputed.
The detail: why long context costs so much.
Step 4: the rest of the computation, where the parameters sleep
Attention moves information between words. Yet it holds only a minority of the model's parameters. The bulk of the stock sits in the layers that follow each attention block, transforming every vector independently of the others.
Those layers account for two thirds of the parameters of a classic model.
And here the work can finally be put in figures. On a classic model, producing one token costs roughly two operations per parameter crossed. A 70 billion parameter model therefore asks for about 140 billion operations per token, at every token.
A FLOP is one floating point operation: an amount of work. You count FLOPs the way you count kilometres.
A FLOPS is a rate, operations per second: a machine capability. You measure FLOPS the way you measure kilometres per hour.
The first says how much computation a request demands. The second says how fast the chip can supply it. Confusing them means confusing the length of a journey with the speed of the car, and the two words get swapped for each other all the time.
This unit matters for two practical reasons. The European AI Act explicitly accepts an energy consumption estimate by FLOP proxy, for want of a standardised method. And a provider publishing a footprint per PFLOPS is publishing an intensity, not a total: the figure can fall while absolute consumption rises.
These dense layers cost in proportion to the length of the text, not to its square, which is why attention's quadratic term takes so long to take over.
Then comes the architecture that scrambled every intuition about model size.
Instead of one big single layer, the model holds dozens of specialised ones. A router picks which to wake for each token and leaves the rest switched off.
Result: a model announced at 2,800 billion parameters may cross only a hundred billion or so per token. The computation collapses. But every expert has to stay loaded in memory, since there is no telling in advance which ones will be needed.
Direct consequence for anyone comparing two models: the headline parameter count no longer says anything about consumption. The question has become "how many active parameters per token", and the answer is sometimes thirty times smaller than the press release figure.
The detail: what Mixture of Experts really changes.
These two layers form the block called a transformer
Attention, then dense layer, with a few normalisations in between: that is the pattern. It is stacked eighty times on a 70 billion class model, and each storey picks up the work of the one below.
The name comes from the 2017 paper that set out the architecture, and it refers to exactly that repeated pattern. Every model this course talks about is a variant of it. What changes from one to the next is the number of storeys, the width of the vectors, how attention shares its keys, and whether the dense layer is single or split into experts.
At the output of the last storey, the model produces a score for every entry in its dictionary. The next token is drawn from those scores, and that is where temperature and the other generation settings apply.
Step 5: all of that runs twice, in two opposite regimes
A stack of eighty blocks, then. It gets crossed in two very different ways depending on whether the model is reading or writing.
The machine takes in your whole message as one block. All the tokens cross the stack at the same time, in parallel.
The compute units work at full capacity here. It is the cheapest phase per token in the whole process.
The machine produces its answer one token after another. Parallelising is impossible: each word depends on the one before it.
To produce a single token, it has to reread the model's entire set of weights from memory. The compute units spend most of their time waiting.
Hence the asymmetry that shapes the rest of the course: a token the model writes costs about 35 times a token it reads, measured at batch size one. The billed price shows a ratio of only 5. If you are looking for where to act on consumption, look at what comes out, not at what goes in.
The detail and the measurements: why an output token costs 35 times an input token.
Step 6: what decoding keeps to hand
To write its fiftieth word, the model needs everything that came before it. Recomputing step 3 over the whole conversation at every new word would be absurd. And it does not have to, since those representations stop moving once computed.
So it keeps them in memory.
The keys and values of every token already seen, held on to so they never have to be recomputed.
This store grows with the length of the conversation and has to stay in the accelerator's fast memory for the whole generation. On a 70 billion parameter model, a single one million token request asks for about 328 GB, against 140 GB for the entire model.
That sharing of the desk is the single biggest driver of consumption in the whole chain. An accelerator handling thirty-two conversations at once reads the model weights once for thirty-two answers. The same accelerator handling one conversation reads them for a single answer. Energy per token varies by a factor of 17 to 25 between those two situations.
This lever belongs to the provider, almost never to the customer. It is why self-hosting a model at low volume burns more than calling a shared API.
Question
You write a 2,000 word prompt and the model answers in 100 words. Where did most of the energy go?
Choisissez une réponse pour voir l'explication.
What to take away before moving on
Splitting text into tokens fixes the unit of account, and it penalises the languages that were poorly served. Each token becomes a frozen vector, which attention then moves according to its context. Most of the parameters sleep in the layers that follow, and expert architectures wake only a fraction of them.
That stack is crossed in parallel to read, one position at a time to write, in a ratio of 35. And the working memory of decoding decides how many conversations an accelerator runs at once, hence the energy of each one.
The rest of the course consists of putting sourced figures on those six mechanisms, then deriving an emission factor that holds up in an audit.
The logical next step: why count tokens rather than requests.
FAQ
How does an LLM work, step by step?
Your text is first cut into tokens, word fragments drawn from a fixed dictionary. Each token is replaced by a vector, a list of several thousand numbers that places it in a space of meaning, and its position is added to it. The attention mechanism then moves that vector according to the words around it. The layers that follow, where two thirds of the parameters sleep, transform each vector. This stack is crossed once to read the whole prompt in parallel, then once per token produced to write the answer.
What is the attention mechanism in a language model?
The mechanism that lets each word adjust its meaning according to the others. Each token puts out a query, what it is looking for, and a key, what it offers. The model compares every query against every key, turns the result into weights, and each token gets back a weighted average of what its neighbours pass on. The word "bank" enters with the same vector in "the bank approved the loan" and "the bank flooded after the storm": attention is what tips it towards the branch or the riverside.
Why is attention said to be quadratic?
Because each token compares itself to all the others. A hundred tokens produce ten thousand comparisons, two hundred produce forty thousand. Doubling the length of the text therefore quadruples the number of scores to compute. In practice this term stays a minority cost for a long time: on a 70 billion parameter class model it only becomes dominant past 100,000 tokens, because the layers after attention grow linearly.
What is an embedding and why does the model need one?
A line number in a dictionary carries no meaning a calculation can use. The embedding swaps that number for a list of several thousand numbers, learned during training, which puts fragments of close meaning at close positions. On a 70 billion parameter model that table alone weighs more than a billion parameters. The input vector is frozen: all the disambiguation by context happens afterwards, inside attention.
Why does writing cost more than reading for an AI?
Because the two phases do not have the same physical regime. Reading handles all the prompt's tokens in parallel and saturates the compute units. Writing is sequential: each token depends on the previous one, and the model has to reread all of its weights from memory to produce a single one. The processor spends most of its time waiting on memory. The ratio measured at batch size one is about 35 to 1 in energy, against 5 to 1 in billed price.