Until 2023, a model with N parameters cost roughly 2N operations per token generated. Parameter count was a direct measure of the work, and models were compared on it.
Since Mixture-of-Experts went mainstream, that is wrong. A model advertised at 2,800 billion parameters activates only 104 of them per token, or 3.7%. The only figure that says anything about cost is the active parameter count, and it is not always the one put forward.
What a Mixture-of-Experts does
In a classic model, every token passes through the entire network. On any given input, almost all of the part that stores knowledge fires for a negligible contribution.
One large knowledge block is replaced by N smaller ones, the experts, and a small network called the router picks which ones to activate for each token. Typically 8 experts out of 256.
Until 2023 the rule was simple: a dense model of N parameters cost roughly 2N FLOP per token produced. Parameter count therefore measured the work directly, and two models could be compared on that single line.
Expert architectures broke the link. Compute per token now follows the active parameters, not the advertised total. The mechanism is described at step 4 of what happens to a request.
Compute per token falls, but every expert has to stay loaded in memory, since there is no way to know in advance which ones the router will pick. MoE computes less. It still loads everything.
The number of parameters a token passes through, across the whole network, attention included. Not the size of one expert, and not the active parameters of a single layer.
It is the only figure that predicts the compute cost of an inference. Ask for it every time, next to the total.
One vocabulary trap worth knowing: 8 × 7 billion does not make 56 billion. Mixtral 8x7B weighs 47 billion parameters, because the attention layers are shared between experts and only the knowledge blocks are duplicated. A document that writes 56 is wrong from its first line.
The trend, in numbers
| Model | Date | Total | Active per token | Ratio |
|---|---|---|---|---|
| Mixtral 8x7B | Jan 2024 | 47bn | 13bn | 27.7% |
| Grok-1 | Mar 2024 | 314bn | 25% of weights | 25% |
| DeepSeek-V3 | Dec 2024 | 671bn | 37bn | 5.5% |
| Llama 4 Maverick | Apr 2025 | 400bn | 17bn | 4.3% |
| Qwen3-235B-A22B | May 2025 | 235bn | 22bn | 9.4% |
| Kimi K2 | Jul 2025 | 1,000bn | 32bn | 3.2% |
| gpt-oss-120b | Aug 2025 | 117bn | 5.1bn | 4.4% |
| DeepSeek-V4-Pro | Apr 2026 | 1,600bn | 49bn | 3.1% |
| Kimi K3 | Aug 2026 | 2,800bn | 104bn | 3.7% |
From 25 to 28% in 2024 down to 3 to 5% in 2026, while the totals went from 47 billion to 2,800 billion. Compute per token is flat. The mass of weights to host has exploded.
Two models not to cite, for lack of data: Grok 3 and Grok 4, for which no parameter count has ever been published. And one frequent confusion to avoid: DeepSeek-R2 does not exist. The line runs R1, then V3, then V4. Several aggregators invent it.
The gain, measured by a regulator
The best independent measurement available comes from PEReN, working for the French digital regulator Arcep, published in May 2026. Protocol: 22 open models from 3 to 123 billion parameters, measured on the Jean Zay supercomputer, consumer use cases.
| Lever | Average measured gain |
|---|---|
| MoE architecture against dense, at equivalent parameter count | −45% |
| 8-bit or 4-bit quantisation | −39% |
| Reasoning mode switched on | +92%, up to +849% on code |
Limits the authors state themselves, to be quoted alongside the figure: open models only, nothing above 125 billion parameters, and GPU consumption alone, when host processors account for around a third of the total.
The regulator's own conclusion is worth repeating verbatim in front of a client: capping consumption does not necessarily mean trading away performance, and frugal models match large ones on given use cases.
The price paid elsewhere
MoE cuts compute, not the memory you have to provision. Marketing material stays quiet on that point.
The DeepSeek-V3 technical report says it flatly: the minimum deployment unit in the generation phase is 320 GPUs, for a model that activates only 37 billion parameters per token. A classic 37 billion model would fit on one or two accelerators.
In real production, the company published 24 hours of measurements: an average of 1,814 H800 GPUs running permanently, for $87,072 of infrastructure a day, serving 608 billion input tokens and 168 billion output tokens.
The nuance that stops you being taken apart in a meeting: "you need 320 GPUs to load the model" is false. Open inference engines serve it on 8 H200 accelerators at reduced precision. The 320 GPUs are the unit DeepSeek deploys to hit its latency and throughput target, an operational trade-off tied to the latency it wants, not a memory constraint.
And it is quantisation, not MoE, that makes a large model fit on little hardware: Llama 4 Scout runs on a single H100 in 4-bit, gpt-oss-120b on 80 GB thanks to a post-trained 4-bit format. Without aggressive compression, neither fits.
Question
A vendor compares two models: a dense 70 billion parameter model, and a 671 billion MoE with 37 billion active. What can you conclude about inference cost?
Choisissez une réponse pour voir l'explication.
What is missing before you can compare
Here is the trade-off as you would want to be able to state it: MoE divides energy per token by X and multiplies the required fleet by Y, and the manufacturing footprint allocated to each request follows from that.
Both halves exist separately. Nobody has joined them up. No source publishes a quantified comparison of the energy gain against the hardware overhead, brought back to the same unit. Filling the gap would take a server life cycle assessment applied to an MoE fleet with per-request allocation, and that work has not been done.
You can guess the order of magnitude of the missing term from the only manufacturer data available, published in July 2025: a board of eight H100 accelerators carries 1,312 kgCO2e of manufacturing, 42% of it memory, and the next generation 2,274 kgCO2e with 49% memory.
Memory is therefore the largest manufacturing line, ahead of the compute die itself. And MoE is precisely what shifts load from compute onto memory. That makes the missing figure awkward rather than anecdotal. Present it as reasoning, never as a measured result.
FAQ
Does a model's parameter count tell you what it costs?
Not since Mixture-of-Experts became standard. A model can advertise 2,800 billion parameters and activate only 104 billion per token, or 3.7%. Inference compute follows active parameters, not the total. Without both numbers, total and active, a size figure will mislead you every time, and always towards overestimation.
What are a model's active parameters?
The number of parameters each token actually passes through, across the whole network, attention included. On DeepSeek-V3, 8 routed experts out of 256 are activated per layer, which gives 37 billion active parameters out of 671 billion in total. It is the only value that predicts the compute cost of an inference.
Does a Mixture-of-Experts model consume less than a dense model?
Less compute per token, yes. The PEReN study for Arcep, published in May 2026 on 22 open models measured on the Jean Zay supercomputer, puts the gain at 45% on average at equivalent parameter count. But the architecture does not reduce the memory you have to provision, since every expert stays loaded, which raises the amount of hardware you need.
Why does Mixtral 8x7B weigh 47 billion parameters rather than 56?
Because only the knowledge blocks are duplicated across experts. The attention layers are shared and counted once. Multiplying the number of experts by the advertised size of one expert therefore gives a wrong answer, and it is a common mistake in comparison tables.
Do you really need 320 GPUs to run DeepSeek-V3?
No. That number is the minimum deployment unit in the generation phase published by DeepSeek to meet its latency and throughput targets, with expert redundancy. Open inference engines serve the same model on 8 H200 accelerators at reduced precision. It is an operational trade-off, tied to the latency being targeted.