TL;DR: A mixture-of-experts (MoE) model replaces one large feed-forward block in each layer with many smaller "experts" and a router that sends each token to a few of them. Two numbers now describe the model. Total parameters decide how much memory you must hold, because every expert has to be loaded. Active parameters decide how much arithmetic each token costs. An MoE model is cheap per token for a provider with heavy, steady traffic and expensive for a team that self-hosts it at low volume, because the memory bill does not shrink with the traffic.
What the router does
In a dense transformer, every token passes through every weight in every layer. An MoE layer keeps attention as it is and splits the feed-forward part into several expert networks. A small learned router scores the experts for each token and sends the token to the top few, then combines their outputs, weighted by the router's scores. The next token may go to a different pair.
Read diagram description
A token enters a router, which scores eight experts and sends the token to two of them, expert 3 and expert 6. Their outputs are combined as a weighted sum. All eight experts are held in memory, but only two run for this token.
The idea is old: Shazeer and colleagues described a sparsely gated mixture-of-experts layer in 2017, and Google's Switch Transformer (2021) routed each token to a single expert. Every model in the table below uses it, because it buys capacity without paying for it on every token.
Two numbers, two bills
Published model cards now report both counts. These are the developers' own figures, as of November 2026:
| Model | Total parameters | Active per token | Active share |
|---|---|---|---|
| gpt-oss-120b (OpenAI, open weights, August 2025) | 117B | 5.1B | 4.4% |
| DeepSeek-V3 (open weights) | 671B | 37B | 5.5% |
| Qwen3.5-397B-A17B (open weights, February 2026) | 397B | 17B | 4.3% |
| DeepSeek-V4-Pro (open weights, April 2026) | 1.6T | 49B | 3.1% |
| Qwen4-Max (Alibaba, hosted only, 16 November 2026) | 2.4T | 71B | 3.0% |
The share column is our arithmetic. Two bills follow from the two numbers.
Memory follows total parameters. Every expert must sit in accelerator memory, because the router may choose any of them for the next token. At one byte per parameter (FP8), DeepSeek-V3's weights alone need about 671 GB before the KV cache, activations and runtime buffers. That is more than any single accelerator holds, so serving it means splitting experts across several chips (expert parallelism), which adds network traffic between them.
Compute follows active parameters. A forward pass costs roughly two floating-point operations per active parameter per token. For DeepSeek-V3 that is about 2 × 37B = 74 billion operations per token. A dense 70B model needs about 140 billion. The MoE model holds about ten times the weights of the dense one and does about half the arithmetic per token.
Why the same model is cheap for one team and expensive for another
The arithmetic saving only turns into a cost saving when the memory is busy.
A provider serving millions of requests keeps every expert loaded and fed. Different tokens in a large batch go to different experts, so the hardware stays occupied and the low compute per token shows up in the price. This is why a provider can charge less for a large MoE model than its parameter count suggests.
A team self-hosting the same model for an internal tool at a few requests a minute pays for all the memory, all day, while most experts sit idle. Using the self-host break-even, the fixed cost is set by the hardware needed to hold the total parameters, and utilisation is low. A dense model a tenth the size might serve that workload on one accelerator for less, at similar quality on a narrow task.
Illustrative comparison, not a benchmark: if the MoE model needs eight accelerators to hold its weights and a dense 70B model at FP8 fits on one, the MoE deployment costs eight times as much per hour before it serves a single request. It wins only when the traffic is high enough that its lower compute per token lets those eight accelerators do far more than eight times the work.
What changes in product decisions
Headline sizes stop predicting cost or speed. "A one-trillion-parameter model" tells you its memory footprint, not its price per token or its latency. Ask for active parameters, and then ask for measured tokens per second at your expected concurrency.
API prices hide the architecture, which is fine. If you buy tokens from an API you pay the provider's price, set by their utilisation and margin. MoE matters to you mainly when you self-host, choose an open-weight model, or plan capacity on dedicated hardware.
Latency can vary with routing. When many tokens in a batch choose the same expert, that expert becomes a queue. Serving stacks cap how many tokens each expert accepts and balance load during training, but tail latency still deserves measurement under realistic traffic, not a single-user demo.
Fine-tuning and quantization behave differently. Adapters and lower precision work on MoE models, but quality can shift by expert, and an evaluation set that misses a rarely routed topic can miss a regression. Test on your own task mix, as the quantization concept argues for dense models.
Smaller deployments may favour dense models. On a laptop, a phone or a single accelerator, memory is the binding limit. A dense model that fits beats a larger MoE model that does not.
In the interview
MoE comes up in platform and technical rounds as "why is this huge open model cheap?" or "should we self-host it?". A weak answer treats parameters as one number. A strong answer separates the two bills, says the model is cheap per token when its experts stay busy, works out the memory floor at a stated precision, and asks for throughput at the team's real concurrency before choosing between a large MoE model, a smaller dense model and an API.
Next: GPU economics for PMs and self-host vs API.