DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Compress the KV cache into a latent, make the experts fine-grained, and get a stronger model for less money.
arXiv:2405.04434Where this sits
The second-era opener. It keeps the transformer backbone of DeepSeek-LLM and the fine-grained expert idea of DeepSeekMoE, then attacks the two costs that dominate the lifecycle of a model: the KV cache that makes attention memory-bound at inference, and the dense FFN that makes every extra parameter expensive to train and serve.
- MLA compresses keys and values into a single 512-dim latent per token per layer, plus a 64-dim decoupled key that carries RoPE, shrinking the cache to the equivalent of GQA with 2.25 groups while beating MHA on quality.
- DeepSeekMoE uses 160 fine-grained routed experts (6 activated per token) plus 2 always-on shared experts, so total parameters grow without per-token compute growing with them.
- Training cost drops 42.5% versus the dense DeepSeek 67B: 172.8K versus 300.6K H800 GPU-hours per trillion tokens.
- KV cache falls 93.3% and maximum generation throughput rises to 5.76x (over 50K tokens per second on a single 8-GPU H800 node).
- Device-limited routing plus expert, device, and communication balance losses keep expert-parallel training efficient, and YaRN (a position-encoding extension method) extends the context window from 4K to 128K.
What it is
DeepSeek-V2 is a 236B-parameter mixture-of-experts language model that activates only 21B parameters per token. It pairs Multi-head Latent Attention (MLA), which caches a small latent vector instead of full keys and values, with DeepSeekMoE, which splits the feed-forward network into many fine-grained experts plus a few shared ones. Against the dense DeepSeek 67B it saves 42.5% of training cost, cuts the KV cache by 93.3%, and raises maximum generation throughput 5.76x. It was pretrained on 8.1T tokens with a 128K context window.
1Theory from first principles
The bottleneck at inference time
A transformer generates one token at a time. To predict token t, every attention layer compares that token's query against the keys of all earlier tokens and then mixes their values. Recomputing keys and values from scratch at every step would repeat work, so serving systems cache them: the KV cache. The arithmetic is unforgiving. A cache entry holds 2 vectors (one key, one value) of size d_head for each head, across n_heads heads and L layers, and it grows by that amount for every token of context. A 60-layer model with 128 heads of width 128 stores 2 * 128 * 128 * 60 = 1.97M numbers per token before any batching, so at long context the cache, not the weights, dominates memory and bandwidth.
Store a latent, rebuild the keys and values
MLA never stores keys and values. It stores one small latent vector per token per layer, produced by a down-projection: c_t = W_DKV * h_t, with dimension 512 instead of 2 * 128 * 128 = 32768. Keys and values are reconstructed by two up-projections, k = W_UK * c and v = W_UV * c. Because those projections are linear, W_UK can be folded into the query projection and W_UV into the output projection at inference, a trick the paper calls absorption, so attention can run against the compressed vectors directly. Positions are the one thing that cannot survive compression: RoPE would sit between the projections and break absorption. MLA therefore keeps a separate 64-dimensional key that carries RoPE. Total cached per token per layer: 512 + 64 = 576 numbers, the same as grouped-query attention (GQA) with only 2.25 groups, but with quality above standard multi-head attention (MHA).
MLA: compress once, reconstruct in attention
c_t = W_DKV * h_t
k_t = W_UK * c_t, v_t = W_UV * c_t
cache per token per layer = 512 + 64 = 576 << 2 * 128 * 128 = 32768The down-projection is where compression happens. The up-projections happen inside attention, and at inference they fold into the query and output weights, so only c_t and the small RoPE key are ever written to memory. The win is memory traffic, which is exactly what decode is bound by.
Make the feed-forward layer sparse
Attention is only half the compute. The other half is the feed-forward network, normally one wide MLP applied to every token. A mixture-of-experts layer replaces it with N smaller expert MLPs plus a router: score every expert for each token, keep the top K, and add their outputs weighted by the scores. Total parameters grow with N; per-token FLOPs grow only with K. DeepSeekMoE adds two choices. First, finer granularity: 160 small routed experts with 6 activated, so a token's computation is assembled from many small pieces and each expert can specialize. Second, shared experts: 2 experts that every token uses, absorbing common patterns so the routed experts do not all relearn them. DeepSeek-V2 replaces every FFN except the first with an MoE layer.
Top-K routed MoE layer
h' = u + sum_i FFN_i^shared(u) + sum_i g_i * FFN_i^routed(u)
g_i = s_i if s_i in TopK({s_j}, K_r) else 0
s_i = softmax(u^T * e_i)s_i is the affinity between token u and expert centroid e_i. TopK keeps only the K_r largest scores, so only those expert MLPs run, and g_i weights their outputs. Everything else in a MoE layer is about which experts get picked and how evenly the traffic is spread.
Keeping a sparse layer trainable and shippable
Two practical problems come with sparsity. Unbalanced routing means some experts never train, a failure called routing collapse, and under expert parallelism it also means some devices idle while others queue. The paper attacks this with three auxiliary losses (expert-level, device-level, communication-level) and a token-dropping rule during training: each device keeps only its share of tokens, dropping the lowest-affinity ones, with about 10% of sequences exempt. Device-limited routing bounds communication by sending each token to at most M devices, which the authors find matches unrestricted routing once M >= 3. Without rules like these, a sparse model is theoretically cheap and practically slow.
What the design buys
Against DeepSeek 67B: 42.5% less training cost (172.8K versus 300.6K H800 GPU-hours per trillion tokens), 93.3% smaller KV cache, and 5.76x higher maximum generation throughput, over 50K tokens per second on one 8-GPU H800 node. Pretraining ran on 8.1T tokens with a 100K-vocabulary tokenizer, then YaRN extended the window from 4K to 128K. The report is candid that none of this is a free win: compression adds projections, routing adds communication, and the balance losses trade routing quality against model quality unless tuned carefully.
def dense_cache(n_heads, d_head, layers, ctx, bytes_per=2):
return 2 * n_heads * d_head * layers * ctx * bytes_per
def mla_cache(d_latent, d_rope, layers, ctx, bytes_per=2):
return (d_latent + d_rope) * layers * ctx * bytes_per
dense = dense_cache(128, 128, 60, 8192)
sparse = mla_cache(512, 64, 60, 8192)
print(dense // sparse) # 56x fewer bytes at the same context- V2 caches 576 numbers per token per layer: the 512-dim latent plus the 64-dim decoupled RoPE key.
- Decode is memory-bound, so a smaller cache buys a longer context or a bigger batch on the same hardware.
2Inside the paper
Problem
Serving a strong open model is dominated by two costs: KV cache memory at long context, and the FLOPs of a dense FFN at every token. Training a good model is dominated by how many parameters you can afford to update and how evenly the work spreads across devices.
Key idea
Answer both with architecture. MLA compresses each token's keys and values into a 512-dim latent plus a 64-dim RoPE key and reconstructs them inside attention, cutting the cache 93.3% while improving quality over MHA. DeepSeekMoE keeps the model sparse: 236B total parameters, 21B active per token, with 160 fine-grained routed experts (6 active) and 2 shared experts, bounded by device-limited routing.
Evidence
Trained on 8.1T tokens, context extended to 128K with YaRN. Versus dense DeepSeek 67B: 42.5% cheaper training (172.8K versus 300.6K H800 GPU-hours per trillion tokens), 93.3% less KV cache, 5.76x maximum throughput (over 50K tokens per second on one 8-GPU node). A 15.7B-total / 2.4B-active V2-Lite is released so the architecture can be studied at a small scale.
What the authors admit
The stated limitations are the usual LLM ones: no knowledge updates after pretraining, a risk of non-factual output, and a corpus that is predominantly Chinese and English, so other languages should be used with caution. The architecture does not remove the need for balance losses, token dropping rules, or communication management; those are what make sparsity pay off in practice.
Practitioner takeaway
If you serve a model at long context, the KV cache is your real budget, and low-rank compression is the most direct lever: store a latent, not the keys and values. If you train sparse, the router is the hard part, so design the balance and communication rules together with the architecture rather than bolting them on later.
3Implementation check
0 of 5 questions answered
1.An MLA layer caches a latent vector plus a small RoPE key. What is actually stored per token per layer?
2.Why does MLA keep a separate 64-dim key for RoPE instead of putting position inside the compressed latent?
3.DeepSeek-V2 activates 21B of 236B parameters per token. Which experts does the router actually select?
4.What does device-limited routing (at most M devices per token) buy the training system?
5.The headline 42.5% training-cost saving compares which two runs?
4Build it yourself
Compress the KV cache into an MLA latent
Intermediate3-5 hoursRebuild the core of Multi-head Latent Attention in miniature: project each token's keys and values down to a shared latent, reconstruct them for attention, and sweep the latent width to find the smallest cache that meets a fixed reconstruction-error budget. The toy uses random projections and a low-rank token state, so the compression arithmetic matches the real one.
Milestones
- Build the toy model: a random 64-number key and value projection from a 32-dim token state, with the state itself low-rank (8 signal dimensions) so real activations are only approximated.
- Implement the MLA round trip: down-project the state to a d_c-number latent, up-project it back to a reconstructed state, then to keys and values.
- Count the cache: 2 * 4 * 8 = 64 numbers per token for full MHA against d_c + 8 for MLA, where the 8 is the decoupled RoPE key.
- Sweep latent widths 2, 4, 6, 8 and record key and value relative reconstruction error at each width.
- Pick the smallest latent whose error is at most 0.05 and report the cache compression ratio.
- Repeat over 200 random tokens and report the mean error so one sample cannot set the result.
- Write a short note comparing the measured trade curve with the paper's 93.3% cache reduction.
Success criteria
- The sweep covers at least 4 latent widths and prints cache size in numbers plus key and value relative error for each.
- The chosen latent reaches mean relative key and value error at or below 0.05 over 200 random tokens.
- The final report includes the compression ratio versus the 64-number MHA cache and counts the 8-number RoPE key in the MLA cache.
- The script runs under 5 seconds with one python3 command using only the standard library.
Stretch goals
- Replace the hand-aligned latent with learned projections: fit W_D by power iteration on the sample covariance (PCA) and re-run the sweep.
- Quantize the latent to 8 bits and measure how much extra error a production cache would pay.
- Run the round trip inside attention: score 64 keys, take the top-1 key per query with full and reconstructed keys, and report the argmax agreement rate.