DeepSeek-V3 Technical Report
671B parameters, 37B active, FP8 training and a loss-free router: frontier quality for 2.788M GPU-hours.
arXiv:2412.19437Where this sits
The era's capstone, and a return to the text main line after the VL2 vision detour. It keeps V2's MLA and DeepSeekMoE, adopts the auxiliary-loss-free router from the methods paper that precedes it in this era, and adds FP8 training plus multi-token prediction. The training-cost table, not a benchmark score, is the headline.
- Load balancing drops the auxiliary loss: a per-expert bias steers top-K selection, with a complementary sequence-wise loss at a tiny weight only to bound per-sequence extremes, and no token is dropped.
- FP8 mixed precision is validated at 671B scale for the first time: 1x128 activation tiles and 128x128 weight blocks with FP32 promotion every 128 products, keeping GEMM error under 0.25% relative.
- Multi-token prediction (MTP) with one extra token densifies the training signal and doubles as a speculative-decoding draft: 85-90% acceptance and 1.8x tokens per second at inference.
- Training economics: 14.8T tokens for 2.664M H800 GPU-hours of pretraining (180K GPU-hours per trillion tokens, 3.7 days on 2048 H800s), plus 119K for context extension and 5K for post-training.
- Infrastructure co-design: DualPipe overlap, no tensor parallelism needed, 16-way pipeline and 64-way expert parallelism over 8 nodes, and a run with no irrecoverable loss spikes and no rollbacks.
What it is
DeepSeek-V3 is a 671B-parameter mixture-of-experts model that activates 37B parameters per token, pretrained on 14.8T tokens. It keeps V2's MLA and DeepSeekMoE, adopts the auxiliary-loss-free balancing from earlier in this era, adds an FP8 mixed-precision training framework, and trains on a multi-token prediction objective. The full run took 2.788M H800 GPU-hours, about $5.576M at $2 per GPU-hour, and the report documents the training framework, the routing rules, and the inference deployment.
1Theory from first principles
Three costs left on the table
V2 showed that sparsity plus latent attention makes a strong model cheap to serve. At V3 scale three costs remain. Precision is the first: keeping 671B parameters and their activations in BF16 burns memory and bandwidth, and moving to 8-bit floats naively destroys accuracy because activations contain outliers. Balance is the second: V2's auxiliary balance losses add gradients that push against the language-modeling objective. Supervision is the third: next-token prediction gives one training signal per position, and nothing supervises longer horizons.
FP8 training, from first principles
An 8-bit float spends bits on sign, exponent, and mantissa: E4M3 means 4 exponent bits and 3 mantissa bits, a wide range with coarse relative precision. What makes it usable is a scale. Divide a block of numbers by its largest absolute value, multiply by the format's largest representable value, round to FP8, and remember the scale to dequantize later. Outliers are the enemy: one huge value forces a tiny scale on everything else in the tensor. So V3 quantizes finely, activations in 1x128 tiles (per token, per 128 channels) and weights in 128x128 blocks, with the scale recomputed online each step. Two further fixes: accumulation stays exact by promoting partial sums to CUDA cores every 128 products, because H800 tensor cores keep only about 14 bits of accumulation; and master weights and gradients stay in FP32. V3 uses E4M3 on all tensors rather than the usual E5M2-for-backward split.
Per-tile FP8 scaling
scale = max_fp8 / amax(X)
X_fp8 = round_to_e4m3(X * scale)
X_hat = X_fp8 / scaleamax is the largest absolute value in the tile, so dividing by it maps the tile exactly into the format's range. The round-trip error is then set by the 3-bit mantissa, not by the tensor's global dynamic range, and smaller tiles shrink amax. Weights use 128x128 blocks because weight matrices are smoother than activations.
from math import floor, log2
def to_e4m3(x):
if x == 0.0:
return 0.0
sign = -1.0 if x < 0 else 1.0
a = abs(x)
e = max(-6, min(8, floor(log2(a))))
step = 2.0 ** (e - 3)
return sign * round(a / step) * step
def quantize_tile(xs, max_fp8=448.0):
scale = max_fp8 / max(abs(x) for x in xs)
return [to_e4m3(x * scale) / scale for x in xs]
print(quantize_tile([0.004, -0.011, 2.5, 0.008]))- 448.0 is the largest magnitude E4M3 can represent in the OCP FP8 spec; three mantissa bits mean values snap to eight steps per exponent.
- The real implementation is hardware: tile scale recomputed online, GEMM in FP8, FP32 promotion every 128 products.
Balancing without a balance loss
V3 routes with the bias-only trick: rank experts by s_i + b_i, weight their outputs by s_i, and move each bias one step in the sign of the previous batch's load error. One refinement matters at this scale: a complementary sequence-wise balance loss with a very small weight, there only to stop pathological imbalance inside a single sequence. With balance handled this way, V3 drops no tokens at all, during training or inference. Communication is bounded by node-limited routing: each token is sent to at most 4 nodes, chosen by the sum of each node's top K_r / M expert scores. Each MoE layer has 1 shared expert and 256 routed experts, 8 of them activated per token; the first three layers stay dense.
Predict the next two tokens
Instead of supervising only the next token, V3 adds a multi-token prediction module that, at position i, predicts token i+2 from the depth-1 hidden state combined with the embedding of token i+1. The module is one transformer block with a projection and a shared embedding and output head, trained with its own cross-entropy loss; the total objective is the main loss plus lambda times the MTP loss. Two things come out of it: denser training signal, which the ablations show improves benchmarks, and a free speculative-decoding draft at inference. The draft proposes a second token, the main model verifies it, and the report measures an 85-90% acceptance rate and 1.8x tokens per second.
MTP objective
L_MTP = (lambda / D) * sum_k L_MTP^k, with D = 1
L_MTP^k = -(1 / T) * sum_i log P_i^k[t_i]Each depth contributes a cross-entropy term over the sequence, averaged over depths and weighted by lambda. V3 uses one extra token (D = 1), enough to improve benchmarks and to serve as a draft model. Keeping the complete causal chain at each depth is what makes the modules reusable at inference: the draft conditions on tokens the main model has already produced.
2Inside the paper
Problem
V2's architecture was validated, but scaling it to frontier quality raises three questions: can training run in 8-bit precision without losing accuracy, can load balancing stop taxing the language objective, and can the training objective itself be made richer than one-token prediction. And the whole thing has to be affordable without a giant cluster.
Key idea
Layer the fixes. MLA plus DeepSeekMoE at 671B total and 37B active; a bias-only router with a tiny sequence-wise complement and no token dropping; an FP8 mixed-precision framework with fine-grained scaling, online scales, and FP32 promotion; and a one-token MTP objective that also accelerates inference. Everything is trained with the DualPipe framework (a pipeline schedule that overlaps communication with computation) on 2048 H800s, with no tensor parallelism.
Evidence
14.8T training tokens. Full training costs 2.788M H800 GPU-hours: 2.664M pretraining, 119K context extension, 5K post-training, about $5.576M at $2 per GPU-hour, and about 180K GPU-hours per trillion tokens, 3.7 days on the 2048-GPU cluster. The run had no irrecoverable loss spikes and no rollbacks. Ablations at 15.7B and 228.7B scale show the loss-free router and MTP each improve benchmarks over strong baselines, and the MTP draft is accepted 85-90% of the time for a 1.8x decoding speedup. The model reports 88.5 MMLU, 75.9 MMLU-Pro, and 59.1 GPQA, comparable to leading closed models of the period.
What the authors admit
The conclusion names two deployment limits: the recommended deployment unit is large, which is a burden for small teams, and generation speed, while more than twice V2's, still has room to improve. The FP8 section is also explicit that the H800's limited accumulation precision is a hardware deficiency partially worked around in software, and the hardware-design chapter lists what future chips should do natively.
Practitioner takeaway
Three transferable moves: quantize fine and accumulate wide if you want 8-bit training to hold; move balancing out of the loss and into the router; and treat a training objective change as a potential inference optimization (MTP as a draft model). Then measure cost per token, not just benchmark scores, because that is what decides whether the next scale-up is affordable.
3Implementation check
0 of 6 questions answered
1.Why does V3 scale per 1x128 tile for activations instead of using one scale per tensor?
2.Why does the GEMM promote partial sums to CUDA cores every 128 products?
3.What does the MTP module do at inference time?
4.How does V3 keep its 256 routed experts balanced?
5.Which number is the full training cost of DeepSeek-V3?
6.Why does V3 limit each token to at most 4 nodes during routing?
4Build it yourself
Tile-scale FP8 plus a two-token MTP head
Advanced5-8 hoursTwo halves of the V3 report, built small. First, an FP8 quantizer with per-tile scaling: show that one activation outlier forces a per-tensor scale that crushes small values, and that 128-number tiles rescue them. Second, a multi-token prediction head in n-gram form: train a 1-step head and a 2-step head whose context includes the intermediate token, then compare held-out accuracy as a stand-in for draft acceptance.
Milestones
- Emulate E4M3 rounding with three mantissa bits and a 448 maximum, as in the paper.
- Build an activation vector with one large outlier and small values near 1e-5, and quantize it per tensor and per 128-number tile.
- Measure relative error on the non-outlier tile for both scaling schemes and explain the gap.
- Sweep tile sizes 32, 64, 128, 256 and report error against the number of scale recomputations.
- Build a synthetic token stream with local motif structure and hold out the tail.
- Train a 1-step next-token counter and a 2-step counter that conditions on the intermediate token.
- Report held-out accuracy for both heads and the acceptance rate a draft would see.
- Estimate a full-tensor FP8 GEMM against the tiled scheme: error versus metadata bytes.
Success criteria
- Per-tensor scaling shows relative error above 0.5 on the non-outlier tile; per-128-tile scaling brings it below 0.05.
- The 2-step MTP head lands within 5 accuracy points of the 1-step head on the same held-out stream.
- The write-up states the E4M3 mantissa limit (roughly 6-12% per-element relative error) and why tiling helps only when the dynamic range is wide.
- Both experiments run in one fixed-seed script with no third-party imports.
Stretch goals
- Add FP32 promotion every 128 products to the GEMM estimate and show how much accumulation error it removes.
- Quantize the residual stream as well as the activations and measure how error compounds across simulated layers.
- Replace the n-gram heads with a tiny learned linear softmax trained by SGD and compare 1-step and 2-step accuracies.