DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
The 7B and 67B base models that opened the DeepSeek line, configured by scaling laws the authors re-measured from scratch.
arXiv:2401.02954Where this sits
This is the root of the whole curriculum. Every later DeepSeek model inherits its trunk, its tokenizer, and its multi-step learning rate schedule.
- A calibrated scaling law for hyperparameters: fitted power laws predict the near-optimal batch size and learning rate for a given compute budget.
- A new model-scale unit, non-embedding FLOPs per token M, which includes attention cost and excludes the vocabulary projection, making the model/data split fit more accurate at small scale.
- A dataset and pipeline that reached 2 trillion tokens, with cross-dump deduplication removing 89.8% of documents versus 22.2% for single-dump dedup.
- A multi-step learning rate schedule that matches cosine's final loss while letting early training be reused when a run is extended.
- A full alignment recipe (about 1.5M SFT instances, then DPO) that produced a 67B chat model rated above GPT-3.5 in open-ended evaluations.
What it is
DeepSeek's first public language model family: two dense decoder-only Transformers, 7B and 67B parameters, trained from scratch on a 2 trillion token corpus of mostly English and Chinese text. The paper's real subject is budgeting: it re-measures scaling laws for batch size, learning rate, and the model/data split, then spends its compute budget according to those curves. Supervised fine-tuning and DPO (direct preference optimization) turn the base models into chat models that beat LLaMA-2 70B on many benchmarks and GPT-3.5 on open-ended comparisons.
1Theory from first principles
Why another language model
By early 2024 the open-source world had settled on LLaMA-2 as the de facto baseline, and most teams trained fixed-size models without asking how compute should be divided between model and data. Two famous scaling-law papers, Kaplan et al. and Chinchilla, disagreed on that split, and neither described how they chose the batch size and learning rate for each run. DeepSeek LLM starts by treating those disagreements as the problem: before training anything large, work out how to allocate compute, then follow the answer. Keep that framing in mind, because the benchmark tables are the smaller half of the paper.
What a scaling law actually claims
A scaling law is a fitted curve: train many small models at known compute budgets, measure validation loss, and extrapolate. The DeepSeek team uses IsoFLOP profiles, the Chinchilla method, which fixes a compute budget and tries several model/data splits to find the loss minimum. The output is not a law of nature but a budget planner. If the curve is right, a run costing 1e20 FLOPs can tell you how to spend 1e23, which is exactly what a lab with limited GPUs needs.
Compute as model times data
C = M * DC is the training compute budget in FLOPs, M is model scale measured as non-embedding FLOPs per token, and D is the number of training tokens. Older work wrote C = 6ND; the authors argue 6N ignores attention cost and, in its full-parameter form, includes the vocabulary projection that contributes little capacity. At small scale those approximations are off by up to 50%, which distorts the fitted curve.
The fitted split
M_opt = 0.1715 * C^0.5243 D_opt = 5.8316 * C^0.4757M_opt is the model scale and D_opt the token budget that minimize loss for compute C. The exponents are close to one half, so extra compute is split almost evenly between a bigger model and more data. They sum to 1.0, which keeps M_opt * D_opt = C consistent. The coefficients are empirical, not constants of nature: refit them on your own data.
Data quality moves the split
The finding that outlasted the paper: the optimal split depends on the dataset. On their early in-house data the exponents were a = 0.450, b = 0.550; after cleaning, a = 0.524, b = 0.476; on the carefully curated OpenWebText2, a = 0.578, b = 0.422. Chinchilla's MassiveText gave roughly 0.49 and 0.51. As data gets better, more of each new FLOP should go to the model, which is a clean explanation for why previous studies disagreed: they were measuring different corpora.
def lr_at(step, total_steps, peak):
# 2000 warmup steps, then drops at 80% and 90% of training.
if step < 2000:
return peak * step / 2000
progress = step / total_steps
if progress < 0.8:
return peak
if progress < 0.9:
return peak * 0.316
return peak * 0.1
for s in (0, 1000, 2000, 7000, 8500, 9500):
print(s, round(lr_at(s, 10000, 4.2e-4), 8))- The 7B model peaked at 4.2e-4 and the 67B model at 3.2e-4, both chosen from the scaling law for hyperparameters rather than from a grid search at full size.
- The 0.316 factor is the square root of 0.1: each drop cuts the learning rate to about a third, then to a tenth of the peak.
- The authors report this schedule reaches essentially the same final loss as cosine decay, with one practical advantage: the first stage can be reused when training is later extended.
The recipe around the math
The micro design follows LLaMA: pre-norm with RMSNorm, SwiGLU feed-forward layers, rotary position embeddings, and a 100K-token byte-level BPE (byte-pair encoding) tokenizer padded to 102400 entries. The macro design differs: 30 layers for 7B, 95 for 67B, depth instead of extra width, and grouped-query attention with 8 key-value heads only in the 67B model. Training ran in bf16 with fp32 gradient accumulation, FlashAttention, ZeRO-1, and pipeline parallelism. Alignment used about 1.5 million instruction instances (1.2M helpful, 300K safety) for SFT (supervised fine-tuning), then DPO. The 7B model was fine-tuned for 4 epochs and the 67B for 2, because the larger model overfit quickly. One side effect worth knowing: adding math SFT data increased the rate at which chat responses looped, and DPO reduced it.
What to watch for while reading
Read Section 3 for the scaling laws and treat Section 5 as receipt-checking. Notice how many choices depend on the fitted curves, and how much rests on a single validation set of 100M tokens distributed like the training data. The limitation section is short and honest: a chat model that cannot update knowledge after pretraining, hallucinates, covers Chinese unevenly, and is shaky in languages beyond English and Chinese.
2Inside the paper
The problem it attacks
Open-source models were being trained at fixed sizes with no shared method for choosing hyperparameters or allocating compute, and published scaling laws disagreed with each other. DeepSeek wants to show that a smaller lab can train 7B and 67B models from scratch if it first works out how to spend the budget.
The key idea
Measure everything small, fit curves, and let the curves configure the big runs. Concretely: refit the batch-size and learning-rate laws against compute, replace 6N with non-embedding FLOPs per token, and refit the model/data split on the actual corpus, because data quality changes the optimal exponents.
What the evidence showed
The pre-training corpus reached 2 trillion tokens. Validation loss for the 7B and 67B runs landed where the curve predicted, extrapolated over roughly a 1000x compute range. DeepSeek LLM 67B surpassed LLaMA-2 70B on a range of benchmarks, with the largest gaps in code, mathematics, and reasoning, and the 67B chat model outperformed GPT-3.5 in open-ended evaluations in both Chinese and English. Cross-dump deduplication removed 89.8% of documents across 91 dumps, against 22.2% for a single dump.
Limits the authors admit
The chat model shares the familiar failure modes: no knowledge updates after pretraining, non-factual advice, and hallucination. The Chinese data is not exhaustive, so some Chinese-specific topics are weak, and because the corpus is basically Chinese and English, other languages should be treated with caution. The scaling analysis also ignores factors beyond the compute budget, and the optimal hyperparameters shift slightly between models with the same budget but different data allocations.
For the practitioner
Two transferable lessons. First, a schedule that is reusable matters: if you may extend a run, the multi-step schedule costs nothing in final loss and saves the whole early phase. Second, never borrow a scaling law; the exponents depend on your corpus, and the correct response to better data is to spend marginal compute on parameters, not more tokens.
3Implementation check
0 of 6 questions answered
1.The paper represents model scale as M (non-embedding FLOPs per token) instead of the usual 6N. What does that change?
2.The fitted exponents are a = 0.5243 and b = 0.4757. If the compute budget grows by 100x, what roughly happens to the optimal model scale and token count?
3.The hyperparameter law says eta_opt = 0.3118 * C^-0.1250 and B_opt = 0.2920 * C^0.3271. As the compute budget grows, what should you do?
4.Why did the team switch from cosine decay to a multi-step learning rate schedule?
5.Deduplicating one Common Crawl dump removed 22.2% of documents; deduplicating across 91 dumps removed 89.8%. What is the practical consequence for a pretraining pipeline?
6.When the training data gets better, how does the optimal model/data allocation shift?
4Build it yourself
Fit a Compute-Optimal Scaling Law
Intermediate3-5 hoursReproduce the paper's budgeting move at toy scale: generate synthetic IsoFLOP profiles from a hidden loss law, recover its exponents by least squares in log space, and use the fitted curve to predict the optimal model/data split at 100x the compute.
Milestones
- Implement synthetic_loss as the hidden ground-truth law and generate IsoFLOP profiles over a grid of model sizes and token budgets.
- Take logs of both sides and fit a least-squares line to recover alpha and beta from the loss-versus-N and loss-versus-D slices.
- Refit the full law (A, alpha, B, beta, E) with a small coordinate search and report the residual at every grid point.
- For each compute budget, find the loss-minimizing split and check that the best model size grows with C.
- Extrapolate to C = 1e10 from the fitted law and compare the predicted optimal N and D with the closed-form optimum.
- Add the paper's multi-step learning-rate schedule and compare its toy loss trace against a cosine decay.
Success criteria
- The refitted exponents land within 0.03 of the synthetic ground truth (alpha = 0.34, beta = 0.28).
- The predicted optimal N at C = 1e10 is within 20% of the closed-form optimum, and the run prints both numbers.
- Every grid point's residual from the refitted law is under 1% of its loss.
- The multi-step schedule finishes within 2% of cosine's final toy loss.
Stretch goals
- Refit a second synthetic corpus with different exponents and show the optimal split shifting toward the model, the paper's 0.450/0.550 to 0.524/0.476 story.
- Swap the synthetic law for a real curve: train a bigram model on the same text at several data sizes and fit loss versus tokens.
- Print a text histogram of residuals to see where the power law bends.