DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
A 7B math model built on web data, plus GRPO (Group Relative Policy Optimization): the critic-free RL method that later powered DeepSeek's reasoning era.
arXiv:2402.03300Where this sits
Fourth paper of the founding era. It shows that code training transfers to mathematical reasoning and introduces GRPO, the algorithm the reasoning era builds on.
- A web-scale math corpus: 120.2B tokens and 35.5M web pages selected from Common Crawl by an iteratively retrained fastText classifier, roughly 7 times the math web pages Minerva used.
- Evidence that code pretraining improves mathematical reasoning, with and without tools, and that arXiv papers, contrary to common practice, brought no notable benchmark gains.
- GRPO, a PPO variant that drops the critic and estimates the baseline from group rewards, cutting the memory cost of RL fine-tuning.
- A unified account of SFT (supervised fine-tuning), rejection sampling, DPO (direct preference optimization), PPO (proximal policy optimization), and GRPO as direct or simplified forms of one objective, distinguished by their gradient coefficients.
- Strong results on a 7B budget: 51.7% on MATH with chain-of-thought and no voting, 60.9% with self-consistency over 64 samples, and out-of-domain gains such as CMATH 84.6% to 88.8%.
What it is
Two contributions in one paper. Starting from DeepSeek-Coder-Base-v1.5 7B, the authors continue pretraining on 120B math tokens filtered out of Common Crawl and reach 36.2% on the MATH benchmark as a base model, then 46.8% after instruction tuning. Reinforcement learning with GRPO lifts that to 51.7%, close to Gemini Ultra and GPT-4 at the time. GRPO itself is the lasting idea: instead of training a value network to estimate advantages, it samples a group of answers to the same question and scores each answer relative to the mean and standard deviation of its group.
1Theory from first principles
Why math is a hard test case
Mathematical reasoning is verifiable: the final answer is right or wrong, so you can build a reward signal without collecting human preferences. It is also unforgiving, because a single dropped sign breaks a long chain of steps. Before this paper, open models trailed closed ones badly on MATH, and the common assumption was that web text had little to offer. DeepSeekMath challenges both halves of that assumption: first, mine the web with a classifier sharp enough to find math; second, reward the model for producing correct final answers and let it discover the reasoning.
The data pipeline, in plain language
Seed the loop with OpenWebMath, a small high-quality collection. Train a fastText classifier (a lightweight text-classification library) on 500K positive pages from the seed and 500K random Common Crawl negatives. Score the web, keep the top pages, then look at which domains contributed heavily (over 10% of a domain's pages collected marks it as math-related), have humans annotate more URLs inside those domains, and retrain. Four iterations produced 35.5M pages and 120.2B tokens; the fourth pass recovered almost nothing new, so collection stopped. Deduplication by URL and near-duplicates shrank the raw crawl to 40B HTML pages before scoring, and a 10-gram filter removes benchmark questions and answers.
Group-relative advantage
A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)For one question, the policy samples G answers. Each answer gets a reward r_i from the reward model, and A_i is how much better or worse that answer was than its siblings. Subtracting the group mean removes the question's overall difficulty, and dividing by the group standard deviation removes how spread out the group happened to be, so one question cannot dominate the gradient simply because its rewards vary more.
import math
def group_advantages(rewards, eps=1e-8):
n = len(rewards)
mean = sum(rewards) / n
var = sum((r - mean) ** 2 for r in rewards) / n
std = math.sqrt(var)
if std < eps:
return [0.0] * n
return [(r - mean) / (std + eps) for r in rewards]
def clipped_objective(ratios, advantages, eps=0.2):
total = 0.0
for ratio, adv in zip(ratios, advantages):
clipped = max(1.0 - eps, min(1.0 + eps, ratio))
total += min(ratio * adv, clipped * adv)
return total / len(ratios)- A degenerate group (every sample scores the same) carries no signal, so its advantages are zeroed instead of dividing by a tiny number.
- ratio is the new policy's probability of a token divided by the old policy's; eps bounds how far one update may move that ratio.
- The min of the unclipped and clipped products is what makes the update a trust region: if moving the ratio further would only help, the clipped term wins and the gradient shrinks.
What GRPO removes, and why it matters
PPO, the standard RL method for language models, trains a value network alongside the policy to estimate advantages, which roughly doubles the memory and adds a moving target to debug. GRPO replaces that value network with the group mean: the baseline for an answer is simply how its siblings scored. The KL (Kullback-Leibler) penalty also moves: PPO shapes per-token rewards with a reference-model term, while GRPO adds the KL divergence between policy and reference directly to the objective, using an unbiased estimator that stays positive. In the paper's runs the KL coefficient is 0.04, the policy learning rate 1e-6, and each question samples 64 outputs, so a group is large enough for the mean and standard deviation to mean something.
The GRPO objective
J = (1/G) * sum_i (1/|o_i|) * sum_t min(ratio * A_i, clip(ratio, 1-eps, 1+eps) * A_i) - beta * KL(pi || pi_ref)G is the number of sampled answers o_i for one question; |o_i| is an answer's token count, so longer answers do not dominate. ratio is pi(o_i,t | q, o_i,<t) / pi_old(...) for the t-th token, A_i is the group-relative advantage, eps bounds the ratio, and beta weights the KL term against the reference model. The first term in the min is the plain policy gradient; the second cancels the gain from moving the ratio outside the trust region.
What the RL runs showed
RL ran on 144K chain-of-thought questions drawn only from GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Both benchmarks improved despite the training data being a subset of what instruction tuning already used, and out-of-domain benchmarks improved too, which is the interesting part: the model did not merely memorize the two training distributions, it got better at math in general. Self-consistency over 64 samples (sample 64 answers and take the majority) reaches 60.9% on MATH, showing the tuned model's answer distribution contains more correct solutions than a single greedy decode reveals. The paper also reports iterative GRPO, where the reference model is reset to the current policy and the reward model continues training on replayed and fresh data; the details are in Algorithm 1 and worth reading alongside the code above.
Two surprising findings and what to watch
First, code pretraining helps math: starting from a code model beats starting from a general LLM, both with and without tools. Second, contrary to common practice, adding arXiv papers brought no notable improvement on these benchmarks, likely because the pipeline's web pages already covered the useful mathematical text. When reading, separate the two stories. The data story is a pipeline paper with careful ablations. The algorithm story is the one that mattered later, and it is short, with the entire GRPO description fitting in a few pages.
2Inside the paper
The problem it attacks
Open models were far behind closed models on competition mathematics. The standard assumptions were that web text was too noisy to teach math and that reinforcement learning for language models required an expensive value network.
The key idea
Mine web math with an iteratively improved classifier, then improve reasoning with a critic-free RL algorithm that uses the average reward of a sampled group as the baseline. Both halves are about extracting signal from cheap resources: web pages instead of curated textbooks, group statistics instead of a learned value function.
Evidence
DeepSeekMath-Base 7B reaches 64.2% on GSM8K and 36.2% on MATH, ahead of Minerva 540B on both despite being 77 times smaller. Instruction tuning brings MATH to 46.8% and GSM8K to 82.9%. GRPO then lifts them to 51.7% and 88.2%, with 60.9% on MATH under self-consistency over 64 samples. The 51.7% figure, without tools or voting, approaches Gemini Ultra and GPT-4 on the same benchmark (53.2 and 52.9). A 1.3B controlled study shows the DeepSeekMath Corpus produces a steeper, longer-lasting learning curve than MathPile, OpenWebMath, or Proof-Pile-2.
Limits the authors admit
Geometry and theorem proving are weak spots: the authors report the model failing on triangle and ellipse problems, which they attribute to selection bias in the pretraining and fine-tuning data. Few-shot prompting barely helps; performance is similar zero-shot and few-shot, unlike GPT-4. The RL gains are measured on two in-domain benchmarks, and the out-of-domain improvements, while consistent, are smaller.
For the practitioner
GRPO is small enough to implement from the equations, and the two details people get wrong are the degenerate group (guard the standard deviation) and the trust region (keep the clipped minimum, not just the ratio times advantage). On the data side, the transferable trick is domain annotation: let the classifier point at the domains it likes, then let humans label within them, and repeat. That loop is what turns a noisy web into 120B usable tokens.
3Implementation check
0 of 6 questions answered
1.In GRPO, why divide the centered rewards by the standard deviation of the group?
2.What replaces the critic (value) network in GRPO?
3.How does GRPO handle the KL penalty relative to PPO?
4.Why does GRPO sample a group of outputs for each question instead of one?
5.A group of 64 sampled answers all receive the same reward. What do the group-relative advantages become, and what should the implementation do?
6.The paper reports 51.7% on MATH single-sample and 60.9% with self-consistency over 64 samples. What does the gap indicate?
4Build it yourself
Make GRPO Beat REINFORCE on a Bandit
Intermediate3-5 hoursImplement group-relative advantages and the clipped surrogate on a small multi-armed bandit, then compare against REINFORCE at an equal number of reward pulls.
Milestones
- Implement the softmax policy and the sampler, then check empirical arm frequencies against the policy's probabilities.
- Implement REINFORCE with a plain score-function update and record its reward after 2000 pulls.
- Implement group sampling and group_advantages with the epsilon guard for zero-variance groups.
- Update the logits with advantage * (onehot - probs) per sampled arm and match REINFORCE's pull budget.
- Add the clipped ratio objective and run several inner epochs per group, tracking the KL from the sampling policy.
- Average both algorithms over 20 seeds and print the comparison table.
Success criteria
- At 2000 reward pulls, GRPO's mean expected reward over 20 seeds exceeds REINFORCE's over the same seeds (reference: 0.846 versus 0.840).
- A group with identical rewards yields all-zero advantages and leaves the policy unchanged.
- With clipping on and multiple inner epochs, the measured KL from the sampling policy stays bounded; the run reports the maximum.
Stretch goals
- Move from a bandit to toy arithmetic: sample short expressions, reward exact answers, and parse the final answer with a string rule.
- Ablate the standard-deviation normalization and report which seeds become unstable.
- Print a text learning curve of reward against pull count for both algorithms.