ReasoningPaperCore2025~18 min theory

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Reward only the final answer and the reasoning trace teaches itself: R1-Zero grows long chains of thought, self-checks, and an aha moment, with no supervised reasoning data at all.

arXiv:2501.12948

Where this sits

DeepSeek-V3 Technical ReportDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-R1-0528 ReportDeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition

The RL algorithm, GRPO, comes straight from the founding era (it was introduced in the DeepSeekMath paper, id deepseek-math), and the base model is DeepSeek-V3. R1 is the moment the two lines meet: a frontier-scale base model plus group-relative RL on verifiable rewards.

Demonstrates that reasoning behaviors (long chains of thought, self-verification, backtracking) emerge from pure RL without any supervised reasoning traces.
Reaches 71.0% pass@1 on AIME 2024 (86.7% with majority voting) from a 15.6% starting point, using only outcome and format rewards.
Reintroduces a small cold-start SFT stage plus a language-consistency reward to fix R1-Zero's readability and language mixing, yielding R1 at 79.8% on AIME 2024 and 97.3% on MATH-500.
Shows distillation beats direct RL for small models: the 32B distilled student reaches 72.6% on AIME 2024 while a 32B model trained with large-scale RL reaches 47.0%.
Releases six dense distilled models (1.5B to 70B) trained on 800k curated samples, putting strong reasoning within reach of ordinary hardware.

What it is

DeepSeek-R1 is the paper that showed reasoning can be incentivized with pure reinforcement learning on verifiable outcomes. DeepSeek-R1-Zero starts from DeepSeek-V3-Base and trains with GRPO using only rule-based rewards for answer correctness and output format, with no supervised fine-tuning stage. The follow-up model, DeepSeek-R1, adds a small cold-start dataset and a multi-stage pipeline to fix readability, then distills the resulting traces into six dense models.

1Theory from first principles

What a reasoning trace is, and why outcome rewards can work

A reasoning trace is text the model writes to itself before answering: restating the problem, sketching a plan, doing algebra, checking intermediate results, abandoning an approach and trying another. In RL terms the whole trace plus the final answer is one action sequence, and the reward function only inspects the last few tokens. That sounds too weak to teach good process, but it works in verifiable domains because the environment can check whether the final answer is correct, and the policy gradient raises the probability of every token that was present when a correct answer was produced. Over many problems, steps that tend to lead to correct answers get reinforced even though no one labeled the steps. The paper deliberately avoids a neural process reward model: the authors report that such models can be reward-hacked at scale and cost extra training cycles.

The GRPO objective R1 maximizes

J(theta) = E[ (1/G) * sum_{i=1..G} ( min( rho_i * A_i, clip(rho_i, 1 - eps, 1 + eps) * A_i ) - beta * KL(pi_theta || pi_ref) ) ]
rho_i = pi_theta(o_i | q) / pi_theta_old(o_i | q)

For each question q the old policy samples G answers; rho_i is how much more (or less) likely the current policy makes answer i. A_i is that answer's advantage (next formula). The min-and-clip term is the PPO-style safety: it stops the update from chasing one lucky answer too far, since the clipped ratio removes the incentive to move probability mass violently. The KL penalty with coefficient beta keeps the policy from drifting away from the reference model, which is the guard against degenerate text that scores well but reads badly. R1 reports beta = 0.001, a clip ratio eps of 10, learning rate 3e-6, temperature 1, and 16 outputs per question.

Group-relative advantage

A_i = (r_i - mean({r_1, ..., r_G})) / (std({r_1, ..., r_G}) + eps)

GRPO has no value network. It uses the other answers to the same question as the baseline: r_i is the reward of answer i, mean(r) is the average reward of the group, and subtracting it turns absolute rewards into 'better or worse than my siblings'. Dividing by the group's standard deviation makes the scale comparable across questions: a group with rewards 0 and 1 spreads as much as a group with rewards 0, 0.5, 1 after normalization, so easy and hard prompts contribute similar-sized updates. eps guards the division when all rewards are equal (then every advantage is 0 and the group teaches nothing).

Group advantages in ten lines

python

def group_advantages(rewards, eps=1e-8):
    n = len(rewards)
    mean_r = sum(rewards) / n
    var_r = sum((r - mean_r) ** 2 for r in rewards) / n
    std_r = var_r ** 0.5
    return [(r - mean_r) / (std_r + eps) for r in rewards]

# a math prompt verified by a rule: two of four answers are correct
print(group_advantages([1.0, 0.0, 1.0, 0.0]))
# a prompt the model always solves: no learning signal
print(group_advantages([1.0, 1.0, 1.0, 1.0]))

Rewards are just the verifier's verdict, so the same code works for math boxes and unit tests.
All-equal rewards give all-zero advantages: hard or trivial prompts contribute no gradient.
The sign, not the magnitude, is what matters most: correct answers are pushed up, wrong ones down.

The GRPO loop: sample a group of answers per question, score them with rule-based rewards, normalize within the group to get advantages, then update the policy with the clipped objective minus a KL term.

The cold-start problem and the aha moment

R1-Zero's traces are effective but often unreadable: languages mix, formatting meanders, and answers are hard to extract. R1 fixes this by starting from a few thousand curated cold-start examples that model a readable thinking format, then running reasoning RL with two extra ingredients: a format reward that requires the thinking between special tags, and a language-consistency reward equal to the proportion of words written in the target language. The paper notes this alignment slightly lowers benchmark scores but buys readability, an explicit trade. Meanwhile the reward curve itself contains the paper's most quoted phenomenon: at some intermediate checkpoint the model spontaneously starts re-examining its own steps ('wait, wait. Wait. That's an aha moment I can flag here' appears in a sample trace), and average response length climbs throughout training as the model learns to think longer.

AIME 2024 accuracy during RL: pass@1 climbs from 15.6% to 71.0% (the paper's later revision reports 77.9% for the same curve) and majority voting lifts it to 86.7%, while average response length grows in step with accuracy.

Distillation: 800k samples from R1 (about 600k reasoning plus 200k non-reasoning) are used to fine-tune six dense students. The 32B student scores 72.6% on AIME 2024 versus 47.0% for a 32B base model trained with large-scale RL, and the 7B student (55.5%) beats QwQ-32B-Preview (44.0%).

2Inside the paper

Problem the paper sets out to solve

Reasoning capability in LLMs had been bought with human-written reasoning traces. Can it instead be incentivized by outcome rewards alone, without any supervised reasoning data, and what does that change about how models are built?

Key idea to look for while reading

Two experiments in one paper. Section 2 is R1-Zero: pure RL from the base model, rule-based rewards, no SFT. Section 3 is R1: cold start, reasoning RL, rejection sampling plus SFT, then a final RL stage with model-based preference rewards for helpfulness and safety. Section 4 distills R1 into smaller dense models.

Evidence the paper must muster

R1-Zero on AIME 2024 goes from 15.6% pass@1 to 71.0%, and 86.7% with majority voting, with no supervised reasoning data. R1 scores 79.8% on AIME 2024, 97.3% on MATH-500, 71.5% on GPQA Diamond, 90.8% on MMLU, a 2029 Elo rating on Codeforces (96.3rd percentile of human competitors), and 49.2% on SWE-bench Verified. The distilled students cover 1.5B, 7B, 8B, 14B, 32B, and 70B parameters: the 1.5B student reaches 28.9% on AIME 2024 and 83.9% on MATH-500, and the 32B student reaches 72.6%, 94.3%, and 57.2% on LiveCodeBench. Distillation clearly beats RL on small base models in the paper's own comparison.

Limits the authors admit

The rule-based RL stage is narrow: it targets math, code, and logic with checkable answers, and R1-Zero is weak on writing and open-domain QA (its instruction-following and writing scores trail the instruct models of the time). Language consistency slightly hurts benchmark accuracy. The cold-start and rejection-sampling stages depend on the existing DeepSeek-V3 pipeline. The paper is also upfront that a neural reward model was rejected on reward-hacking grounds, which leaves general, non-verifiable reasoning as unfinished business.

Practitioner takeaway

If your task has a verifier, you can teach process with outcome rewards, and you probably do not need a process reward model. Use a group-relative baseline to avoid training a critic, keep a KL leash to a reference model, and seed readability with a small curated cold start. When compute is short, distill a strong reasoner into the model size you can actually serve.

3Implementation check

0 of 5 questions answered

1.Why can DeepSeek-R1-Zero forgo SFT entirely?
2.What does the KL term in the GRPO objective guard against?
3.Two prompts yield reward groups [0, 0, 0, 1] and [0.4, 0.5, 0.5, 0.6]. Why does GRPO divide by the group standard deviation?
4.What is the cold-start dataset for?
5.The paper rejects a neural process reward model for R1-Zero. What reason does it give?

4Build it yourself

GRPO vs REINFORCE on a Verifiable Bandit

Intermediate3-5 hours

Implement group-relative advantages on a contextual bandit with 0/1 rule rewards, then run the same sample budget through REINFORCE with a running baseline. The toy isolates the variance reduction the group baseline buys without a critic.

Milestones

Build the bandit: six contexts, four arms, one correct arm per context, and a 0/1 reward rule.
Write softmax sampling from per-context logits.
Implement grpo_update: standardize the group's rewards into advantages and apply the score-function update.
Implement reinforce_update with a running mean baseline.
Run both methods for the same 1600 sampled actions across five seeds.
Average rewards and confirm GRPO leads on every seed, not only on average.

Starter code

python · editable

import math, random
BEST = [0, 2, 1, 3, 2, 0]
ARMS, CONTEXTS, GROUP, BUDGET, SEEDS = 4, 6, 8, 1600, 5
def reward(ctx, arm, rng):
    return 1.0 if rng.random() < (0.9 if arm == BEST[ctx] else 0.1) else 0.0

def softmax(logits):
    top = max(logits)
    exps = [math.exp(x - top) for x in logits]
    total = sum(exps)
    return [x / total for x in exps]

def sample(probs, rng):
    r, acc = rng.random(), 0.0
    for i, p in enumerate(probs):
        acc += p
        if r < acc:
            return i
    return len(probs) - 1

def grpo_update(logits, ctx, rewards, actions, probs, lr=0.1):
    # TODO: turn rewards into group-relative advantages, then update each
    # sampled action by lr * advantage * (onehot - probs).
    return logits

def reinforce_update(logits, ctx, action, r, baseline, probs, lr=0.1):
    # TODO: single-sample update with advantage r - baseline.
    return logits

def train(kind, seed):
    rng = random.Random(seed)
    logits = [[0.0] * ARMS for _ in range(CONTEXTS)]
    total, b_sum, b_n, batch = 0.0, 0.0, 0, GROUP if kind == "grpo" else 1
    for _ in range(BUDGET // batch):
        ctx = rng.randrange(CONTEXTS); probs = softmax(logits[ctx])
        # TODO: sample a batch of actions, score each with reward(), update
        # the matching policy, and track a running baseline for REINFORCE.
    return total / BUDGET

def main():
    for kind in ("grpo", "reinforce"):
        mean = sum(train(kind, s) for s in range(SEEDS)) / SEEDS
        print(kind, "mean reward:", round(mean, 3))
    # EXPECTED: after the TODOs, grpo ~0.78 and reinforce ~0.71 over 5 seeds.
main()

Runs locally in your browser.

Success criteria

GRPO mean reward is at least 0.05 above REINFORCE over five seeds.
GRPO wins on all five seeds in the printed output.
Setting the group size to 1 zeroes every advantage, so the policy stops improving.

Stretch goals

Add a cold-start phase that pretrains logits on a few correct trajectories, then run RL.
Swap the bandit for a two-state gridworld with terminal reward and repeat the comparison.
Add a KL penalty toward the initial logits and trace the reward-drift tradeoff.

Problems

5Keep going

previousDeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search nextCodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

Problems

Articles

Post-Training: RLHF → DPO → GRPO

Where this sits

What it is

1·Theory from first principles

What a reasoning trace is, and why outcome rewards can work

The cold-start problem and the aha moment

2·Inside the paper

Problem the paper sets out to solve

Key idea to look for while reading

Evidence the paper must muster

Limits the authors admit

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going