EfficiencyPaperAdvanced2024~15 min theory

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Continue-pretrain a general MoE model on code and math, and open-source code intelligence reaches closed-source territory.

arXiv:2406.11931

Where this sits

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelDeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

A case study in specialization. It continues the DeepSeek-Coder line of code models, but instead of training from scratch it resumes from an intermediate 4.2T-token checkpoint of DeepSeek-V2 and spends 6T mostly-code tokens moving capability into one domain without giving up general language skill.

Total pretraining exposure grows to 10.2T tokens (4.2T inherited plus 6T new), with the new mix 60% source code, 10% math, and 30% natural language.
Language coverage expands from 86 to 338 programming languages and context from 16K to 128K tokens, using YaRN in two 1000-step extension stages.
The 16B (2.4B active) model beats the older DeepSeek-Coder-33B, and the 236B model reaches 90.2% HumanEval, 76.2% MBPP+, 43.4% LiveCodeBench, and 75.7% MATH.
It becomes the first open-source model past 10% on SWE-bench (12.7%) and tops the Aider editing benchmark at 73.7%, above GPT-4o's 72.9%.
The recipe details matter: fill-in-the-middle at rate 0.5 for the 16B variant, and a reversion from exponential normalization to standard normalization to stop gradient spikes.

What it is

DeepSeek-Coder-V2 is an open-source MoE code model built by resuming DeepSeek-V2 from an intermediate checkpoint and training on 6T additional tokens of code, math, and language. It ships in 236B (21B active) and 16B (2.4B active) sizes, supports 338 programming languages and a 128K context window, and reaches closed-source-level scores on several coding benchmarks, including 90.2% on HumanEval for the instruct model. The paper documents the data pipeline, the two-stage context extension, and the GRPO alignment that follows.

1Theory from first principles

A model is a summary of its training text

Pretraining compresses the statistics of a huge corpus into weights, so when the corpus changes, the weights are wrong in a specific way. A general model has seen plenty of prose and some code, but not the long-range structure of repositories, the grammar of rare languages, or the shape of an edit that closes a function. You can train a code model from scratch, as DeepSeek-Coder 33B did, or start from a model that already understands language and specialize it. The second route is cheaper, and this paper shows it can also be stronger.

Specialization is a data problem first

The new corpus is 6T tokens: 60% source code, 10% math, 30% natural language. Code comes from public GitHub repositories created before November 2023, filtered and near-deduplicated, plus code-related web text recalled with a fastText classifier seeded from StackOverflow and documentation sites. The result is 1,170B code tokens covering 338 languages, up from 86. Math adds 221B tokens, roughly doubling the DeepSeekMath corpus. Keeping 30% natural language is deliberate: it defends general ability while the model learns to code.

The data pipeline: raw GitHub and Common Crawl, filtering and deduplication, classifier-based domain recall, then a 60/10/30 mix that yields 1,170B code tokens in 338 languages plus 221B math tokens.

Fill-in-the-middle as a sequence

PREFIX | <fim_hole> | SUFFIX | <fim_end> | MIDDLE
loss = -log P(MIDDLE | PREFIX, SUFFIX)

FIM trains the model to complete a gap given both sides, which is the shape of real editor completions. The 16B variant trains with FIM at rate 0.5; the 236B variant uses next-token prediction only.

Long context, in two stages

Code needs context because definitions live far from their uses. The model starts at a 16K window and is extended to 128K with YaRN (scale 40, alpha 1, beta 32). The extension runs two stages of 1000 steps each: first at 32K sequence length with batches of 1152 sequences, then at 128K with batches of 288, keeping roughly the same number of tokens per stage while upsampling long documents. Needle-in-a-haystack checks pass across the full window.

16K to 128K in two 1000-step stages, alongside the jump from 86 to 338 supported languages, on a total pretraining budget of 10.2T tokens.

Context stages and the token mix

python

def stage_tokens(seq_len, batch_seqs, steps):
    return seq_len * batch_seqs * steps

stage_one = stage_tokens(32_768, 1152, 1000)
stage_two = stage_tokens(131_072, 288, 1000)
print(stage_one == stage_two)  # about 37.7B tokens each

def token_mix(total):
    return {"code": 0.60 * total, "math": 0.10 * total, "language": 0.30 * total}

mix = token_mix(6_000_000_000_000)
print(mix["code"])  # 3.6T code tokens

Both extension stages spend roughly 37.7B tokens, so the context window quadruples without a token-budget spike.
The 4.2T-token V2 checkpoint plus the 6T new tokens gives the 10.2T total reported for both model sizes.

What the specialization buys, and what it costs

The instruct model scores 90.2% on HumanEval, 76.2% on MBPP+, 43.4% on LiveCodeBench, and 75.7% on MATH, and solves 4 of 30 AIME 2024 problems (5 of 30 with maj@64, a majority vote over 64 samples). On the Aider editing benchmark it reaches 73.7%, above GPT-4o's 72.9%, and its 12.7% on SWE-bench is the first open-source result above 10%. Two honest notes from the paper: the reasoning gap on CRUXEval is attributed to having only 21B active parameters, and the run suffered gradient spikes that the authors traced to exponential normalization and fixed by returning to conventional normalization.

2Inside the paper

Problem

Open code models trail the best closed models on coding and math tasks, and training a hundred-billion-parameter code model from scratch is expensive. The question is whether specialization can be bought cheaply by continuing pretraining instead.

Key idea

Resume DeepSeek-V2 from an intermediate 4.2T-token checkpoint and train 6T more tokens that are mostly code and math but keep 30% natural language. Keep the MLA plus DeepSeekMoE architecture unchanged, extend context to 128K, then align with SFT and GRPO using compiler and test-case signal, with a reward model to smooth noisy binary feedback.

Evidence

A 1B proxy ablation shows the new corpus adds 6.7 points on HumanEval and 9.4 points on MBPP over the old DeepSeek-Coder corpus (30.5% to 37.2% and 44.6% to 54.0% at 2T tokens). The 16B/2.4B-active model outperforms the older 33B dense code model; the 236B model posts 90.2% HumanEval, 76.2% MBPP+, 43.4% LiveCodeBench, 75.7% MATH, 12.7% SWE-bench, and 73.7% Aider. The 16B FIM variant averages 86.4% on single-line infilling across Python, Java, and JavaScript.

What the authors admit

Specialization has a price. General-language scores dip slightly against the base V2 (TriviaQA 82.3 versus 86.7, NaturalQuestions 47.5 versus 53.4), and the CRUXEval gap is attributed to the 21B active-parameter budget rather than the data. The training run was also not frictionless: gradient spikes forced the team to revert an exponential normalization scheme.

Practitioner takeaway

Continued pretraining on a domain mix is the cheapest path to specialization, but the mix is the product decision: keep enough general data to avoid regressing, plan a long-context stage if your task needs it, add FIM if you want an editor-grade completion model, and budget for alignment with execution-based feedback.

3Implementation check

0 of 5 questions answered

1.The 236B instruct model was produced by continuing from which starting point?
2.Why does the data mix keep 30% natural language when the goal is code?
3.What does fill-in-the-middle change about the training objective?
4.Both long-context stages train for 1000 steps. How do they differ?
5.Which claim is supported by the paper's numbers?

4Build it yourself

Fill in the middle with a tiny n-gram model

Starter2-3 hours

Build the fill-in-the-middle pipeline DeepSeek-Coder-V2 trains on: reorder documents into prefix, hole, and suffix, train a small counting model on the reordered stream, and measure how often it completes the hole correctly against a plain left-to-right model trained on the same corpus. The corpus is a synthetic motif language where the middle token is predictable only when both neighbors are visible.

Milestones

Generate a synthetic corpus by concatenating random motifs, so tokens have local structure and some positions are ambiguous from one side only.
Hold out a test split and compute majority-token accuracy as a floor.
Implement the FIM transform: cut a one-token hole at a random interior position and emit (previous token, next token) -> middle.
Train a left-to-right baseline that counts (previous) -> middle pairs from the original documents.
Train the FIM model that counts (previous, next) -> middle pairs from the transformed data.
Evaluate both on held-out holes and report exact-match accuracy.
Widen the hole to 2 or 3 tokens and check how accuracy changes.

Starter code

python · editable

import random
from collections import Counter
random.seed(7)
# EXPECTED before TODOs: 400 train docs, 1600 holes, majority accuracy 0.259.
# EXPECTED after TODOs: left-to-right 0.418, fim 0.644.

MOTIFS = ["abc", "adb", "cda", "cbd"]

def make_corpus(n_docs, rng):
    return ["".join(rng.choice(MOTIFS) for _ in range(6)) for _ in range(n_docs)]

def fim_sample(doc, rng):
    """TODO: pick 1 <= i < len(doc) - 1 and return ((doc[i-1], doc[i+1]), doc[i])."""
    raise NotImplementedError("TODO: fim_sample")

def train(rows):
    """TODO: rows are (context_tuple, middle) pairs; return a dict of Counters."""
    raise NotImplementedError("TODO: train")

def predict(counts, context):
    """TODO: return the most common middle for a context, or None."""
    raise NotImplementedError("TODO: predict")

def accuracy(counts, holes, use_suffix):
    hits = 0
    for prev, mid, nxt in holes:
        context = (prev, nxt) if use_suffix else (prev,)
        if predict(counts, context) == mid:
            hits += 1
    return hits / len(holes)

train_docs = make_corpus(400, random)
test_docs = make_corpus(100, random)
holes = [(d[i - 1], d[i], d[i + 1]) for d in test_docs for i in range(1, len(d) - 1)]
print("train docs:", len(train_docs), "test holes:", len(holes))
print("majority-token accuracy:",
      round(Counter(h[1] for h in holes).most_common(1)[0][1] / len(holes), 3))
try:
    left_to_right = train([((d[i - 1],), d[i]) for d in train_docs for i in range(1, len(d))])
    fim = train([fim_sample(d, random) for d in train_docs])
    print("left-to-right accuracy:", round(accuracy(left_to_right, holes, False), 3))
    print("fim accuracy:", round(accuracy(fim, holes, True), 3))
except NotImplementedError as exc:
    print("unimplemented:", exc)

Runs locally in your browser.

Success criteria

The held-out holes number at least 1000 and come from documents the models never trained on.
The FIM model beats the left-to-right baseline by at least 10 accuracy points.
Both accuracies and the majority-token floor are printed from one run with a fixed seed.
The hole is chosen uniformly from interior positions, not hand-picked.

Stretch goals

Score completions by mean negative log-likelihood instead of exact match and compare the two models on that scale.
Add the paper's token mix: dilute the code corpus with 30% prose-like sequences and show which model degrades less.
Extend the counting model to order 3 and find where more context stops helping on held-out documents.

5Keep going

previousDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model nextLet the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Where this sits

What it is

1Theory from first principles

A model is a summary of its training text

Specialization is a data problem first

Long context, in two stages

What the specialization buys, and what it costs

2Inside the paper

Problem

Key idea

Evidence

What the authors admit

Practitioner takeaway

3Implementation check

4Build it yourself

Fill in the middle with a tiny n-gram model

Milestones

Success criteria

Stretch goals

5Keep going

Concepts

Problems

Articles

Where this sits

What it is

1·Theory from first principles

A model is a summary of its training text

Specialization is a data problem first

Long context, in two stages

What the specialization buys, and what it costs

2·Inside the paper

Problem

Key idea

Evidence

What the authors admit

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

5·Keep going

Concepts

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going