DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence
A code model family trained from scratch on 2 trillion tokens, with repository structure and fill-in-the-middle objectives built into pretraining.
arXiv:2401.14196Where this sits
Third paper of the founding era. It takes the DeepSeek LLM trunk and specializes it on code; the base checkpoint it produces is then the starting point for DeepSeekMath.
- Repository-level data assembly: file dependencies are parsed and topologically sorted so a training sample sees a file's imports before the file, and near-deduplication happens per repository instead of per file.
- Fill-in-the-middle pretraining at a 50% rate in PSM order, chosen from an ablation that measured the trade-off between infilling and ordinary code completion.
- A 16K-token context window, obtained by RoPE linear scaling (factor 1 to 4, base 10000 to 100000) plus 1000 extra training steps at 16K.
- Openly released weights from 1.3B to 33B under a permissive license, where DeepSeek-Coder-Instruct 33B beat GPT-3.5-Turbo on HumanEval and the 6.7B model matched CodeLlama 34B.
- A decontamination pass that removes code containing 10-gram matches against HumanEval, MBPP, GSM8K, and MATH.
What it is
DeepSeek's code models, from 1.3B to 33B parameters, trained from scratch on 2 trillion tokens that are 87% source code across 87 programming languages. Two design choices separate it from earlier code models: the corpus is assembled at repository level, ordering files so dependencies come before their dependents, and half the training documents are rearranged with fill-in-the-middle sentinels so the model learns to write code from the signature above it and the code below it. It reached state of the art among open code models and beat Codex and GPT-3.5 on prominent benchmarks.
1Theory from first principles
Why code completion needs a different objective
Autocomplete in an editor is not next-token prediction from the start of a file. A developer types inside an existing function with code above and code below, and the model must produce what belongs in the gap. Plain left-to-right training gives the model no practice with a known suffix, and file-level pretraining gives it no sense of a project's structure: which imports exist, which module defines the helper two files away. DeepSeek-Coder attacks both gaps in the data pipeline rather than in the architecture.
Fill-in-the-middle in PSM order
<fim_start> f_pre <fim_hole> f_suf <fim_end> f_midA document is cut into prefix, middle, and suffix. In PSM (prefix-suffix-middle) order the prefix comes first, then the suffix behind a hole sentinel, then the middle behind an end sentinel. Loss is computed on f_mid only, so the model learns to generate the missing span from the prefix and the suffix together. SPM (suffix-prefix-middle) is the same idea with suffix and prefix swapped; the paper chooses PSM.
import random
def build_fim(doc, rate=0.5, rng=None):
# Half of all documents get the PSM rearrangement.
rng = rng or random.Random(0)
if rng.random() > rate or len(doc) < 3:
return doc
p = rng.randrange(1, len(doc))
s = rng.randrange(p, len(doc))
prefix, middle, suffix = doc[:p], doc[p:s], doc[s:]
return ("<fim_start>" + prefix + "<fim_hole>" +
suffix + "<fim_end>" + middle)- Three sentinel tokens mark the hole, the end of the context, and the start of the answer; the model is supervised only on the middle segment.
- At inference you place the hole where the missing line belongs and let the model fill it, which is why the ordering at train time has to match the prompt format at serve time.
- The ablation ran a 1.3B model on Python at 0%, 50%, and 100% FIM: 100% maximized infilling but produced the weakest completion, so 50% PSM became the default.
Repository-level assembly, step by step
Start with public repositories created before February 2023 and keep 87 programming languages; StarCoder-style filters (line length, alphabetic character ratio, visible-text ratio for HTML, size bounds for JSON and YAML) cut the raw crawl to 32.8% of its original size. Next, parse invocation relationships with regular expressions (import, using, include) and sort the files of each repository so dependencies precede dependents; because real projects contain cycles, the sort picks the minimum in-degree node instead of requiring zero. A comment with the file path is prepended to every file, then files are concatenated in dependency order into one training sample. Deduplication runs on the concatenated repository, not on individual files, so dedup cannot delete half a project and break the structure the model is supposed to learn.
The numbers behind the corpus
After cleaning, the code corpus is 798 GB across 603 million files; it covers 87 languages, from Java and Python down to exotic ones, with the largest slices in Java, Python, C++, C#, JavaScript, and TypeScript. The training mixture is 87% source code, 10% English code-related text such as GitHub Markdown and StackExchange, and 3% Chinese natural language, a deliberate bet that natural-language understanding of code concepts matters for instructions and bug reports. The tokenizer is BPE with a 32,000-token vocabulary, smaller than the general model's because code token counts dominate. An n-gram filter removes files containing 10-gram matches against HumanEval, MBPP, GSM8K, and MATH, and exact-matches shorter test strings of at least 3 grams.
The ablation that picked 50%
The FIM section is a model of how to choose a pretraining knob. Using a 1.3B model on a Python subset, the authors compare FIM rates of 0%, 50%, and 100%, plus a masked-span variant. The result is a clean trade-off: 100% FIM peaks on the HumanEval-FIM infilling benchmark but is the worst at ordinary code completion, because every document is rearranged and the prefix-to-continuation statistics shift. The 50% PSM setting best balances the two abilities, and the masked-span strategy underperforms plain PSM at the same rate. When you read the benchmark tables later, remember that this single number governs the model's most distinctive skill.
Extending the window, and what to watch for
The models train on 4K context, then get 1000 extra steps at 16K with RoPE's linear scaling factor raised from 1 to 4 and its base frequency from 10000 to 100000. The authors say the modification theoretically supports 64K but that the model is most reliable inside 16K; treat advertised context as a ceiling and measured reliability as the real number. One caution appears in the paper itself: on the LeetCode contest benchmark, scores are higher for contests close to the data collection window, and the authors admit contamination cannot be fully ruled out. That is the kind of honest footnote worth internalizing, because it applies to every benchmark you will meet later.
2Inside the paper
The problem it attacks
Strong code models were closed. Open models were trained on file-level code with next-token prediction only, so they had neither project context nor infilling skill, and the gap to GPT-3.5 looked structural rather than a matter of scale.
The key idea
Treat the repository as the unit of data and infilling as a first-class objective. Order files by dependency before concatenation, deduplicate whole repositories, and rearrange half the documents into prefix-suffix-middle form with sentinel tokens.
Evidence
Base 33B reached 56.1% pass@1 (the fraction of problems solved by one generated program) on HumanEval Python and 66.0% on MBPP, with a multilingual average of 50.3% against CodeLlama 34B's 41.0%. Base 6.7B matched CodeLlama 34B overall. Instruct 33B reached 79.3% on HumanEval Python, above GPT-3.5-Turbo's 76.2%, and was the only open model to beat GPT-3.5-Turbo on the LeetCode contest set (27.8% versus 23.3%). On the single-line infilling benchmark, the 7B model averaged 80.7% against StarCoder's 69.7%. A v1.5 model was later produced by continuing from a DeepSeek LLM 7B checkpoint on 2B mixed tokens, improving natural language without losing code skill.
What the authors admit
The LeetCode comparison may be contaminated, and the authors say so rather than hiding it. The 1.3B model is far behind its larger siblings, and the reliable context is 16K even though 64K is theoretically reachable. The strongest results also come from instruction-tuned models, which is a different comparison than the base-model tables that dominate the paper.
For the practitioner
If you train a code model, the objective layout at training time must match the prompt layout at inference time; a mismatch quietly wastes the whole FIM investment. Order matters inside the sample, so preserve file boundaries and paths. And when you evaluate on a public benchmark, assume contamination until proven otherwise, including in your own runs.
3Implementation check
0 of 5 questions answered
1.The FIM ablation found that a 100% FIM rate maximized the infilling score. Why did the authors still choose 50%?
2.In PSM order, the document becomes <fim_start> prefix <fim_hole> suffix <fim_end> middle. Where should the cross-entropy loss be computed?
3.Why does the paper deduplicate at repository level instead of file level?
4.The dependency sort concatenates files so that a file's imports appear before it. Real repositories contain import cycles, so the algorithm does not require zero in-degree. What does it do instead?
5.How did the models get to 16K context, and what was the actual result?
4Build it yourself
Build a Repository-Level FIM Pipeline
Starter2-3 hoursImplement the paper's data pipeline for real: shingle-based repository fingerprints, near-duplicate detection that drops whole projects instead of files, dependency-ordered concatenation, and PSM fill-in-the-middle examples.
Milestones
- Turn every file into n-gram shingles and compute Jaccard similarity between files and between repositories.
- Implement repository-level dedup so a project edited in a few places is dropped while unrelated projects survive.
- Implement a relaxed topological sort that emits the remaining file with the fewest unmet dependencies at each step.
- Concatenate files in dependency order behind a path comment and raise the FIM rate to 50% in PSM order.
- Check that the hole and end sentinels land between the prefix and the moved middle span, then measure the observed FIM rate over 1000 documents.
Success criteria
- Repo-level dedup drops the edited near-duplicate repository and keeps the unrelated one, printing the dropped index.
- Dependency order places every import before its consumer on a sample project that contains a cycle.
- Over 1000 synthetic documents the observed FIM rate is within 0.03 of the requested 0.5.
Stretch goals
- Swap Jaccard shingles for MinHash and compare runtime and recall on 10,000 synthetic files.
- Score files the way the paper's filters do (line length, alphabetic ratio) and reject low-quality files before shingling.
- Round-trip an infilling example: hide a function body, rebuild the prompt, and assert the middle comes back.