Skip to content
DeepForgeby svx
FrontierPaperAdvanced2025~13 min theory

DeepSeek-OCR: Contexts Optical Compression

Render text as an image and let a 570M-active MoE read it back: ten text tokens per vision token is near-lossless, twenty is still 60%.

arXiv:2510.18234

Where this sits

DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal UnderstandingDeepSeek-OCR: Contexts Optical CompressionDeepSeek-OCR 2: Visual Causal Flow

The multimodal line's most utilitarian branch. Where VL2 and Janus aim at general understanding or generation, DeepSeek-OCR uses the same ingredients - a window-attention perception encoder, a global-attention semantic encoder, a small MoE decoder - to attack the cost of long text, and it is the first DeepSeek release whose subject is context compression rather than a capability benchmark.

  • Introduces DeepEncoder: SAM-base window attention for perception, a 16x convolutional compressor, and CLIP-large global attention for knowledge, keeping activation memory low at high resolution.
  • Compresses a 1024 x 1024 page from 4096 patch tokens to 256 vision tokens before the language decoder ever runs.
  • Quantifies the trade curve rather than a single point: 96%+ OCR precision at 9-10x text-to-vision compression, about 90% at 10-12x, and about 60% at 20x.
  • Beats GOT-OCR2.0 on OmniDocBench with 100 vision tokens against its 256, and MinerU2.0 with fewer than 800 against its 6000-plus per page.
  • Ships as a practical data engine: over 200k pages per day on one A100-40G, with code and weights open.

What it is

DeepSeek-OCR is a two-part system built to test one hypothesis: that vision can serve as a compression medium for text. The encoder, DeepEncoder, pairs a SAM-base block that reads patches with window attention against a CLIP-large block that adds global context, bridged by a 16x convolutional compressor, so a 1024 x 1024 page becomes 4096 patch tokens and then 256 vision tokens while activations stay small. The decoder is a 3B mixture-of-experts model with only 570M parameters active. The experiments measure how much text can be recovered from how few vision tokens: near 97% OCR precision at compression ratios under 10x, about 60% at 20x. On OmniDocBench it beats GOT-OCR2.0's 256-token pages using 100 vision tokens, and MinerU2.0's 6000-plus-token pages using fewer than 800. In production it generates training data at 200k-plus pages per day on a single A100-40G.

1Theory from first principles

The context bill, paid in a different currency

Long-context language models pay for text twice: once in attention compute, which grows with the square of the sequence, and once in memory traffic, because decoding streams the KV cache on every step. The paper asks whether text can be stored and read in a cheaper representation. An image of a page is two-dimensional and dense: a paragraph that costs a thousand tokens as text costs a few hundred patches as pixels, and the layout, fonts, and tables come along for free. The catch is that the reader must be able to recover the text, which is an OCR problem, and the quality of that recovery is what the whole paper measures.

DeepEncoder: two attention patterns, one compressor

The encoder is assembled from parts with complementary strengths. SAM-base processes 16 x 16 patches with window attention, so its cost stays local and its 80M parameters keep activations manageable at high resolution - it handles perception, the pixels. CLIP-large contributes the visual knowledge of a contrastively pretrained model and runs dense global attention, but it cannot afford to see 4096 tokens. Between them sits the compressor: two convolutional layers with kernel 3, stride 2, and padding 1, with channels rising from 256 to 1024, which downsamples the token grid by a factor of 16. A 1024 x 1024 page therefore reaches CLIP as 256 tokens instead of 4096. The design is a division of labor: window attention for seeing, a bottleneck for paying, global attention for understanding.

Compression ratio and what survives

compression = text_tokens / vision_tokens
< 10x  ->  about 97% OCR precision
~ 20x  ->  about 60% OCR precision

The ratio is the design objective and the quality knob at once. Under 10x the reconstruction is effectively lossless for documents; at 20x the model is losing detail, and the paper attributes part of the drop to long texts blurring when rendered at 512 x 512 or 640 x 640, and part to complex layouts. The honest reading is that optical compression has a knee, and the paper's contribution is locating it rather than hiding it.

DeepEncoder: SAM-base window attention reads patches, a 16x convolutional compressor cuts the token count, and CLIP-large adds global context; the 3B MoE decoder with 570M active parameters reconstructs the text.
Counting the tokens a page costs
python
def page_tokens(pixels=1024, patch=16, compression=16):
    patches = (pixels // patch) ** 2          # 64 x 64 = 4096
    return patches // compression              # 256 vision tokens

def compression_ratio(text_tokens, vision_tokens):
    return text_tokens / vision_tokens

# A dense page of about 2560 text tokens rendered at 1024 x 1024
print(page_tokens())                                  # 256
print(compression_ratio(2560, 256))                   # 10.0x, near-lossless
print(compression_ratio(5120, 256))                   # 20.0x, about 60%

# Production mode (Gundam) tiles the page into crops and reaches ~1156 tokens,
# trading vision tokens for fidelity on dense or small-print documents.
  • The 16x figure is the compressor's downsampling factor, which is what turns 4096 patches into 256 tokens.
  • Real deployments choose a mode: fewer tokens for bulk data generation, more crops for hard pages.
  • The same arithmetic applies to any rendered text - chat history, logs, retrieved documents - which is why the paper frames OCR as a proof of concept for context compression in general.

The decoder and the data

The decoder is DeepSeek3B-MoE-A570M: a 3B-parameter mixture-of-experts model activating 570M parameters per token, chosen because the reconstruction task is narrow and does not need a large dense model. Training data is dominated by OCR tasks across document types, with about 20 percent general vision data - captions, detection, grounding - following the DeepSeek-VL2 recipe, included to keep the vision interface usable for future work rather than to make this a general VLM. The paper is explicit that it is not one: the point is the compression experiment, and the general data is insurance.

Production numbers and the honest limits

The system's practical claim is throughput: over 200k pages per day on a single A100-40G, which makes it a data engine for pretraining corpora rather than just a benchmark entry. The scientific claim is the compression curve. Both come with caveats the paper states. OCR is a proof of concept: decoding text from a rendered page is not the same as compressing arbitrary context, and the paper defers digital-optical interleaved pretraining and needle-in-a-haystack tests to future work. Beyond 10x compression quality falls, and long documents rendered at low resolution blur. The comparison models are from the same release window, and the 20 percent general-vision data means the model is deliberately not optimized as a general-purpose VLM. Read it as the opening argument for optical context compression, with the compression curve as the evidence.

2Inside the paper

Problem the paper sets out to solve

Text context is expensive to store and read. The paper asks whether a page of text can be rendered as an image, compressed to a small number of vision tokens, and decoded back accurately enough to be useful - and where the accuracy knee lies.

Key idea to look for while reading

The encoder architecture and the compression curve. Section 2 covers DeepEncoder's three stages and the decoder; Section 4 is the quantitative study of ratios against precision, plus the OmniDocBench comparison and the production numbers. The framing to keep in mind is LLM-centric: the question is how few vision tokens can carry how much text.

Evidence the paper musters

Compression results: 96%+ OCR precision at 9-10x, about 90% at 10-12x, about 60% at 20x, measured on document benchmarks with diverse layouts. OmniDocBench: better than GOT-OCR2.0 with 100 vision tokens versus its 256 per page, and better than MinerU2.0 with under 800 versus its 6000-plus. Production: 200k-plus pages per day on one A100-40G. A 1024 x 1024 input becomes 4096 patch tokens and 256 vision tokens through a 16x compressor.

Limits the authors admit

OCR is explicitly a proof of concept for context compression, not the full claim; interleaved digital-optical pretraining and long-context retrieval tests are left for later. Quality degrades past 10x, and rendering long text at 512 or 640 pixels blurs it. The model is not a general VLM by design, and all comparisons are vendor-run on the authors' benchmark selection.

Practitioner takeaway

If you own a document pipeline, resolution policy is a cost decision you can measure: pick a compression ratio from the curve rather than a fixed resolution, and budget vision tokens the way you budget context. Optical storage wins when layout matters and the reader is good; it loses when text must be edited or exactly searched, so keep the text for those paths.

3Implementation check

0 of 5 questions answered

  1. 1.What does the 16x compressor in DeepEncoder actually compress?

  2. 2.Why does DeepEncoder use SAM-base for the first stage and CLIP-large for the second?

  3. 3.At what compression ratio does OCR precision fall to roughly 60%, and what does the paper say about why?

  4. 4.What makes DeepSeek-OCR useful as a production component rather than just a benchmark model?

  5. 5.Which statement correctly describes the paper's own framing of its contribution?

4Build it yourself

Find the compression knee of a toy optical reader

Intermediate2-4 hours

Model a page as a stream of symbols and every vision token as a fixed 30-bit budget. A bin of R words must share the budget, so each word keeps only min(5, 30/R) bits of its 5-bit code; decode each truncated code to the most frequent word that shares its prefix and score word accuracy at ratios 1 through 16.

Milestones

  1. Fill to_vision: set b = min(BITS, CAP // ratio) and turn each bin into (b, codes) with code = w >> (BITS - b).
  2. Fill recover: count the words in the document, expand each code to the 2^(BITS-b) words under it, and emit the most frequent one per position.
  3. Run the ratio sweep 1, 2, 4, 8, 16 and check that accuracy stays 1.0 while b remains 5.
  4. Print the vision-token count at every ratio and verify text tokens divided by vision tokens equals the ratio.
  5. Report the first ratio where accuracy falls below 0.9 and compare it with the paper's knee at 9-10x compression.
  6. Replace the frequency prior with the lowest symbol id and record how much accuracy changes at ratio 16.
Starter code
python · editable
Runs locally in your browser.

Success criteria

  • The sweep prints accuracy and vision-token counts for all five ratios plus the first ratio below 0.9.
  • Accuracy is 1.0 whenever the per-word bit budget is at least 5 and falls below 0.5 at the first ratio where it is not.
  • The script runs in under 5 seconds with the standard library only.

Stretch goals

  • Add rendering blur: mix one random extra word into every bin before decoding and re-measure the knee.
  • Make CAP a parameter and report how the knee ratio moves with the vision-token budget.
  • Give each code a shared prefix correction and compare a joint decoder against the independent one.

5Keep going

previousDeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse AttentionnextDeepSeek-V3.2: Pushing the Frontier of Open Large Language Models