Skip to content
DeepForgeby svx
FoundingPaperAdvanced2024~13 min theory

DeepSeek-VL: Towards Real-World Vision-Language Understanding

A vision-language family whose hybrid encoder reads 1024 x 1024 images into 576 tokens, trained so the language model never forgets how to read.

arXiv:2403.05525

Where this sits

DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek-VL: Towards Real-World Vision-Language UnderstandingDeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

Last paper of the founding era. It applies the DeepSeek LLM trunk to a second modality and sets the pattern (hybrid encoder, joint training, taxonomy-driven fine-tuning) that later multimodal work reuses.

  • High-resolution visual input inside a fixed token budget: 1024 x 1024 pixels compressed to 576 visual tokens, where comparable open models worked at 336 or 448 pixels.
  • A hybrid encoder that pairs SigLIP-L for semantics with SAM-B for low-level detail, aimed at dense OCR (optical character recognition), charts, and tiny objects where a single CLIP-family encoder struggles.
  • Joint vision-and-language pretraining with a roughly 7:3 language-to-multimodal mix plus modality warm-up, which preserves language benchmark scores that pure multimodal training degrades.
  • Instruction data organized by a use-case taxonomy built from real GPT-4V and Gemini test cases, instead of an unweighted pile of academic datasets.
  • Open weights at two scales with strong small-model results: DeepSeek-VL-1.3B scores 64.6 on MMBench and 87.6 on POPE.

What it is

DeepSeek-VL adapts the DeepSeek LLM into a vision-language model at 1.3B and 7B scale. Its vision side is a hybrid: a SigLIP encoder takes a coarse semantic view at 384 x 384, a SAM-B encoder adds fine detail at 1024 x 1024, and a small adaptor compresses both into 576 visual tokens. The authors also tackle a practical failure mode, multimodal training eroding language ability, by keeping at least 70% text in the mix and warming the model onto vision gradually. Both sizes are fully open.

1Theory from first principles

The vision-language problem

A language model reads tokens; an image is a grid of pixels. The standard bridge is a vision encoder that turns the image into a sequence of vectors, plus an adaptor that maps those vectors into the language model's embedding space. Everything else is a data and training question. Two failure modes dominate the field: encoders that see the image at low resolution and miss small text, and training that spends so much time on image-caption pairs that the language half of the model forgets its skills. DeepSeek-VL is built around avoiding both.

Two encoders, one image: SigLIP-L gives the gist at 384 x 384, SAM-B gives the fine print at 1024 x 1024, and the adaptor fuses them into 576 tokens the language model can read.

Why one encoder was not enough

CLIP-family encoders, SigLIP included, are trained to align an image with a caption, which rewards semantic gist and punishes attention to pixel detail. The paper cites the CLIP-blind-pairs problem: visually distinct images can land on similar embeddings, and the low input resolution (224 to 384 pixels) is too coarse for dense OCR or visual grounding. A vision-only encoder is the complement: SAM-B is trained for segmentation, so it preserves location and texture. DeepSeek-VL runs both and lets the adaptor learn what each contributes. The cost is managed by keeping the output at a fixed token count rather than by shrinking the input.

From 1024 pixels to 576 tokens

576 = (96 / 2 / 2)^2

SAM-B sees the image at 1024 x 1024 with 16-pixel patches, which is a 64 x 64 grid of 4096 tokens. The adaptor interpolates that grid to 96 x 96 and applies two convolutions with stride 2, each halving the grid: 96, then 48, then 24. Reshaped, the 24 x 24 grid is 576 vectors. The low-resolution branch already produces 576 vectors after SigLIP-L, so the two are concatenated into 576 tokens of 2048 dimensions each.

Building a 7:3 joint batch
python
def joint_batch(text_pool, vision_pool, vision_fraction=0.3):
    # Keep the language-to-multimodal ratio near 7:3 in every batch.
    size = 1000
    n_vision = int(size * vision_fraction)
    n_text = size - n_vision
    batch = text_pool[:n_text] + vision_pool[:n_vision]
    return batch, n_text / size
  • The paper's experiments show a 100% multimodal batch causes a stark decline in language metrics, while 70:30 both preserves language and lets multimodal scores rise.
  • The warm-up starts text-heavy and ramps the vision share toward 30%, so the model eases into the new modality instead of switching abruptly.
  • The ratio is a real training decision with a cost curve: more vision per batch buys vision ability and spends language ability, and the paper argues for keeping language above 70%.

Three training stages

Stage 1 freezes the vision encoder and the language model and trains only the adaptor, on 1.25M ShareGPT4V captions plus 2.5M rendered document OCR pairs. This aligns the embedding spaces cheaply; the authors also test scaling the adaptor's data and find it plateaus, which is why stage 2 unfreezes the language model. Stage 2 is joint vision-language pretraining: the vision encoder stays frozen while the language model and adaptor learn together on a mixed batch. Stage 3 is supervised fine-tuning on instruction data, where the SigLIP-L encoder, adaptor, and language model train and only SAM-B stays frozen because of GPU memory limits.

What went into the data

The pretraining mix is dominated by text: the DeepSeek-LLM 2T corpus is 70% of it. The rest is spread across interleaved image-text (MMC4 at 13.1%), image captions (11.1%), tables and charts (2.1%), document OCR (2.1%), scene text (1.2%), and web code (0.4%). The document OCR slice is worth a look: 1.4 million arXiv articles compiled and rendered into image-text pairs with Nougat tooling, plus 860K English and 180K Chinese e-books and millions of education exam questions rendered from HTML. The web-code slice came from 1.46 million Jupyter notebooks, whose plots were paired with the code that preceded them, about 2M pairs narrowed to 1.1M. Instruction tuning follows a taxonomy of real use cases, from recognition and OCR to image-to-code conversion, charts, commonsense and logical reasoning, evaluation, multi-image comparison, and safety.

Results, limits, and what to watch

DeepSeek-VL-7B reaches 73.2 on MMBench and 72.8 on MMBench-CN, 36.6 on MMMU, 456 on OCRBench, and 88.1 on POPE, beating open models of similar size on most of these and coming close to much larger ones. The 1.3B model scores 64.6 on MMBench and 87.6 on POPE. Two limits are stated plainly: the work covers visual understanding only, with the loss computed on the language side, so the model does not generate images; and SAM-B stays frozen during fine-tuning, so the fine-detail encoder never adapts to the tasks. A third, implicit limit is scale, and the authors say the next step is a larger model with mixture-of-experts blocks. While reading, give the training-strategy section the most attention: the encoder is a composition of known parts, but the 7:3 ratio and the warm-up are where the real-world behavior comes from.

2Inside the paper

The problem it attacks

Open vision-language models often win academic benchmarks yet disappoint in real use, because they under-pretrain the multimodal stages, train on low-resolution inputs, and let language ability degrade during vision training. DeepSeek-VL targets all three.

The key idea

Spend the vision budget where it matters: a hybrid encoder gives both semantic gist and high-resolution detail at a fixed 576-token cost, and a joint language-plus-vision training mixture protects the language half while the vision half learns.

Evidence

At 7B the model reaches 73.2 on MMBench, 72.8 on MMBench-CN, 37.9 on CMMMU, 456 on OCRBench, and 88.1 on POPE, ahead of open models of the same size and competitive with larger ones. At 1.3B it still manages 64.6 on MMBench, 61.3 on MMBench-CN, 409 on OCRBench, and 87.6 on POPE. The ablations show that scaling the adaptor alone saturates, that a 100% multimodal batch collapses language metrics, and that the 7:3 mixture keeps both modalities strong.

What the authors admit

This is an understanding model, not a generator: the training loss is computed only on the language part, so image synthesis is out of scope. SAM-B remains frozen in the final fine-tuning stage due to GPU memory. The 1.3B model needed special handling (multiple-choice perplexity evaluation plus a small amount of SFT data mixed into pretraining) because generative metrics were too noisy to steer the run, which the authors attribute to limited capacity rather than to the data.

For the practitioner

If you build a multimodal model, two numbers deserve attention before architecture: the text share in your training batches and the visual token count per image. The first protects the language model from forgetting; the second decides your serving cost. DeepSeek-VL fixes the second at 576 and lets the encoder work hard to make those tokens informative, which is a better trade than shrinking the input image.

3Implementation check

0 of 5 questions answered

  1. 1.The hybrid encoder combines SigLIP-L at 384 x 384 with SAM-B at 1024 x 1024. Which task most needs the SAM-B branch?

  2. 2.SAM-B processes a 1024 x 1024 image and produces a 64 x 64 feature grid. After the adaptor interpolates to 96 x 96 and applies two stride-2 convolutions, how many visual tokens reach the language model?

  3. 3.Why keep 70% of each training batch as pure text when the goal is vision understanding?

  4. 4.In stage 1, the vision encoder and the language model are frozen and only the adaptor trains. Why does the pipeline move past this stage?

  5. 5.The 1.3B model had unstable generative benchmark scores during pretraining. What did the authors do?

4Build it yourself

Compress an Image into a Fixed Token Budget

Starter2-4 hours

Reproduce the vision side of DeepSeek-VL at toy scale: patchify synthetic images, encode patches into small token vectors, and pool the grid down the way the adaptor does without losing the classifier.

Milestones

  1. Generate 16x16 images whose class is defined by a horizontal or vertical bright bar plus a noise floor.
  2. Patchify each image into an 8x8 grid and encode each patch as a 2-d token (mean brightness, variance).
  3. Implement 2x2 average pooling with stride 2 and run it twice to turn 64 tokens into 4.
  4. Fit a nearest-centroid classifier on raw and pooled tokens and compare accuracy at each budget.
  5. Sweep the number of pooling steps and report the smallest token budget that keeps accuracy within 0.05 of raw.
Starter code
python · editable
Runs locally in your browser.

Success criteria

  • Pooled-token accuracy is within 0.05 of raw-token accuracy while using at most 16 tokens per image.
  • An 8x8 patch grid becomes a 2x2 token grid after two stride-2 pools, matching the paper's (96 / 2 / 2)^2 pattern at toy scale.
  • The pooling sweep prints accuracy for 64, 16, and 4 tokens and names the smallest budget that holds the within-0.05 bar.

Stretch goals

  • Add a second encoder branch (blur versus edge features) and concatenate its tokens with the first while keeping the token count fixed.
  • Replace nearest-centroid with a tiny softmax classifier trained by SGD in pure Python.
  • Implement the paper's 7:3 text/vision batch sampler and show the vision share holds between 0.28 and 0.32 across 100 batches.

5Keep going

previousDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsnextDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model