DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Drop the fixed view: cut the image into 384-pixel tiles, add a thumbnail, and let an MLA-plus-MoE trunk read the grid.
arXiv:2412.10302Where this sits
The vision-language line's second step. DeepSeek-VL fused a semantic encoder with a segmentation encoder at a fixed 1024 x 1024; VL2 keeps one encoder but changes what it is shown, and swaps the dense trunk for the V2 architecture so multimodal serving inherits MLA's small cache and the MoE's low activated compute.
- Dynamic tiling replaces the fixed-resolution view: candidate grids up to 3 x 3 tiles are scored by padding area, and the best grid plus a global thumbnail is what the encoder sees.
- One shared SigLIP-SO400M-384 encoder serves every tile, producing 27 x 27 = 729 embeddings per tile that a pixel shuffle (a space-to-depth fold of 2 x 2 patches into one position) compresses to 14 x 14 = 196 tokens.
- The language model is DeepSeekMoE with MLA at three scales (3B/16B/27B total, 1.0B/2.8B/4.5B activated), so the family inherits V2's inference economics.
- Document and chart understanding jump well beyond DeepSeek-VL: DocVQA 93.3, ChartQA 86.0, InfoVQA 78.1, and OCRBench 811, against 456 on OCRBench for the 7B predecessor.
- A three-stage recipe (adaptor alignment on 2B tokens, about 800B image-text tokens of joint training, then SFT) with expert bias correction enabled in stage 2.
What it is
DeepSeek-VL2 is the second generation of DeepSeek's vision-language family, shipped at three MoE scales: 3B, 16B, and 27B total parameters with 1.0B, 2.8B, and 4.5B activated. The vision side replaces DeepSeek-VL's hybrid SigLIP-plus-SAM encoder with dynamic tiling: one SigLIP-SO400M-384 encoder reads a global thumbnail plus up to nine 384 x 384 local tiles whose grid is chosen to fit the image's aspect ratio, and a 2 x 2 pixel shuffle plus a two-layer MLP compresses each tile to 196 tokens. The language side is DeepSeekMoE with MLA, so the architecture that made V2 cheap to serve now reads images. All three sizes are open, and the 27B model reaches DocVQA 93.3, ChartQA 86.0, OCRBench 811, and MMBench 83.1.
1Theory from first principles
Why a fixed resolution breaks on real images
DeepSeek-VL fed every image through a 1024 x 1024 path because that is what its SAM-B branch required. Real documents, infographics, and screenshots do not cooperate: they arrive as tall scans, wide charts, or dense pages, and forcing them into a square wastes pixels on padding and shrinks the text you actually need to read. The tiling idea is the standard answer, and VL2's version is worth studying because the grid is chosen per image rather than fixed. The encoder stays small and unchanged; what varies is how many times it is run and in what arrangement.
Choosing a grid, then reading it
The candidate set is every grid (m x 384, n x 384) with m and n at least 1 and m * n at most 9. For an input image the pipeline resizes to each candidate and measures the padding area, then keeps the candidate that pads least. That image is cut into m * n local tiles of 384 x 384, and a single global thumbnail of the whole image is added. One SigLIP-SO400M-384 encoder processes all 1 + m * n views. Each view becomes 729 embeddings of width 1152; a 2 x 2 pixel shuffle turns that into a 14 x 14 = 196-token patch. Newline tokens mark rows, a view-separator token splits thumbnail from tiles, and a two-layer MLP projects the sequence into the language model's embedding space.
Visual token budget
N_visual = 210 + 1 + m * 14 * (n * 14 + 1), with 1 <= m, n and m * n <= 9The thumbnail contributes 14 x 15 = 210 tokens (14 rows plus a newline per row), the view separator adds 1, and each of the m * 14 tile rows contributes n * 14 content tokens plus a newline. A single 384 x 384 tile therefore costs 421 tokens, and the largest 3 x 3 layout costs 2017. The budget is the knob that trades resolution for context length, and it is set by the image's shape rather than by a constant.
def choose_grid(h, w, base=384, max_tiles=9):
"""Return (m, n) rows x cols whose padding area is smallest."""
best, best_pad = None, float("inf")
for m in range(1, max_tiles + 1):
for n in range(1, max_tiles + 1):
if m * n > max_tiles:
continue
target_h, target_w = m * base, n * base
# resize until the long side fits, then pad the short side
scale = min(target_h / h, target_w / w)
rh, rw = h * scale, w * scale
pad = target_h * target_w - rh * rw
if pad < best_pad:
best, best_pad = (m, n), pad
return best
print(choose_grid(1200, 800)) # (3, 2): tall image, least wasted area
print(210 + 1 + 3 * 14 * (2 * 14 + 1)) # 1429 visual tokens for that grid- The paper defines the candidate set the same way: (m * 384, n * 384) with m, n >= 1 and m * n <= 9, then picks the minimum padding area.
- Padding is measured after scaling so the image keeps its aspect ratio; a 1200 x 800 photo prefers 3 rows by 2 columns over a square.
- When a conversation carries more than two images, the pipeline disables tiling and processes each image as one view, because the token count would otherwise crowd out the dialogue.
Why the trunk is the V2 architecture
A vision-language model is prefill-heavy: the model reads a long sequence of visual tokens once and then writes a comparatively short answer. That makes the two V2 choices directly relevant. MLA compresses the KV cache into a latent, so the long visual prefix does not balloon memory during decoding, and DeepSeekMoE keeps the activated parameter count near 4.5B for the 27B model, so the FLOPs per token track the small active budget rather than the total. The paper's deployment note follows from the same arithmetic: the 3B, 16B, and 27B models fit on single GPUs with roughly 10GB, 40GB, and 80GB of memory.
Three training stages
Stage 1 trains only the adaptor, with the vision encoder and language model frozen, on 2B tokens - enough to align the spaces and no more. Stage 2 is joint vision-language pretraining on about 800B image-text tokens with the language model unfrozen; this is where the model learns to read documents and charts. Stage 3 is supervised fine-tuning on in-house vision-language instructions, mixing in text-only dialogue data from DeepSeek-V2 so language ability does not regress. One detail carried from the efficiency era: the 27B model turns on the expert bias correction from the loss-free balancing work during stage 2, which is the same router fix used in the V3 line.
Results and the honest limits
The 27B model posts DocVQA 93.3, ChartQA 86.0, InfoVQA 78.1, TextVQA 84.2, OCRBench 811, MMBench 83.1, MMMU 51.1, MMStar 61.3, and MathVista 62.8, competitive with dense open models several times larger while activating 4.5B parameters. The Tiny model's 1.0B active budget still reaches DocVQA 88.9 and MMBench 73.3. The limits are structural: tiling multiplies the number of encoder passes, so cost scales with the grid; the encoder itself is frozen from a contrastive pretraining that never saw the tiling layout; and MMMU-class reasoning remains far behind the best closed models of the period. Read the paper for the tiling arithmetic and the stage-2 data recipe, not for a claim about multimodal reasoning.
2Inside the paper
Problem the paper sets out to solve
Vision-language models of the time either fixed their input resolution, which wastes pixels on odd aspect ratios, or scaled the encoder itself. DeepSeek-VL2 asks whether one small encoder plus a per-image tiling policy can match much larger dense models at a fraction of the activated compute.
Key idea to look for while reading
Two substitutions. The vision side substitutes dynamic tiling for a fixed high-resolution encoder. The language side substitutes DeepSeekMoE plus MLA for a dense trunk. Read Section 2 for the tiling and token arithmetic, Section 3 for the three-stage schedule and the data, and the tables for the per-scale comparison.
Evidence the paper musters
DeepSeek-VL2 (27B total, 4.5B activated): DocVQA 93.3, ChartQA 86.0, InfoVQA 78.1, TextVQA 84.2, OCRBench 811, MMBench 83.1, MMBench-CN 79.6, MMMU 51.1, MMStar 61.3, MathVista 62.8, RealWorldQA 68.4, AI2D 81.4, MME 2253. Small (16B/2.8B): DocVQA 92.3, OCRBench 834. Tiny (3B/1.0B): DocVQA 88.9, OCRBench 809. Grounding is evaluated on RefCOCO, RefCOCO+, and RefCOCOg, where the models beat other VLMs at similar scale.
Limits the authors admit
The vision encoder is frozen and pretrained for contrastive alignment, not for this tiling layout. Tiling is disabled for more than two images per conversation, a context-length compromise. The benchmark suite is the authors' selection, and the comparison models come from the same release window. Nothing here closes the reasoning gap on MMMU-style tasks, and the report is a technical report rather than a peer-reviewed paper.
Practitioner takeaway
For document-heavy multimodal work, resolution policy matters more than encoder size: one shared encoder run over a per-image grid is cheaper than a bigger encoder run once. Then budget the visual tokens explicitly, because that number sets both context length and serving cost, and keep enough text-only data in the final stage that the language half does not forget how to talk.
3Implementation check
0 of 5 questions answered
1.An image is assigned a 2 x 2 tiling grid. How many encoder views does SigLIP process?
2.Each tile yields 729 embeddings of width 1152. After the pixel shuffle and newline tokens, roughly how many tokens per tile enter the language model?
3.The 27B model activates 4.5B parameters per token. Which pair of V2 ideas does it inherit, and why do they matter for vision?
4.Why does the pipeline disable dynamic tiling when a conversation contains more than two images?
5.Which result best supports the claim that a small activated budget can still read documents well?
4Build it yourself
Dynamic tiling with content-routed experts
Intermediate3-5 hoursBuild the vision half of DeepSeek-VL2 on synthetic images: choose the tile grid that wastes the least padding, cut the image into tiles, route each tile to a small content expert, and stitch the tile features before classifying. The baseline squeezes the whole image into one tile; the tiled pipeline should find a small high-contrast patch that the global average dilutes.
Milestones
- Generate synthetic 48x32 images, half containing a 16x16 bright patch aligned to a tile boundary, and keep the labels.
- Implement the VL2 grid rule: try every (m, n) with m * n <= 9, resize to m*TILE by n*TILE preserving aspect ratio, and keep the least-padding grid.
- Implement tile features: one (mean, std) pair per tile in row-major order.
- Implement content routing: flat tiles go to the amplify expert, textured tiles to the identity expert.
- Classify the stitched features and report accuracy for the single-tile baseline and the tiled pipeline.
- Sweep the classification threshold and report the best accuracy for both pipelines.
- Shift the bright patch off the tile grid and report how much accuracy drops.
Success criteria
- The single-tile baseline lands near chance (about 0.5), showing the global average hides the patch.
- The tiled pipeline reaches at least 0.95 accuracy on the same 200 images with a fixed seed.
- The grid selected for a 48x32 image is (3, 2) and the script prints it.
- Both pipelines and the threshold sweep run in one standard-library script under 5 seconds.
Stretch goals
- Add a third expert for high-variance noise tiles and test whether routing by std still separates the content types.
- Disable tiling when more than two images share a context and check the token-count arithmetic from the paper's formula.
- Quantize each tile feature and measure how much accuracy survives before the stitched vector is classified.