Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Understanding wants semantic features, generation wants discrete tokens. Use two visual encoders, two adaptors, and keep one transformer in the middle.
arXiv:2410.13848Where this sits
Janus opens a second front in the DeepSeek family: not language-model scaling, but unified multimodal modeling. It builds on the DeepSeek-LLM backbone rather than on the V-series chat models, and it is a research line about representation, not about serving cost.
- Separates visual encoding into an understanding path (SigLIP semantic features) and a generation path (VQ discrete IDs) instead of forcing one encoder to do both.
- Keeps a single autoregressive transformer with two adaptors and two prediction heads, so decoupling adds no second language model.
- Shows the cost of a shared encoder directly: swapping to decoupled encoding lifts MMBench from 35.0 to 69.4 and POPE from 60.1 to 87.0 in the paper's ablation.
- Demonstrates that unified training does not have to trade tasks off: the jointly trained model matches understanding-only and generation-only variants on their own benchmarks.
- Releases a 1.3B model whose scores (MMBench 69.4, SEED-Bench 63.7, POPE 87.0, GenEval 61%) beat several much larger task-specific and unified baselines of its time.
What it is
Janus is DeepSeek's answer to a structural problem in unified multimodal models: a single visual encoder cannot serve both understanding and generation well. Chameleon-style models tokenize images with one VQ (vector-quantized) encoder for both, which starves understanding of semantic features. Janus decouples the paths: images for understanding go through a SigLIP-Large-Patch16-384 encoder, images for generation through a VQ tokenizer with a 16,384-entry codebook and 16x downsampling, each followed by its own two-layer MLP adaptor into a shared autoregressive transformer (DeepSeek-LLM, 1.3B). Text uses the LLM's built-in head; images use a separately initialized prediction head over codebook IDs. The ablation is the argument: with one VQ encoder for both tasks, MMBench sits at 35.0 and POPE at 60.1; with decoupled encoders the 1.3B model reaches MMBench 69.4, POPE 87.0, SEED-Bench 63.7, and a GenEval score of 61% with COCO FID (Frechet inception distance) 8.53.
1Theory from first principles
Two tasks, two kinds of visual information
Understanding and generation want opposite things from an image representation. Understanding needs high-level semantics: what objects are present, their attributes, relations, and scene meaning - the kind of feature a contrastive encoder like SigLIP produces after training on image-text pairs. Generation needs fine-grained, reconstructable detail: exact textures, positions, and colors, which is why VQ tokenizers produce discrete IDs that can be decoded back into pixels. A VQ tokenizer optimized for reconstruction carries semantics only implicitly, and a SigLIP encoder discards the low-level detail generation needs. Force one encoder to serve both and it must compromise, and the compromise shows up wherever the tasks disagree.
The conflict, measured
Janus does not assert the trade-off; it measures it. With a single VQ tokenizer encoding images for both tasks (the Chameleon-style baseline), understanding collapses: POPE 60.1, MMBench 35.0, SEED-Bench 34.9. The generation side is fine - VQ tokenizers are built for it. Building a stronger shared alternative, a SigLIP-distilled semantic tokenizer that emits discrete IDs, helps understanding (MMBench 52.7) but generation suffers relative to the decoupled design. The cleanest evidence is the control: the same semantic tokenizer scores far better on understanding when trained for understanding only (MMBench 62.1) than when it must also support generation (52.7). The gap is the tax a shared encoder charges.
The architecture: two doors, one room
The design is deliberately minimal. On the input side, text goes through the LLM's own tokenizer and embedding table; images for understanding go through SigLIP, are flattened from a 2D grid into a sequence, and pass through the understanding adaptor (a two-layer MLP) into the LLM's input space; images for generation are converted to VQ IDs, and their codebook embeddings pass through the generation adaptor into the same space. All three streams concatenate into one sequence and the transformer processes them together, with no special attention masks. On the output side, the built-in language head predicts text, and a randomly initialized head predicts image tokens over the codebook. Training proceeds in three stages: adaptors and image head first, then unified pretraining of everything except the two encoders, then supervised fine-tuning that also unlocks the understanding encoder.
The image token budget
N_img = (H / 16) * (W / 16) -> 384 x 384 images become 24 x 24 = 576 tokensThe VQ tokenizer downsamples by 16, so an image becomes a fixed-length sequence of discrete IDs the transformer predicts one at a time. This number is the currency of the design: too few tokens and generation loses detail, too many and the context fills with image tokens that crowd out text. It also explains why 384x384 resolution is a hard boundary in this line - quadrupling pixels quadruples tokens - and why the paper's stated limitation about small faces being under-detailed is really about token budget per region.
def janus_forward(text_ids, image, task):
"""text_ids: tokenizer output; image: raw image; task: 'understand' or 'generate'."""
text_emb = embed_text(text_ids)
if task == "understand":
feats = siglip(image) # continuous semantic features
feats = flatten_2d(feats)
img_emb = understand_adaptor(feats) # 2-layer MLP into LLM space
seq = concat(text_emb, img_emb)
return lm_head(transformer(seq)) # built-in text head
vq_ids = vq_tokenizer.encode(image) # discrete IDs, 16x downsample
img_emb = generate_adaptor(embed_codebook(vq_ids))
seq = concat(text_emb, img_emb)
return image_head(transformer(seq)) # separate head over codebook IDs
# Same weights for the transformer; the task picks the input encoding and the
# output head. No mask tricks, no duplicated language model.- The adaptors exist because the two encoders produce different spaces: continuous SigLIP features versus codebook embeddings.
- The image head is randomly initialized and trained jointly; the language head is the LLM's own.
- Because generation is autoregressive over codebook IDs, Janus needs no diffusion stack and no special attention pattern.
One transformer can hold both if the doors differ
A natural worry is that joint training drags either task down. Janus checks this with single-task controls: the jointly trained decoupled model (POPE 87.0, MMBench 69.4, COCO FID 8.53) is statistically comparable to the understanding-only variant (POPE 85.9, MMBench 70.6) and to the generation-only variant (FID 8.92). Whatever cost unified training adds is smaller than the cost the shared encoder was imposing. The interpretation the authors offer is that the transformer is a general sequence model and the conflict lived in the encoder, not in the weights: once each modality enters the sequence in its own natural representation, the language model can learn to reason over image tokens and text tokens in one space.
2Inside the paper
Problem the paper sets out to solve
Unified multimodal models typically share one visual encoder for understanding and generation, and performance in understanding suffers because generation-friendly tokenizers are semantically weak. Can a unified model keep the benefits of one transformer while giving each task the visual representation it needs?
Key idea to look for while reading
Decoupling as the organizing principle: two encoders, two adaptors, one transformer, two heads. The ablation study is the heart of the paper - read it as a controlled experiment about representation conflict, not as a leaderboard.
Evidence the paper musters
The 1.3B Janus model reports POPE 87.0, MME-Perception 1338.0, MMBench 69.4, SEED-Bench 63.7, VQAv2 77.3, GQA 59.1, MMMU 30.5, and MM-Vet 34.3 for understanding; GenEval 61% and COCO-30K FID 8.53 for generation. In the ablation, single-encoder VQ encoding lands at POPE 60.1, MMBench 35.0, SEED 34.9 with COCO FID 8.72; the semantic-tokenizer variant reaches MMBench 52.7, and the same encoder without generation training reaches 62.1. The decoupled model beats LLaVA-v1.5 (7B) on POPE, MMBench, SEED-Bench, and MM-Vet, though it trails it on VQAv2, GQA, and MMMU.
Limits the authors admit
Everything runs at 384x384, the model is validated at a single 1.3B scale, and the ablation's strongest baseline is a distilled semantic tokenizer rather than a full alternative architecture. The paper explicitly frames the contribution as showing the importance of decoupling, not as claiming the architecture is finished - the scaling and data questions are left to the next paper.
Practitioner takeaway
When two objectives fight over one representation, the fix may be to give each its own encoder and share the model that reasons, not the model that perceives. Also notice the training-staging pattern: freeze the expensive encoders while adaptors learn to speak the LLM's language, then open the encoder later. That is the same adaptor-first discipline used across the multimodal line.
3Implementation check
0 of 5 questions answered
1.Why does Janus use different encoders for understanding and generation?
2.What do the two adaptors do?
3.A 384x384 image goes through Janus's VQ tokenizer with 16x downsampling. How many image tokens does the transformer see?
4.The comparison between the semantic-tokenizer variant trained jointly (Exp-B) and the same encoder trained for understanding only (Exp-C) shows what?
5.Which training-staging choice matches Janus?
4Build it yourself
Two doors, one core: decouple understanding and generation codes
Intermediate3-4 hoursGive each task its own tiny quantization of synthetic tokens - a semantic code for understanding and a fine detail code for generation - and let one shared predictor serve both. Compare against a single shared codebook to measure exactly what sharing costs each door.
Milestones
- Generate 400 seeded tokens, each with an independent semantic class and a continuous detail value.
- Implement the two decoupled encoders and the shared 32-code book: 4 coarse semantic groups times 8 detail bins.
- Implement score(tokens, encode_u, encode_g): fit a majority-class table and a detail-mean table on the first half, then measure accuracy and MAE on the second half.
- Run the decoupled row and the shared row through the identical predictor code path.
- Explain which task the shared book sacrifices and why a single code cannot stay sharp on both.
Success criteria
- Decoupled understanding accuracy is at least 0.95 and generation MAE at most 0.02.
- The shared book's understanding accuracy falls below 0.35 while its generation MAE stays worse than the decoupled one.
- Both rows run through the same predictor function; only the encoding functions differ.
Stretch goals
- Build the semantic-tokenizer control with 8 shared codes and compare it with the paper's Exp-B/Exp-C gap.
- Learn the shared book with k-means (k = 32) instead of the hand partition and rerun both metrics.
- Replace the tables with a small linear model and check whether shared codes still lose both tasks.