Skip to content
DeepForgeby svx
FrontierPaperAdvanced2025~14 min theory

Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

Same decoupled design, a 7B transformer, 90M more understanding samples and 72M synthetic images: both doors of the architecture get better.

arXiv:2501.17811

Where this sits

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and GenerationJanus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

Janus validated the decoupled design at a single 1.3B scale and left two open problems: unstable generation and understanding below specialist models. Janus-Pro is the follow-up that asks whether those were architecture problems or resource problems, and answers them by changing everything except the architecture. It is the last entry of the Janus line for now; after it, the frontier era returns to the language backbone with the sparse-attention and V3.1 work.

  • Scales the LLM backbone from 1.5B to 7B while keeping the decoupled design, showing the architecture scales rather than only working at small size.
  • Reworks the three-stage recipe: longer Stage I training on ImageNet with the LLM frozen, no ImageNet in Stage II, and a rebalanced SFT mix (5:1:4 versus 7:3:10).
  • Adds roughly 90M multimodal understanding samples following the DeepSeek-VL2 recipe, including chart, table, and document data.
  • Adds about 72M synthetic aesthetic images so pretraining sees a 1:1 real-to-synthetic mix, which the paper credits for stability and aesthetics.
  • Raises the headline numbers substantially: GenEval from 0.61 to 0.80 and DPG-Bench from 79.68 to 84.19, with MMBench moving from 69.4 to 79.2.

What it is

Janus-Pro is the scaling study on top of Janus. It keeps the architecture identical - SigLIP for understanding, a 16,384-entry VQ tokenizer for generation, two MLPs, one autoregressive transformer - and improves three things: the training strategy (longer Stage I on ImageNet with the LLM frozen, no ImageNet in Stage II, and an SFT mix of 5:1:4 instead of 7:3:10), the data (+90M multimodal understanding samples in the DeepSeek-VL2 style and about 72M synthetic aesthetic images, bringing real to synthetic to 1:1), and the model scale (two sizes, 1B and 7B, with the LLM grown from 1.5B to 7B). Janus-Pro-7B reports MMBench 79.2, MMMU 41.0, MM-Vet 50.0, POPE 87.4, SEED 72.1, GenEval 0.80, and DPG-Bench 84.19, outperforming TokenFlow-XL (13B) on understanding benchmarks except GQA and DALL-E 3 on GenEval. The conclusion is honest about the ceiling: at 384x384, OCR and fine detail remain weak, and the VQ tokenizer's reconstruction losses still limit small-object quality.

1Theory from first principles

What a scaling study is for

Janus proved the decoupling idea at 1.3B parameters but shipped with visible flaws: image generation from short prompts was unstable, and understanding lagged similarly sized specialists. Janus-Pro asks whether those are architectural limits or resource limits. It changes nothing about the architecture and spends on three axes instead - the training schedule, the data, and the model size - so any improvement can be attributed to scale rather than to a new mechanism. This is a useful pattern to recognize: when a paper's contribution is 'same design, more of everything', read it as evidence about which constraints were resource constraints all along.

The training-strategy fix

Janus's Stage II spent two thirds of its text-to-image steps on ImageNet-style category prompts to model pixel dependencies, which the authors later judged inefficient. Janus-Pro moves that work earlier: Stage I trains longer with the LLM frozen, where the paper finds the model can learn reasonable category-conditioned generation on its own, and Stage II goes straight to dense-caption text-to-image data. Stage III then rebalances the supervised mix from 7:3:10 to 5:1:4 (understanding, text, generation), trading a little generation data for better understanding. The lesson generalizes: a schedule is a hypothesis about where the model is bottlenecked, and moving cheap objectives earlier frees expensive stages for the data that actually needs them.

Data: semantics on one side, aesthetics on the other

Two data decisions map neatly onto the two doors of the architecture. For understanding, Janus-Pro follows the DeepSeek-VL2 recipe and adds about 90M samples - captions plus table, chart, and document understanding, and later chat-style SFT data - widening what the SigLIP path can reason about. For generation, the paper diagnoses the real-data problem as quality and noise, and adds about 72M synthetic aesthetic images, making the pretraining mix 1:1 real to synthetic. Synthetic data converges faster and produces more stable, better-looking outputs at low resolution. The asymmetry is the point: each encoding path gets the data its objective needs, which is only possible because the paths are decoupled.

Scale: 1B and 7B

Janus-Pro ships in two sizes with the same architecture: a 1B and a 7B variant, with the language model growing from a 1.5B to a 7B DeepSeek-LLM and the rest of the stack unchanged. The larger model converges faster on both understanding and generation losses and moves every headline metric: MMBench 75.5 for the 1B jumps to 79.2 for the 7B, MMMU from 36.3 to 41.0, GenEval from 0.73 to 0.80. The 7B also outperforms TokenFlow-XL, a 13B unified model, on nearly every understanding benchmark. The reported training cost is 9 and 14 days on 16 and 32 nodes of 8 A100 40GB GPUs for the two sizes, a useful reminder of how modest the compute is compared with the frontier language models in this curriculum.

GenEval, read per capability

GenEval_overall = (1/6) * ( single_obj + two_obj + counting + colors + position + color_attr )

GenEval averages six category accuracies, so the overall score hides which skill improved. Janus to Janus-Pro-7B: position 0.46 to 0.79, color attributes 0.42 to 0.66, counting 0.30 to 0.59, colors 0.84 to 0.90. The gains are largest exactly where a weak model fails compositionally - placing objects, binding attributes, counting - which is what better instruction-following data and a larger transformer should fix. Reading the sub-scores instead of the average tells you whether a generation model is actually better or just better at the easy categories.

The three-stage schedule, restated
python
def janus_pro_stages():
    # Janus:  stage1 = adaptors + image head, 180K steps stage2, 24K steps stage3
    # Pro:    longer stage1 on ImageNet (LLM frozen), stage2 drops ImageNet,
    #         stage3 uses a 5:1:4 mix of understanding : text : generation data.
    stage1 = {"trains": ["understand_adaptor", "generate_adaptor", "image_head"],
              "data": "ImageNet categories",
              "note": "longer than Janus; pixel dependence while the LLM is frozen"}
    stage2 = {"trains": ["llm", "adaptors", "image_head"],
              "data": "dense captions + 90M new understanding samples",
              "note": "no ImageNet; 1:1 real to synthetic generation data"}
    stage3 = {"trains": ["llm", "adaptors", "image_head", "understanding_encoder"],
              "ratio_understanding_text_generation": (5, 1, 4)}
    return stage1, stage2, stage3

# Janus used 10K / 180K / 24K steps for the 1B model; Pro trains the shared
# stages longer and reports 9 (1B) and 14 (7B) days on 16/32 A100 nodes.
  • The reordering matters more than the step counts: cheap category modeling moves to the frozen-LLM stage so dense-caption training is not diluted.
  • The 5:1:4 ratio is a deliberate trade: slightly less generation data in exchange for measurably better understanding.
  • Stage 3 is the only stage where the understanding encoder is unfrozen, the same pattern Janus used.
Janus-Pro keeps Janus's two-door architecture - SigLIP understanding path, VQ generation path, one autoregressive transformer - and scales the transformer, the data, and the training schedule rather than the design.

2Inside the paper

Problem the paper sets out to solve

Janus validated decoupled visual encoding at 1B scale, but its generation was unstable on short prompts and its understanding trailed dedicated models. The paper asks whether training strategy, data, and model size can close those gaps without changing the architecture.

Key idea to look for while reading

Three axes, one fixed architecture. The method section is short by design: optimized three-stage training, scaled data with explicit real-to-synthetic balance, and 1B/7B model sizes. Read the ablations as a scaling argument: what improves when you only add resources.

Evidence the paper musters

Janus-Pro-7B: MMBench 79.2, MMMU 41.0, MM-Vet 50.0, POPE 87.4, MME-Perception 1567.1, SEED 72.1, GQA 62.0; GenEval 0.80 and DPG-Bench 84.19. Janus-Pro-1B: MMBench 75.5, MMMU 36.3, GenEval 0.73, DPG-Bench 82.63. Relative to Janus (MMBench 69.4, GenEval 0.61, DPG-Bench 79.68), the largest gains are in compositional generation categories: position 0.46 to 0.79, counting 0.30 to 0.59, color attributes 0.42 to 0.66. The 7B model beats TokenFlow-XL (13B) on all listed understanding benchmarks except GQA, and beats DALL-E 3 and SD3-Medium on GenEval overall.

Limits the authors admit

Resolution is capped at 384x384, which limits fine-grained tasks such as OCR, and the VQ tokenizer's reconstruction losses mean generated images can lack fine detail - small faces occupying little image space may appear under-detailed. The evaluation suite is the authors' selection, the generation comparison is against models from the same period rather than a live leaderboard, and the data-scaling result depends on synthetic images whose license and diversity properties are not dissected in the paper.

Practitioner takeaway

If a design works at small scale and you believe in it, the cheapest next experiment is often not a new mechanism but a disciplined scaling pass: fix the schedule so each stage does only the work it is good at, add data separately for each objective the architecture exposes, and grow the shared component. Report the sub-scores, not just the averages - they tell you which part of the journey the resources actually bought.

3Implementation check

0 of 5 questions answered

  1. 1.Which three axes does Janus-Pro scale?

  2. 2.What changed in Stage II text-to-image training compared to Janus?

  3. 3.Why did the authors add about 72M synthetic images?

  4. 4.Janus-Pro-7B's GenEval overall is 0.80 versus Janus at 0.61. Which category improved most, and why does reading sub-scores matter?

  5. 5.Which limitation does the Janus-Pro conclusion state plainly?

4Build it yourself

Scale the decoupled design without changing it

Intermediate3-4 hours

Keep the two-door architecture fixed and scale only data and codebooks: 400 to 4000 tokens and 32 to 128 detail bins. Score both doors before and after, then run the scaled shared codebook as the control to see which metric scale fixes and which stays limited.

Milestones

  1. Build the small run: 400 tokens, 32 detail bins, decoupled encoders with the shared predictor.
  2. Build the large run: 4000 tokens and 128 detail bins, architecture unchanged.
  3. Add the scaled shared control: one 128-code book holding 16 semantic groups times 8 detail bins.
  4. Score all three rows with half the tokens for fitting and half for evaluation.
  5. Track understanding accuracy and generation MAE separately, as the paper reads GenEval per capability.
  6. Write down which metric scale fixed and which stayed limited, and tie the second to the 384x384 and tokenizer caveat.
Starter code
python · editable
Runs locally in your browser.

Success criteria

  • Scaling cuts decoupled generation MAE by at least 3x, from about 0.008 to about 0.002, with understanding accuracy fixed at 1.0.
  • The scaled shared codebook reaches 1.0 understanding but keeps generation MAE above 0.02, an order of magnitude worse.
  • The script exits 0 and prints all three rows from one command.

Stretch goals

  • Add synthetic tokens sampled from the training distribution at a 1:1 ratio and measure how much of the gap they close.
  • Break understanding accuracy down by semantic class and find the classes the shared book cannot separate.
  • Give the predictor a second table layer and check whether capacity, not codes, was the limit.

5Keep going

previousJanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and GenerationnextNative Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention