Skip to content
DeepForgeby svx
FrontierPaperAdvanced2024~14 min theory

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

Keep Janus's two doors, swap the generation door from discrete tokens to rectified flow: the same transformer predicts text autoregressively and image velocity vectors for an ODE (ordinary differential equation) solver.

arXiv:2411.07975

Where this sits

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and GenerationJanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and GenerationJanus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

Janus decoupled the two visual paths but generated images as discrete tokens, which caps quality at what a VQ codebook can represent. JanusFlow keeps the decoupling and replaces the tokenizer with a flow model, testing whether the same transformer can drive both next-token prediction and a continuous generative process.

  • Replaces autoregressive generation over a VQ codebook with rectified flow in a continuous latent space, using the language model itself as the velocity predictor.
  • Keeps the architecture minimalist: only a lightweight ConvNeXt encoder and decoder plus a long skip connection are added to adapt the LLM for flow, with causal attention throughout.
  • Retains decoupled encoders (SigLIP for understanding, ConvNeXt for generation) and adds representation alignment that pulls the generation encoder's features toward the understanding encoder's, improving generation quality.
  • Reports generation scores that beat much larger specialists of its time: GenEval 0.63, DPG-Bench 80.09, and MJHQ FID-30k 9.51, above SDv1.5 and SDXL.
  • Keeps understanding competitive at 1.3B: MMBench 74.9, SEED-Bench 70.5, GQA 60.3, ahead of LLaVA-v1.5 and Qwen-VL-Chat on the listed benchmarks.

What it is

JanusFlow is the sibling of Janus that asks what happens if image generation is a continuous flow instead of a sequence of codebook tokens. The architecture stays deliberately minimal: a 1.3B DeepSeek-LLM backbone, a SigLIP-Large-Patch/16 encoder for understanding, and a small ConvNeXt encoder-decoder pair, initialized from scratch, that adapts the language model for rectified flow in the latent space of a pretrained variational autoencoder (SDXL-VAE). The language model's output at the image positions is read as a velocity field, and an Euler solver integrates it from noise to image. Two ideas make it work: decoupled encoders for the two tasks, and a representation-alignment loss that pulls generation features toward the understanding encoder's semantics during unified training. It reaches GenEval 0.63, DPG-Bench 80.09, and MJHQ FID-30k 9.51 for generation, and MMBench 74.9, SEED-Bench 70.5, and GQA 60.3 for understanding, all from a 1.3B backbone. Published at CVPR 2025.

1Theory from first principles

Two generative families, one transformer

Autoregressive models write a sequence one token at a time, which is exactly what a language model already does, so a unified model can simply add image tokens to the vocabulary. Diffusion and flow models instead start from noise and iteratively refine a whole continuous sample, which is a different computation but a much better fit for pixel-level detail. Janus chose the first route; JanusFlow chooses the second and asks a narrow question: can an ordinary decoder-only transformer predict the velocity field a flow model needs, without special architecture or attention masks? The paper's answer is yes, and the modifications required turn out to be surprisingly small.

Rectified flow in one paragraph

Rectified flow defines a path from noise to data and trains a network to predict its instantaneous direction, the velocity. At generation time you start with Gaussian noise and take small Euler steps along the predicted field until you reach a sample. The training signal is simple: pick a time, interpolate between noise and the real latent, and regress the network's output onto the difference between them. Because the network conditions on the text through the same attention stack, classifier-free guidance applies directly: run the model with and without the caption and extrapolate away from the unconditional direction to sharpen prompt alignment.

Flow matching and Euler sampling

z_t = (1 - t) * noise + t * latent,   target velocity = latent - noise
v = v(z_t, t | caption) + w * ( v(z_t, t | caption) - v(z_t, t | empty) )
z_{t+dt} = z_t + v * dt

The first line defines the training target: the velocity that moves a point along the straight path from noise to data. The second is classifier-free guidance with scale w, which extrapolates between the conditional and unconditional predictions; the reported images use w = 2. The third is the Euler update the sampler iterates, and the number of steps is a cost dial that a token-by-token generator does not have.

The sampling loop
python
def sample(llm, text_emb, latent_shape, steps=30, w=2.0):
    z = gaussian_noise(latent_shape)
    dt = 1.0 / steps
    for step in range(steps):
        t = step * dt
        # the LLM predicts a velocity for every latent position at once
        v_cond = llm.velocity(z, t, cond=text_emb)
        v_uncond = llm.velocity(z, t, cond=None)
        v = v_cond + w * (v_cond - v_uncond)
        z = z + v * dt
    return vae_decode(z)   # SDXL-VAE latent back to pixels

# Understanding uses the same weights with next-token prediction instead.
  • The reported FID uses w = 2 and 30 sampling steps; both are swept in the appendix, so quality and latency trade off explicitly.
  • Unlike a discrete-token generator, the model emits a velocity for the whole latent grid per step, which is why the ConvNeXt decoder exists.
  • The same transformer runs both modes; the task decides whether its output is read as logits or as a velocity field.

What minimal actually means

The integration is three pieces. A generation encoder turns the noisy latent into a sequence of embeddings; a time embedding is concatenated so the model knows where it is on the path; a generation decoder turns the transformer's output back into a latent-shaped velocity. The encoder and decoder are ConvNeXt blocks trained from scratch, with a long skip connection between them, and generation happens in an SDXL-VAE latent space so the resolution of the transformer's sequence stays manageable. The paper tested attention-masking schemes used by other unified models and found causal attention sufficient - no special mask, no second attention stack. That simplicity is the contribution: the language model does not need to be rebuilt to become a flow model.

Two doors again: SigLIP for understanding, a ConvNeXt encoder-decoder pair for flow generation, one transformer in the middle. The generation path is continuous, not a codebook sequence.

Representation alignment

Decoupling removes the conflict between tasks but leaves the generation path without semantic supervision: its ConvNeXt encoder sees latents, not concepts. JanusFlow adds a regularization term during unified training that aligns the generation encoder's intermediate features with the understanding encoder's features for the same image. The generation branch then learns representations that are not only reconstructable but semantically structured, which the paper credits for better prompt following. It is a small loss term with a large effect, and it only exists because the encoders are decoupled - a shared encoder would have nothing to align.

Results, and what the paper leaves open

The 1.3B model reports MJHQ FID-30k 9.51, GenEval 0.63, and DPG-Bench 80.09, ahead of SDv1.5 and SDXL on those generation benchmarks despite being a unified model, and MMBench 74.9, SEED-Bench 70.5, and GQA 60.3 for understanding, ahead of LLaVA-v1.5 and Qwen-VL-Chat on the listed tasks. The limits are structural: everything is validated at one 1.3B scale and 384 x 384 resolution; flow sampling is iterative, so generation costs many forward passes; the VAE latent space is inherited from SDXL rather than learned; and the evaluation suite is the authors' selection. Janus-Pro later scales the token-based line rather than this one, so JanusFlow is best read as the controlled experiment that shows a flow objective can live inside a language model.

2Inside the paper

Problem the paper sets out to solve

Unified multimodal models usually generate images as discrete tokens because that is what a language model already predicts, but VQ tokenization loses fine detail. Can a decoder-only transformer instead drive a continuous flow model, keeping both tasks in one set of weights?

Key idea to look for while reading

Minimal adaptation plus alignment. Section 3 describes the flow formulation, the ConvNeXt encoder-decoder, and the causal-attention finding; the representation-alignment loss is the second contribution. Read the ablations for what each piece buys, and the sampling sweep for the step-count trade-off.

Evidence the paper musters

Generation: GenEval 0.63, DPG-Bench 80.09, MJHQ FID-30k 9.51 (w = 2, 30 steps), beating SDv1.5 and SDXL. Understanding: MMBench 74.9, SEED-Bench 70.5, GQA 60.3 at 1.3B parameters, ahead of LLaVA-v1.5 and Qwen-VL-Chat. Ablations cover the decoupled encoders, the alignment loss, and the attention-mask choice. Published at CVPR 2025.

Limits the authors admit

One model scale and one resolution; iterative sampling means generation costs multiple forward passes; the latent space comes from a pretrained SDXL VAE; and the benchmark comparisons are the authors' selection from the same release window. The paper positions itself as evidence that the approach works, not as a finished generation system.

Practitioner takeaway

You do not need a diffusion-specific network to run a flow model: a language transformer with a small latent-space encoder and decoder is enough. If you decouple representations for two tasks, add a loss that reconnects them semantically, because the decoupled branch loses the supervision the shared one used to get for free. And treat sampling steps as a serving dial, not a constant.

3Implementation check

0 of 5 questions answered

  1. 1.What does the language model predict when JanusFlow generates an image?

  2. 2.Which components does JanusFlow add to the language model for generation?

  3. 3.What does the representation-alignment loss do?

  4. 4.Why does classifier-free guidance appear in a rectified-flow model?

  5. 5.Which limitation does the JanusFlow paper's scope impose on its claims?

4Build it yourself

Rectified flow vs a DDPM baseline on a 1D Gaussian

Intermediate3-5 hours

Train two small fields on the same standard normal target: a velocity field for the straight rectified-flow path and a noise predictor for a variance-preserving diffusion chain. Sample both with the same budget and measure which one recovers the target mean and standard deviation in fewer steps.

Milestones

  1. Fill fit(kind): draw noise e, data x, and t; for flow regress x - e on the features [z, t*z] with z = (1-t)*e + t*x; for ddpm regress e with z = sqrt(ab)*x + sqrt(1-ab)*e at a random schedule index.
  2. Use fit2, the closed-form 2x2 least-squares solver, to return two weights per kind, and confirm both fits finish in under a second.
  3. Implement flow sampling: start z ~ N(0,1), then take steps Euler updates z += (w0*z + w1*t*z)/steps with t = s/steps.
  4. Implement ddpm sampling: start z ~ N(0,1), map the step count onto the 200-entry schedule, and apply the ancestral posterior mean and variance around the predicted noise.
  5. Print mean and std for flow at 32 and 4 steps and ddpm at 200 and 16 steps; find the step count where the flow std error passes 0.2.
  6. Change the seed and confirm that the four printed rows keep the same ordering.
Starter code
python · editable
Runs locally in your browser.

Success criteria

  • At the fixed seed with 1000 samples, flow at 32 steps has |mean| below 0.1 and |std - 1| below 0.1, while flow at 4 steps has |std - 1| above 0.25.
  • The ddpm sampler runs the full 200-step schedule and reports mean and std within 0.1 of the target.
  • One python3 run prints all four rows in under 5 seconds using only the standard library.

Stretch goals

  • Add a third feature t*t*z to the flow model with a 3x3 solver and check whether the 4-step std error drops.
  • Sweep step counts 4, 8, 16, and 32 and print the std error trend.
  • Train a second flow field on a two-component Gaussian mixture and report where the straight path fails.

5Keep going

previousJanus: Decoupling Visual Encoding for Unified Multimodal Understanding and GenerationnextJanus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling