FoundingPaperCore2024~14 min theory

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Split every expert into smaller pieces and pin a few experts as always-on, so parameters grow without compute growth and each expert stays specialized.

arXiv:2401.06066

Where this sits

DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelAuxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

Second paper of the founding era. DeepSeek LLM showed how to train a good dense trunk; this paper asks how to grow that trunk's capacity without growing its bill, an idea the efficiency era then scales into DeepSeek-V2.

Fine-grained expert segmentation: each expert shrinks by a factor m and mK experts activate, keeping expert parameters and per-token compute constant while multiplying the combinations the router can form.
Shared expert isolation: Ks experts run for every token and the top-k routed count drops by Ks, so common knowledge lives in one place instead of being duplicated across experts.
Two balance losses, expert-level and device-level, that keep the router from collapsing onto a few experts and keep computation even across devices.
Empirical validation at three scales: 2B matching GShard 2.9B with 1.5x expert parameters and compute, 16B matching LLaMA2 7B at about 40% of the compute, and 145B matching DeepSeek 67B at 28.5% (possibly 18.2%) of the compute.
A deployable checkpoint: DeepSeekMoE 16B fits on a single 40GB GPU without quantization.

What it is

An architecture paper about mixture-of-experts (MoE) language models. Instead of routing a token to one or two large feed-forward experts, DeepSeekMoE chops each expert into m finer experts and routes to mK of them, giving the router a much richer menu. A small set of shared experts is always active to absorb knowledge every token needs. Validation runs at 2B, 16B, and a preliminary 145B scale show the design matching dense models of comparable quality with a fraction of the compute.

1Theory from first principles

Where experts come from

A Transformer block does attention, then a feed-forward network. Most of the parameters and most of the compute per token sit in that feed-forward network. Mixture-of-experts replaces it with N expert networks plus a tiny router: every expert holds a full copy of the layer's weights, but each token only runs through the K experts the router selects. Total parameters scale with N while per-token compute scales with K, which is the entire economic argument for MoE. The catch is training the router: if it keeps picking the same experts, the others never learn.

Routing in a classic MoE layer

s_i = softmax_i(u^T * e_i)    g_i = s_i if s_i in TopK(s_1..s_N, K) else 0    h = sum_i g_i * FFN_i(u) + u

u is the token's hidden vector after attention, e_i is a learned centroid for expert i, and s_i is the affinity between the token and that expert. The softmax turns the affinities into a distribution; TopK keeps only the K largest and zeroes the rest, so g is sparse and only K experts run. h is the layer output plus the residual u.

One token, one layer: the router scores every expert, keeps the top K, and blends their outputs. Everything about MoE design is about making those choices useful.

The two failure modes the paper names

Knowledge hybridity: with only 8 or 16 big experts, each expert receives tokens from wildly different topics and is forced to jam unrelated knowledge into one set of weights. Knowledge redundancy: tokens routed to different experts often need the same common facts, so several experts all learn them. Both problems reduce specialization, and specialization is the only reason sparse models should beat dense ones at equal active compute.

Keep the expert budget fixed and slice each feed-forward width into quarters. Now a token can pick 8 small experts instead of 2 big ones, and the space of combinations explodes.

Top-k routing, from affinities to gates

python

import math


def route_token(u, centroids, k):
    # Affinity of this token to every expert centroid.
    logits = [sum(a * b for a, b in zip(u, e)) for e in centroids]
    weights = [math.exp(s - max(logits)) for s in logits]
    total = sum(weights)
    # Softmax over all experts first, then keep only the top-k gates.
    picked = sorted(range(len(logits)), key=lambda i: -logits[i])[:k]
    return {i: weights[i] / total for i in picked}


def moe_layer(u, centroids, experts, k):
    gates = route_token(u, centroids, k)
    out = [0.0] * len(u)
    for i, g in gates.items():
        out = [o + g * v for o, v in zip(out, experts[i](u))]
    return out

The softmax is taken over all experts before the top-k cut, exactly as in the paper's equations; the kept gates therefore do not sum to one, which is intended.
A production implementation groups tokens per expert and fuses the weight matrices, but the logic is the same: score, select, scale, sum.
With fine-grained segmentation you would pass mN centroids and activate mK, changing only two integers in this code.

Expert-level balance loss

L_ExpBal = alpha_1 * sum_i f_i * P_i

For routed expert i, f_i is the fraction of tokens that selected it and P_i is its average affinity across the batch. Multiplying them means the loss grows only when an expert is both popular and confident, which is the onset of routing collapse. The factor alpha_1 stays small (0.01 in the 2B runs, 0.001 at 16B) so the loss nudges the router without distorting the language objective.

What the 16B configuration looks like

Twenty-eight layers, hidden width 2048, sixteen attention heads, and MoE layers everywhere except the first, because that layer's load balance converges slowly. Each layer holds 2 shared experts and 64 routed experts, each expert one quarter the width of a standard feed-forward network. A token goes to both shared experts and to 6 of the 64 routed ones, so 8 expert passes in total. That is 16.4B total parameters but only about 2.8B activated, and the checkpoint fits on one 40GB GPU. At 145B the recipe widens: 4 shared and 128 routed experts, 12 routed active, 144.6B total and 22.2B active.

Reading the evidence carefully

Three scales, three different kinds of claim. The 2B validation is a controlled comparison: five architectures, the same 100B tokens, the same training settings, with DeepSeekMoE 2B matching GShard 2.9B that has 1.5 times the expert parameters and compute. The 16B model is a real 2T-token run matching LLaMA2 7B and DeepSeek 7B at about 40% of the compute. The 145B number is explicitly preliminary: 245B training tokens, a constant learning rate, and performance comparable to their own DeepSeek 67B at 28.5% (maybe 18.2%) of the compute. Watch the qualifiers; they are the authors' own.

2Inside the paper

The problem it attacks

Sparse MoE models should beat dense models at equal active compute, but standard top-K routing produces experts that each contain a jumble of topics and duplicate each other's common knowledge. More parameters then buy less than they should.

The idea in two moves

First, spend the same expert-parameter budget on many smaller experts and activate more of them per token, so knowledge can be decomposed more finely and combined more precisely. Second, reserve a few always-on shared experts for common knowledge, subtracting them from the routed budget so activated compute stays fixed.

Evidence

At 2B, the model matches GShard 2.9B despite 1.5x less expert parameter and compute, and nearly reaches the dense model with the same total parameters, which is the experimental upper bound for any MoE. DeepSeekMoE 16B was trained on 2T tokens and lands near LLaMA2 7B at roughly 40% of the compute, beating models with similar activated parameters on the Open LLM Leaderboard. Ablations confirm both moves separately, and the analysis finds lower redundancy between routed experts and shared experts that cannot be replaced by routed ones.

What the authors admit

The 145B results are preliminary: fewer than a tenth of the tokens used for the smaller models, a constant learning rate schedule, and no alignment work. Even at 16B the authors keep expert granularity at 1/4 and the first layer dense, both practical compromises; they note that finer cuts hurt computational efficiency and that the first layer's balance converges slowly. Load balance remains a tuned hyperparameter, not a solved problem.

For the practitioner

Think in ratios, not counts. The useful quantities are expert width relative to a dense feed-forward block, routed experts per token, shared experts per layer, and the balance factors. If you build your own MoE, start from a dense model of the same activated size, then move capacity into experts while watching two curves: validation loss and per-expert token counts. If the token counts collapse onto a few experts, the dense baseline will win.

3Implementation check

0 of 5 questions answered

1.Fine-grained expert segmentation keeps expert parameters and per-token compute constant. How?
2.In the DeepSeekMoE 16B configuration, how many experts actually process one token?
3.When Ks shared experts are added, the number of nonzero routed gates becomes mK - Ks rather than mK. Why subtract?
4.In L_ExpBal = alpha_1 * sum_i f_i * P_i, f_i is the fraction of tokens routed to expert i and P_i is its mean affinity. Why multiply the two?
5.The paper compares 120 possible 2-of-16 expert choices with 4,426,165,368 possible 8-of-64 choices. What is that number supposed to show?

4Build it yourself

Route Tokens Through a Tiny MoE

Intermediate4-6 hours

Build a top-1 mixture-of-experts layer from scratch and race it against a dense logistic baseline on a task that is not linearly separable, then add the paper's expert-level balance loss and watch per-expert token counts.

Milestones

Generate the ring dataset and train the dense logistic baseline with SGD until its accuracy plateaus near 0.68.
Implement top-k routing over expert affinities and the gate-weighted expert combination for one token.
Train each selected expert by SGD on the tokens routed to it and print per-expert token counts.
Add the balance loss alpha * sum_j f_j * P_j and sweep alpha over 0, 0.01, and 0.05.
Add a shared expert that runs for every token and subtract it from the routed budget so per-token expert passes stay fixed.
Compare dense and MoE accuracy, then count expert passes per token to show where the compute goes.

Starter code

python · editable

# DeepSeekMoE: top-1 routed experts with a balance loss vs a dense baseline.
import math, random
def make_data(n=240, seed=0):
    # Inner disc is label 1, outer ring is label 0: not linearly separable.
    rng = random.Random(seed)
    pts = [(rng.random() * math.tau, rng.choice((0.0, 2.0)) + rng.gauss(0, 0.3))
           for _ in range(n)]
    return [((r * math.cos(a), r * math.sin(a)), 1.0 if r < 1.2 else 0.0)
            for a, r in pts]
def predict(model, x):
    w, b = model
    return 1.0 / (1.0 + math.exp(-(w[0] * x[0] + w[1] * x[1] + b)))
def train_dense(data, steps=300, lr=0.5):
    w, b = [0.0, 0.0], 0.0
    for _ in range(steps):
        for x, y in data:
            g = predict((w, b), x) - y
            w[0] -= lr * g * x[0]
            w[1] -= lr * g * x[1]
            b -= lr * g
    return (w, b)
def route(x, routers, k=1):
    scores = [r[0] * x[0] + r[1] * x[1] for r in routers]
    return sorted(range(len(scores)), key=lambda i: -scores[i])[:k]
def train_moe(data, n_experts=2, balance=0.01, steps=300, lr=0.5):
    # TODO: return (experts, routers). Train each routed expert by SGD and
    # update routers with cross-entropy plus balance * f_j * P_j per expert.
    return ([([0.0, 0.0], 0.0) for _ in range(n_experts)],
            [[0.0, 0.0] for _ in range(n_experts)])
def accuracy(model, data):
    return sum((model(x) >= 0.5) == (y >= 0.5) for x, y in data) / len(data)
# EXPECTED: dense about 0.68; the untrained MoE sits at chance with usage
# [240, 0]; after the TODO routed MoE clears 0.90 and usage splits near even.
def main():
    data = make_data()
    dense = train_dense(data)
    experts, routers = train_moe(data)
    print("dense accuracy: %.3f" % accuracy(lambda x: predict(dense, x), data))
    print("MoE accuracy: %.3f (todo: train experts with a balance loss)"
          % accuracy(lambda x: predict(experts[route(x, routers)[0]], x), data))
    print("expert usage:", [sum(1 for x, _ in data if route(x, routers)[0] == j)
                            for j in range(len(experts))],
          "(todo: balance to roughly [120, 120])")
if __name__ == "__main__":
    main()

Runs locally in your browser.

Success criteria

Routed MoE with 2 experts beats the dense baseline by at least 0.10 accuracy on the ring task.
With the balance loss on, expert usage lands within 25% of even across 20 seeds; the run prints the usage vector for each alpha.
The shared-expert variant matches routed-only accuracy while keeping the same number of expert passes per token.

Stretch goals

Move to 4 experts with top-2 gates and compare the combination count against 2-of-2.
Replace hard top-1 assignment with soft gates and check whether usage stays balanced without the extra loss.
Add unit tests for the router: ties, k equal to the number of experts, and a single-expert layer.

Problems

5Keep going

previousDeepSeek LLM: Scaling Open-Source Language Models with Longtermism nextDeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence

Concepts

Iterative Methods & Step Sizes

Where this sits

What it is

1·Theory from first principles

Where experts come from

The two failure modes the paper names

What the 16B configuration looks like

Reading the evidence carefully

2·Inside the paper

The problem it attacks

The idea in two moves

Evidence

What the authors admit

For the practitioner

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Concepts

Problems

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going