FrontierReportCore2026~17 min theory

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

A million tokens as the default: compress the cache, select sparsely, and keep the residual stream stable with manifold-constrained hyper-connections.

arXiv:2606.19348

Where this sits

DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceDeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

V3.2-Exp proved that sparse attention could be attached to a production backbone with parity; the V3.2 report then spent the year's remaining budget on RL and agentic data over that architecture. V4 asks the next question: if the goal is a million tokens, top-k selection over a full-resolution cache is not enough - the cache itself must shrink - so the attention story becomes compression plus sparsity, and the residual stream gets rebuilt for stability at depth.

Raises supported context to one million tokens for both models and cuts single-token inference cost at 1M context to 27% of the FLOPs and 10% of the KV cache versus V3.2 for V4-Pro, and to 10% / 7% for V4-Flash.
Replaces plain attention with hybrid CSA plus HCA layers, where CSA compresses then selects and HCA keeps a heavily compressed dense view of the whole context.
Introduces mHC, widening the residual stream into parallel streams mixed by a doubly stochastic matrix so signal propagation stays non-expansive across deep stacks.
Switches the optimizer to Muon for faster convergence and stability, and moves routed expert weights to FP4 in the instruct checkpoints.
Adds a two-stage post-training pipeline - specialized domain experts consolidated by on-policy distillation - with a maximum reasoning-effort mode (V4-Pro-Max).

What it is

DeepSeek-V4 is the April 24, 2026 preview release that makes one-million-token context routine. Two MoE models ship: V4-Pro with 1.6T parameters (49B activated) and V4-Flash with 284B (13B activated). The architecture keeps the DeepSeekMoE framework and multi-token prediction from V3 and adds three things: a hybrid attention that interleaves Compressed Sparse Attention (CSA, which compresses the KV cache along the sequence and then runs DSA over the compressed entries) with Heavily Compressed Attention (HCA, which compresses much more aggressively and keeps dense attention); Manifold-Constrained Hyper-Connections (mHC), which replace plain residual connections with several parallel streams mixed by a doubly stochastic matrix; and the Muon optimizer. Pretraining covers more than 32T tokens (32T for Flash, 33T for Pro). At 1M-token context, V4-Pro needs only 27% of single-token inference FLOPs and 10% of the KV cache of V3.2. Post-training cultivates domain experts and consolidates them by on-policy distillation; V4-Pro-Max is the maximum reasoning-effort mode. Weights are open, and the API offers V4-Pro and V4-Flash in thinking and non-thinking modes with 1M context.

1Theory from first principles

What actually breaks at a million tokens

Two costs scale with context, and they fail in different places. Attention compute grows with the number of query-key pairs, so pretraining and prefilling pay quadratically. KV cache memory grows linearly with context per layer and head, so decoding - which is memory-bound - pays for every cached token it must stream on every step. At a million tokens the memory term is the one that decides whether a service exists at all: you cannot hold hundreds of gigabytes of cache per request and also batch. V4's design responds to both terms at once, and the reported numbers are stated in exactly those units: percent of single-token inference FLOPs and percent of KV cache versus V3.2.

CSA: compress first, then select

Compressed Sparse Attention starts by shrinking the sequence dimension: groups of consecutive KV entries are projected into a single compressed entry, with overlapping windows so information at block boundaries is not lost. That alone cuts the cache by the compression rate. Then DSA - the top-k lightning-indexer selection from V3.2 - runs over the compressed entries, so the main attention reads only a few thousand compressed entries per query instead of the whole history. The two operations attack different terms: compression shrinks what must be stored, selection shrinks what must be read. This is the direct lineage move from V3.2-Exp: the same selector, now pointed at a coarser cache.

HCA: a dense, heavy summary

Heavily Compressed Attention takes the other extreme. It compresses the KV cache far more aggressively - a much larger compression rate than CSA, without overlapping windows - and then keeps attention dense over the compressed entries, with no indexer at all. The design bet is that a very coarse global view is cheap enough to read in full, and that fine detail is better handled by the uncompressed sliding-window path concatenated alongside it. CSA and HCA are interleaved across layers: some layers get fine-grained sparse attention, others get a coarse global summary. This interleaving is the hybrid: instead of one attention recipe for all layers, the stack mixes two cost profiles with different jobs.

Compression and selection

compressed entries:  n_comp = n / m
CSA: small m, overlapping windows, then top-k selection of compressed entries
HCA: large m, no overlap, dense attention over all n_comp entries
KV cache: n_comp * entry_size instead of n * entry_size

n is the sequence length and m the compression rate. Compressing by m divides both the cache size and the number of entries attention can read. CSA then pays a second constant - the top-k selection over compressed entries - to read only k of them; HCA skips selection and reads all n/m, which is affordable because m is large and the read is dense and hardware-friendly. The layer pattern decides how much of each you pay, and the reported end-to-end result at 1M context is the sum of those choices.

mHC: making the residual stream safe at depth

Residual connections are the reason deep transformers train at all: the identity path lets a signal reach layer 60 without being multiplied by sixty arbitrary matrices. Hyper-Connections generalize this by widening the residual stream into several parallel streams, with learned matrices deciding how to read from the streams into a layer, how to mix the streams forward, and how to write the layer output back. That buys expressivity but breaks the identity guarantee: the forward mix matrix is now learned and unconstrained, and at scale the multiplication of many such matrices amplifies or attenuates signals, which is training instability. mHC constrains the mix matrix to the manifold of doubly stochastic matrices (the Birkhoff polytope). Every row and column sums to one, so the spectral norm is bounded by one: the residual transformation can mix and rotate but never expand. The set is closed under multiplication, so the bound survives arbitrary depth, and the read/write maps are squashed through a sigmoid to keep them non-negative and bounded.

The mHC residual update

X(l+1) = B(l) X(l) + C(l) F(l)(A(l) X(l))
B(l) is doubly stochastic:  B >= 0,  B 1 = 1,  1^T B = 1^T

X(l) is a small stack of parallel residual streams, A(l) reads one d-dimensional vector into layer F, C(l) writes the layer output back into the streams, and B(l) mixes the streams forward. The doubly stochastic constraint on B makes the forward path a convex combination of the streams: the average signal across streams is conserved and the norm cannot grow, so the identity-mapping property that plain residuals give for free is restored in the generalized setting. During training, B is projected back onto the manifold with Sinkhorn-Knopp iterations, which is differentiable and cheap because the stream count is small.

The rest of the training stack

V4 keeps DeepSeekMoE routing and multi-token prediction from V3 with only minor adjustments, and changes the optimizer to Muon for faster convergence and greater stability - a natural pairing with mHC, since both are about keeping trillion-parameter training well-conditioned. Precision is mixed: base checkpoints are FP8 throughout, while instruct checkpoints use FP4 for routed expert parameters and FP8 for most of the rest, with the report noting that FP4 x FP8 peak throughput on then-current hardware matched FP8 x FP8 but was expected to improve on future silicon. Pretraining ran on 32T tokens for V4-Flash and 33T for V4-Pro; the models support 1M context natively after pretraining rather than through a late extension stage.

Post-training: specialists, then one student

The post-training pipeline has two stages. First, domain experts are cultivated independently - math, coding, agentic tasks, instruction following - each starting from SFT and then RL with GRPO against reward models tailored to that domain. Then the experts are consolidated into one deployable model by on-policy distillation: the unified model is the student, the domain experts are teachers, and training minimizes a reverse KL loss on the student's own rollouts. This is a clean answer to the old tension between specialists and a single generalist: train specialists where reward design is easy, then transfer their behavior into one set of weights. The reasoning-effort modes, including V4-Pro-Max, then expose how much test-time computation to spend per request.

Compress, then select - CSA in miniature

python

def csa_forward(entries, m=4, topk=2):
    """entries: KV vectors along the sequence. Compress overlapping groups of
    size m, then keep the top-k compressed entries per query."""
    compressed = []
    stride = max(1, m // 2)          # overlap so boundaries are not lost
    for start in range(0, len(entries) - m + 1, stride):
        block = entries[start:start + m]
        compressed.append([sum(v[i] for v in block) / m for i in range(len(block[0]))])

    query = entries[-1]
    scores = [sum(a * b for a, b in zip(query, c)) for c in compressed]
    order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    selected = order[:topk]          # per query, shared across query heads
    return [compressed[i] for i in selected]

# The cache stores compressed entries, not originals: memory drops by the
# compression rate, and attention reads top-k entries instead of all of them.

Real CSA learns the compression weights rather than averaging, and HCA uses a much larger compression rate with no overlap and dense attention over everything.
Selection happens on compressed entries, which is why the cache and the read set shrink together.
The sliding-window branch keeps recent uncompressed entries for local detail; the snippet omits it for brevity.

V4's hybrid attention: CSA compresses the sequence dimension and then selects top-k compressed entries; HCA compresses far more aggressively but attends densely over the coarse summary; interleaved layers mix both cost profiles.

2Inside the paper

Problem the report sets out to solve

Long-horizon agents and cross-document analysis want contexts far beyond what attention and the KV cache can serve affordably. The report asks what a frontier model looks like when a million tokens is the design point rather than an extension: attention that compresses as well as selects, a residual scheme that stays stable at depth and scale, and a training stack to match.

Key idea to look for while reading

Three axes, one target. Section 2 covers the hybrid CSA plus HCA attention and mHC; the compression math is where the efficiency comes from. The optimizer and infrastructure sections explain why the recipe trains. The post-training section is where the model becomes deployable: specialist cultivation plus on-policy distillation, and the reasoning-effort modes.

Evidence the report musters

Reported efficiency at 1M context: V4-Pro at 27% of V3.2's single-token inference FLOPs and 10% of its KV cache; V4-Flash at 10% of the FLOPs and 7% of the KV cache. Pretraining on 32T (Flash) and 33T (Pro) tokens; both models natively support 1M context. Open checkpoints for Pro and Flash, base and instruct, at FP8 (base) and FP4 plus FP8 (instruct). The report positions V4-Pro-Max as defining open-model state of the art, and the release note claims agentic-coding leadership among open models and world knowledge trailing only one named closed model. The Hugging Face community blog independently describes the headline as 'benchmark numbers competitive, but not SOTA', with the real innovation being efficient long-context support for agentic work.

Limits and open questions

This is a preview release: DeepSeek itself calls the series a preview, and the GA rollout of V4-Pro and the multimodal variant followed months later. Headline capability comparisons are vendor reported with selected benchmarks; the efficiency ratios, while stated in precise units, come from DeepSeek's own estimates of equivalent FP8 FLOPs rather than a third-party audit. Many architectural details (how many layers of each attention type, exact compression rates) live in the report rather than the release note, and mHC's stability argument leans on a companion paper. The API transition also carries real cost: the long-standing deepseek-chat and deepseek-reasoner names were retired in favor of V4 model names, with a three-month overlap.

Practitioner takeaway

When context grows by an order of magnitude, attack memory and traffic, not just arithmetic: compress what you store and select what you read. Keep a dense coarse path for global awareness so selection mistakes stay survivable. And if you generalize residual connections for expressivity, constrain them so the identity property survives: stability techniques like doubly stochastic mixing and Muon are what make depth at this scale trainable at all.

3Implementation check

0 of 6 questions answered

1.In V4's hybrid attention, what distinguishes CSA from HCA?
2.What is the reported efficiency of V4-Pro versus V3.2 at 1M-token context?
3.Why does mHC constrain the residual mix matrix B to be doubly stochastic?
4.Which V3 components does V4 explicitly retain?
5.How are domain specialists consolidated into the final V4 model?
6.A team needs a model for 900K-token document analysis on a fixed GPU budget. Per the V4 report, which property matters most?

4Build it yourself

Spend effort where block selection is uncertain

Advanced4-6 hours

Run a miniature version of the V4 pipeline: blocks of tokens are selected sparsely, a clue-routed expert reads the selected values, and an effort knob reads a second block only when the top saliences are too close to call. Report the accuracy/blocks-read frontier of fixed budgets against the adaptive policy.

Milestones

Generate sequences with one +0.9 and one -0.9 token, a hidden reading direction, and a clue that is wrong 20% of the time.
Reproduce the full-cache clue-router baseline with 2 blocks and 1 expert.
Implement run(seq, clue, effort): fixed k = 1 or 2 block reads, plus the adaptive policy that adds a second block only when the top two block saliences differ by at most MARGIN.
Measure accuracy, average blocks read, and expert calls for the three policies.
Confirm the adaptive policy matches k = 2 accuracy at fewer average blocks, and separate attention errors from the clue's 20% router ceiling.

Starter code

python · editable

import random
random.seed(34)
# EXPECTED before TODOs: clue-router 0.820 acc at 2 blocks, then
# "unimplemented: TODO: run".
# EXPECTED after TODOs: k=1 0.623/1.0, k=2 0.820/2.0, adaptive 0.820/1.545
# block reads: same quality on 23% fewer blocks.
N, L, B, FLIP, MARGIN = 400, 16, 8, 0.2, 0.3
def make_example(rng):
    seq = [rng.uniform(-0.45, 0.45) for _ in range(L)]
    pos, neg = rng.sample(range(L), 2)
    seq[pos], seq[neg] = 0.9, -0.9
    domain = rng.randrange(2)
    clue = domain if rng.random() > FLIP else 1 - domain
    return seq, clue, int((pos < neg) == (domain == 0))
def blocks_of(seq): return [seq[b * B:(b + 1) * B] for b in range(L // B)]
def select(seq, k): return sorted(range(L // B), key=lambda b: -max(abs(v) for v in blocks_of(seq)[b]))[:k]
def experts(seq, chosen):
    vals = [v for b in sorted(chosen) for v in blocks_of(seq)[b]]
    return [int(vals.index(max(vals)) < vals.index(min(vals))), int(vals.index(max(vals)) > vals.index(min(vals)))]
def run(seq, clue, effort):
    """TODO: return (prediction, blocks_read). effort 1 or 2 reads that many
    blocks. effort 0 is adaptive: read one, then a second only when the top
    two block saliences differ by at most MARGIN, the cheap difficulty cue.
    Answer with the clue's expert either way."""
    raise NotImplementedError("TODO: run")
DATA = [make_example(random) for _ in range(N)]
def main():
    print("clue-router acc", round(sum(experts(s, select(s, 2))[c] == y for s, c, y in DATA) / N, 3))
    try:
        for name, effort in (("k=1", 1), ("k=2", 2), ("adaptive", 0)):
            rows = [run(s, c, effort) for s, c, y in DATA]
            print(name, "acc", round(sum(1.0 if p == y else 0.5 if p == 0.5 else 0.0 for (p, _), (_, _, y) in zip(rows, DATA)) / N, 3), "blocks", sum(b for _, b in rows) / N)
    except NotImplementedError as exc:
        print("unimplemented:", exc)
main()

Runs locally in your browser.

Success criteria

The adaptive policy reaches the k = 2 accuracy within 0.01 while reading at least 20% fewer blocks on average.
The script exits 0 and prints accuracy and average blocks for all three policies.
The report separates attention errors from router errors and shows no policy beats the clue's 80% ceiling.

Stretch goals

Fit the margin threshold on a train split instead of fixing it at 0.3.
Run both experts at high effort and arbitrate disagreement with a confidence score.
Quantize cached block values to FP4-style levels and measure accuracy lost per byte saved.

Problems

5Keep going

previousDeepSeek-OCR 2: Visual Causal Flow nextDeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

Where this sits

What it is

1·Theory from first principles

What actually breaks at a million tokens

CSA: compress first, then select

HCA: a dense, heavy summary

mHC: making the residual stream safe at depth

The rest of the training stack

Post-training: specialists, then one student

2·Inside the paper

Problem the report sets out to solve

Key idea to look for while reading

Evidence the report musters

Limits and open questions

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going