DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention
A lightning indexer scores every past token with a few cheap heads, keeps the top 2048, and lets the main attention read only those: quadratic becomes O(L k).
SourceWhere this sits
V3.1 stabilized the hybrid model and long context; the next cost wall was attention itself. V3.2-Exp is explicitly an experimental step toward the next architecture: it keeps the same MoE backbone and MLA and changes only how attention reads the cache, so the delta from V3.1-Terminus is measurable.
- Introduces DeepSeek Sparse Attention, reducing core attention complexity from O(L^2) to O(L k) with k = 2048 selected entries per query.
- Adds a lightning indexer that scores tokens in FP8 with few heads and ReLU, keeping selection cheap enough to run every layer at every step.
- Aligns the sparse model's training configuration with V3.1-Terminus so benchmark parity can be read as the price of sparsity rather than a confound of extra training.
- Cuts API prices by more than 50% on release day, the first public evidence that the sparsity translates into serving cost.
- Releases the kernels (TileLang prototyping plus CUDA in DeepGEMM and FlashMLA) so outside teams can reproduce the efficiency claims.
What it is
DeepSeek-V3.2-Exp is the September 29, 2025 experimental release that attaches DeepSeek Sparse Attention (DSA) to V3.1-Terminus by continued training. DSA has two parts: a lightning indexer - a small set of indexer heads running in FP8 that scores each query against every preceding token - and a fine-grained selector that keeps the top 2048 key-value entries per query, shared across query heads. The main attention then runs only over the selected entries, so its cost drops from O(L^2) to O(L k). DeepSeek deliberately aligned training with V3.1-Terminus and reports benchmark parity, plus API prices cut by more than 50%. The release includes an open technical report, model weights, and the indexer and sparse-attention kernels in TileLang and CUDA (DeepGEMM and FlashMLA). A later, expanded write-up appeared as the DeepSeek-V3.2 report on arXiv (2512.02556), describing the same architecture.
1Theory from first principles
Why attention cost grows with context
Self-attention compares every query with every earlier key. Doubling the context quadruples the number of query-key pairs in the core computation, and the KV cache you must hold - and, while decoding, stream from memory - grows linearly with context on top of that. The two phases fail differently: training and prefilling are compute-bound (arithmetic dominates), while autoregressive decoding is memory-bound (each token costs a pass over the KV cache). Sparse attention attacks both by reading fewer keys, but only if the choice of which keys to read is cheap and hardware-friendly, otherwise you pay the selection cost and save nothing.
Post-hoc sparsity versus native sparsity
One family of methods takes a trained dense model and, at inference time, guesses which keys matter - attention sinks, recent windows, heuristic scores. These are easy to deploy and fragile: the model was never told it would be evaluated under that mask, and the heuristic competes with the learned attention pattern. The other family trains the sparsity into the model, so selection and attention co-adapt. The Native Sparse Attention research line made this case first; DSA is the production descendant, and it adds a twist: rather than training the selection mechanism from scratch end to end, it trains a dedicated indexer to approximate the main attention distribution, then uses that indexer to select at serving time.
The complexity DSA changes
dense core attention: O(L^2 * d)
DSA core attention: O(L * k * d) with k = 2048
indexer: O(L^2 * H_I * d_I) with H_I small and FP8 arithmeticL is the context length, d the head dimension, k the number of selected entries per query. The core attention cost becomes linear in L for fixed k, which is what makes million-token ambitions conceivable. Note what does not change: the indexer still compares each query with every key, so it is quadratic. It is affordable only because it runs with a handful of heads (and a small indexer head dimension) in FP8 - the point of the design is that you pay twice for keys, once cheaply and once expensively, and only the expensive pass is sparse.
The lightning indexer
For each query token, the indexer computes a score against every preceding token: a sum over a small number of indexer heads of a learned per-head weight times a ReLU (a rectifying activation that zeroes negative values) of the indexer query-key dot product. ReLU is chosen for throughput - it needs no softmax normalization and runs well in low precision - and the whole module is small enough to execute in FP8. The top 2048 scoring entries are kept for that query, and crucially the selection is shared across all query heads, which allows a single gather of KV entries instead of one per head. The design is a purpose-built approximation of the question 'which keys would dense attention have cared about?', answered cheaply enough to run everywhere.
How the selector learns to be right
Selection is a discrete operation, so you cannot backpropagate through 'which 2048 tokens were kept' in the obvious way. DSA's answer: supervise the indexer directly. The indexer's input is detached from the main computation graph, and it receives its own training signal that aligns its scores with the attention distribution the main model would have produced; the main model keeps training on the language-modeling loss alone. This decouples the two problems - attention quality and selection quality - and gives each a clean gradient. In the reported stage, the model selects 2048 entries per query, trains for 15000 steps of 480 sequences at 128K tokens (about 943.7B tokens) at a learning rate of 7.3e-6, and both the main model and the indexer are updated.
Sparsity that survives contact with kernels
A sparse pattern that ignores GPU memory hierarchies loses its theoretical win. Two decisions make DSA's sparsity real: the selection happens at entry granularity over the MLA latent, which is shared across query heads (the MQA, multi-query attention, mode of MLA), so one gather serves many queries; and the released kernels - indexer logits in DeepGEMM, sparse attention in FlashMLA, with TileLang versions for research - are built for that shared-entry pattern. DeepSeek also notes a subtle numerical trap found later: the indexer's RoPE (rotary position embedding) needs a non-interleaved layout while MLA's RoPE uses an interleaved one, and the original demo code mixed them up, degrading performance until a November 17, 2025 fix. That is worth remembering: in sparse attention, the selector is part of the model, and a bug in the selector is a quality bug, not just an efficiency bug.
def index_scores(q, k, w):
"""q: [H_i, d_i] indexer query, k: [L, d_i] indexer keys,
w: [H_i] learned per-head weights. ReLU keeps it cheap in FP8."""
total = [0.0] * len(k)
for h in range(len(w)):
for s in range(len(k)):
dot = sum(q[h][i] * k[s][i] for i in range(len(q[h])))
total[s] += w[h] * max(0.0, dot)
return total
def topk_mask(scores, k=2048):
"""Keep the k highest-scoring entries; every other position gets -inf."""
order = sorted(range(len(scores)), key=lambda s: scores[s], reverse=True)
keep = set(order[:k])
return [0.0 if s in keep else float("-inf") for s in range(len(scores))]
# the mask is added to the attention logits, so unselected keys vanish after
# softmax: attention reads k entries per query instead of all L.- This is the concept, not the kernel: a real implementation keeps scores in FP8, selects with a radix Top-K, and reuses the selection across query heads and layers.
- Selection is per query token, so queries do not share masks in the general case - though all query heads of one token do share the selected KV entries.
- If L <= k, every past token is kept and selection is a no-op; production kernels special-case this with a dense fast path.
2Inside the paper
Problem the report sets out to solve
Long-context training and serving were dominated by attention cost. The experiment asks a narrow question with a clean comparison: if you bolt a trainable sparse attention onto an unchanged backbone by continued training, how much efficiency do you buy and how much quality do you pay?
Key idea to look for while reading
Two modules and one alignment strategy. The indexer (small, FP8, ReLU, few heads) produces scores; the selector keeps the top k entries per query; the training configuration is deliberately aligned with V3.1-Terminus so differences read as the effect of sparsity. Look for the detached-indexer training scheme and the choice to instantiate DSA under the MQA mode of MLA.
Evidence the report and release muster
Parity with V3.1-Terminus across public benchmarks while cutting price by more than 50% at the API. Training details: k = 2048, 15000 steps x 480 sequences x 128K tokens per step, about 943.7B tokens, learning rate 7.3e-6. The complexity reduction O(L^2) to O(L k) is argued from the architecture and supported by deployed-service cost curves in the report; kernels for indexer logits and sparse attention ship in open source. The follow-up V3.2 report on arXiv keeps the same architecture and extends the story to RL training and agentic tasks, including gold-medal-level IMO and IOI results for its Speciale variant under the 2025 contest conditions.
Limits admitted and visible
The indexer is still quadratic in context length; only the main attention is linear, so the savings are a constant factor that grows with L but never eliminates the L^2 term. Efficiency gains depend on the released kernels and on the shared-entry (MQA) layout - a different implementation can lose them. Benchmark parity is vendor reported, and an experimental model that is 'on par' is a cost optimization, not a capability jump. The November 2025 RoPE-layout fix shows that the indexer's numerical details are easy to get subtly wrong. And the report's most honest signal is structural: the flagship claimed quality came from continued training on an existing model, not from training sparse attention from scratch.
Practitioner takeaway
When you add sparsity, separate the question 'what should be read' from 'how should it be read'. Make the selector cheap (few heads, low precision), train the selector against the signal it is approximating, and keep the main model's objective untouched. Then verify the win end to end, because arithmetic intensity and memory layout decide whether theoretical sparsity becomes wall-clock speed.
3Implementation check
0 of 6 questions answered
1.DSA reduces core attention cost from O(L^2) to O(L k). What remains quadratic?
2.Why does DSA instantiate its sparse attention under the MQA mode of MLA?
3.How is the indexer trained, given that top-k selection is not smoothly differentiable?
4.A serving stack applies DSA with k = 2048 to sequences of 1024 tokens. What happens in the selector?
5.The November 17, 2025 note about RoPE in the indexer warns implementers that...
6.Why did DeepSeek align V3.2-Exp's training configuration with V3.1-Terminus?
4Build it yourself
Top-k block selection with MQA attention
Starter2-4 hoursBuild DeepSeek Sparse Attention at toy scale. A tiny indexer scores every full block of shared keys by the max dot with the mean query, the top blocks are selected, and all query heads attend only to those keys. Verify the O(L k) main-attention work against dense attention as the sequence grows from 64 to 256.
Milestones
- Build sequences of multi-head queries with one shared key and value per position, the MQA layout DSA selects under.
- Print the dense query-key pair count at L = 64, 128, 256 as the baseline.
- Implement block scoring: max dot between the mean query and each full preceding block of keys.
- Let every head attend the top-k selected blocks plus the most recent block, and count the pairs read.
- Confirm sparse work grows linearly in L while the ratio to dense falls, and note that the indexer itself stays quadratic.
Success criteria
- The script exits 0 and prints sparse and dense pair counts at all three lengths.
- Sparse work grows roughly linearly: doubling L roughly doubles sparse pairs while dense pairs roughly quadruple.
- At L=256 the sparse/dense ratio is below 0.12 with TOPK = 1.
Stretch goals
- Count the indexer's scalar block scores separately and show the quadratic term survives through a small constant.
- Compare shared per-token selection against per-head selection and report the output difference it causes.
- Add a positional key component so block selection cannot cheat on content similarity alone.