Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
MoE spends capacity on computation; Engram spends it on memory. Hash N-grams to a giant embedding table, look facts up in O(1), and split the sparse budget between the two.
arXiv:2601.07372Where this sits
A research satellite that grows out of the MoE line rather than the model line. It uses DeepSeekMoE models as the baseline it reallocates against, and its finding - that lookup is a complement to computation, not a replacement - is the kind of architectural claim the V-series kept testing at scale.
- Names a new sparsity axis: conditional memory, where static knowledge is retrieved by hash lookup instead of recomputed by experts.
- Modernizes N-gram embeddings with tokenizer compression, multi-head hashing, contextualized gating, and multi-branch integration so a huge table can be trained and served.
- Formulates the sparsity allocation problem and measures a U-shaped scaling law: reallocating about 20 to 25 percent of sparse parameters from experts to memory minimizes loss, and pure MoE is suboptimal.
- Scales to Engram-27B, which beats a strictly iso-parameter and iso-FLOPs MoE baseline on knowledge (MMLU about +3), reasoning (BBH +5.0, ARC-Challenge +3.7), and code and math (HumanEval +3.0, MATH +2.4).
- Improves long-context retrieval substantially (multi-query needle-in-a-haystack 84.2 to 97.0) and enables host-memory offload through deterministic addressing with negligible runtime overhead.
What it is
Engram is DeepSeek's proposal for a second axis of sparsity. Mixture-of-experts scales capacity by choosing which parameters compute a token, but the model still has to reconstruct static knowledge - names, formulas, idioms - through dynamic computation. Engram instead modernizes the classic N-gram embedding table into a conditional memory module: the token's recent N-grams are hashed to indices, the corresponding rows of a large embedding table are looked up in constant time, and a contextual gate fuses the retrieved memory into the hidden state. The paper frames the design question as a sparsity allocation problem - how much of a fixed sparse parameter budget should be computation and how much memory - and finds a U-shaped curve whose optimum is roughly 75 to 80 percent MoE and 20 to 25 percent memory. An Engram-27B model built by converting experts into a 5.7B-parameter memory table beats its strictly iso-parameter, iso-FLOPs MoE baseline across knowledge, reasoning, code, and math benchmarks, with the largest gains on reasoning rather than recall, and it lifts long-context retrieval from 84.2 to 97.0 on multi-query needle-in-a-haystack. The code is open.
1Theory from first principles
Two ways to spend capacity
A transformer stores knowledge in its weights, and at inference it retrieves that knowledge by computing: attention routes information and the feed-forward network reconstructs facts from distributed parameters. That is powerful and flexible, but it is also wasteful for knowledge that is fixed and local - a frequent phrase, a named entity, a formula, a code idiom. A lookup table answers those in one step, and a lookup is exactly what the model does not have. The paper's argument is that computation and memory are different resources with different costs, and a sparse model should allocate capacity to both rather than putting everything into experts.
What Engram is, mechanically
The module takes the last few tokens and forms their N-grams, compresses them through the tokenizer's structure so semantically equivalent suffixes share keys, hashes each compressed N-gram with several independent hash heads, and gathers the corresponding rows from a very large embedding table. Because the keys are hash indices, the lookup is constant time and, crucially, deterministic: the indices are known as soon as the token sequence is, before the layer that consumes them runs. A learned gate conditioned on the current hidden state decides how much retrieved memory to mix into the stream, and multiple branches let different N-gram orders contribute in parallel. The retrieved vectors are static, but how much they matter is contextual, which is the difference between a memory module and a lookup bolted onto a language model.
The sparsity allocation problem
rho = memory_params / (memory_params + expert_params)
loss(rho) is U-shaped: minimum near rho ~ 0.20-0.25
pure MoE (rho = 0) is suboptimalThe total sparse budget is fixed, so every parameter given to memory is a parameter taken from experts. If memory were a substitute for computation, loss would fall monotonically as rho grows; if it were useless, loss would rise monotonically. The measured U shape says it is a complement: some memory frees experts from static reconstruction, and too much memory starves the computation that handles the dynamic parts. In the paper's 10B-scale experiments the optimum sits at roughly rho = 0.20 to 0.25, stable across compute regimes.
def ngram_keys(tokens, n=3, heads=8, table_size=10**7):
"""Deterministic keys: same suffix -> same key, computable before the
layer runs, which is what allows host-memory prefetch."""
keys = []
for start in range(max(0, len(tokens) - n), len(tokens)):
suffix = tokens[start:]
for h in range(heads):
hsh = h + 1 # seed each head differently
for token in suffix:
hsh = (hsh * 31 + token) % table_size
keys.append((h, hsh))
return keys
def engram_read(embedding_table, keys, gate):
# each key gathers one row; the gate weights the fused result
rows = [embedding_table[k] for k in keys]
fused = [sum(r[i] for r in rows) / len(rows) for i in range(len(rows[0]))]
return [gate * f for f in fused]
# table_size can be very large: the memory is the point, and static addressing
# means it can live off the GPU with prefetching.- The rolling hash is written out on purpose: Python's built-in hash() is randomized per process, so it would break the promise that the same suffix always maps to the same row.
- Multi-head hashing is a collision mitigation: several independent hashes make it unlikely that all heads collide for the same N-gram.
- Tokenizer compression groups suffixes that decode to the same text, so the table is not wasted on byte-level variants.
- The paper instantiates Engram at selected layers (2 and 15 in the 27B model) with N-gram order up to 3, 8 heads, and dimension 1280.
The 27B experiment
The main result is a controlled conversion. Start from an MoE-27B model with 72 routed experts per layer, remove 17 of them, and spend the freed parameters on a 5.7B-parameter Engram table, giving an allocation ratio of 74.3 percent computation and 25.7 percent memory at the same total size of 26.7 billion parameters. Train both with the same tokens and the same FLOP budget. Engram-27B improves on the baseline across every reported domain: MMLU about +3, CMMLU +4.0, MMLU-Pro +1.8, BBH +5.0, ARC-Challenge +3.7, DROP +3.3, HumanEval +3.0, MBPP +1.6, GSM8K +2.2, MATH +2.4. The pattern is the interesting part: gains on reasoning and code exceed the gains on knowledge tests, which is the opposite of what a pure lookup table should buy.
Why reasoning improves
The mechanistic analysis offers an explanation: with static patterns handled by lookup, the early layers no longer spend their capacity reconstructing them, which effectively deepens the network available for reasoning, and attention is freed from local bookkeeping to handle global context. The long-context numbers support the second half: multi-query needle-in-a-haystack rises from 84.2 to 97.0, and variable tracking from 77.0 to 89.0. The paper also trains an Engram-40B variant, which continues to reduce loss and improve most benchmarks, though the authors note it is under-trained at the current token budget.
Systems: deterministic addressing
A giant embedding table is only practical if it does not have to sit in HBM next to the experts. Engram's keys are static functions of the token sequence, so the next lookup's indices can be computed before the corresponding layer executes, which makes prefetching from host memory possible and decouples storage from compute. The paper reports negligible overhead from the offload and treats this infrastructure-aware property as a first-class design goal rather than a side effect. It is the same instinct as the sparse-attention line: the arithmetic is only half the design; the memory hierarchy decides whether the arithmetic's advantage is real.
Limits and how to read it
The evidence comes from 27B and 40B models trained at research scale, not from a frontier release, so the allocation law is measured in a range where the U shape could still shift. The optimum is stable across the compute regimes tested, but those regimes are small by the standards of the V-series. The memory table is large, and while offload makes it affordable, it is still memory that must be stored and served. The U shape is empirical: the paper does not derive the optimum from theory, so treat it as a measured regularity rather than a constant. Read this alongside the V4.1-Flash entry, which ships a 196B-parameter Engram in a production model, as the first evidence that the idea scales beyond the paper.
2Inside the paper
Problem the paper sets out to solve
Sparse models allocate all their conditional capacity to computation, forcing experts to reconstruct static knowledge that a lookup could return directly. The paper asks whether memory should be a second sparse axis, and how a fixed budget should be split between the two.
Key idea to look for while reading
Three pieces: the Engram module's design (tokenizer compression, multi-head hashing, contextual gating, multi-branch integration), the sparsity allocation problem and its U-shaped curve, and the mechanistic analysis of why reasoning improves. Read the iso-parameter, iso-FLOPs comparison, because that is the control that makes the result meaningful.
Evidence the paper musters
A U-shaped validation-loss curve with the optimum near 20 to 25 percent memory allocation, stable across the tested compute regimes; in the 10B regime, loss improves from 1.7248 at pure MoE to 1.7109 near the optimum. Engram-27B (5.7B memory, 55 routed experts, 26.7B total) improves MMLU about +3, CMMLU +4.0, MMLU-Pro +1.8, BBH +5.0, ARC-Challenge +3.7, DROP +3.3, HumanEval +3.0, MBPP +1.6, GSM8K +2.2, MATH +2.4, and lifts multi-query NIAH from 84.2 to 97.0 and variable tracking from 77.0 to 89.0. An Engram-40B variant continues improving but is under-trained.
Limits the authors admit
The allocation law is empirical and measured at research scale; the largest model is 40B and under-trained. The memory table adds storage and serving complexity even when offloaded, and the gains are reported against one baseline architecture. The paper frames conditional memory as an indispensable primitive for future sparse models, which is a hypothesis its own experiments support but do not prove at frontier scale.
Practitioner takeaway
If a fixed fact is cheap to look up, do not make a network recompute it. When you have a sparse budget, try spending part of it on memory and measure the allocation curve rather than assuming more computation is always better. Prefer deterministic addressing, because it converts a memory-capacity problem into a prefetch scheduling problem, which is much easier to solve.
3Implementation check
0 of 5 questions answered
1.What does Engram add to a sparse model?
2.The allocation curve is U-shaped with an optimum near 20 to 25 percent memory. What does the shape mean?
3.In the Engram-27B comparison, what exactly is held constant?
4.Which result is most surprising, given that Engram is a memory mechanism?
5.Why does deterministic addressing matter operationally?
4Build it yourself
Hash trigram memory and measure the collision cost
Intermediate3-4 hoursBuild a conditional memory for a synthetic 48-symbol stream: hash each trigram to a bucket of next-symbol counts, look it up through one or two hash heads, and compare the bucket hit rate and perplexity against a smoothed unigram baseline.
Milestones
- Fill lookup: gather the count vector from every hash head, sum them, and return all zeros when no bucket was seen.
- Fill evaluate: build the memory from the training tokens, smooth the unigram backoff with ALPHA, and walk the held-out tokens.
- Track the hit rate as the fraction of test positions whose summed counts are nonzero, and the perplexity.
- Run one head and two heads at 4096 buckets and check that two heads lowers perplexity and raises the hit rate.
- Sweep bucket counts 256, 1024, and 4096 and report the perplexity trend.
- Compare the best memory perplexity with the unigram baseline and state the improvement factor.
Success criteria
- Two heads at 4096 buckets reach hit rate at or above 0.97 and perplexity at or below 10, against a unigram baseline near 48.
- The one-head run at the same size reports a lower hit rate or a higher perplexity, showing the collision cost.
- The full evaluation runs in under 3 seconds with the standard library.
Stretch goals
- Add a bigram branch to the trigram memory, weight the branches, and report whether perplexity drops.
- Average the per-head distributions instead of summing counts and compare.
- Grow the vocabulary to 200 symbols and find the bucket count where collisions stop dominating.