mHC: Manifold-Constrained Hyper-Connections
Widen the residual stream for expressivity, but keep the identity property: project the mixing matrix onto the manifold of doubly stochastic matrices so the signal can mix but never amplify.
arXiv:2512.24880Where this sits
An architecture paper about the residual path rather than the attention or the experts. It trains MoE models built in the V3 style, uses the DualPipe schedule from the V3 infrastructure, and produces a component that the V4 generation then adopts - the frontier era reaching back into the trunk's oldest design decision.
- Identifies the failure mode of Hyper-Connections precisely: the unconstrained forward mixing matrix can amplify signals exponentially across layers, measured at up to about 3000x gain versus roughly 1.6x for mHC.
- Projects the residual mixing matrix onto the Birkhoff polytope with Sinkhorn-Knopp, restoring the identity-mapping property in generalized form: a convex combination of streams, with spectral norm bounded by one.
- Keeps expressivity while bounding it, because the doubly stochastic set is closed under matrix multiplication, so composite mappings across arbitrary depth stay non-expansive.
- Delivers the improvement at system cost: 6.7 percent additional training time at expansion rate n = 4 after TileLang kernel fusion, mixed precision, selective recomputation, and overlap inside DualPipe.
- Validates across 3B, 9B, and 27B MoE models plus a 1T-token token-scaling run, with a final loss reduction of 0.021 versus the baseline and gains such as +2.1 on BBH and +2.3 on DROP versus HC.
What it is
Hyper-Connections (HC) generalize the residual connection by widening the residual stream into several parallel streams with learned read, write, and forward-mixing matrices. The expressivity gain is real, but the forward matrix is unconstrained, so when many of them multiply across layers the signal can explode or vanish, and training becomes unstable at scale. mHC fixes this by projecting the forward matrix onto the Birkhoff polytope - the set of doubly stochastic matrices - using the Sinkhorn-Knopp algorithm. Rows and columns sum to one, so the transform is a convex combination of streams: the mean is preserved and the spectral norm is bounded by one, and because the set is closed under multiplication the bound survives any depth. On MoE models up to 27B parameters the method removes HC's instabilities, improves on both the baseline and HC on downstream benchmarks, and costs only 6.7 percent extra training time after kernel fusion, mixed-precision kernels, selective recomputation, and pipeline overlap.
1Theory from first principles
The residual connection is a stability contract
Every deep transformer is trainable because of one line: x_{l+1} = x_l + F(x_l). The identity path means a gradient or a signal can travel from layer 1 to layer 60 without being multiplied by sixty learned matrices, so neither explodes nor vanishes. The cost is that the residual stream is one vector wide, and every layer reads and writes through the same channel. Hyper-Connections widen it: the stream becomes several parallel vectors, and learned matrices decide how to read from the streams into a layer, how to write the layer's output back, and how to mix the streams forward. More routes between layers means more expressive topology, and that is the whole appeal.
Where HC breaks, in numbers
The forward mixing matrix is learned and unconstrained. If its spectral norm is slightly above one, the product of many such matrices grows exponentially with depth; if slightly below, it decays. The paper measures this directly with the maximum gain magnitude of the residual mapping and reports values up to roughly 3000 in HC, against about 1.6 for its constrained version. A network with a 3000x gain somewhere in its depth is a network whose loss spikes, whose gradients diverge, and whose training requires rescues. This is the same problem the original identity mapping solved for free, reappearing because the architecture generalized away the guarantee.
Constrain to doubly stochastic matrices
The fix is geometric. Restrict the forward matrix B to the Birkhoff polytope: non-negative entries with every row and every column summing to one. Three properties follow. First, B is a convex combination operator: each stream's next value is an average of the current streams, so the mean is conserved and no stream can be amplified. Second, the spectral norm of a doubly stochastic matrix is at most one, so the mapping is non-expansive. Third, the set is closed under multiplication, so a product of such matrices is itself doubly stochastic and the bound holds at any depth. The read and write maps A and C are kept non-negative and bounded with a sigmoid instead, because they must be able to emphasize and suppress features; only the forward path needs the strict constraint.
The mHC residual update
X_{l+1} = B_l * X_l + C_l * F_l(A_l * X_l)
B_l = SinkhornKnopp( exp(H_l) ), B_l >= 0, B_l * 1 = 1, 1^T * B_l = 1^TX_l is the stack of parallel residual streams, A_l reads one d-dimensional vector into layer F_l, C_l writes the output back into the streams, and B_l mixes the streams forward. The exponential makes the learned values positive and the Sinkhorn-Knopp iteration alternately normalizes rows and columns until both sum to one, producing a differentiable projection onto the manifold. The update keeps HC's structure and replaces its unconstrained forward map with a convex combination, which is what restores stability without collapsing the streams into one.
def sinkhorn(logits, iters=20):
"""Project a positive matrix onto the doubly stochastic manifold by
alternately normalizing rows and columns."""
m = [[pow(2.718281828, x) for x in row] for row in logits]
n = len(m)
for _ in range(iters):
for i in range(n): # row normalization
s = sum(m[i])
m[i] = [x / s for x in m[i]]
for j in range(n): # column normalization
s = sum(m[i][j] for i in range(n))
for i in range(n):
m[i][j] /= s
return m
B = sinkhorn([[0.1, 0.2, 0.3, 0.4]] * 4)
print([round(sum(row), 6) for row in B]) # rows sum to 1
print([round(sum(B[i][j] for i in range(4)), 6) for j in range(4)])- The paper uses 20 iterations, which makes the projection approximate rather than exact; the residual deviation in the backward gain stays bounded and small.
- Because the stream count is tiny (expansion rate 4 in the experiments), the iteration is cheap and fully differentiable inside the graph.
- In the fused kernel the whole projection runs as one operation, which is part of how the overhead stays at 6.7 percent.
Making it fast enough to use
A constraint that doubles training time is not a fix. The paper's systems half is what makes mHC practical: kernel fusion that merges the RMSNorm (the normalization layer), the mixing, and the complete Sinkhorn iteration into unified kernels; mixed-precision implementations written with TileLang; selective recomputation to cut the memory footprint of the intermediate activations; and communication overlapped inside the DualPipe schedule borrowed from the V3 stack. With all of it, expansion rate n = 4 costs 6.7 percent additional training time, and the paper reports the scaling behavior across 3B, 9B, and 27B models so the overhead is not a single measurement.
Results, limits, and why it matters here
On the 27B MoE model, mHC eliminates the loss spikes and gradient-norm blowups seen in HC, achieves a final loss 0.021 lower than the baseline, beats the baseline on all eight reported benchmarks, and beats HC on most, with +2.1 on BBH and +2.3 on DROP. The limits are stated in the paper's own terms: the Sinkhorn projection is approximate at 20 iterations, so the backward gain deviates slightly from one; the overhead is real, if small; and validation stops at 27B, far below the frontier scale where the instability it fixes was most feared. The reason this paper sits in the curriculum is the last mile: V4 adopts mHC for its million-token stack, which makes this the rare methods paper whose component ships in the next flagship release.
2Inside the paper
Problem the paper sets out to solve
Hyper-Connections buy expressivity by widening and mixing the residual stream, but the unconstrained mixing matrix destroys the identity-mapping guarantee and makes large-scale training unstable. The paper asks whether the expressivity can be kept while the stability guarantee is restored by construction.
Key idea to look for while reading
The manifold argument in Section 2: why doubly stochastic matrices give conservation, a spectral-norm bound, and closure under composition. Then read the systems section, because the 6.7 percent overhead is the difference between a theoretical fix and a usable one. The signal-propagation analysis with the gain measurements is the evidence for the diagnosis.
Evidence the paper musters
MoE models at 3B, 9B, and 27B parameters, plus a 3B run on 1T tokens for token scaling. Composite gain magnitude about 1.6 for mHC versus up to about 3000 for HC. Final loss 0.021 below the baseline, stable gradient norms, wins on eight downstream benchmarks including +2.1 BBH and +2.3 DROP over HC. Training overhead 6.7 percent at expansion rate n = 4 with the full optimization stack, Sinkhorn at 20 iterations.
Limits the authors admit
The projection is approximate, so the gain bound holds only up to the residual Sinkhorn error; the overhead, while small, is not zero; and the largest validated model is 27B, so extrapolation to frontier scale is an argument rather than a measurement. The doubly stochastic constraint deliberately narrows the architecture's expressive range, which the authors accept as the price of stability.
Practitioner takeaway
When you generalize a stabilizing mechanism, write down the property that made it stable and enforce it geometrically. Identity mapping is a special case of a convex combination; the Birkhoff polytope is the set where that property survives composition. Then budget for kernels, because a constraint that is mathematically clean but five times slower will not ship.
3Implementation check
0 of 5 questions answered
1.What problem do Hyper-Connections introduce?
2.Why is the Birkhoff polytope the right manifold for the forward matrix?
3.What does the Sinkhorn-Knopp step do in mHC?
4.What did the engineering work buy, and why does it matter?
5.Which limit does the paper itself acknowledge?
4Build it yourself
Bound the residual mixing with a 2x2 Sinkhorn projection
Advanced4-6 hoursBuild an 8-layer residual stream with two parallel channels per layer and compare three forward mixings on a sin(x) regression: identity, a learned unconstrained 2x2 matrix, and a matrix projected to the doubly stochastic manifold. Report the final loss and the spectral norm of the composite mixing matrix.
Milestones
- Implement the forward pass in train: X' = B X + c*tanh(X) from X0 = [x, 0], loss = (sum(X_L) - y)^2, with B from build_B (identity, raw, or Sinkhorn).
- Backprop manually through all 8 layers, accumulating gradients for c and the mixing parameters over the 64 training points.
- Add the Sinkhorn chain rule for mhc: B = [[p, 1-p], [1-p, p]], so the diagonal gets p*(1-p)/2 and the off-diagonal its negative.
- Run full-batch SGD for 800 steps at learning rate 0.15 for all three variants.
- Implement gain: multiply the B matrices in order, then return the square root of the top eigenvalue of M-transpose times M.
- Print mse and gain per variant and check that mhc gain is 1.0 while hc gain drifts away from it.
Success criteria
- Plain residual ends near mse 0.31 with gain 1.0, and mhc ends below mse 0.01 with gain between 0.999 and 1.001.
- The hc composite gain is printed and is not 1.0 (amplified or decayed), and hc final mse is worse than mhc.
- Raising the learning rate to 0.2 makes hc non-finite while mhc still finishes with a finite loss.
Stretch goals
- Expand to three streams and implement the general Sinkhorn-Knopp iteration with 20 rounds; check rows and columns sum to one.
- Track gain every 100 steps and report whether hc's gain crosses 1 during training.
- Scale B toward identity and measure how much the stability bound costs in final loss.