Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

What scaling actually hits

The paper opens with the three walls every large training run meets: memory capacity, compute efficiency, and interconnect bandwidth. Model architectures answer the first two - MLA compresses the KV cache, MoE keeps compute per token low, FP8 raises arithmetic throughput per byte. The third is not the model's to fix, because it is a property of the machine. V3 was trained on 2,048 H800s, a GPU whose NVLink bandwidth was cut for export compliance, and the paper's argument is that several of V3's most distinctive choices are direct responses to that one fact.

The 4:1 ratio and what follows from it

Inside an H800 node, NVLink delivers about 200 GB/s per direction, of which roughly 160 GB/s is achievable. Between nodes, each 400G InfiniBand NIC delivers a nominal 50 GB/s and an effective 40 GB/s on realistic message sizes. That is a 4:1 gap, and it means the model should be shaped so that most communication stays inside the node. Node-limited routing is the answer: the 256 routed experts are split into 8 groups of 32, each group living on one node, and the router may send a token to at most 4 nodes. Inter-node traffic then depends on the number of nodes a token visits, not on the number of experts it uses - which is why the paper can say the same communication budget would support 13 experts instead of 8.

The SM tax nobody budgets for

Communication is not free even when the network is fast: somebody has to move the bytes. The paper reports that during training, up to 20 of the H800's streaming multiprocessors are occupied with network message handling - filling queue pairs and work requests - and with forwarding data over NVLink, leaving fewer SMs for matrix math. At inference the same problem is worse, because latency matters more. V3's serving stack therefore runs expert-parallel all-to-all entirely through NIC RDMA, whose asynchronous model overlaps transfer with computation and returns the SMs to the model. This is a concrete example of a software decision made to work around a hardware deficiency, and the paper says so plainly.

Multi-plane networking and the storage plane

The cluster fabric is a two-layer multi-plane fat-tree. Each node has 8 GPUs and 8 InfiniBand NICs, and each GPU-NIC pair is assigned to its own network plane, so a node spreads its traffic across eight independent paths instead of contending for one. A separate 400G RoCE (RDMA over Converged Ethernet) NIC connects each node to a storage plane that carries 3FS traffic, keeping checkpoint and data loading off the compute fabric. With 64-port 400G switches, the topology can address up to 16,384 GPUs while keeping the cost and latency of two layers; the deployed cluster was just over two thousand GPUs. The paper is candid that the deployed fabric does not fully realize the design, because the ConnectX-7 NICs of the time expose separate ports rather than one bonded logical interface, so packets from one queue pair cannot be sprayed across planes.

Precision and the request list

The FP8 section repeats the V3 findings - fine-grained scaling, FP32 promotion every 128 products, low-precision communication - and then makes the hardware case: current units make you trade exponent range against mantissa precision, so training stacks compensate in software. The paper's recommendation is precise low-precision computation units that preserve accuracy for accumulation, alongside the other three asks: a unified scale-up and scale-out fabric with dedicated network co-processors, lower-latency interconnects that do not consume SM cycles, and direct CPU-GPU links such as NVLink or Infinity Fabric to remove the PCIe bottleneck during parameter, gradient, and KV cache transfers.

How to read it

This is not a model report and contains no new benchmark results; it is an engineering reflection with numbers, published at ISCA 2025. Read it alongside the V3 technical report, because it explains the why behind choices the report states as configuration: why routing is node-limited, why the cluster is two-layer, why the serving stack insists on RDMA, and why the paper believes the next gains come from co-designing chips and models together rather than from either alone. The stated limits are the flip side: the recommendations are informed by one vendor's hardware generation and by regulatory constraints specific to that deployment, and several suggestions require silicon changes that no vendor had shipped at the time.

Problem the paper sets out to solve

Training frontier models on export-restricted hardware exposes mismatches between what architectures want (cheap, fast, uniform communication; precise low-precision math) and what chips provide (a scale-up/scale-out bandwidth gap, SM cycles consumed by networking, coarse low-precision units). The paper documents those mismatches from the V3 run and proposes fixes.

Key idea to look for while reading

The paper is organized by layer: model architecture (MLA, MoE, MTP), low-precision compute and communication, scale-up interconnect, scale-out network, then broader reflections. Read the bandwidth arithmetic in Sections 4 and 5 first; the architectural choices in Section 2 are the V3 report's material restated with hardware motivation.

Evidence the paper musters

Concrete numbers from the deployed system: H800 NVLink 400 GB/s total, about 200 GB/s per direction and 160 GB/s achievable; 8 x 400G IB NICs per node at 50 GB/s nominal and 40 GB/s effective each; a 4:1 ratio; up to 20 SMs consumed by communication; 256 experts in 8 node-groups with M <= 4; a 64-port 400G two-layer fabric sized for 16,384 GPUs; 2,048 GPUs actually deployed; a separate 400G storage plane for 3FS.

Limits the authors admit

The reflection is tied to one hardware generation and one deployment's regulatory constraints. The multi-plane design is not fully realized in the deployed NICs, so some of its projected benefits are argued rather than measured. And the recommendations are proposals to vendors, not results: whether converged fabrics or precise low-precision units arrive is outside the authors' control.

Practitioner takeaway

Read your hardware's bandwidth ratio before you design your routing. If inter-node bandwidth is a quarter of intra-node, shape the model so communication stays inside nodes, and remember that moving bytes consumes compute cycles unless the network stack can do it asynchronously. When you write the postmortem of your own training run, quantify the workarounds: that list is what hardware teams need.

Milestones

1. 1.Fix the machine constants: 60 TFLOP/s peak and 3 TB/s bandwidth, giving a ridge point of 20 FLOP/byte.
2. 2.Write the intensity function: 2*m*n*k FLOPs over twice the bytes of A, B, and C.
3. 3.Implement the GEMM cost model: seconds = max(flops / peak, bytes / bandwidth) plus a bound label.
4. 4.Print intensity and bound for small, square, and large-k shapes.
5. 5.Implement a bisection search for the smallest k that is compute-bound at a fixed (m, n).
6. 6.Verify the crossing analytically: solve flops/bytes = ridge for k and compare with the search.
7. 7.Re-run with a second machine profile (for example 100 TFLOP/s at 2 TB/s) and show the crossover moves.

Success criteria

• The model labels 64x64x16 memory-bound and 64x64x64 compute-bound, matching the intensity print.
• The bisection returns k = 54 for (64, 64), and the analytic solve returns the same value.
• Switching to a higher ridge point changes the crossover in the direction the formula predicts.
• The whole script is standard library, uses fixed constants, and runs in under a second.

Stretch goals

• ·Add a batch dimension and show how batching raises intensity until the shape hits the compute roof.
• ·Model FlashAttention-style tiling: keep a block of keys on chip and recompute bytes to see when attention becomes compute-bound.
• ·Add a latency term for the memory-bound case and find the batch size where latency is hidden by arithmetic.

Problems

• Throughput Estimate
• Latency Estimate
• Roofline Bound Check
• Throughput Saturation

Articles

• KV Cache & FlashAttention
• Quantization: INT8 to FP8