EfficiencyPaperAdvanced2024~15 min theory

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

Build the cluster around the workload: PCIe A100s, CPU allreduce, and a network that carries storage and compute without congestion.

arXiv:2408.14158

Where this sits

The ground layer of the efficiency era. The model papers above it (MLA, DeepSeekMoE, FP8) are algorithmic answers to cost; this is the systems answer. Its patterns, integration of storage and compute on one fabric, overlapping communication with computation, and squeezing cheap PCIe interconnects with software, recur in the training stacks of the later frontier runs.

10,000 PCIe A100 GPUs deliver about 83% of DGX-A100 GEMM (dense matrix multiply) throughput at roughly 60% of node cost and power, which the abstract rounds to half the cost and 40% less energy.
A two-zone, two-layer fat-tree needs 122 switches where a DGX-style three-layer fabric needs 1,320, cutting networking cost about 40% versus an equivalent three-layer design.
HFReduce reduces gradients inside each node on the CPU before the network and launches no GPU kernel: 6.3-8.1 GB/s inter-node at 186 MiB versus NCCL's 1.6-4.8 GB/s, and over 10 GB/s once NVLink (NVIDIA's GPU-to-GPU link) bridges are added.
The software stack, HaiScale for parallelism, 3FS for storage traffic, and the open-sourced HAI-Platform scheduler, keeps computation and communication overlapped across 10,000 GPUs.
The paper publishes a year of production failure data (Xid errors, the driver's GPU error reports, and network flash cuts) alongside the checkpoint and validation machinery that keeps month-long runs alive.

What it is

This is the systems paper behind DeepSeek's training cluster. Fire-Flyer 2 is a 10,000-GPU PCIe A100 cluster whose two-layer fat-tree merges storage and compute traffic, built to reach near-DGX-A100 throughput at roughly half the cost and 40% less energy. The paper documents the co-design: HFReduce, the allreduce library that reduces inside the node on CPUs before touching the network; HaiScale, the parallelism and overlap stack; 3FS, the distributed file system; HAI-Platform, the scheduler; and a year of real failure data from production.

1Theory from first principles

Compute demand grows faster than hardware

The paper opens with a scaling mismatch. Deep learning compute demand has grown roughly 10x per year, while Moore's Law adds about 3x of arithmetic every two years, DRAM bandwidth about 1.6x, and interconnect bandwidth about 1.4x. The gap is closed by buying more machines, and at 10,000-GPU scale the GPUs are no longer the whole bill: switches, optics, NICs, power, and cooling are comparable line items. Cost-effectiveness has to be designed, not bought.

The interconnect decides the design

Training is a loop of matrix multiplications plus a gradient synchronization. The allreduce at the end of each step (a collective that sums every worker's gradients) moves every gradient across the network, so the ratio of computation to communication sets the ceiling on how many GPUs can be used efficiently. NVIDIA's SXM and DGX systems attack this with fast NVLink and many NICs (network interface cards) per node. The Fire-Flyer bet is that cheaper PCIe A100 nodes with a single 200Gbps NIC can reach a similar place if software does more, which requires raising work per byte: larger batches, gradient accumulation, and MoE layers that keep most computation local.

Cost, power, and fabric: PCIe A100 nodes reach about 83% of DGX-A100 GEMM throughput at roughly 60% of node cost and power, and the two-zone fat-tree replaces a 1,320-switch DGX-style fabric with 122 switches.

HFReduce: reduce on the CPU, not the GPU

In a ring allreduce each unit of data passes through the ring, and on a PCIe node every hop competes for the same bus, while NCCL also launches GPU kernels that steal cycles from compute. HFReduce splits the work differently. First, gradients are copied from the node's 8 GPUs into CPU memory and reduced there with SIMD instructions (wide vector math on the CPU), supporting FP32, FP16, BF16, and FP8. Second, the cross-node allreduce runs on CPUs over RDMA (remote direct memory access, network transfers that bypass the CPU's involvement per byte) using a double binary tree. Third, results are copied back to the GPUs. One transfer down and one up, and because the copies use the GPU's copy engine, no compute kernel is launched.

PCIe traffic per unit of allreduced data

ring allreduce: (2n - 1) / n PCIe bandwidth units per data unit
HFReduce: 1 bidirectional unit (one device-to-host, one host-to-device)

The paper's accounting says a ring allreduce consumes (2n - 1) / n units of PCIe bandwidth per unit of data, while HFReduce touches the bus once down and once up regardless of scale and runs the inter-node tree over the network instead. Measured outcome: 6.3-8.1 GB/s inter-node for HFReduce versus 1.6-4.8 GB/s for NCCL on a 186 MiB message, scaling from 16 to 1440 GPUs.

The stack above the collective

HaiScale is the parallelism toolkit: data, pipeline, tensor, expert, and fully sharded data parallel, with scheduling that overlaps communication with computation. 3FS is a distributed file system for the all-flash era, because training reads huge shards and writes checkpoints, and that traffic shares the fabric with gradients, so the network was built to carry both. HAI-Platform schedules jobs, handles faults, and is open-sourced. The paper also reports what actually breaks over a year in production, GPU Xid errors and network flash cuts, and the checkpoint manager and validator that keep a run recoverable.

The stack: HaiScale for parallelism and overlap, 3FS for storage traffic, HAI-Platform for scheduling and fault handling, all riding one two-zone fat-tree that carries storage and compute together.

PCIe traffic model: ring allreduce versus HFReduce

python

def ring_pcie_units(n):
    return (2 * n - 1) / n

def hfreduce_pcie_units():
    return 1.0  # one device-to-host plus one host-to-device per unit

for gpus in (16, 128, 1440):
    print(gpus, round(ring_pcie_units(gpus), 2), hfreduce_pcie_units())

This is the paper's bandwidth accounting, not a latency model; the measured result is the 6.3-8.1 GB/s versus 1.6-4.8 GB/s comparison.
Adding NVLink bridges between paired GPUs cuts the host traffic further and pushes HFReduce past 10 GB/s inter-node.

What the numbers say, and what they cost

The cluster holds 10,000 PCIe A100s across about 1,250 compute nodes and nearly 200 storage servers. Per GPU, TF32 GEMM runs at 107 versus 131 TFLOPS for DGX-A100, about 83% of the throughput, while a node draws 2500W instead of 4200W and costs about 60% as much. The paper is explicit about the trade: PCIe cards are slower than SXM, so part of the win is bought with software engineering, and its next-generation plan targets MoE training with multi-NIC, multi-plane networking.

2Inside the paper

Problem

AI infrastructure costs are inflating with model scale, traditional HPC machines were built for double-precision workloads and are poorly suited to deep learning, and long-term cloud rental can exceed the cost of owning a cluster.

Key idea

Co-design cheap hardware with custom software. Use PCIe A100 nodes with one NIC, wire them into a two-zone two-layer fat-tree that carries storage and compute traffic together, and compensate for the slower interconnect with CPU-side collectives (HFReduce), compute-communication overlap (HaiScale), a purpose-built file system (3FS), and scheduling plus fault handling (HAI-Platform).

Evidence

About 83% of DGX-A100 GEMM throughput per GPU (107 versus 131 TF32 TFLOPS) at about 60% of node cost and 2500W versus 4200W per node; the abstract states roughly half the cost and 40% less energy at cluster level. HFReduce reaches 6.3-8.1 GB/s inter-node versus NCCL's 1.6-4.8 GB/s on a 186 MiB message, and over 10 GB/s with NVLink bridges. The fabric uses 122 switches versus 1,320 for a DGX-style three-layer design, about 40% cheaper networking than an equivalent three-layer network.

What the authors admit

PCIe is slower than SXM and the paper does not pretend otherwise: part of the performance gap is accepted and repaid in software complexity, and HFReduce itself becomes memory-bound without NVLink bridges. The design leans on in-house engineering (HaiScale, 3FS, HAI-Platform) and on hardware choices made in 2021, and RDMA congestion control remains an open discussion.

Practitioner takeaway

Pick the interconnect for your workload's communication-to-computation ratio, not for its spec sheet. If the ratio is low, cheap PCIe nodes plus strong collectives and overlap can win on cost. And treat storage and checkpoint traffic as part of the training network, not as an afterthought.

3Implementation check

0 of 5 questions answered

1.Why does HFReduce beat NCCL on this cluster?
2.What is the point of the two-zone, two-layer fat-tree?
3.What fraction of DGX-A100 GEMM throughput does a PCIe A100 node reach, and at what cost?
4.Why do gradient accumulation and MoE layers help a PCIe cluster specifically?
5.What distinguishes the paper's treatment of hardware failures?

4Build it yourself

Tune a prefetch policy across SSD, DRAM, and HBM

Intermediate3-5 hours

Build a three-tier cache simulator with a small cost model: shards live on SSD, promote through DRAM into HBM, and every demand reads from wherever it currently sits. Add a speculative prefetch policy, then tune prefetch distance against a cost budget: more lookahead buys hits, but each speculative load costs bandwidth and can evict a shard you needed.

Milestones

Generate an access trace with locality: mostly local walks over the shard space plus a fraction of random jumps.
Implement the three-tier simulator with LRU at each tier and a demand-only baseline.
Define the cost model: SSD fetch 40 units, DRAM fetch 4, and 2 extra per speculative prefetch.
Collect the demand-only hit count and cost.
Implement a locality prefetch policy that guesses the next shards from the current one.
Sweep prefetch distances 1 through 4 and record hits and total cost.
Choose the largest distance that stays inside the budget and report the hit gain per extra 1000 cost units.
Compare against an oracle that prefetches exactly the next demands and note the gap.

Starter code

python · editable

import random
random.seed(5)
# EXPECTED before TODOs: trace 4000, demand-only 325 hits at cost 91884.
# EXPECTED after TODOs: local-ahead prefetch 1/2/3/4 gives 1281/2264/3300/3304
# hits at cost 171964/226052/250126/274152; 2 is best under the 240000 budget.
SHARDS, TRACE_LEN, BUDGET = 128, 4000, 240000
HBM_SLOTS, DRAM_SLOTS, SPEC_OVERHEAD = 16, 64, 2
COST = {"dram": 4, "ssd": 40}
trace, cur = [], 0
for _ in range(TRACE_LEN):
    cur = (cur + random.randint(1, 3)) % SHARDS if random.random() < 0.8 else random.randrange(SHARDS)
    trace.append(cur)
def simulate(trace, policy=None):
    """3-tier LRU: HBM -> DRAM -> SSD. A miss promotes one tier at a time:
    SSD 40 units, DRAM 4, and each speculative prefetch adds 2. Returns
    (hits, cost)."""
    hbm, dram, hits, cost = [], [], 0, 0
    for i, shard in enumerate(trace):
        for s in [shard] + [x for x in (policy(i) if policy else []) if x != shard]:
            if s in hbm:
                hbm.remove(s); hbm.append(s)
                hits += s == shard
                continue
            if s in dram:
                cost += COST["dram"]; dram.remove(s)
            else: cost += COST["ssd"]
            if s != shard:
                cost += SPEC_OVERHEAD
            dram.append(s)
            if len(dram) > DRAM_SLOTS: dram.pop(0)
            hbm.append(s)
            if len(hbm) > HBM_SLOTS: hbm.pop(0)
    return hits, cost
def prefetch_policy(i, trace, ahead):
    """TODO: return up to ahead shards to preload at step i, e.g. the
    locality guess [(trace[i] + d) % SHARDS for d in range(1, ahead + 1)]."""
    raise NotImplementedError("TODO: prefetch_policy")
hits, cost = simulate(trace)
print("demand-only LRU:", len(trace), "accesses, hits", hits, "cost", cost, "budget", BUDGET)
try:
    for ahead in (1, 2, 3, 4):
        h, c = simulate(trace, lambda i: prefetch_policy(i, trace, ahead))
        print("prefetch", ahead, "hits", h, "cost", c)
except NotImplementedError as exc:
    print("unimplemented:", exc)

Runs locally in your browser.

Success criteria

The demand-only run and at least three prefetch distances are reported with hits and cost.
The chosen distance is the best hit count under the stated budget, not the largest distance.
Prefetching beyond the chosen distance costs more per extra hit than the chosen one (the curve turns).
The full script runs in under 5 seconds with a fixed seed.

Stretch goals

Add a prefetch queue with a confidence threshold so speculative loads that miss the next few accesses are charged a penalty.
Replace LRU with a scan-resistant admission filter and compare hits at the same cost.
Give SSD, DRAM, and HBM different latencies and optimize p99 latency instead of mean cost.

Problems

5Keep going

previousLet the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models nextAuxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

Where this sits

What it is

1·Theory from first principles

Compute demand grows faster than hardware

The interconnect decides the design

HFReduce: reduce on the CPU, not the GPU

The stack above the collective

What the numbers say, and what they cost

2·Inside the paper

Problem

Key idea

Evidence

What the authors admit

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going