EfficiencyPaperAdvanced2026~13 min theory

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

In agentic serving the KV cache comes from disk, and the prefill nodes' storage NICs saturate while the decode nodes' NICs sit idle. Pool them: let decode engines read the cache and hand it over.

arXiv:2602.21548

Where this sits

DeepSeek-V3 Technical ReportDualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

The serving-side counterpart of the long-context line. V3.2 and V4 made million-token contexts possible and agent loops made them common; DualPath is about the infrastructure that has to move those caches around, and it belongs to the same systems tradition as Fire-Flyer and the V3 hardware reflections.

Introduces dual-path KV cache loading: storage-to-prefill as before, plus storage-to-decode followed by an RDMA transfer to prefill over the compute network.
Pools the storage bandwidth of all engines instead of only the prefill side, removing the asymmetry where prefill NICs saturate while decode NICs idle.
Adds NIC-centric traffic management that isolates cache traffic from latency-critical model-execution communication, so the extra path does not interfere with collectives.
Schedules requests across both paths with a global policy that balances computation and network load jointly across prefill and decode engines.
Reports up to 1.87x offline and 1.96x average online throughput, with an ablation attributing the gains: layerwise prefill -17.21% JCT (joint completion time), dual-path loading -38.19%, scheduling -45.62% versus the baseline.

What it is

DualPath is an inference-systems paper about the storage side of agentic workloads: multi-turn sessions with long contexts and heavy cache reuse, where the model repeatedly reloads a KV cache that no longer fits in HBM. In a prefill-decode disaggregated cluster, the conventional path is storage to prefill engine, which saturates the prefill nodes' storage NICs while decode nodes' NICs go unused. DualPath adds a second path - load the cache into a decode engine, then forward it to the prefill engine over RDMA (remote direct memory access) on the compute network - and a global scheduler that picks between the two paths per request. On production agentic traffic it reports up to 1.87x offline throughput and an average 1.96x online serving throughput without violating latency SLOs (service-level objectives).

1Theory from first principles

What changed when agents arrived

Classic chat inference is one turn: read a prompt, generate an answer, forget the cache. Agentic inference is a loop: the model calls a tool, reads the result, thinks again, and each turn appends to a context that can reach hundreds of thousands of tokens. The cache from previous turns is valuable, because recomputing it wastes prefill compute, so serving systems spill it to external storage and reload it on the next turn. That makes storage I/O, not arithmetic, the resource that limits throughput. The paper's motivating observation is an imbalance in the standard disaggregated architecture: prompt processing (prefill) and token generation (decode) run on separate pools, and only the prefill pool is wired to load the cache from storage.

The asymmetry, stated as a bandwidth budget

Storage is reachable through NICs, and every engine has them. If only prefill engines ever read the cache, the effective storage bandwidth of the deployment is the prefill pool's NIC bandwidth, no matter how much capacity the decode pool has. When agent traffic is cache-heavy, prefill NICs become the bottleneck while decode NICs sit at low utilization, and the cluster's aggregate storage bandwidth is wasted. DualPath's fix is not faster storage or a bigger cache; it is routing: let decode engines read the cache too, then send it to the prefill engine that needs it over the compute network, which is typically under-used relative to the storage plane and has its own bandwidth.

Effective storage bandwidth under one path and two

single path:  B_eff = B_nic * N_prefill
DualPath:     B_eff = B_nic * (N_prefill + N_decode), bounded by the compute network

B_nic is the per-engine storage NIC bandwidth. Under the conventional path only prefill engines contribute, so adding decode capacity does not raise the ceiling. DualPath makes every engine a potential loader, and the compute network's RDMA capacity becomes the new limit. The scheduler's job is to decide when the extra hop is worth it, since a transfer that crosses the compute network consumes bandwidth that collectives also need.

Choosing a loading path per request

python

def choose_path(request, prefill_load, decode_load, storage_nic_load):
    """Pick the path with the lowest estimated transfer time. load values are
    current bytes-in-flight per pool, capacity is bytes/s per pool."""
    prefill_capacity = 8 * 40e9   # 8 engines, 40 GB/s effective per NIC
    decode_capacity = 8 * 40e9
    direct = request.kv_bytes / max(1.0, prefill_capacity - prefill_load)
    via_decode = (
        request.kv_bytes / max(1.0, decode_capacity - decode_load)
        + request.kv_bytes / request.compute_rdma_bandwidth
    )
    if via_decode < direct and storage_nic_load < 0.9:
        return "storage-to-decode"
    return "storage-to-prefill"

# The scheduler recomputes this per request and rebalances as load moves.

The constants are illustrative; the paper's scheduler estimates the same quantities online and accounts for the extra RDMA hop.
The compute-network transfer is the new cost, so the decision is only taken when the decode-side storage read is sufficiently cheaper.
Layerwise prefill streams the cache in chunks so the transfer overlaps with the first layers of computation, which is why the ablation attributes a 17% reduction to it alone.

The bottleneck and the fix: single-path loading caps effective storage bandwidth at the prefill pool's NICs, while DualPath pools prefill and decode NICs and pays one RDMA hop over the compute network.

Interference is the hard part

Adding a second path introduces traffic where none existed, and that traffic shares the compute network with tensor, pipeline, and expert-parallel collectives that are latency-critical. The paper's second contribution is NIC-centric traffic management: cache transfers are scheduled and shaped so they do not collide with model execution, and the scheduler avoids paths that would congest the fabric. This is the same lesson the Fire-Flyer and V3 hardware work arrived at from the training side: the network carries several kinds of traffic, and treating them as one undifferentiated stream is what turns a bandwidth advantage into a latency problem.

What the evaluation shows

The evaluation runs three models - a 660B DeepSeek model, a 27B DeepSeek model, and Qwen 32B - on production agentic traces with long contexts and high cache reuse, across prefill-to-decode engine ratios (1P1D means one prefill engine per decode engine; 2P1D, 1P2D scale those counts). DualPath reports up to 1.87x offline throughput and an average 1.96x online serving throughput without SLO violations, and an average 1.64x across all P/D configurations, peaking at 2.46x. The ablation separates the contributions: layerwise prefill accounts for 17.21% of the joint-completion-time reduction, dual-path loading for 38.19%, and the scheduling policy for 45.62%, all relative to the baseline stack. The small model shows the design's boundary: with limited storage bandwidth in a 1P1D layout, DualPath still trails an oracle that never waits on storage by 1.09x to 1.85x.

Limits and how to read it

The system is evaluated on the authors' in-house inference stack, not a public benchmark, so the multipliers depend on that baseline and its storage configuration. The benefits shrink when storage bandwidth is scarce or the P/D ratio is skewed. The paper assumes disaggregated prefill and decode, which not every deployment has. And it is a systems paper: it does not claim model quality gains, only throughput at fixed SLOs. Read it for the bandwidth-pooling argument and the interference-management design, both of which generalize beyond this stack.

2Inside the paper

Problem the paper sets out to solve

Agentic, multi-turn workloads turn the KV cache into storage I/O. In disaggregated inference, the conventional storage-to-prefill path makes prefill NICs the bottleneck while decode NICs are idle, capping throughput at a fraction of the cluster's aggregate storage bandwidth.

Key idea to look for while reading

Two paths and one scheduler. Section 3 describes the dual-path data flow and why the decode-side read followed by RDMA does not inherently congest the fabric; Section 4 covers NIC-centric traffic isolation; Section 5 covers the scheduling policy. Read the ablation table, which separates the contributions of layerwise prefill, the second path, and scheduling.

Evidence the paper musters

Up to 1.87x offline throughput and 1.96x average online throughput on production agentic workloads, across three models (660B, 27B DeepSeek and Qwen 32B) and three P/D ratios, with an average 1.64x and a peak 2.46x across configurations. Joint-completion-time reductions from the ablation: 17.21% for layerwise prefill, 38.19% adding dual-path loading, 45.62% adding the scheduler. Online results are reported at fixed SLOs.

Limits the authors admit

Evaluation is on an in-house stack with production traces, so absolute multipliers depend on the baseline. Gains shrink in storage-constrained or unbalanced P/D layouts, and the design assumes disaggregated serving. Interference management is a mitigation, not a proof that cache traffic never competes with collectives.

Practitioner takeaway

When one resource saturates while an equivalent one idles, the cheapest fix is usually routing, not capacity. In agent serving, pool every engine's storage bandwidth and decide per request whether the extra network hop is worth it, while keeping cache traffic isolated from latency-critical collectives. And measure with the metric that matches the workload: joint completion time and SLO-qualified throughput, not raw tokens per second.

3Implementation check

0 of 5 questions answered

1.What imbalance does DualPath target?
2.What exactly is the second path DualPath adds?
3.Why is interference management essential to the design?
4.The ablation adds techniques one at a time. Which ordering and attribution matches the paper?
5.Where does the reported improvement get weakest, and why?

4Build it yourself

Pool KV cache reads with an LRU/LFU simulator

Intermediate3-4 hours

Simulate the storage side of agentic serving: multi-turn sessions re-read a growing KV context that no longer fits in cache, and every miss is either a cheap storage reload or an expensive recompute. Implement LRU and LFU eviction, compare hit rates and recompute bytes saved, and see why the paper's reuse pattern favors recency over frequency.

Milestones

Generate a multi-turn trace: sessions whose context grows every turn, with occasional tool truncation, across more unique blocks than the cache can hold.
Implement the cache simulator: hits update recency; misses pay storage cost if the block was seen before and recompute cost otherwise.
Implement least-recently-used eviction and report hit rate, total cost, and recompute bytes saved.
Implement least-frequently-used eviction on the same trace and cost model.
Add an unbounded-cache replay to get the reuse upper bound and compare LRU's headroom against it.
Sweep cache capacity and report the capacity where LRU captures 80% of the upper-bound hits.
Explain the LFU collapse: which blocks it holds and why they are never reused.

Starter code

python · editable

import random
random.seed(13)
# EXPECTED before TODOs: 480 turns, 43008 accesses, 7680 unique blocks, upper
# bound 0.821, then "unimplemented". EXPECTED after: lru hit 0.445, lfu 0.011.
BLOCK_KB, CAPACITY, RECOMPUTE_COST, STORAGE_COST = 1024, 96, 10, 1
def make_trace(rng, sessions=40, turns=12, per_turn=16):
    trace = []
    for s in range(sessions):
        context = []
        for t in range(turns):
            context = context[-64:] if rng.random() < 0.15 else context
            context.extend((s, k) for k in range(t * per_turn, (t + 1) * per_turn))
            trace.append((s, list(context)))
    return trace
def choose_victim(entries, policy):
    """TODO: entries are (block, freq, last_step). lru -> smallest last_step;
    lfu -> smallest freq, ties broken by smallest last_step."""
    raise NotImplementedError("TODO: choose_victim")
def simulate(trace, policy):
    cache, seen, cost, hits = {}, set(), 0, 0
    for step, (_s, blocks) in enumerate(trace):
        for block in blocks:
            if block in cache:
                hits += 1
                freq, _ = cache[block]
                cache[block] = (freq + 1, step)
            else:
                cost += STORAGE_COST if block in seen else RECOMPUTE_COST; seen.add(block)
                cache[block] = (1, step)
            if len(cache) > CAPACITY:
                entries = [(b, f, last) for b, (f, last) in cache.items()]
                del cache[choose_victim(entries, policy)]
    return hits, cost
trace = make_trace(random)
accesses = sum(len(blocks) for _, blocks in trace)
unique = len({b for _, blocks in trace for b in blocks})
print("turns", len(trace), "accesses", accesses, "unique", unique, "capacity", CAPACITY,
      "unbounded-reuse upper bound", round(1 - unique / accesses, 3))
try:
    for policy in ("lru", "lfu"):
        hits, cost = simulate(trace, policy)
        print(policy, "hit rate", round(hits / accesses, 3), "cost units", cost,
              "recompute MB saved", round(hits * BLOCK_KB / 1024, 1))
except NotImplementedError as exc:
    print("unimplemented:", exc)

Runs locally in your browser.

Success criteria

LRU hit rate lands between 0.40 and 0.50 and LFU below 0.05 on the fixed trace, with both printed.
The script reports recompute MB saved and storage cost units, so the policies are compared on bytes, not only hits.
The capacity sweep shows a monotone hit-rate curve for LRU and identifies the 80% point.
One standard-library run with a fixed seed finishes in under 10 seconds.

Stretch goals

Age LFU counts (halve them every N steps) and see whether it recovers from cache pollution.
Add a size-2 admission filter and test whether scan resistance helps on the same trace.
Simulate dual-path loading: charge an extra compute-network cost for decode-side loads and pick the cheaper path per miss.

Problems

5Keep going

previousInsights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures nextDeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data

Problems

Articles

KV Cache & FlashAttention

Where this sits

What it is

1·Theory from first principles

What changed when agents arrived

The asymmetry, stated as a bandwidth budget

Interference is the hard part

What the evaluation shows

Limits and how to read it

2·Inside the paper

Problem the paper sets out to solve

Key idea to look for while reading

Evidence the paper musters

Limits the authors admit

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going