Decode Latency Estimate

Problem

Estimate the per-token decode latency when generation is memory-bandwidth bound: (model_bytes + kv_cache_bytes) / (bandwidth_gbps * 1e6) milliseconds.

The signature is decode_latency_ms(model_bytes, kv_cache_bytes, bandwidth_gbps).

Hint

One decode step must stream all weights and the whole KV cache through the GPU.

Test cases

case 1

in: [1000000000,100000000,100]

expected: 11

case 2

in: [1000000000,0,1000]

expected: 1

case 3

in: [2000000000,200000000,50]

expected: 44

case 4

in: [1000000,0,10]

expected: 0.1

Starter codePython

def decode_latency_ms(model_bytes, kv_cache_bytes, bandwidth_gbps):
    # Your code here
    pass

4 test cases · run your code against them in the editor