Quantization Latency Model

Problem

Model quantized inference latency as the sum of weight-loading time, dequantization time, and compute time: weight_bytes / (bandwidth_gbps * 1e6) + dequant_ms + compute_ms. Bandwidth is in GB/s and the memory term is in milliseconds.

The signature is quantization_latency_model(weight_bytes, bandwidth_gbps, dequant_ms, compute_ms).

Hint

At small batch sizes weight loading dominates, so lower bits cut latency directly.

Test cases

case 1

in: [1000000000,1000,1,2]

expected: 4

case 2

in: [0,1000,0.5,0]

expected: 0.5

case 3

in: [500000000,500,0.25,3]

expected: 4.25

case 4

in: [100000,100,0,0]

expected: 0.001

Starter codePython

def quantization_latency_model(weight_bytes, bandwidth_gbps, dequant_ms, compute_ms):
    # Your code here
    pass

4 test cases · run your code against them in the editor