Quantized Inference Latency

Problem

Estimate quantized inference latency as max(memory time, compute time), where memory time = weight_bytes / (bandwidth_gbps * 1e6) milliseconds.

The signature is quantized_latency_ms(weight_bytes, bandwidth_gbps, compute_ms).

Hint

4-bit weights move 8x fewer bytes, which is why quantization speeds up decode so much.

case 1

in: [1000000000,1000,1]

expected: 1

case 2

in: [1000000000,100,0.5]

expected: 10

case 3

in: [500000000,500,2]

expected: 2

case 4

in: [2000000000,1000,0.1]

expected: 2

Starter codePython

def quantized_latency_ms(weight_bytes, bandwidth_gbps, compute_ms):
    # Your code here
    pass

4 test cases · run your code against them in the editor