Per-Tensor vs Per-Channel Error

Problem

Compute the mean squared quantization error of a weight matrix.

Quantize each value with q = round(v / scale) * scale. When per_channel is False use one scale = max|v| / 127 over all rows; when True use a separate scale per row. Return the mean squared error rounded to 4 decimal places. Rows that are all zeros quantize to 0.

Hint

Per-channel scaling uses the row maximum to build each scale.

Test cases

case 1

in: [[[100,-100],[10,-10]],false]

expected: 0.0279

case 2

in: [[[100,-100],[10,-10]],true]

expected: 0

case 3

in: [[[1.5,-1.5]],false]

expected: 0

Starter codePython

def quant_error(rows, per_channel):
    # Your code here
    pass

3 test cases · run your code against them in the editor