Per-Channel vs Per-Tensor MSE

Problem

Compare symmetric int quantization MSE for a weight matrix with one per-tensor scale versus one scale per row. Quantize with round(x/scale) clamped to +/-(2^(bits-1)-1), dequantize, and return [mse_per_tensor, mse_per_channel] averaged over all elements. Rows or matrices of zeros contribute zero error.

The signature is per_channel_vs_tensor_mse(W, bits=8).

Hint

Per-channel scales help most when rows have very different dynamic ranges.

Test cases

case 1

in: [[[1,0],[100,0]],8]

expected: [0.011299522599045193,0]

case 2

in: [[[0.7,-0.3],[0.3,-0.7]],8]

expected: [0.0000027900055800111765,0.0000027900055800111765]

case 3

in: [[[0,0],[0,0]],8]

expected: [0,0]

case 4

in: [[[1,0],[2,1]],8]

expected: [0.00003100006200012389,0.000015500031000061944]

Starter codePython

def per_channel_vs_tensor_mse(W, bits=8):
    # Your code here
    pass

4 test cases · run your code against them in the editor