GPTQ Hessian Layer Error

Problem

Compute the GPTQ layer-wise squared output error of quantizing a weight row: (w - w_q) H (w - w_q)^T, where H is the input-feature Hessian from calibration and is symmetric. Return the scalar error.

The signature is gptq_layer_error(w, w_q, H).

Hint

The Hessian weights errors in directions the calibration inputs excite most.

Test cases

case 1

in: [[1,0],[0,0],[[1,0],[0,1]]]

expected: 1

case 2

in: [[1,1],[0.5,0.5],[[1,0.5],[0.5,1]]]

expected: 0.75

case 3

in: [[2,-1],[2,-1],[[1,0],[0,1]]]

expected: 0

case 4

in: [[1,2],[0,0],[[2,0],[0,1]]]

expected: 6

Starter codePython

def gptq_layer_error(w, w_q, H):
    # Your code here
    pass

4 test cases · run your code against them in the editor