Normalized SGD Scale

Problem

Scale a gradient to unit norm before stepping.

Return lr * g_i / (||g|| + eps) for each coordinate. A zero gradient returns all zeros.

Hint

Normalized SGD makes the step length independent of the gradient magnitude.

case 1

in: [[3,4],0.1,1e-8]

expected: [0.05999999988000001,0.07999999984]

case 2

in: [[1,0],0.5,0]

expected: [0.5,0]

case 3

in: [[0,0],1,1e-8]

expected: [0,0]

Starter codePython

def normalized_sgd_scale(grad, lr, eps):
    # Your code here
    pass

3 test cases · run your code against them in the editor