NovoGrad Second Moment

Problem

Perform one NovoGrad update on a scalar parameter.

Update m = beta1 * m + (1 - beta1) * g and the scalar second moment v = beta2 * v + (1 - beta2) * g ** 2, then return [w - lr * m / (sqrt(v) + eps), m, v]. NovoGrad uses the gradient norm rather than its square as the second moment.

Hint

Without bias correction the first step is scaled by sqrt(0.001).

Test cases

case 1

in: [1,0,0,0.5,0.1,0.9,0.999,1e-8]

expected: [0.6837724339830358,0.04999999999999999,0.0002500000000000002]

case 2

in: [1,0.1,0.01,0.5,0.01,0.9,0.999,1e-8]

expected: [0.9861650366039507,0.14,0.01024]

case 3

in: [-2,-0.3,0.4,1.2,0.05,0.8,0.99,1e-8]

expected: [-2,-5.551115123125783e-17,0.41040000000000004]

Starter codePython

def novograd_update(w, m, v, g, lr, beta1, beta2, eps):
    # Returns [w_new, m, v]
    # Your code here
    pass

3 test cases · run your code against them in the editor