Adam Epsilon Effect

Problem

Compare the two common placements of the Adam epsilon.

Bias-correct to m_hat and v_hat, then return [lr * m_hat / (sqrt(v_hat) + eps), lr * m_hat / sqrt(v_hat + eps)]: once with eps outside the square root and once inside it.

Hint

Adding eps before the root changes the effective step most when v_hat is tiny.

Test cases

case 1

in: [0.5,0.25,1,0.1,0.9,0.999,1e-8]

expected: [0.03162277658168381,0.03162277660105136]

case 2

in: [0.1,0.01,5,0.01,0.9,0.999,0.0001]

expected: [0.001724866624462107,0.0017249454321074265]

case 3

in: [-0.3,0.4,10,0.05,0.8,0.99,0.001]

expected: [-0.008212004764114034,-0.008215037972232048]

Starter codePython

def adam_epsilon_effect(m, v, t, lr, beta1, beta2, eps):
    # Returns [eps_outside, eps_inside]
    # Your code here
    pass

3 test cases · run your code against them in the editor