AdaDelta Accumulated Update

Problem

AdaDelta updates the gradient accumulator G_new = rho * G + (1 - rho) g^2 and the update accumulator D_new = rho * D + (1 - rho) delta^2, where delta = -sqrt(D + eps) / sqrt(G_new + eps) * g and x_new = x + delta.

Given x, g, G, D, rho, and eps, return [x_new, G_new, D_new].

Hint

Use the old update accumulator in this step, then update it with the new delta.

Test cases

case 1

in: [1,0.5,0,0,0.9,0.000001]

expected: [0.9968377855834876,0.024999999999999994,9.99960001599936e-7]

case 2

in: [0,-2,1,0.25,0.5,1e-8]

expected: [0.6324555434178754,2.5,0.3250000072]

case 3

in: [2,1,4,1,0.9,0]

expected: [1.4801247550899637,3.7,0.927027027027027]

Starter codePython

def adadelta_accumulated_update(x, g, g_acc, d_acc, rho, eps):
    # Your code here
    pass

3 test cases · run your code against them in the editor