RAdam Rectification Term

Problem

Compute the RAdam rectification term.

With rho_inf = 2 / (1 - beta2) - 1 and rho_t = rho_inf - 2 * t * beta2^t / (1 - beta2^t), return [rho_t, rect] where rect = sqrt(((rho_t - 4) * (rho_t - 2) * rho_inf) / ((rho_inf - 4) * (rho_inf - 2) * rho_t)) when rho_t > 4, and 0.0 otherwise. The term scales the adaptive step early in training.

Hint

The rectification term stays zero until rho_t exceeds 4.

Test cases

case 1

in: [1,0.999]

expected: [1,0]

case 2

in: [5,0.999]

expected: [4.995998000395048,0.017311503166315034]

case 3

in: [10,0.99]

expected: [9.83419764817532,0.15516525667773864]

case 4

in: [100,0.9]

expected: [18.994687579117155,0.999806454677178]

Starter codePython

def radam_rectification(t, beta2):
    # Returns [rho_t, rect]
    # Your code here
    pass

4 test cases · run your code against them in the editor