LR Transfer Estimate

Problem

Estimate an equivalent SGD learning rate from an Adam learning rate.

Return adam_lr / sqrt(1 - beta2), the common heuristic that rescales the adaptive step back to the raw gradient scale.

Hint

Dividing by sqrt(1 - beta2) undoes the RMS scaling of the adaptive step.

case 1

in: [0.001,0.999]

expected: 0.031622776601683784

case 2

in: [0.01,0.9]

expected: 0.0316227766016838

case 3

in: [0.1,0.99]

expected: 0.9999999999999996

case 4

in: [0.001,0]

expected: 0.001

Starter codePython

def adam_to_sgd_lr_transfer(adam_lr, beta2):
    # Your code here
    pass

4 test cases · run your code against them in the editor