SGD With Decay

Problem

Take one SGD step with inverse-time learning-rate decay.

Compute lr = lr0 / (1 + decay * step) and return [w - lr * grad, lr], so the caller also sees the decayed rate.

Hint

The effective learning rate shrinks as 1 / (1 + decay * step).

case 1

in: [1,0.5,0.1,0.01,100]

expected: [0.975,0.05]

case 2

in: [1,0.5,0.1,0.1,9]

expected: [0.9736842105263158,0.052631578947368425]

case 3

in: [0,-2,0.05,0.5,5]

expected: [0.028571428571428574,0.014285714285714287]

case 4

in: [2,0,0.1,1,3]

expected: [2,0.025]

Starter codePython

def sgd_with_decay(w, grad, lr0, decay, step):
    # Returns [w_new, lr]
    # Your code here
    pass

4 test cases · run your code against them in the editor