Natural Gradient 1D Step

Problem

Take one natural gradient step for a logistic model parameterized by logit z.

With p = 1 / (1 + exp(-z)), the Fisher information is p * (1 - p); return x - lr * grad / fisher. If the Fisher information is 0, return x unchanged.

Hint

Dividing by the Fisher information rescales the gradient by curvature.

case 1

in: [1,0.5,0,0.1]

expected: 0.8

case 2

in: [0,1,2,0.05]

expected: -0.4762195691083627

case 3

in: [2,-1,-1,0.2]

expected: 3.017232253926098

Starter codePython

def natural_gradient_step(x, grad, z, lr):
    # Your code here
    pass

3 test cases · run your code against them in the editor