Natural Gradient Fisher Diagonal

Problem

Take one natural gradient step for a Gaussian model with parameters x = [mu, log_sigma].

The diagonal Fisher information is [1 / sigma ** 2, 2] with sigma = exp(x[1]). Return [x_0 - lr * grad_0 / f_0, x_1 - lr * grad_1 / f_1].

Hint

The Fisher diagonal rescales each parameter by its own curvature.

case 1

in: [[0,0],[1,1],0.1]

expected: [-0.1,-0.05]

case 2

in: [[1,0.5],[0.5,0.5],0.2]

expected: [0.7281718171540954,0.45]

case 3

in: [[-1,-0.5],[2,1],0.05]

expected: [-1.0367879441171441,-0.525]

Starter codePython

def natural_gradient_fisher_diagonal(x, grad, lr):
    # Your code here
    pass

3 test cases · run your code against them in the editor