BatchNorm Backward (Simplified)

Problem

Backward pass through training-mode batch normalization for one feature with gamma = 1 and beta = 0. Given the upstream gradient dout and the input batch x, return dx_i = (1 / (n * std)) * (n * dout_i - sum(dout) - x_hat_i * sum(dout * x_hat)), where x_hat is the normalized input and std = sqrt(var + eps).

The signature is batchnorm_backward(dout, x, eps=1e-5).

Hint

There are three terms: the upstream gradient, a mean correction and a variance correction.

Test cases

case 1

in: [[1,1,1],[1,2,3]]

expected: [0,0,0]

case 2

in: [[1,-1],[0,2]]

expected: [0.0000099999,-0.0000099999]

case 3

in: [[2,-1,0.5],[1,3,5]]

expected: [0.4592801879,-0.9185569313,0.4592767433]

case 4

in: [[1,2],[-1,1]]

expected: [-0.0000049999,0.0000049999]

Starter codePython

import math
def batchnorm_backward(dout, x, eps=1e-5):
    # Your code here
    pass

4 test cases · run your code against them in the editor