LayerNorm Forward

Problem

Layer normalization over the last dimension. For each row compute the mean and biased variance, normalize, then scale by gamma and shift by beta.

The signature is layernorm_forward(x, gamma, beta, eps=1e-5) where x is a 2D matrix and gamma/beta are vectors with the row length.

Hint

Normalize each row independently so rows with the same shape normalize identically.

case 1

in: [[[1,2,3]],[1,1,1],[0,0,0]]

expected: [[-1.2247356859,0,1.2247356859]]

case 2

in: [[[0,0],[0,0]],[1,1],[0,0]]

expected: [[0,0],[0,0]]

case 3

in: [[[1,-1]],[2,3],[1,0]]

expected: [[2.9999900001,-2.9999850001]]

case 4

in: [[[1,2],[3,4]],[1,1],[0,0]]

expected: [[-0.9999800006,0.9999800006],[-0.9999800006,0.9999800006]]

Starter codePython

import math
def layernorm_forward(x, gamma, beta, eps=1e-5):
    # Your code here
    pass

4 test cases · run your code against them in the editor