LayerNorm vs RMSNorm Outputs

Problem

Compute LayerNorm and RMSNorm outputs for the same input so the difference is visible. LayerNorm subtracts the mean and divides by the standard deviation, while RMSNorm divides by the root mean square without centering. Both scale by weight and add eps inside the square root.

The signature is norm_compare(x, weight, eps=1e-5). Return [layernorm_output, rmsnorm_output].

Hint

A constant input makes LayerNorm output exactly zero but leaves RMSNorm nonzero.

Test cases

case 1

in: [[1,3],[1,1],0]

expected: [[-1,1],[0.4472135955,1.3416407865]]

case 2

in: [[0,0],[1,1],1]

expected: [[0,0],[0,0]]

case 3

in: [[2,2,2],[1,2,3],0.00001]

expected: [[0,0,0],[0.99999875,1.9999975,2.99999625]]

case 4

in: [[1,2,3],[1,1,1],0.00001]

expected: [[-1.2247356859,0,1.2247356859],[0.4629095539,0.9258191078,1.3887286617]]

Starter codePython

import math
def norm_compare(x, weight, eps=1e-5):
    # Your code here
    pass

4 test cases · run your code against them in the editor