Decoupled vs Coupled Decay Delta

Problem

Measure the per-step difference between decoupled and coupled weight decay under an adaptive optimizer.

Coupled decay is scaled by the adaptive denominator, giving lr * wd * w / denom, while decoupled decay gives lr * wd * w. Return decoupled - coupled, where denom is sqrt(v_hat) + eps.

Hint

The adaptive denominator changes how much shrinkage a coupled penalty actually applies.

Test cases

case 1

in: [2,0.1,0.01,2]

expected: 0.001

case 2

in: [1,0.5,0.1,1]

expected: 0

case 3

in: [-3,0.2,0.05,4]

expected: -0.022500000000000006

case 4

in: [0,0.1,0.1,1]

expected: 0

Starter codePython

def decoupled_vs_coupled_delta(w, lr, wd, denom):
    # Your code here
    pass

4 test cases · run your code against them in the editor