Gradient Clipping (Global Norm)

Problem

Clip a list of gradient rows by their global L2 norm: if the norm of all entries exceeds max_norm, scale every entry by max_norm / norm; otherwise return the gradients unchanged.

The signature is clip_global_norm(grads, max_norm). grads is a list of equal-length rows.

Hint

Square every element to get the global norm, then rescale the whole gradient by one shared factor.

Test cases

case 1

in: [[[3,4]],5]

expected: [[3,4]]

case 2

in: [[[3,4]],2.5]

expected: [[1.5,2]]

case 3

in: [[[1,1],[1,1]],10]

expected: [[1,1],[1,1]]

case 4

in: [[[0,0]],1]

expected: [[0,0]]

case 5

in: [[[6,8],[0,0]],5]

expected: [[3,4],[0,0]]

Starter codePython

import math
def clip_global_norm(grads, max_norm):
    # Your code here
    pass

5 test cases · run your code against them in the editor