GELU Derivative (Approx)

Problem

Derivative of the tanh-approximated GELU: with u = sqrt(2/pi) * (x + 0.044715 * x^3) and t = tanh(u), the derivative is 0.5 * (1 + t) + 0.5 * x * (1 - t^2) * du/dx, where du/dx = sqrt(2/pi) * (1 + 3 * 0.044715 * x^2).

The signature is gelu_derivative(x).

Hint

Differentiate the product 0.5*x*(1+tanh(u)) with the chain rule on u.

Test cases

case 1

in: [[0]]

expected: [0.5]

case 2

in: [[1,-1]]

expected: [1.0829640838,-0.0829640838]

case 3

in: [[2,-2,0.5]]

expected: [1.0860992566,-0.0860992566,0.8673699035]

case 4

in: [[-3]]

expected: [-0.0115841666]

Starter codePython

import math
def gelu_derivative(x):
    # Your code here
    pass

4 test cases · run your code against them in the editor