GeGLU Forward Value

Problem

Evaluate a GeGLU feed-forward layer on one input vector: out[j] = GELU(W[j] . x) * (V[j] . x), using the exact GELU = 0.5 * a * (1 + erf(a / sqrt(2))).

The signature is geglu_forward(x, W, V). W and V are (hidden, d) matrices and x has length d; the result has length hidden.

Hint

GeGLU is SwiGLU with the erf-based exact GELU instead of SiLU as the gate.

Test cases

case 1

in: [[1],[[0]],[[3]]]

expected: [0]

case 2

in: [[1],[[1]],[[1]]]

expected: [0.8413447461]

case 3

in: [[0,1],[[1,0],[0,1]],[[1,1],[1,-1]]]

expected: [0,-0.8413447461]

case 4

in: [[1,2],[[1,1]],[[1,0]]]

expected: [2.9959503059]

Starter codePython

import math
def geglu_forward(x, W, V):
    # Your code here
    pass

4 test cases · run your code against them in the editor