SwiGLU Forward Value

Problem

Evaluate a SwiGLU feed-forward layer on one input vector: out[j] = SiLU(W[j] . x) * (V[j] . x), where SiLU(a) = a / (1 + exp(-a)).

The signature is swiglu_forward(x, W, V). W and V are (hidden, d) matrices and x has length d; the result has length hidden.

Hint

One projection produces the gate that is passed through SiLU, the other produces the value that multiplies it.

case 1

in: [[1],[[0]],[[2]]]

expected: [0]

case 2

in: [[1],[[1]],[[1]]]

expected: [0.7310585786]

case 3

in: [[0,1],[[1,0],[0,1]],[[1,1],[1,-1]]]

expected: [0,-0.7310585786]

case 4

in: [[1,2],[[1,1]],[[1,0]]]

expected: [2.8577223805]

Starter codePython

import math
def swiglu_forward(x, W, V):
    # Your code here
    pass

4 test cases · run your code against them in the editor