Sinusoidal Positional Encoding

Problem

Compute sinusoidal positional encodings for a sequence.

For position pos and pair index i, PE[pos][2i] = sin(pos / 10000^(2i/d_model)) and PE[pos][2i+1] = cos(pos / 10000^(2i/d_model)). Return a seq_len x d_model matrix and assume d_model is even.

Hint

Even indices use sin, odd indices use cos, and the wavelength grows with i.

Test cases

case 1

in: [1,4]

expected: [[0,1,0,1]]

case 2

in: [2,4]

expected: [[0,1,0,1],[0.8414709848078965,0.5403023058681398,0.009999833334166664,0.9999500004166653]]

case 3

in: [0,4]

expected: []

case 4

in: [3,2]

expected: [[0,1],[0.8414709848078965,0.5403023058681398],[0.9092974268256817,-0.4161468365471424]]

Starter codePython

import math
def positional_encoding(seq_len, d_model):
    # Your code here
    pass

4 test cases · run your code against them in the editor