Sinusoidal Positional Encoding

Problem

Build the sinusoidal positional encoding matrix of shape (seq_len, d_model): PE[pos, 2k] = sin(pos / 10000^(2k / d_model)) and PE[pos, 2k + 1] = cos(pos / 10000^(2k / d_model)).

The signature is sinusoidal_positional_encoding(seq_len, d_model).

Hint

Even feature indices use sine, odd indices use cosine, with the frequency halving every pair.

Test cases

case 1

in: [3,4]

expected: [[0,1,0,1],[0.8414709848,0.5403023059,0.0099998333,0.9999500004],[0.9092974268,-0.4161468365,0.0199986667,0.9998000067]]

case 2

in: [1,2]

expected: [[0,1]]

case 3

in: [2,6]

expected: [[0,1,0,1,0,1],[0.8414709848,0.5403023059,0.0463992235,0.998922976,0.002154433,0.9999976792]]

case 4

in: [4,2]

expected: [[0,1],[0.8414709848,0.5403023059],[0.9092974268,-0.4161468365],[0.1411200081,-0.9899924966]]

Starter codePython

import math
def sinusoidal_positional_encoding(seq_len, d_model):
    # Your code here
    pass

4 test cases · run your code against them in the editor