MediumNLPnlp-070

Multi-Head Attention Projection Parameter Count

Problem

Return the parameter count of the attention projections: query, key, value, and output projections, each d_model x d_model with a bias vector.

That is 4 * d_model ** 2 + 4 * d_model.

Hint

Each of the four projections contributes d squared weights plus d biases.

case 1

in: [4]

expected: 80

case 2

in: [8]

expected: 288

case 3

in: [1]

expected: 8

case 4

in: [768]

expected: 2362368

Starter codePython

def multihead_attention_param_count(d_model):
    # Your code here
    pass

4 test cases · run your code against them in the editor