HardNLPnlp-375

Attention Weights Entropy Nats

Problem

The entropy of an attention row after softmax is -sum_i p_i ln p_i, measured in nats. Compute the softmax from the scores first.

Given the raw attention scores, return the entropy.

Hint

Uniform attention maximizes the entropy at ln(number of keys).

case 1

in: [[0,0]]

expected: 0.6931471805599453

case 2

in: [[1,2,3]]

expected: 0.8323955818399389

case 3

in: [[10,0]]

expected: 0.0004993775862411646

Starter codePython

def attention_weights_entropy_nats(scores):
    # Your code here
    pass

3 test cases · run your code against them in the editor