Attention Row Entropy

Problem

Compute the entropy of each attention weight row: H = -sum(p * log(p)) over positive entries only. A uniform row has maximum entropy and a one-hot row has zero entropy.

The signature is attention_row_entropy(weights). Return one float per row.

Hint

Skip zero weights so log(0) never appears, and expect ln(row_length) for a uniform row.

Test cases

case 1

in: [[[0.5,0.5]]]

expected: [0.6931471806]

case 2

in: [[[1,0]]]

expected: [0]

case 3

in: [[[1]]]

expected: [0]

case 4

in: [[[0.25,0.25,0.25,0.25]]]

expected: [1.3862943611]

Starter codePython

import math
def attention_row_entropy(weights):
    # Your code here
    pass

4 test cases · run your code against them in the editor