MediumNLPnlp-073

Attention Entropy

Problem

Return the Shannon entropy -sum(p * log(p)) of a row of attention weights using the natural log.

Zero-probability entries contribute nothing, and an empty row yields 0.0.

Hint

A peaked attention row has zero entropy; a uniform row is maximally uncertain.

case 1

in: [[1]]

expected: 0

case 2

in: [[0.5,0.5]]

expected: 0.6931471805599453

case 3

in: [[0.25,0.25,0.25,0.25]]

expected: 1.3862943611198906

case 4

in: [[]]

expected: 0

case 5

in: [[1,0,0]]

expected: 0

Starter codePython

import math
def attention_entropy(weights):
    # Your code here
    pass

5 test cases · run your code against them in the editor