Cross-Entropy of a Language Model

Problem

Given the log2-probabilities the model assigned to each observed token, return the average negative log2-probability (cross-entropy in bits):

H = -mean(log2_probs)

Assume the list is non-empty and every value is <= 0.

Hint

Perplexity is 2 raised to this cross-entropy.

case 1

in: [[-1,-1,-1]]

expected: 1

case 2

in: [[0,0]]

expected: 0

case 3

in: [[0,-2]]

expected: 1

case 4

in: [[-0.5,-1.5,-2]]

expected: 1.3333333333333333

Starter codePython

def lm_cross_entropy(log2_probs):
    # Your code here
    pass

4 test cases · run your code against them in the editor