Cross-Entropy Between Distributions

Problem

Compute the cross-entropy in bits between a true distribution p and a model distribution q:

H(p, q) = -sum(p_i * log2(q_i))

Skip terms with p_i = 0. Assume q_i > 0 wherever p_i > 0.

Hint

Cross-entropy equals H(p) + D(p || q).

case 1

in: [[0.5,0.5],[0.5,0.5]]

expected: 1

case 2

in: [[0.5,0.5],[0.25,0.75]]

expected: 1.207518749639422

case 3

in: [[1,0],[0.5,0.5]]

expected: 1

case 4

in: [[0.25,0.25,0.25,0.25],[0.25,0.25,0.25,0.25]]

expected: 2

Starter codePython

import math
def cross_entropy(p, q):
    # Your code here
    pass

4 test cases · run your code against them in the editor