KL Divergence (Discrete)

Problem

Compute the Kullback-Leibler divergence D(p || q) in bits:

D = sum(p_i * log2(p_i / q_i))

Skip terms with p_i = 0. Assume q_i > 0 wherever p_i > 0.

Hint

KL divergence is always >= 0 and is 0 only when p equals q.

case 1

in: [[0.5,0.5],[0.5,0.5]]

expected: 0

case 2

in: [[0.5,0.5],[0.25,0.75]]

expected: 0.20751874963942185

case 3

in: [[0.9,0.1],[0.5,0.5]]

expected: 0.5310044064107189

case 4

in: [[0.25,0.25,0.25,0.25],[0.5,0.25,0.125,0.125]]

expected: 0.25

case 5

in: [[1,0],[0.25,0.75]]

expected: 2

Starter codePython

import math
def kl_divergence(p, q):
    # Your code here
    pass

5 test cases · run your code against them in the editor