Information Bottleneck Objective

Problem

Compute the information bottleneck objective in bits for the joint pxy and the encoder pt_given_x where pt_given_x[x][t] = P(t | x):

L = I(X; T) - beta * I(T; Y)

Build p(x, t, y) = pxy[x][y] * P(t | x), marginalize, and evaluate both mutual informations in bits.

Hint

A constant encoder destroys all information, so I(X; T) and I(T; Y) both vanish.

case 1

in: [[[0.5,0],[0,0.5]],[[1,0],[0,1]],0.5]

expected: 0.5

case 2

in: [[[0.5,0],[0,0.5]],[[1,0],[0,1]],2]

expected: -1

case 3

in: [[[0.5,0],[0,0.5]],[[0.5,0.5],[0.5,0.5]],1]

expected: 0

case 4

in: [[[0.4,0.1],[0.1,0.4]],[[0.9,0.1],[0.1,0.9]],1]

expected: 0.35775077890333673

Starter codePython

import math
def information_bottleneck(pxy, pt_given_x, beta):
    # Your code here
    pass

4 test cases · run your code against them in the editor