HardNLPnlp-061

Log-Likelihood Ratio Collocation

Problem

Return Dunning's log-likelihood ratio for the (w1, w2) collocation. Build the 2x2 table O11 = adjacent bigram count, O12 = c1 - O11, O21 = c2 - O11, and O22 = N - c1 - c2 + O11, with expected counts E_ij = row_total * col_total / N.

Return 2 * sum of O_ij * log(O_ij / E_ij) over cells with O_ij > 0. Return 0.0 for an empty corpus or if such a cell has zero expectation.

Hint

The four expected counts are row_total times col_total over N; multiply O by log of O times N over that product.

Test cases

case 1

in: [["new","york","new","york","city"],"new","york"]

expected: 6.730116670092565

case 2

in: [["the","cat","sat","on","the","mat"],"the","cat"]

expected: 2.6341457841558746

case 3

in: [[],"a","b"]

expected: 0

case 4

in: [["a","b","c"],"a","c"]

expected: 1.0464962875290957

Starter codePython

import math
def log_likelihood_ratio(tokens, w1, w2):
    # Your code here
    pass

4 test cases · run your code against them in the editor