MediumNLPnlp-059

PMI Collocation Score

Problem

Return the pointwise mutual information score log((O * N) / (c1 * c2)), where O is the number of times w1 is immediately followed by w2, c1 and c2 are the token counts of w1 and w2, and N is the corpus length.

Return 0.0 when any of those counts is 0 or the corpus is empty.

Hint

O * N / (c1 * c2) measures how much more often the pair co-occurs than chance.

Test cases

case 1

in: [["new","york","new","york","city"],"new","york"]

expected: 0.9162907318741551

case 2

in: [["the","cat","sat","on","the","mat"],"the","cat"]

expected: 1.0986122886681098

case 3

in: [["a","b"],"a","z"]

expected: 0

case 4

in: [[],"a","b"]

expected: 0

case 5

in: [["a","b","c"],"a","c"]

expected: 0

Starter codePython

import math
def pmi_score(tokens, w1, w2):
    # Your code here
    pass

5 test cases · run your code against them in the editor