HardNLPnlp-060

T-Score Collocation

Problem

Return the t-score (O - E) / sqrt(O), where O is the number of adjacent (w1, w2) bigrams and E = c1 * c2 / N is the expected count under independence.

Return 0.0 when O is 0 or the corpus is empty.

Hint

The t-score divides the excess over chance by the square root of the observed count.

Test cases

case 1

in: [["new","york","new","york","city"],"new","york"]

expected: 0.8485281374238569

case 2

in: [["the","cat","sat","on","the","mat"],"the","cat"]

expected: 0.6666666666666667

case 3

in: [["a","b","c"],"b","a"]

expected: 0

case 4

in: [[],"a","b"]

expected: 0

Starter codePython

import math
def t_score(tokens, w1, w2):
    # Your code here
    pass

4 test cases · run your code against them in the editor