MediumNLPnlp-311

Length-Normalized Logprob Flip Check

Problem

Check whether length normalization changes the ranking of two hypotheses. Normalize each log probability as logprob / (length ** alpha), using the raw logprob when the length is not positive.

Return 'tie' when the normalized scores are equal, 'same' when the normalized winner also has the higher raw logprob, and 'flipped' when normalization reverses the raw ranking.

Hint

Because log probabilities are negative, longer sequences get a boost from normalization.

Test cases

case 1

in: [-2,10,-1.5,1,1]

expected: "flipped"

case 2

in: [-2,10,-1.5,1,0]

expected: "same"

case 3

in: [-1,2,-2,4,1]

expected: "tie"

case 4

in: [-0.5,2,-2,4,1]

expected: "same"

case 5

in: [-1,0,-2,4,1]

expected: "flipped"

Starter codePython

def length_norm_flip_check(logprob_a, length_a, logprob_b, length_b, alpha):
    # Your code here
    pass

5 test cases · run your code against them in the editor