MediumNLPnlp-310

Perplexity Comparison from Logprobs

Problem

Compare two language models from token log probabilities. perplexity is exp of the negative mean log probability, and an empty list is treated as perplexity 1.0.

Return [perplexity_a, perplexity_b, winner], where winner is 'a' for the lower perplexity, 'b' for the higher perplexity of the other model, or 'tie' when the values are equal.

Hint

Lower perplexity means the model assigned higher average probability to the tokens.

Test cases

case 1

in: [[-1,-2],[-0.5,-0.5]]

expected: [4.4816890703380645,1.6487212707001282,"b"]

case 2

in: [[-1],[-1]]

expected: [2.718281828459045,2.718281828459045,"tie"]

case 3

in: [[],[]]

expected: [1,1,"tie"]

case 4

in: [[-0.5],[]]

expected: [1.6487212707001282,1,"b"]

case 5

in: [[-2,-2],[-1,-1,-1]]

expected: [7.38905609893065,2.718281828459045,"b"]

Starter codePython

import math
def perplexity_comparison(logprobs_a, logprobs_b):
    # Your code here
    pass

5 test cases · run your code against them in the editor