Perplexity From Logits

Problem

Compute perplexity from logits and target token ids: perplexity = exp(mean over positions of the negative log softmax probability of the target). Use the log-sum-exp trick for stability.

The signature is perplexity(logits, targets). logits has shape (seq_len, vocab_size) and targets has length seq_len.

Hint

A uniform distribution over V classes gives perplexity V; confident correct predictions give near 1.

Test cases

case 1

in: [[[1,2,3]],[2]]

expected: 1.5032147244

case 2

in: [[[0,0],[0,0]],[0,1]]

expected: 2

case 3

in: [[[10,0],[0,10]],[0,1]]

expected: 1.0000453999

case 4

in: [[[2,2,2],[0,0,0]],[1,2]]

expected: 3

Starter codePython

import math
def perplexity(logits, targets):
    # Your code here
    pass

4 test cases · run your code against them in the editor