Token-Level F1 for Generation

Problem

Tokenize the prediction and reference on whitespace and compute their multiset overlap. Precision is overlap / len(prediction tokens), recall is overlap / len(reference tokens), and the returned F1 is their harmonic mean. Return 0.0 if either side is empty or the overlap is zero.

Test cases

case 1

in: ["the cat sat","the cat"]

expected: 0.8

case 2

in: ["a b c","a b c"]

expected: 1

case 3

in: ["x","y"]

expected: 0

case 4

in: ["","hello"]

expected: 0

case 5

in: ["the the cat","the cat cat"]

expected: 0.6666666666666666

Starter codePython

def token_f1(pred, ref):
    # Your code here
    pass

5 test cases · run your code against them in the editor