MediumNLPnlp-158

QA Token F1

Problem

Return the QA token F1 between prediction and gold after normalization: lowercase, replace non-alphanumeric characters with spaces, drop the articles a, an, and the, then compute clipped multiset F1.

Precision is matches / predicted tokens, recall is matches / gold tokens, and F1 = 2PR/(P+R). Return 1.0 when both sides normalize to empty and 0.0 when only one side is empty or nothing matches.

Hint

Normalize both sides identically before the clipped multiset matching.

Test cases

case 1

in: ["The cat","cat"]

expected: 1

case 2

in: ["a dog","the dog!"]

expected: 1

case 3

in: ["cat","cats"]

expected: 0

case 4

in: ["the","the"]

expected: 1

case 5

in: ["red car","blue car"]

expected: 0.5

Starter codePython

import re
def qa_token_f1(prediction, gold):
    # Your code here
    pass

5 test cases · run your code against them in the editor