All problems
QA F1 over Multiple References
Problem
Return the best token-overlap F1 of a prediction against several reference answers. prediction is a token list and golds is a list of token lists.
For each reference compute the clipped multiset F1, treating two empty lists as 1.0 and a single empty side as 0.0, then return the maximum score. Return 0.0 when there are no references.
Hint
Score against every reference and keep the maximum, which is the standard QA metric.
Test cases
case 1
in: [["a","b"],[["a","c"],["a","b","c"]]]
expected: 0.8
case 2
in: [[],[[]]]
expected: 1
case 3
in: [["a"],[]]
expected: 0
case 4
in: [["a","b","c"],[["a","b"],["x","y","z"]]]
expected: 0.8
case 5
in: [["a","a"],[["a"]]]
expected: 0.6666666666666666
Starter codePython
def qa_f1_multi(prediction, golds):
# Your code here
pass5 test cases · run your code against them in the editor