EasyNLPnlp-223

Tool Selection Accuracy

Problem

Return the fraction of examples where the highest-scoring tool matches the gold tool. tools[i] lists the candidate names and scores[i] the parallel scores.

Ties keep the earliest candidate, and an empty gold list gives 0.0.

Hint

Argmax per example, then compare with the gold tool name.

Test cases

case 1

in: [[["search","calc"],["search","calc"]],[[0.7,0.3],[0.2,0.8]],["search","calc"]]

expected: 1

case 2

in: [[["search","calc"],["search","calc"]],[[0.7,0.3],[0.2,0.8]],["calc","calc"]]

expected: 0.5

case 3

in: [[],[],[]]

expected: 0

case 4

in: [[["a","b"]],[[0.5,0.5]],["a"]]

expected: 1

Starter codePython

def tool_selection_accuracy(tools, scores, gold):
    # Your code here
    pass

4 test cases · run your code against them in the editor