EasyNLPnlp-320

Dialogue State Slot Accuracy

Problem

Compute slot accuracy over dialogue turns. gold_states and pred_states are parallel lists of state dictionaries mapping slot names to values, and a slot that is missing from the prediction counts as incorrect.

Count every slot in the union of gold and predicted slots for each turn, and return the fraction of slots whose value matches gold. Return 0.0 when there are no slots. Extra predicted slots are counted but never correct.

Hint

Use the union of gold and predicted slots per turn; a missing slot is a miss, an extra slot is not a hit.

Test cases

case 1

in: [[{"a":"1","b":"2"},{"a":"1"}],[{"a":"1","b":"3"},{"a":"1"}]]

expected: 0.6666666666666666

case 2

in: [[{"a":"1"}],[{"a":"1","c":"9"}]]

expected: 0.5

case 3

in: [[],[]]

expected: 0

case 4

in: [[{"a":"1"}],[]]

expected: 0

case 5

in: [[{"a":"x","b":"y"}],[{"a":"x","b":"y"}]]

expected: 1

Starter codePython

def dialogue_slot_accuracy(gold_states, pred_states):
    # Your code here
    pass

5 test cases · run your code against them in the editor