HardNLPnlp-300

WER with Alternative References

Problem

Compute the word error rate when each reference position accepts multiple surface forms. reference_alts is a list of positions, each a list of acceptable tokens for that position, and hypothesis is the recognized token list.

A substitution costs 0 when the hypothesis token is any accepted alternative at that position, otherwise 1. Return the minimum edit distance divided by the number of reference positions, or 0.0 for two empty inputs and 1.0 when only the hypothesis is non-empty.

Hint

Keep the usual dynamic program and make the substitution cost a membership test.

Test cases

case 1

in: [[["a"],["b"]],["a","b"]]

expected: 0

case 2

in: [[["a","b"],["c"]],["b","c"]]

expected: 0

case 3

in: [[["a"],["b"]],["b","a"]]

expected: 1

case 4

in: [[],[]]

expected: 0

case 5

in: [[],["x"]]

expected: 1

Starter codePython

def wer_with_alternatives(reference_alts, hypothesis):
    # Your code here
    pass

5 test cases · run your code against them in the editor