MediumNLPnlp-237

Cross-Lingual Fertility Gap

Problem

Compute tokenizer fertility for two languages and the gap between them. Fertility is the number of tokens divided by the number of words, and is 0.0 when a side has no words.

Return [fert_en, fert_xx, fert_xx - fert_en].

Hint

Compute each side independently, then subtract source from target.

Test cases

case 1

in: [["the","cat"],["the","cat"],["el","gato","corre"],["el","gato"]]

expected: [1,1.5,0.5]

case 2

in: [["a","b","c","d"],["a","b"],["x"],["x"]]

expected: [2,1,-1]

case 3

in: [[],[],[],[]]

expected: [0,0,0]

case 4

in: [["a"],["a"],[],["x"]]

expected: [1,0,-1]

Starter codePython

def fertility_gap(tokens_en, words_en, tokens_xx, words_xx):
    # Your code here
    pass

4 test cases · run your code against them in the editor