MediumNLPnlp-099

Tokenizer Fertility Ratio

Problem

Return the tokenizer fertility: the average number of subword tokens per word.

Tokens prefixed with '##' are continuations and do not start a new word. Return 0.0 when no word starts appear.

Hint

Count words as the tokens that do not begin with the continuation marker.

case 1

in: [["play","##ing","run","##ning"]]

expected: 2

case 2

in: [["hello","world"]]

expected: 1

case 3

in: [[]]

expected: 0

case 4

in: [["##ed"]]

expected: 0

Starter codePython

def tokenizer_fertility(tokens):
    # Your code here
    pass

4 test cases · run your code against them in the editor