EasyNLPnlp-001

Tokenization

Problem

Tokenize text into lowercase alphanumeric tokens.

Lowercase the text, replace any non-alphanumeric non-whitespace character with a space, then split on whitespace.

case 1

in: ["Hello, World!"]

expected: ["hello","world"]

case 2

in: ["The quick brown fox."]

expected: ["the","quick","brown","fox"]

case 3

in: ["Don't stop!"]

expected: ["don","t","stop"]

case 4

in: ["One2Three4"]

expected: ["one2three4"]

Starter codePython

def tokenize(text):
    # Your code here
    pass

4 test cases · run your code against them in the editor