MediumNLPnlp-323

Regex Word Tokenizer Lite

Problem

A word tokenizer extracts maximal runs of letters and digits, discarding punctuation.

Given text, return the list of alphanumeric runs using a regular expression.

Hint

One or more letters or digits per token.

case 1

in: ["Hello, world!"]

expected: ["Hello","world"]

case 2

in: ["a-b_c"]

expected: ["a","b","c"]

case 3

in: ["...123 abc..."]

expected: ["123","abc"]

Starter codePython

def regex_word_tokenizer_lite(text):
    # Your code here
    pass

3 test cases · run your code against them in the editor