EasyNLPproj-016

Query Tokenization

Problem

Normalize and tokenize a search query.

Lowercase the text, replace every non-alphanumeric non-whitespace character with a space, split on whitespace, and drop tokens that appear in the stopwords list. Return the remaining tokens in order.

Hint

Clean the text first, then filter whole tokens against the stopword list.

Test cases

case 1

in: ["The cat sat",["the"]]

expected: ["cat","sat"]

case 2

in: ["Hello, World! hello",[]]

expected: ["hello","world","hello"]

case 3

in: ["a b c",["a","b","c"]]

expected: []

case 4

in: ["Don't stop",["don","t"]]

expected: ["stop"]

Starter codePython

def tokenize_query(text, stopwords):
    # Your code here
    pass

4 test cases · run your code against them in the editor