MediumNLPnlp-007

Whitespace Tokenizer with Offsets

Problem

Return a list of [token, start, end] triples for every whitespace-separated token in text. start is the index of the token's first character and end is one past its last character.

Original casing and punctuation are preserved. Return an empty list for empty or all-whitespace text.

Hint

Walk the string with an index and skip over whitespace runs.

Test cases

case 1

in: ["Hello world"]

expected: [["Hello",0,5],["world",6,11]]

case 2

in: [" a b "]

expected: [["a",2,3],["b",4,5]]

case 3

in: ["one"]

expected: [["one",0,3]]

case 4

in: [""]

expected: []

Starter codePython

def whitespace_tokenize_offsets(text):
    # Your code here
    pass

4 test cases · run your code against them in the editor