MediumNLPnlp-132

Detokenization Spacing Rules

Problem

Join tokens back into a string using simple spacing rules: no space before closing punctuation .,!?;:%)]}, no space after an opening bracket at the end of the output, and no space before the contraction suffix "'s".

All other tokens are separated by a single space.

Hint

Build the string incrementally and inspect the current tail before adding a token.

Test cases

case 1

in: [["Hello",",","world","!"]]

expected: "Hello, world!"

case 2

in: [["I","like","it","."]]

expected: "I like it."

case 3

in: [["(","a",")"]]

expected: "(a)"

case 4

in: [[]]

expected: ""

case 5

in: [["a","'s","b"]]

expected: "a's b"

Starter codePython

def detokenize(tokens):
    # Your code here
    pass

5 test cases · run your code against them in the editor