HardNLPnlp-022

WordPiece Longest-Match Tokenizer

Problem

Tokenize word with the WordPiece longest-match-first strategy. At each position, emit the longest prefix of the remaining text that appears in vocab.

If no prefix matches, emit the literal '[UNK]' and advance one character. Return the resulting list of tokens.

Hint

Shrink the candidate window from the full remaining string down to one character.

case 1

in: ["playing",["play","ing","p","l","a","y","i","n","g"]]

expected: ["play","ing"]

case 2

in: ["unwanted",["un","want","ed"]]

expected: ["un","want","ed"]

case 3

in: ["abc",["a"]]

expected: ["a","[UNK]","[UNK]"]

case 4

in: ["x",["x"]]

expected: ["x"]

case 5

in: ["book",["bo","ok","b","o","k"]]

expected: ["bo","ok"]

Starter codePython

def wordpiece_tokenize(word, vocab):
    # Your code here
    pass

5 test cases · run your code against them in the editor