EasyNLPnlp-096

Byte-Level BPE Token Count

Problem

Return the number of tokens a byte-level tokenizer would produce for text, which equals its UTF-8 byte length.

ASCII characters count as one byte, while accented letters and emoji may use several.

Hint

str.encode returns the UTF-8 bytes of a string.

case 1

in: ["abc"]

expected: 3

case 2

in: [""]

expected: 0

case 3

in: ["é"]

expected: 2

case 4

in: ["😀"]

expected: 4

case 5

in: ["aé"]

expected: 3

Starter codePython

def byte_bpe_count(text):
    # Your code here
    pass

5 test cases · run your code against them in the editor