HardNLPnlp-095

MTLD-lite Lexical Diversity

Problem

Estimate lexical diversity with a simplified MTLD. Scan tokens into segments; whenever a segment's type-token ratio falls to or below threshold, count one factor and start a fresh segment.

Close any leftover segment with partial credit (1 - TTR) / (1 - threshold). Return len(tokens) / factors, or len(tokens) when no factors were counted, and 0.0 for empty input.

Hint

Frequent repetition ends segments sooner, which raises the final score through more factors.

Test cases

case 1

in: [["a","b","c","d"],0.72]

expected: 4

case 2

in: [["a","a","a","a"],0.72]

expected: 2

case 3

in: [["a","b","a","c"],0.72]

expected: 4

case 4

in: [["a","b","c","a","b","c","a"],0.72]

expected: 7

case 5

in: [[],0.72]

expected: 0

Starter codePython

def mtld_lite(tokens, threshold):
    # Your code here
    pass

5 test cases · run your code against them in the editor