MediumNLPnlp-314

Label Smoothing Token Loss

Problem

Compute label-smoothed cross-entropy for one token. logits is the score list, target is the correct class index, and smoothing is the label smoothing factor epsilon in [0, 1).

Using log_softmax values lp, loss = -(1 - epsilon) * lp[target] - (epsilon / K) * sum(lp). Compute the log softmax with the max-subtraction trick. Return 0.0 for empty logits.

Hint

Smoothing blends the target log probability with the mean of all log probabilities.

Test cases

case 1

in: [[0,0],0,0]

expected: 0.6931471805599453

case 2

in: [[0,0],0,0.1]

expected: 0.6931471805599453

case 3

in: [[1,0],0,0]

expected: 0.31326168751822286

case 4

in: [[1,0],0,0.2]

expected: 0.4132616875182229

case 5

in: [[0,1],0,0.5]

expected: 1.0632616875182228

Starter codePython

import math
def label_smoothing_cross_entropy(logits, target, smoothing):
    # Your code here
    pass

5 test cases · run your code against them in the editor