Why Softmax Needs Temperature

A model that has to choose the next word from 50,000 possibilities, or label an email as spam or not spam, first produces one raw score per option. These raw scores are called logits. A logit is just a number the model found convenient: it can be negative, it can be 12, and the scores do not add up to anything in particular.

We want probabilities instead — numbers between 0 and 1 that add up to 1. Why not simply divide each score by their total? Because the scores can be negative, and a total can be zero or negative, which produces nonsense. Softmax fixes this in two moves. First, raise e (about 2.718) to each score, which turns any number into a positive one. Second, divide each result by the total. Bigger scores stay bigger, so the ranking never changes. In symbols, p_i = exp(z_i) / sum_j exp(z_j): the probability of option i is e to the power of its score z_i, divided by the sum of e to the power of every score.

Softmax has a handy property: adding the same number c to every logit changes nothing. The reason is the rule exp(z_i + c) = exp(z_i)·exp(c). The factor exp(c) appears in every term on top and bottom, so it cancels. This is not just trivia. It is how code stays numerically safe: subtract the largest logit before exponentiating, so the biggest exponent becomes 0 and nothing overflows. The answer is identical.

Softmax is not scale-invariant, though. Multiply every logit by 10 and the differences explode, so the winner takes almost all of the probability. Divide them by 10 and everything flattens toward equal odds. Temperature is the dial that sets this scale.

Temperature is a number we divide the logits by before exponentiating: p_i(T) = exp(z_i / T) / sum_j exp(z_j / T). With T = 1 this is ordinary softmax.

Dividing by a small T makes every gap between scores wider. As T approaches 0, one probability approaches 1 and the rest approach 0: the model commits to the single best option. Dividing by a large T shrinks every gap toward zero, and the distribution approaches 1/n — complete indifference, where n is the number of options. Temperature is a smooth dial between 'decide' and 'shrug'.

Here is the fact that surprises people: no positive temperature can change which option is most likely. Dividing by a positive number keeps the order of the scores, and exponentiating keeps it too. So the largest logit always gets the largest probability. Temperature changes how confident the model sounds, never the ranking. If the ranking itself must change, you have to change the scores.

So what does temperature change? It changes what you draw when you sample instead of always taking the top option. Greedy decoding always picks the largest probability; it is repeatable but dull. A common alternative is to draw from softmax(z/T).

The Gumbel-max trick makes that concrete. Draw one random number g_i for each option from the Gumbel distribution, a standard source of noise with a particular shape. Then pick the option with the largest z_i / T + g_i. This gives exactly the same odds as sampling from the softmax, but it turns sampling into a race. At low temperature the scores decide the winner; at high temperature the noise does. The tick marks in the demo show the exact temperatures where the sampled option changes for one fixed set of random numbers.

Temperature scaling is also used after training to repair overconfident probabilities. Take a trained classifier, freeze its weights, and fit one number T on a held-out set by minimizing negative log-likelihood — a score that punishes confident mistakes. Neural networks are often too sure of themselves, so a T above 1 softens their probabilities toward reality. Accuracy does not move, because ranking does not move.

Knowledge distillation uses the same idea in reverse. A large, already-trained teacher model is asked for soft targets at a high temperature, and a smaller student is trained to match them. The soft targets tell the student how the teacher ranked the wrong answers, not just which answer won.

Two practical warnings before you practice. First, never divide by zero. Clamp T to a small positive value; the limit as T reaches zero is only valid in the limit. Second, negative temperatures are not a fix for anything. They reverse the ranking and put the worst option first, which is almost never what you want.

When you write softmax yourself, subtract the largest logit before exponentiating. Apply temperature before that subtraction so the arithmetic stays safe.