Gradient Descent from MSE to Logistic

Training a model means searching for the numbers that make it wrong as little as possible. Those numbers are called parameters, written θ (the Greek letter theta). 'Wrong' needs a number too: a loss function L(θ) that turns all the model's mistakes into a single score, where smaller is better.

Picture standing on a hill in fog, wanting the bottom. The gradient ∇L(θ) is a list of slopes: how the loss changes when you nudge each parameter one way or the other. It points uphill. So step the other way: θ ← θ − η·∇L(θ). The Greek letter η (eta) is the learning rate — how big a step you take. Too big and you leap across the valley; too small and you crawl.

Almost everything in deep learning is a variation on that one sentence: different ways to measure wrongness, different numbers to move, different ways to estimate the slope from a sample of data.

Start with the simplest model, a straight line: ŷ = w·x + b. Here x is the input, w is the slope (called a weight), b is where the line crosses zero (the bias), and ŷ is the prediction. Measure wrongness with mean squared error, the average of (ŷ − y)² over all the data, where y is the true answer. Squaring makes errors in both directions count and makes big misses count extra.

The slope of that loss is refreshingly simple. For each weight it is 2·(ŷ − y)·x, and for the bias it is 2·(ŷ − y). In words: the correction is the error, multiplied by the input. A point where the model is far off and the input is large gets a big nudge; a point the model already gets right gets none. This loss has a single valley and no false bottoms, so with a small enough learning rate, descent finds the best line.

Logistic regression keeps the same straight-line score z = w·x + b but turns it into a probability with the sigmoid function, σ(z) = 1/(1 + e^{−z}). The sigmoid squeezes any number between 0 and 1. Wrongness is then measured with cross-entropy, a loss built from logarithms: L = −[y·log(p) + (1−y)·log(1−p)], where p is the predicted probability of the positive class and y is the true label, 1 for that class and 0 for the other. When the model is confident and right, this loss is near zero. When it is confident and wrong, the logarithm explodes: the penalty has no upper limit. Squared error, by comparison, tops out at 1, so it stops caring exactly when caring matters most.

Then something elegant happens. The slope of the cross-entropy loss with respect to the weights is (p − y)·x — the same shape as the squared-error slope, except the raw prediction is replaced by the probability. The sigmoid's own derivative cancels perfectly against the logarithm.

Click the canvas to add points and watch the boundary line w·x + b = 0 refit after each one. That line is where the model is exactly 50/50.

Points near the boundary pull hardest: their p − y has the largest size. Points far on the correct side are already called correctly with confidence, so their pull nearly vanishes. Switch the update rule between mean squared error and log loss. Squared error picks the same direction but carries an extra p(1−p) factor, so it slows to a crawl when the model is confidently wrong. That is one reason cross-entropy is the default for classification.

The loss curve below the plot tells you whether the learning rate is sane. Falling smoothly and then flattening means converged. Jumping up and down or blowing up means the step size is too large. Still creeping down after hundreds of steps means it is too small, or the inputs sit on wildly different scales.

Updating after every single point gives a noisy staircase. That is stochastic gradient descent, and the noise is useful: it shakes the model out of flat spots and saddle points — places that slope down one way and up another — where full-batch descent could circle forever.

With data that can be split perfectly by a line, logistic regression has no best answer. The weights can keep growing forever, making the sigmoid more certain and the loss smaller. In the demo the boundary stops moving visually long before the weights do.

In practice you stop early or add L2 regularization: a small penalty on large weights that pulls them toward zero and makes the best answer finite again.

For nonlinear models the loss surface has many valleys, so descent can settle into a local minimum. In high dimensions, though, true dead ends are rare; saddle points are the usual obstacle — another reason the randomness of stochastic updates helps.

Two implementation details matter more than they look. First, batch versus stochastic. The true gradient averages the slope over all examples. Stochastic methods estimate it from one example or a small batch. Small batches trade a little noise for much better hardware speed and smoother curves.

Second, scale your features. When one input ranges over thousands and another over fractions, the loss surface becomes a long thin valley and descent zig-zags instead of heading straight down. Standardizing the inputs is often the difference between converging in fifty steps and five thousand.

The problems below ask for the sigmoid, both gradients, and a single descent step.