Post-Training: RLHF → DPO → GRPO

Pretraining teaches a model the statistics of text. It does not teach the model to follow instructions, to prefer helpful answers, or to show its reasoning. Those are behaviors, and behaviors come from a second stage called post-training.

The modern stack has three rungs. Supervised fine-tuning imitates demonstrations: show the model good examples and it copies their style. Preference optimization learns from comparisons: this answer is better than that one. Reinforcement learning from verifiable rewards trains on tasks a program can check, such as a math answer or a passing unit test. Each rung teaches something the one below cannot.

RLHF, reinforcement learning from human feedback, was the original recipe. Step one: supervised fine-tuning on demonstrations. Step two: collect human comparisons between pairs of responses. Step three: train a reward model to predict which response a human would prefer. Step four: adjust the model to earn a high score from that reward model.

The comparison step uses a simple probability model. If the preferred answer scores r_w and the rejected one scores r_l, the chance a human picks the preferred answer is σ(r_w − r_l), where σ is the sigmoid function that squeezes any number between 0 and 1. The reward model is trained to make that probability high.

One guardrail matters. The adjusted model is also penalized for drifting too far from the model it started from, measured by KL divergence, a number that says how different two probability distributions are. Without that penalty, the model — from here on called the policy, because it decides what to do — learns whatever tricks maximize the reward model, including the reward model's own mistakes. That failure is called reward hacking, and closing the gap between reward score and real quality is most of RLHF engineering.

PPO (proximal policy optimization) is the workhorse optimizer for that loop. It is online: sample answers from the current model, score them, update, and repeat. It also carries a critic, a second network that predicts how much reward the rest of an answer is likely to earn. The critic reduces the noise in the update direction.

PPO limits how far one update can move the model with a clipped objective: min(r·A, clip(r, 1−ε, 1+ε)·A). Here r is the ratio between the new and old probability of an answer, A is the advantage — how much better the answer did than the critic expected — and ε (epsilon) is a small cap such as 0.2. The clip says: do not chase a large ratio further than this.

The catch is that the critic is another model to train, tune, and store. Removing it is one reason GRPO took over.

DPO, direct preference optimization, skips the reward model entirely. For the same comparison model, the best possible adjusted policy has an exact mathematical answer, and it implies an implicit reward: β·log(π_θ(y) / π_ref(y)). In words: how much more likely the current model makes answer y than the original model did, multiplied by β (beta), a positive number that controls how far the model may drift.

Substituting that into the comparison probability leaves a loss built from preference pairs alone: L = −log σ(β·[(log π_θ(y_w) − log π_ref(y_w)) − (log π_θ(y_l) − log π_ref(y_l))]). The preferred answer y_w should gain probability relative to the original model more than the rejected answer y_l does. Here π_θ is the model being trained and π_ref is the frozen copy it started from.

No sampling, no critic, no reward model. DPO is a classification loss on log-ratios, which is why it is the stable default. Published comparisons put it within about 0.3 points of PPO on MT-Bench, a standard chat-quality benchmark, at roughly a tenth of the compute.

GRPO (group relative policy optimization) keeps the online loop and drops the critic. For each question it samples a group of G answers, scores them all, and compares each score to the average of the group: A_i = (r_i − mean(r)) / std(r). An answer better than the group average gets a positive advantage; a worse one gets a negative advantage. Dividing by the standard deviation — a measure of how spread out the scores are — keeps the advantages on a consistent scale. The group average replaces the critic's prediction.

When the score comes from a program that can check the answer — a math checker, a unit test, a schema validator — the loop is called RLVR: reinforcement learning from verifiable rewards. There is no reward model to hack, because the checker knows the truth. GRPO with verifiable rewards is the standard way reasoning models are trained, and DAPO and GSPO are refinements that stabilize its clipping and normalization.

The choice depends on the data and the budget. DPO wins when preferences are fixed and pairs already exist, because it is one pass over a fixed dataset. GRPO wins when outcomes can be verified, because fresh attempts explore beyond the demonstrations and the verifier cannot be fooled. Supervised fine-tuning still owns format, tone, and tool-call syntax.

Rankings flip with model size. A controlled 2026 comparison found different winners at different sizes, so 'best method' always depends on where you measure. One practical note applies throughout: a 4-bit base model with 16-bit adapters is how these methods fit on a single machine.

Three failure modes are worth naming. Reward hacking: the model finds the reward model's blind spot, so the score rises while human judges disagree. Length bias: longer answers score higher, so the model learns to ramble. Distribution collapse: too much pressure to stay near the original model, or too little exploration, narrows the model until it gives one safe answer to everything.

The defenses are boring and effective. Hold out a test set graded by humans or by the checker, watch response length, cap how far each update may move the model, and refresh the reward model as the policy changes.

The problems below implement the comparison probability, the reward-model loss, the DPO loss, its implicit reward gap, and the PPO clipped objective. Together they are the arithmetic behind every rung of the stack.