DPO Loss

Problem

Compute the direct preference optimization loss for one pair: -log sigmoid(beta * ((logp_chosen - ref_logp_chosen) - (logp_rejected - ref_logp_rejected))), evaluated stably with softplus. The signature takes log-probabilities, not probabilities.

The signature is dpo_loss(logp_chosen, logp_rejected, ref_logp_chosen, ref_logp_rejected, beta).

Hint

DPO turns the RLHF objective into a simple classification loss on preference pairs.

Test cases

case 1

in: [0,0,0,0,0.1]

expected: 0.6931471806

case 2

in: [1,0,0,0,1]

expected: 0.3132616875

case 3

in: [0,1,0,0,1]

expected: 1.3132616875

case 4

in: [2,1,1,0,0.5]

expected: 0.6931471806

Starter codePython

import math
def dpo_loss(logp_chosen, logp_rejected, ref_logp_chosen, ref_logp_rejected, beta):
    # Your code here
    pass

4 test cases · run your code against them in the editor