ORPO Odds-Ratio Term

Problem

Compute the ORPO preference term added to the supervised loss.

With d = chosen_log_odds - rejected_log_odds, the penalty is -log(sigmoid(d)) = softplus(-d); return sft_nll + lambda_ * softplus(-d) using the stable form max(-d, 0) + log1p(exp(-|d|)).

Hint

ORPO adds this odds-ratio penalty to standard next-token training, avoiding a separate SFT stage.

Test cases

case 1

in: [1,0,0,0.5]

expected: 1.3465735902799727

case 2

in: [1,1,0,0.5]

expected: 1.1566308437591115

case 3

in: [0,-1,0,1]

expected: 1.3132616875182228

Starter codePython

import math
def orpo_loss(sft_nll, chosen_log_odds, rejected_log_odds, lambda_):
    # Your code here
    pass

3 test cases · run your code against them in the editor