IPO Loss

Problem

Compute the mean identity preference optimization loss.

For each pair let h = (policy_chosen - ref_chosen) - (policy_rejected - ref_rejected), then average (h - 1 / (2 * beta)) squared over the batch. An empty batch returns 0.0.

Hint

IPO targets a fixed margin instead of letting the preference gap grow forever.

Test cases

case 1

in: [[0.5],[0],[0],[0],0.1]

expected: 20.25

case 2

in: [[0],[0],[0],[0],0.5]

expected: 1

case 3

in: [[1],[0],[0],[0],1]

expected: 0.25

case 4

in: [[],[],[],[],0.5]

expected: 0

Starter codePython

def ipo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta):
    # Your code here
    pass

4 test cases · run your code against them in the editor