DPO Implicit Reward Gap

Problem

Compute the mean implicit reward gap of DPO.

Each pair contributes beta * ((policy_chosen - ref_chosen) - (policy_rejected - ref_rejected)); return the average over pairs. An empty batch returns 0.0.

Hint

This gap is exactly the reward DPO assigns to the chosen response minus the rejected one.

Test cases

case 1

in: [[0],[-1],[0],[0],0.1]

expected: 0.1

case 2

in: [[-1,-2],[-2,-1],[-1,-2],[-2,-1],0.5]

expected: 0

case 3

in: [[],[],[],[],1]

expected: 0

Starter codePython

def dpo_reward_gap(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta):
    # Your code here
    pass

3 test cases · run your code against them in the editor